Design Sentinel data storage, log types, and retention
5 minutes
5 Questions
Microsoft Sentinel data storage design requires careful planning to optimize costs, performance, and compliance requirements. The architecture revolves around Log Analytics workspaces, which serve as the primary repository for all ingested data.
Log Analytics workspaces store data in Azure Monitor…Microsoft Sentinel data storage design requires careful planning to optimize costs, performance, and compliance requirements. The architecture revolves around Log Analytics workspaces, which serve as the primary repository for all ingested data.
Log Analytics workspaces store data in Azure Monitor Logs, using a columnar storage format optimized for fast queries. When designing storage, consider workspace architecture decisions: single workspace for simplicity, multiple workspaces for geographic distribution, or hybrid approaches for compliance needs.
Sentinel supports various log types categorized by their source and purpose. Security logs include Azure Active Directory sign-in and audit logs, Microsoft 365 Defender data, and Azure Security Center alerts. Infrastructure logs encompass Azure Activity logs, Azure Diagnostics, and virtual machine performance data. Custom logs allow ingestion from third-party sources via CEF, Syslog, or custom connectors.
Data tables in Sentinel fall into different categories: Analytics logs for high-value security data requiring full query capabilities, Basic logs for verbose troubleshooting data with reduced query features at lower cost, and Archive tier for long-term retention needs.
Retention policies are crucial for compliance and cost management. Interactive retention keeps data queryable for 30 to 730 days, configurable per table. Archive retention extends storage up to 7 years total, with data accessible through search jobs or restoration. Different tables can have different retention periods based on regulatory requirements and investigation needs.
Cost optimization strategies include routing high-volume, low-value data to Basic logs tier, implementing data collection rules to filter unnecessary information, and using commitment tiers for predictable workloads. Transform ingestion-time data to reduce storage costs while preserving essential security information.
Proper design ensures efficient threat detection, investigation capabilities, and regulatory compliance while managing operational costs effectively across your security operations environment.
Design Sentinel Data Storage, Log Types, and Retention
Why It Is Important
Understanding how Microsoft Sentinel stores data and manages log retention is critical for security operations analysts. Proper data storage design impacts your organization's security posture, compliance requirements, cost management, and ability to investigate incidents effectively. Misconfigured retention settings can lead to lost evidence during investigations or unnecessary storage costs.
What It Is
Microsoft Sentinel data storage refers to how security logs and events are ingested, stored, and retained within the Azure Log Analytics workspace. Sentinel uses different log types and retention policies to optimize both cost and accessibility of security data.
Key Log Types in Sentinel:
Analytics Logs: The default log type offering full query capabilities and 90 days of interactive retention by default. Best for active security monitoring and investigation.
Basic Logs: A cost-effective option for high-volume, verbose logs that require reduced query capabilities. Offers 8 days of interactive retention with a lower ingestion cost. Ideal for compliance and occasional troubleshooting.
Archived Logs: Long-term storage solution for logs beyond the interactive retention period. Data can be retained for up to 12 years total. Requires search jobs or restore operations for access.
How It Works
1. Data Ingestion: Logs flow into Sentinel through data connectors and are stored in the Log Analytics workspace.
2. Retention Configuration: You can set retention at the workspace level (30-730 days for interactive retention) or configure table-level retention for specific data types.
3. Archive Process: After interactive retention expires, data automatically moves to the archive tier if configured, allowing up to 12 years total retention.
4. Data Access: Analytics logs support full KQL queries, Basic logs have limited query operations, and Archived logs require restoration or search jobs to access.
Cost Considerations: - Analytics logs have higher ingestion costs but full functionality - Basic logs offer up to 80% cost reduction for ingestion - Archive storage is the most economical for long-term retention
Exam Tips: Answering Questions on Design Sentinel Data Storage, Log Types, and Retention
1. Know the retention limits: Remember that interactive retention can be set from 30 to 730 days, and total retention including archive can extend to 12 years (4,383 days).
2. Understand Basic Logs limitations: Basic logs do not support alerts, have 8 days of interactive retention, and support only simple queries. Questions may test whether you know which scenarios are appropriate for Basic logs.
3. Cost optimization scenarios: When questions mention high-volume logs, verbose data, or cost reduction, think Basic logs. When compliance or long-term storage is mentioned, consider Archive.
4. Table-level vs workspace-level retention: Know that you can configure different retention periods for individual tables, which allows granular control over specific log types.
5. Search jobs and restore operations: For archived data access questions, remember that search jobs are used for querying archived data and restore operations bring data back to interactive tier temporarily.
6. Compliance requirements: Questions involving regulatory compliance often require understanding of Archive logs for long-term retention needs.
7. Watch for keywords: 'Cost-effective' suggests Basic logs, 'long-term' suggests Archive, 'full analytics' suggests Analytics logs, and 'compliance' often involves retention period configurations.