Last month Microsoft announced and released into public preview a bunch of new features for Microsoft Sentinel which makes administering data within the solution easier for administrators and security operations teams. Whilst these features are not strictly part of Microsoft Sentinel and exist within Log Analytics, they introduce new ways to manage security logs.
These three additions are:
- Basic Logs
- Archive(d) Logs
- Search & Restore
Another item of note is that ‘standard’ logs are now called ‘analytics’ logs, which is apt as these are the only logs that can be used in Microsoft Sentinel analytics rules.
Archive Logs
I’ve been asked about how to retain logs within Microsoft Sentinel and Log Analytics for some time now, and before this Operations Management Suite! It hasn’t always been easy, until now the main options have been:
- Setup a Logic App to query tables and send the output to a location
- Integrate Log Analytics with Azure Data Explorer
- Use the newly released Data Export feature to export individual tables to a storage account
- Make use of the data retention feature within Log Analytics to store logs past 90 days
All of these options require a significant amount of technical configuration or extra steps to re-hydrate or query logs for analysis, or in certain cases add additional costs and infrastructure to manage outside of the security context.
The new archive tier is currently in public preview and allows for storage of data past the interactive data retention period per table. This provides approx. 80% savings on standard log retention. From the blog post:
Archive will generate a cost per GB/month stored. Currently the price will be $.02 per GB per month.
One of the main benefits of the archive tier is that log data remains within Microsoft Sentinel and can be retrieved using the restore function
Set table retention with archive
At time of writing it’s not possible to manage the archiving of logs at a workspace level or via the Azure Portal, only through the Azure Management API or the Azure CLI. Through the API and CLI it’s worth noting that there’s no ‘retention’ and ‘archive retention’ rather a ‘total retention’. For example:
az monitor log-analytics workspace table update --resource-group mgmt-sentinel-prod --workspace-name mysentinelworkspace --name SignInLogs --retention-time 30 --total-retention-time 90An example command output can be found here.
It’s also possible to show the current configuration:
az monitor log-analytics workspace table show --resource-group mgmt-sentinel-prod --workspace-name mysentinelworkspace --name SignInLogsTo break the command down we’re sending a command to Azure to update the specific table ‘SignInLogs’ (from Azure AD) and setting a retention period of 30 days with a total retention period of 90 days. This would mean that Azure AD Sign In Logs remain as analytics logs (think of this as active hot storage) for 30 days and 60 days in archive tier.
Hot + Archive = Total Retention Time
The new archive tier can be set for up to 7 years. The current maximum standard retention which can be set on a Log Analytics workspace is 2 years.
Search & Restore
Whilst the search and restore features within Microsoft Sentinel are separate they both cover similar use cases.
Search
Querying logs within Microsoft Sentinel is ideal for investigations and fast queries via security analysts alongside complex queries with parsing and data transforms. In some cases searches need to be executed across a large amount of data within a single table, including both hot and archived data.
Starting a search can be done in the Microsoft Sentinel interface, Management API or CLI. Within the interface a time window can be selected alongside a table and then a query entered. For example:
Searches run in an asynchronous fashion, so it’s possible to exit the portal or carry out other tasks whilst they run in the background. It’s worth noting there are some documented limitations such as a timeout of 24 hours, and 100 searches per day.
When a search is complete it’s possible to see statistics of the search and then query the search table alongside any other table within the workspace.
Whilst search is ideal for querying analytics logs and logs in archive it’s best suited to large searches of ‘hot’ logs. It’s worth noting that searches also have a cost:
Search will generate cost only when Search jobs are performed. The cost will be generated per GB scanned (data within the workspace retention does not add to the amount of GB scanned). Currently the price will be $0.005 per GB scanned.
Restore
Where search is ideal for pulling back logs using a query across archive and analytics logs, the restore feature is used to restore tables within a time-frame with all data. Restoring data operates in a ‘job’ fashion, restore tasks are created in the background and can be dismissed when no longer needed. When a table is restored a custom log is created within the workspace with a suffix of _RST.
When the restore job has completed the table can be queried alongside analytics data or for historical purposes.
It’s worth noting that restore jobs and tables must be deleted when actions have been completed with them. So far restore jobs have a number of limitations such as only being able to perform 4 restores per workspace per week. Restore costs are related to the tables being in use:
Restore will generate cost only when a Restore job is performed. The cost will be generated per GB restored/per day that the table is kept within the workspace. Currently the cost will be $.10 per GB restored per day that it is active. To avoid the recurring cost, remove Restore tables once they are no longer needed.
Basic Logs
The final new addition is the availability of the basic log tier. This allows for the ingestion of high-volume or verbose logs at a reduced cost for 8 days. These logs cannot be used in analytics rules and can only be queried using basic functions; where, extend, project, parse and their sub-operations.
Currently the only way to configure basic logs is to convert a standard analytics table to basic using the Management API or CLI:
az monitor log-analytics workspace table update --resource-group mgmt-sentinel-prod --workspace-name mysentinelworkspace --name ContainerLog --plan BasicTo switch back to analytics:
az monitor log-analytics workspace table update --resource-group mgmt-sentinel-prod --workspace-name mysentinelworkspace --name ContainerLog --plan AnalyticsBasic logs are useful where custom data collection rule logs are being ingested. From the documentation the following tables are supported:
- All tables created with the Data Collection Rule (DCR)-based custom logs API.
- ContainerLog and ContainerLogV2, which Container Insights uses and which include verbose text-based log records.
- AppTraces, which contains freeform log records for application traces in Application Insights.
It’s worth noting that basic logs have a different pricing model to standard analytics logs:
Basic log ingestion will work similar to how the current model works. It will generate cost per GB ingested into Azure Log Analytics and also Microsoft Sentinel if it is on the workspace. The new billing addition for basic log ingestion will be a query charge for GB scanned for the query. Data ingested into the Basic tier will not count towards commitment tiers. Currently the price will be $.50 per GB ingested in Azure Log Analytics and $.50 per GB ingested into Microsoft Sentinel.
This was a quick write-up of my experience with the new data management features in Microsoft Sentinel. There are plenty of other new features such as ingestion-time transformations and data collection rules of equal importance! But the new data management features are a fantastic addition to both Log Analytics and the Microsoft Sentinel platform.
