One of the best pratices that have been defined for Log Analytics in Azure is to have a few workspaces as possible. Now with the introduction of Azure Sentinel as well, there more sentralized set of logs you have the easier it is to build hunting queries and such in the data that is ingested. Now one of the limitations that has been with Log Analytics is that retention that is defined on a workspace is regardless of what kind of data that is stored within it.
Now having a single log analytics workspace which you use for Sentinel and it might also contain other data such as performance metrics especially if you are using it for PaaS services or just collecting performance metrics from agents as well you don’t want to have 1 year renention on both Security Logs and Performance Metrics?
Just to give an example, having Linux & Windows Agents you can collect Logs and Metrics. If you have an policy which states that security logs should be stored for 1 year, you will also be storing the performance metrics for a whole year as well.
An overview from a webinar I hosted last week on Azure Sentinel
I also mentioned in the webinar that table based renention was on the roadmap and that it was not far away, and luckily it came out today! so now you can have custom retention time on different tables within a Log Analytics Workspace! as it is now you can change this using the ARM API essentially using the Armclient which you can install using choco (more info here –> https://github.com/projectkudu/ARMClient)
choco install armclient --source=https://chocolatey.org/api/v2/Using this command you can list out all the tables within a workplace
armclient get “/subscriptions/subscriptionid/resourceGroups/resourcegroup/providers/Microsoft.OperationalInsights/workspaces/workspacename/Tables?api-version=2017-04-26-preview”
Then I can change the retention using a PUT command and define the retention time as a string.
armclient put /subscriptions/subscriptionid/resourceGroups/resourcegroup/providers/Microsoft.OperationalInsights/workspaces/workspacename/Tables/SecurityEvent?api-version=2017-04-26-preview” “{properties: {retentionInDays: 7}}”
After I’ve changed the retention I can see the change just by quering the API for the specific table (In this example I choose the table SecurityEvent
armclient get /subscriptions/subscriptionid/resourceGroups/resourcegroup/providers/Microsoft.OperationalInsights/workspaces/workspacename/Tables/SecurityEvent?api-version=2017-04-26-preview”
Now with this change it allows me to have longer retention of specific content such as security logs but not on performance metrics which I only require for a short amount of time or verbose logs for Containers for instance.