The Mysteries of Log Analytics Workspaces

Log Analytics workspaces provide a special way to store log data from multiple sources such as Microsoft Defender for Cloud, Azure Monitor, and so much more. A workspace typically combines data from multiple services and likely has it’s own distinct configuration for retention. I get a lot of questions about what the differences between workspaces within the Log Analytics scope are and why we would use them. Let’s take a look today at some of the information around Microsoft Azure Log Analytics Workspaces and what we can use them for, and what makes a good case to have multiple workspaces.

First up, let’s take a look at why there are different types of Log Analytics workspaces. There are 2 types of workspaces:
1) Log Analytics workspace, and
2) Microsoft Sentinel enabled Log Analytics workspace.

What’s the difference and why make them different at all? Well, pricing is the real difference between the two flavours. Now, there is technically no charge for the workspace…..but there definitely is for data ingestion and how long that data is stored (retention of data).

There is the free tier – yup – a free tier with certain types of data being able to be stored for free in your Azure Log Analytics workspace for up to 8 days like Microsoft 365 and Azure Activity logs. There are some limitations other than data retention, but what a great way to get acquainted with Log Analytics workspaces, retention, log management, and Microsoft Sentinel!

The Data Structure of Log Analytics workspace is shown below with multiple tables each organized into separate columns and then the ingested data is stored as multiple rows in those tables. The columns are shared between the sources and workspace – this let’s the data be ‘pushed’ from the source into our Log Analytics workspace as a destination.

To get data out of this structure we use Log Queries. A log query is essentially a definition statement that states the columns of data you want to see and then provide some other parameters such as time span or originating source to refine that data query or request from the workspace.

Awesome stuff….ok….now onto some other interesting features of Log Analytics workspaces — log filtering controls called Ingestion-Time Transformations.

An ingestion-time transformation is actually a KQL query that runs against the incoming data and modifies that data before it’s stored in the workspace based on the query statement. (Side explanation if you are new to this > KQL is a query language that is quite similar to SQL query statements. I highly recommend becoming familiar with KQL with this awesome Tech Community reference here.)

Alright, back on the rails here…

Data collection rules that define data coming into Azure Monitor can also support these transformations and help to filter that data before being ingested — saving us from that ingestion data charge! Not all workflows are able to support data collection rules today, but we are able to define ingestion-time transformations. Just remember that the transformations can only apply to a workflow that does not have a DCR in place. As an example, if you have Azure Monitor agent using a DCR to define data coming from Azure Firewall, the ingestion-time transformations defined will not apply to this data.

The final piece I want to add to our Log Analytics workspace is two-fold, permissions. We have our usual Azure RBAC that applies to who can access the Log Analytics workspace, but we can go a bit deeper right!

On each workspace, we can configure access control mode that allows us to grant custom or built-in roles a bit more granularly such as allowing access to data collected from resources to users that have access to those particular resources already. This is a great way to avoid duplication of work and allow those roles and permissions to be ‘inherited’ into your Log Analytics workspace. Learn more about this functionality right here.

Well, I didn’t add as many diagrams and screenshots as usual today, but hopefully this article helps to remove some of the mystery around Log Analytics workspaces for you! Thanks for joining me for this little de-mystification!

Source: https://docs.microsoft.com/en-us/azure/azure-monitor/logs/log-analytics-workspace-overview