In my previous post I talked about the impending deprecation of the Log Analytics agent which is to be replaced with the Azure Monitor Agent: AMA? Monitor Servers with the Azure Monitor Agent (and Sentinel!) | by Tom Hind | Oct, 2022 | Medium
That post covered the process of onboarding servers with the new agent making use of Data Collection Rules and Azure Arc to service both Azure virtual machines and virtual machines in other clouds or on-prem. This post will deep-dive into two options which are available for ingesting logs to Microsoft Sentinel (via Log Analytics) by locking down the endpoints to which logs can be sent. The first method is fairly straightforward and fits into any standard deployment of Microsoft Sentinel, Log Analytics and Azure Monitor. The second method makes use of some advanced features within Azure to control log flow. Custom logs and Syslog forwarding is out of scope of this post, and it will focus on server log collection. Those log types with the Azure Monitor Agent are in the works. In this post I’ll show you how to configure these methods although I will be working from an existing ‘landing zone’ hub and spoke network setup with the following resources already deployed:
- Hub virtual network
- Azure Firewall service deployed with some basic OS & Azure rules
- Infrastructure virtual network
- Linux server & Windows server
As is common with Azure, there are many ways and interfaces available to achieve the same goal.
Data Collection Endpoints
Making use of data collection endpoints is a simple way to restrict where server logging traffic is being sent as it provides you with an ‘endpoint’ for control traffic and ingestion traffic. Data collection endpoints are straightforward to setup and provide the endpoint information immediately. Whilst it is possible to set these up when configuring the Data collection rules, it’s not possible to configure an endpoint when configuring security event log data from Microsoft Sentinel. To set the endpoints up beforehand navigate to Azure Monitor and Data Collection Endpoints:
The creation process for an endpoint just requires a name, resource group and region. You will need to create endpoints to match the regions for any resources you wish to collect logs from:
When the endpoint is created by navigating to the resource you will see two separate URLs for configuration access and logs ingestion:
Both of these URLs should be allowed via a firewall or network controls, alongside the URLs at this link: Define Azure Monitor Agent network settings — Azure Monitor | Microsoft Learn
The table on the above page provides the global URLs to allow which perform the configuration of the data collection endpoint to an agent and allow for log ingestion to the targeted log analytics workspace. When the endpoint is created it’s then possible to link it to an existing data collection rule or create a new rule and link the endpoint. For this example, I have a data collection rule created via Microsoft Sentinel which is collecting no data as the firewall is blocking the connection:
I can link the new data collection endpoint created above to the rule:
After a short period, it’s possible to see the agent on the virtual machine attempting the connection to the data collection endpoint:
After adding the data collection endpoint URLs to the Azure Firewall logs are then ingested into the Microsoft Sentinel workspace:
And that’s data collection endpoints! A fairly simple way to lock down log ingestion routes for the Azure Monitor Agent and data collection rules. If you’re operating in a restricted networking environment or wanting to control communications between on-prem <-> cloud (or other clouds!) this is a good first step to providing a structured route. They are also a prerequisite to the next ‘stage’ below. Azure Monitor Private Links aren’t required unless you need private network traversal.
Azure Monitor Private Links
Whilst data connection endpoints provide a singular place for monitor agents to connect and ship logs, private link scopes ensure that the data being sent to Microsoft Sentinel (and Log Analytics) is traversing the virtual networking environment within Azure from virtual machines. In this scenario all virtual machines & servers should be connected to the network via VPN or ExpressRoute. When creating a private link scope its recommended to host this within the hub network of the Azure environment for easier routing configuration.
It’s possible to create the monitor link scope as a new resource, in this case I’m selecting the option to allow for querying to take place but no log ingestion over the internet:
With the monitor link scope created we can then start to create private endpoint connections and link our Azure Monitor resources:
When running through the creation process of the link scope you will have to specify the virtual network to create the links. As mentioned, it’s recommended to create this at the hub level, or in your primary virtual network:
Running through this creation process will also setup a private DNS zone, whilst it's possible to manage this manually using your own DNS servers this allows for Azure to manage additional endpoints, a good description of the DNS configurations is here: Configure your Private Link — Azure Monitor | Microsoft Learn
When the resource is deployed, by navigating to the private link you will be able to inspect the network interface configuration and see which IPs link to which Azure service FQDN:
We can now link our Microsoft Sentinel workspace to the monitor link scope:
When this is complete if you navigate to the workspace settings, within network isolation we have two more options to configure within the workspace itself depending on if we want queries and log ingestion to be available from outside of the link scope:
Whilst this process has created the monitor private link scope and set the foundations of the private endpoint connectivity, we still have no logs being sent into the Microsoft Sentinel workspace. This is still achieved via data collection rules. Following the steps as above I’ll create a data collection endpoint, but before configuring the rule we have to link this to the monitor private link scope as we did the workspace alongside disabling public access:
And then creating a data collection rule specifying the endpoint, or selecting an existing rule:
After some time, logs will be ingested to the Microsoft Sentinel workspace. You’ll still have to add the firewall rules as before. But by viewing the metrics of the monitor link scope private endpoint connection we can see traffic in-out:
And within Microsoft Sentinel we are now receiving logs:
And that’s it! Whilst it’s quite the process to onboard the private link scope, there are quite a few benefits. Firstly, it’s a global service which means multiple networks and regions can be linked. Alongside the core reason for deploying which is making all server log ingestion traffic private.
As a quick aside, it’s possible to validate the private link by running nslookup or hitting the private IP in the DNS zone in a web browser on the network:
