Blog

A Technical Preview of Microsoft Azure’s Network Watcher

A technical preview for configuring Microsoft Azure’s new Network Watcher service for security monitoring.

Microsoft’s Azure team is releasing service called Network Watcher, which contains a set of new monitoring tools. They were kind enough to give us early access to the APIs, so we could see what we could build with them – see below for details of our technical preview. Click here to set up a Demo and Product review

New services

There are several new tools available with Network Watcher. In the general release, we have:

  • Topology: Provides details about how items in a resource group relate to each other. For example, a VM “contains” a NIC, and that NIC is “associated” with an NSG.
  • Packet capture: Provides a means of initiating captures of network traffic going to and from a VM.
  • Security group view: Provides the set of “effective” access rules for a VM, combining the resource’s default rules, its NICs’ rules, and the subnet’s rules for easy auditing.
  • IP flow verify: Provides an answer to the question of whether an IP flow will be allowed or denied by a VM’s access rules, and describes which rule was involved.
  • Next hop: Provides the next resource that packets starting from address A to ending at B will hit, which is helpful in answering routing questions.

Each of these APIs is useful. It’s easy to imagine using the Topology API to draw a zoom-able map of all the items in a resource group. The Security group view API could be used to build an app that identifies which resources aren’t in compliance with a defined security policy.

This post will focus on the Packet Capture API, which has applications ranging from performance monitoring to security monitoring to low-level debugging.

Using the Packet Capture API for endpoint modeling

Observable’s network monitoring service uses endpoint modeling, which is driven by IP flow data. IP flow data is of interest for several reasons – it’s lightweight, it keeps the sensitive parts of network communications private, and is available even for traffic that’s encrypted end-to-end. Below we’ll sketch out how to use the Packet Capture API to generate flow data suitable for use with the Observable service.

Deploying watchable VMs

Next, we’ll spin up some new Windows VMs to monitor. During provisioning, we’ll pick the Network Watcher Agent for Windows extension. This is what enables packet captures.

Setting up an application

While the VMs are deploying, we’ll create an application that can be used to set up captures, write them to Blob storage, and sends them to Observable for modeling.

First, we’ll set up an AD application and a Service principal using these instructions. The application’s role will need to have permissions to read from several resources and write to a few others. We can use a Custom role for this:


For convenience, we’ll also set up a Shared access signature for the Storage account’s container that we’ll be using the hold capture files. That will allow our application to read captures and remove them when they’re no longer needed.

Application architecture

For endpoint modeling, we’d ideally have a continuous source of IP flow data. We can get close to that by setting up “header only” captures for each VM in the resource group, and renewing the captures when they expire (to minimize gaps in coverage). So, our application will thus need to:

  • Check for existing captures (using the “List Packet captures” API call)
  • Find which captures that have completed (using the “Query Packet capture status” API call)
  • Clear completed captures (using the “Delete Packet capture” API call)
  • Create new captures for VMs that don’t have one (using the “Create Packet capture” API call)

We’ll also want to make sure that there is a Network Watcher enabled for the target subscription and region (using the “Create Network Watcher” API call). This only needs to be done once.

Once captures are complete (they can be set to time out after a certain interval, or after they collect a certain amount of data), they write their data to a Blob storage container. So we’ll want to pick them up and ship them to Observable for analysis.

All of these tasks are possible to run as an Observable ONA service. The ONA software is designed to take in network data (from a mirror port, from NetFlow, etc.) and ship it to Observable, so we’ll use that. The overall setup will look something like this:

 

Application details

We can run the ONA service on a small Linux VM running in Azure. To use coordinate access to our Azure resources, we’ll send HTTP requests to the REST API and use the Azure-storage package to retrieve capture files.

Using the Network Watcher API documentation, we can construct the URLs and parameters for each of the API calls above. For example, to check for existing captures, we’ll plug our Subscription ID, Resource group’s name, and Network Watcher’s name into the template:


When we want to download completed captures, we’ll access the Blob storage service, filling in our storage account name and SAS token:


You can see more of the code here.

Results

After the ONA is provisioned and is managing continuous captures, data will make its way into the Observable web portal.

Flow data

We can use the Observable web portal to query the flow data for our monitored VMs. If we glance through the logs, we’ll see some addresses that might be unfamiliar:


Some research shows that 168.63.129.16 is an internal address for the Azure service – see this Azure Support Team Blog post for more information.

We can filter for interesting protocols like Remote Desktop. If we do that, we can see my address (198.51.100.18), but also another address that seems to geolocate to China:


Since I don’t expect RDP traffic from China, this prompts me to look at the NSG rules for the VM. In the Azure portal, I can see that the circled rule is the culprit:



This rule is allowing inbound Remote Desktop connections from the whole Internet (0.0.0.0/0). I can restrict that to my IP range to decrease my exposure would-be hackers.

Observations and alerts

The Observable service will take note of interesting traffic and patterns. As it builds up a model of each device, it will use these observations to generate alerts for notable changes in behavior.

We can see one observation for some failed RDP access attempts from my IP address. For example:


We can also see a “Watchlist” alert, for interaction with a user-specified list of resources – in this case the list of Tor exit nodes:


Observable alerts cover a variety of categories. We’ve seen intrusion detection here, but endpoint modeling is also good at detecting potential data exfiltration, “persistent threats” that play out over time, and more. See some of our previous blog posts for more detailed examples.

Future work and conclusions

As we’ve illustrated here, the Network Watcher APIs can readily be put to practical use. This post focused on security monitoring, but we can also easily imagine using the data to trace performance issues and debug problems.

Our sample application shows how we can use packet captures to feed the Observable service. This probably isn’t a fully satisfactory way to implement monitoring on a production network. For example:

  • There are periodic gaps in coverage when the packet captures expire and need to be renewed - there is a tradeoff between time-to-analyze and the frequency of gaps.
  • The impact of the Network Watcher Agent on a VM’s performance should be evaluated - perhaps a better place to monitor would be at the network level, e.g. with passive flow logs that keep session counters.

Nonetheless, this initial release gives us reason to be excited for what’s coming next.

See the Microsoft Azure Network Watcher Blog post.


Experience Dynamic Endpoint Modeling on Your Own Network

Getting better visibility into your network and improving your security couldn’t be easier. Sign up for a free, no-risk trial of Observable’s Endpoint Modeling solution, and change the way you see security.


Detect Threats Faster – Start Your Free, No-Risk Trial