Automatic Remediation with Stealthwatch Cloud and AWS

A proof-of-concept approach to automate IP blocking using alerts from our Stealthwatch Cloud service (formerly Observable Cloud)

A common response to an attack is to block the attacker’s IP address ranges. This isn’t a great long-term solution – attackers can switch addresses and legitimate users can be mistakenly affected by blocks. However, this approach can temporarily mitigate problems while a better strategy is worked out.

This post outlines a proof-of-concept method of automating IP blocking (in an AWS environment) based on an alert from the Stealthwatch Cloud service. There are several caveats, so don’t “try this at home” – I’m a trained professional, this is a closed course, etc.

Auto-blocking pipeline

Our goal is to prevent a remote IP address associated with an alert from interacting with our AWS resources. We might consider Security Groups for this, but they don’t accept “deny” rules. VPC ACLs are more suitable since they accept both “allow” and “deny” rules.

By default, the Stealthwatch Cloud service sends an e-mail when an alert is generated. However, it can also deliver information about alerts to several other services. The one we’ll use for this exercise is Amazon SNS, which is convenient because of its integration with AWS Lambda.

We can use a Lambda function to accept the Stealthwatch Cloud alert data from SNS and write a “deny” rule to the appropriate VPC ACL. That will give us this pipeline:

Delivering alerts to SNS

For the first stage of our pipeline, we’ll configure Amazon SNS to receive data. From the SNS Console, we’ll create a topic. After creating, we should note the Topic ARN, which we’ll need below.

Next, we’ll configure the Stealthwatch Cloud service to deliver alerts to SNS. From the Stealthwatch Cloud web portal, go to Settings > Services/Webhooks > Amazon SNS. Here we’ll fill out the form:

  • Region: The AWS topic where the target SNS topic lives
  • Topic ARN: This is the ID of the SNS topic from above.
  • Active: Select in order to enable delivery
  • Notes: Optional

As the screenshot shows, we’ll need to set the SNS topic to permit messages from the Stealthwatch Cloud service. Back in the SNS Console, under Other topic actions > Edit topic policy, enter the Stealthwatch Cloud web portal’s ARN:

Download A New Way to Look at AWS Security whitepaper.

Download White Paper

Triggering a Lambda function

For the second stage of our pipeline, we’ll create an AWS Lambda function to react to messages from SNS. From the Lambda Console, we’ll Create a Lambda function, choose the sns-message-python blueprint, and then select the SNS topic from above. We’ll choose Enable trigger and then continue.

On the next page we’ll need to name the function and create a new custom role for it. Since it will need to be able to modify resources, it will need special permissions. We’ll use some of the pre-made Managed Policies for this and attach them to a role from the IAM Console.

A memory limit of 128 MB and a timeout of 60 seconds should be sufficient for this function. We will discuss the code in more detail below.

Adding ACL rules programmatically

The data that the Stealthwatch Cloud service delivers to SNS includes information about the alert event, such as which resource is affected, what time the events took place, etc. We’ll supplement that by using the Stealthwatch Cloud API to pull associated observation data, which will include things like which remote IP addresses to block. (We can get an API key from the Stealthwatch Cloud web portal under Settings > User.)

We’ll have several things to do in our Lambda function:

  • Decode the message from SNS.
  • Make sure the alert type is one that includes blockable remote IPs, such as Excessive Access Attempts (External).
  • Query the Stealthwatch Cloud API for observation details, collecting a set of remote IPs to block.
  • Query the AWS API (DescribeInstances) for the alert “source” to see which VPC it belongs to.
  • Query the AWS API (DescribeNetworkAcls) to see what rules are in place for the VPC’s ACL. We’ll need to block using a low-numbered rule, since the default ACL includes an “allow all” rule at position 100.
  • Put the blocking rules in place with the AWS API (CreateNetworkAclEntry).

Fortunately, the default libraries are suitable for performing all these functions, so we can avoid creating a deployment package and just edit the code in the Lambda console. Annotated Python code is here.

Our final process diagram looks like this:


Once we’ve saved and tested our code, we can look at results when new alerts are generated. In this example, the IP address is observed to be attempting to access the EC2 instance with the identifier i-0a1b2c3d:

The alert will fire a message off to SNS, which will in turn trigger our Lambda function. The Lambda function will add a new DENY rule for We can see the results in the VPC Console:

Future work and closing comments

The above shows that we can automatically implement mitigations for security incidents with the Stealthwatch Cloud service and AWS tools. However, don’t go copying this into your production environment  -- there are some obvious limitations with this example:

  • The Lambda function assumes all resources are in the same region.
  • The generated rules only work on IPv4 addresses.
  • You could run out of rule numbers if many remote addresses were trying to break in.

Resolving and testing solutions for these issues would be critical for a serious deployment.

It would be fun to work out some follow-up activities. For example, we could implement a temporary IP block (a “shun”) by having our Lambda function write rule details to an SQS Delay Queue, which is then polled periodically by a periodically scheduled Lambda function. This would prevent a human from having to clear out block rules.

In putting together this post, I found some places where the Stealthwatch Cloud service’s output could be extended to make tasks like this more convenient. If you try any of this out and have suggestions, do let us know.

Detect Threats Faster – Start Your Free, No-Risk Trial