Blog

Monitoring Serverless Architectures in AWS

An article that discusses the methods for auditing and monitoring serverless architectures in AWS.

The introduction of serverless architectures is a positive development from a security perspective – splitting up services into single-purpose functions with well-defined inputs and outputs helps reduce exposure to many types of threats. However, the security of serverless architectures is an under-studied topic; many practices and tools that are available in more traditional architectures don’t apply.

This article will discuss methods of auditing and monitoring of AWS Lambda functions (a key component of serverless architectures in AWS) and how the Observable Networks service puts these methods into practice.

Lambda security topics

We can break down questions about Lambda security into a few different areas:

  • Access to the function: Who/what can modify and/or invoke a function?
  • Access from the function: What can the function access when it is running?
  • Unintended behavior: Can the function be made to do something unexpected?

For each area there is a question of auditing, or looking to see whether configurations match expectations, and monitoring, or keeping track of activities on an ongoing basis.

Auditing

The Lambda docs describe how to use the IAM API and Lambda API for auditing policies related to function access. It’s important to audit policies periodically to check:

  • Whether some policies are outdated – they made sense in the past, but haven’t been updated to reflect changes (e.g., a user that has since moved to another team still has rights to change a Lambda function’s code).
  • Whether some policies allow access to more resources than necessary (e.g., a Lambda function that can write to any S3 bucket).

The IAM API can be used to show which users (or roles) in an AWS account have the ability to change a Lambda function or its configuration. For example, we could query the IAM API to see who has the UpdateFunctionCode permission – this would show who is allowed to change the code a function executes.

The Lambda API can be used to show who (or what) is allowed to invoke a function. It’s common to allow other AWS services like S3 or API Gateway call a function. An example of something to look for would be cross-account access for S3 – the AWS docs provide this warning for this problem (reference):

If you add a permission for the Amazon S3 principal without providing the source ARN, any AWS account that creates a mapping to your function ARN can send events to invoke your Lambda function from Amazon S3.

The Lambda and IAM APIs together can show what a Lambda function is allowed to do. For example, to see what permissions a particular Lambda function is granted (e.g., writing to an S3 bucket) you can use the Lambda API’s GetFunction call to determine which role to look up with the IAM API.

Monitoring

The Lambda docs describe how to use CloudTrail and CloudWatch for monitoring.

CloudTrail tracks API activities. We can use CloudTrail logs to both look for changes to functions (e.g., the UpdateFunctionCode call described above) and changes made by functions. This helps keep track of access between audits.

CloudWatch Logs tracks function invocations and their output. This can be used for billing purposes (they include information about duration and memory usage, which are part of Lambda pricing) as well as for monitoring purposes. For example, a function can be made to write its input to the log – this could be used to help look for injection attacks.

CloudWatch Metrics also tracks function usage – the number of invocations, the duration of each run, and other metrics. Manual alarms can be set for things like excessive invocations. This can be used to monitor for unintended behavior and abuse – for example, if the function can be invoked from the outside, an alarm could show whether it is being abused and wasting resources.

Extending monitoring with VPC Flow Logs

The auditing and monitoring resources described in the AWS documentation help answer some of our security-related questions, but miss some others. For example, it would be difficult to use the services above to determine whether a Lambda function is accessing an internal RDS or ElastiCache instance inside a VPC.

We can get more detailed monitoring of Lambda function activity by understanding how functions communicate. As the AWS documentation describes, Lambda functions are allocated temporary network interfaces (ENIs) and IP addresses when they need to communicate with a VPC resource. These ENIs can be enumerated with the EC2 API and matched to the requesting Lambda function.

Once the ENIs and IP addresses are known, a Lambda function’s activity can tracked using VPC Flow Logs. Flow logs describe all of the communication to, from, and within a VPC, so if a Lambda function interacts with an internal resource, the associated network activity will be logged.

Furthermore, Lambda functions need to use a NAT instance or VPC NAT gateway to gain Internet access. This means that their Internet activity will also go through the VPC and be logged as Flow Log records.

This level of detail helps answer our other security questions about Lambda. More types of unintended behavior besides function invocation and API calls can be tracked; and the effects of something exotic like container escape (a breach of the isolated environment that runs the function) could be detected if it was followed by access to VPC resources.

Observable integration

The Observable Networks service uses each of the resources described above to track Lambda functions.

VPC Flow Logs allow Lambda functions to be monitored with Observable’s endpoint modeling process; this enables a wide variety of security alert types. The screenshot below (from the Observable web portal) shows a set of VPC Flow Log records for a Lambda function accessing an RDS database running MySQL:


Lambda functions are tracked by name, so as they change ENIs and IP addresses, their activity will be updated:


Lambda functions are good examples of “Static devices,” or entities that express a very narrow set of behaviors. This makes identifying even subtle changes that may indicate a security concern possible. The screenshot below shows an alert for a Lambda function that normally connects to two internal resources connecting to an unexpected third:


The Observable service also reads from CloudTrail, so if a Lambda function executes an unusual API call (due to code injection, for example) that can generate an alert:


Additionally, the Observable service polls CloudWatch Metrics, so if the function is invoked many more times than normal (due to abuse from the outside), that would generate an alert:


Note that the “normal” level of invocations is determined automatically, so a user doesn’t need to specify up-front what the expected invocation rate is. 

Conclusions and recommendations

In this article we’ve:

  • Described the standard tools for auditing and monitoring Lambda function security – the Lambda and IAM APIs, CloudTrail, and CloudWatch.
  • Explored details of how Lambda function access internal resources to discover another way to monitor behavior – VPC Flow Logs.
  • Explained how the Observable Networks service uses these tools to enable endpoint modeling-based security alerts.

There are some simple steps Lambda users can take to improve their security stance with standard tools:

  • Auditing: At the very least, use the AWS Console to look through the list of the Lambda function’s execution roles and triggers. Check whether any seem overly broad (e.g., allowing “Full Access” to some service like S3) and set a reminder to do the same again in three months.
  • Monitoring: Set up some CloudWatch Metrics alarms for any Lambda functions that react to outside (i.e., untrusted) input. This will let you know if resources are being wasted or if there was some latent behavior in the function’s code that’s now being expressed. Set a reminder to do this for new functions.

If you’re using the Observable service already, you can make sure that the Observable policy for your account is up-to-date (the latest version is here) since this will let it use the AWS APIs for Lambda monitoring. If you’re new to Observable, you’ll be able to check on Lambda function activity shortly after you enable monitoring.

This article is based on a talk given to the Seattle AWS Architects and Engineers Meetup group. See the accompanying slides for more on serverless security.


Experience Dynamic Endpoint Modeling on Your Own Network

Getting better visibility into your network and improving your security couldn’t be easier. Sign up for a free, no-risk trial of Observable’s Endpoint Modeling solution, and change the way you see security.


Detect Threats Faster – Start Your Free, No-Risk Trial