Blog

Our Open Source VPC Flow Logs Tool Version 1.0

Since the 0.1 release we've added a number of features, and are blessing the latest version as 1.0.

Amazon introduced VPC Flow Logs last June, which have become an important source of network data for Observable. In August we released the first version of our command line tool and Python library for working with VPC Flow Logs, flowlogs-reader. Since the 0.1 release we've added a number of features, and are blessing the latest version as 1.0. It's a small project, but makes working with flow logs programmatically a snap.

You can install flowlogs-reader from the Python Package Index with pip install flowlogs-reader . As always, this will install the command line tool, which allows for quick access to flow logs:

$ flowlogs_reader flowlog_group print 5
2 234567890123 eni-efac0123 198.51.100.17 192.0.2.71 10574 22 6 12 1687 1466366527 1466366583 ACCEPT OK
2 234567890123 eni-efac0123 192.0.2.71 204.2.134.162 123 123 17 1 76 1466366527 1466366583 ACCEPT OK
2 234567890123 eni-efac0123 203.0.113.218 10.0.0.71 56674 22 6 15 1843 1466366527 1466366583 ACCEPT OK
2 234567890123 eni-efac0123 192.0.2.71 198.51.100.17 22 10574 6 14 3330 1466366527 1466366583 ACCEPT OK
2 234567890123 eni-efac0123 203.0.113.162 192.0.2.71 123 123 17 1 76 1466366527 1466366583 ACCEPT OK

The latest version also allows for aggregation over time from the command line:

$ flowlogs_reader flowlog_group aggregate
bytes    dstaddr    dstport    end    packets    protocol    srcaddr    srcport    start
3045    203.0.113.20    59606    2016-06-19 20:37:04    11    6    192.0.2.71    22    2016-06-19 20:36:15
228    192.0.2.71    123    2016-06-19 20:42:04    3    17    198.51.100.151    123    2016-06-19 20:07:04
1791    192.0.2.72    22    2016-06-19 20:37:50    14    6    203.0.113.227    53051    2016-06-19 20:36:50
3349    203.0.113.17    22909    2016-06-19 20:18:03    13    6    192.0.2.71    22    2016-06-19 20:16:04
78    192.0.2.108    137    2016-06-19 20:20:27    1    17    198.51.100.210    50591    2016-06-19 20:20:07

The library and command line tool have benefited from contributions from a handful of community members. The latest version has support for:

  • boto3 configuration profiles and cross-account log access
  • Server-side filtering and record limiting
  • CPython 3.5 and PyPy (in addition to CPython 2.7 and 3.4)

It can also be used alongside our more recent kinesis-logs-reader tool for working with VPC Flow Logs in Kinesis. The source for both tools is on our GitHub page.

So far this summer we've been to AWS Meetups in Seattle and Boston talking about VPC Flow Logs and flowlogs-reader. We'll be at other meetups in Denver and Vancouver as well. Check those out, and let us know what you'd like in the next release.



Download A New Way to Look at AWS Security whitepaper.

Download White Paper


Detect Threats Faster – Start Your Free, No-Risk Trial