Disclaimers as before!
It’s not been a great few weeks for data leakage, the most worrying was likely the Accenture Cloud Platform issue where as discussed in the last article, its a really, really bad idea to leave the private encryption keys just lying around.
Fortunately (its hoped) for Accenture, that issue was found by a security consultancy who informed them, giving them time to make the bucket private before someone with less honest motives walked off with them and the data.
And as discussed previously, its extremely unlikely that this data was leaked deliberately, just someone making a typo or clicking on the wrong thing, and almost always done with the best of intentions.
Not that matters if some black hats get a hold of your most critical data.
Knowing its happened
One of the greatest benefits of AWS is how easy it is to integrate the different services together, and to create a monitoring solution around this we will need to add a few additional services to the mix.
CloudTrail — This is Amazon’s audit control plane, almost all API calls (some list operations don’t tend to turn up), and so if someone changes a policy or access control on a bucket, it will be recorded here.
CloudWatch — The AWS monitoring system, we will be sending the events from CloudTrail here to create a report on them, at which point when the threshold for an alert is breached we will send an alert to the…
SNS — The Simple Notification Service, this is part of Amazon’s messaging suite that allows you to send notifications via a Publish/Subscribe mechanism to emails, SMS numbers and so forth.
CloudTrail
In the CloudTrail console, we are going to create a new trail for S3 events like so -
As you can see we are recording down to the individual object level in the S3 buckets to see the API calls even there.
We will need to send the trail into a CloudWatch Log Group where we can create an alert, this is done a bit further down the page -
With this all the events passing through the trail will be filtered via CloudWatch to allow us to monitor for specific behavior.
The events we see in the Trail look like this -
This shows (in a slightly edited manner) that a user in the account has changed the bucket policy of the topsecret bucket, now if this is an approved activity with supporting change and is being checked — no problem, your oncall security incident team know about it.
However if it isn’t, well it needs investigation!
Into CloudWatch
Going to the CloudWatch/Logs we will see the LogGroup as shown below, and we now want to create a Metric Filter based on the behavior we need to monitor
The filter definition we will create is below, and is specifically orientated towards changes of behavior of the monitored buckets
{ ($.eventSource = s3.amazonaws.com) && (($.eventName = PutBucketAcl) || ($.eventName = PutBucketPolicy) || ($.eventName = PutBucketCors) || ($.eventName = DeleteBucketPolicy) || ($.eventName = DeleteBucketCors) ) }We can then assign this to a metric and put in some sensible names
Next, we have the option to review, and just as importantly, create an alarm
We give the alarm a name, and in this example triggering the alarm if so much a single event comes through for in this case a duration of 5 minutes, where all the filtered objects are summed together potentially getting the metric far higher than one (which might not be a good idea).
We can also directly add in an action here, which would be carried out by SNS, the action can be an autoscaling event, such as in the event of a service becoming overloaded, but we need to be notified -
This will automatically setup an SNS topic, but for completeness we will go through the SNS setup process below. This says send a message to my personal email address whenever the state of the metric becomes ALARM.
Any subscriptions to any SNS topic require that the email address, phone number etc be validated before becoming active.
I would strongly suggest that even the smallest of companies have a more robust response plan!
And we now have our monitoring system for S3!
CloudTrail takes up to 15 minutes to update, coupled with CloudWatch being configured for a 5 minute threshold gives a 20 minute alert resolution.
Couple this with your response times to the event and the change to the bucket policy may be active for well over an hour, however you know, and you can do something about it beyond waiting for (hopefully) a security research firm to tell you there is a problem.
Simple Notification Service
SNS is Amazon’s gateway to the wider world, and as shown above when we configured it through the CloudWatch console, it is trivial to setup.
In the Metric above, we created a Topic — ‘S3_Alerts’ that sent an email, however we can send an alert — the Subscription — by several other protocols including SMS -
HTTP(S) and Lambda both allow an easy mechanism to integrate with any monitoring/ticketing software to ensure proper tracking of the incident, and all of these protocols can be stacked to provide a full capability.
With Lambda you could even run a function that automatically makes all buckets in the account private (possibly the topic of a future article).
Likewise additional Topics could be created, if the issue hasn’t been resolved in say 2 hours, send an alert to an escalation point, and on and on to ensure it gets the attention it needs.
We can even send push notification alerts via Application integration, which includes both Android and Apple devices — however your oncall/ops centre works, there is a solution here.
Proof
Just before starting to write the above section, I changed the permissions on the Top-Secret bucket to public, just to show that it does work
And of course the email notification
Conclusion
What has been demonstrated in this series of articles is a whole suite of methods, some process orientated, others purely technical, that can help prevent any data leaks and awkward conversations with the Information Commissioner, or your CEO having a very difficult time at a government hearing.
But this is also the easy, and to be frank, fun stuff. Heck Amazon make it incredibly easy to plug things together and technology is fun!
Let me re-iterate, the Technology is the easy part! Believe me I wouldn’t have written these articles if I wasn’t enjoying doing them!!
The hard part is the ‘soft’ side of things, the processes and responses to events, the categorisation of the data into public, private and worse (although Amazon Macie might help here, its not a service that I currently know well enough to say), and ensuring the validation of all the configurations and that they are doing what they claim (although again there are ‘rumors’ of something in this space for re:invent).
Security is hard, it requires according to the legendary security researcher Joanna Rutkowska an ‘unlovely personality’ and it is the easiest thing in the world to think you’ve got right, and then have an unpleasant surprise.
However increasingly security of your customers data is becoming vital, five years ago it would have been met with a ‘oops, sorry’, now with identity theft and other forms of cyber-crime, data leakage incidents will cause significant damage to your business.
Some might suggest that this is reason enough not to use public clouds, but public clouds are just the latest platform where simple mistakes, poor training and bad practice have massively unintended consequences.
At least with AWS, and their competitors, you have a chance of avoiding and mitigating the damage.
