Mark Needham

Thoughts on Software Development

Archive for the ‘lambda’ tag

AWS Lambda: Programmatically scheduling a CloudWatchEvent

without comments

I recently wrote a blog post showing how to create a Python ‘Hello World’ AWS lambda function and manually invoke it, but what I really wanted to do was have it run automatically every hour.

To achieve that in AWS Lambda land we need to create a CloudWatch Event. The documentation describes them as follows:

Using simple rules that you can quickly set up, you can match events and route them to one or more target functions or streams.

2017 04 05 23 06 36

This is actually really easy from the Amazon web console as you just need to click the ‘Triggers’ tab and then ‘Add trigger’. It’s not obvious that there are actually three steps are involved as they’re abstracted from you.

So what are the steps?

  1. Create rule
  2. Give permission for that rule to execute
  3. Map the rule to the function

I forgot to do step 2) initially and then you just end up with a rule that never triggers, which isn’t particularly useful.

The following code creates a ‘Hello World’ lambda function and runs it once an hour:

import boto3
 
lambda_client = boto3.client('lambda')
events_client = boto3.client('events')
 
fn_name = "HelloWorld"
fn_role = 'arn:aws:iam::[your-aws-id]:role/lambda_basic_execution'
 
fn_response = lambda_client.create_function(
    FunctionName=fn_name,
    Runtime='python2.7',
    Role=fn_role,
    Handler="{0}.lambda_handler".format(fn_name),
    Code={'ZipFile': open("{0}.zip".format(fn_name), 'rb').read(), },
)
 
fn_arn = fn_response['FunctionArn']
frequency = "rate(1 hour)"
name = "{0}-Trigger".format(fn_name)
 
rule_response = events_client.put_rule(
    Name=name,
    ScheduleExpression=frequency,
    State='ENABLED',
)
 
lambda_client.add_permission(
    FunctionName=fn_name,
    StatementId="{0}-Event".format(name),
    Action='lambda:InvokeFunction',
    Principal='events.amazonaws.com',
    SourceArn=rule_response['RuleArn'],
)
 
events_client.put_targets(
    Rule=name,
    Targets=[
        {
            'Id': "1",
            'Arn': fn_arn,
        },
    ]
)

We can now check if our trigger has been configured correctly:

$ aws events list-rules --query "Rules[?Name=='HelloWorld-Trigger']"
[
    {
        "State": "ENABLED", 
        "ScheduleExpression": "rate(1 hour)", 
        "Name": "HelloWorld-Trigger", 
        "Arn": "arn:aws:events:us-east-1:[your-aws-id]:rule/HelloWorld-Trigger"
    }
]
 
$ aws events list-targets-by-rule --rule HelloWorld-Trigger
{
    "Targets": [
        {
            "Id": "1", 
            "Arn": "arn:aws:lambda:us-east-1:[your-aws-id]:function:HelloWorld"
        }
    ]
}
 
$ aws lambda get-policy --function-name HelloWorld
{
    "Policy": "{\"Version\":\"2012-10-17\",\"Id\":\"default\",\"Statement\":[{\"Sid\":\"HelloWorld-Trigger-Event\",\"Effect\":\"Allow\",\"Principal\":{\"Service\":\"events.amazonaws.com\"},\"Action\":\"lambda:InvokeFunction\",\"Resource\":\"arn:aws:lambda:us-east-1:[your-aws-id]:function:HelloWorld\",\"Condition\":{\"ArnLike\":{\"AWS:SourceArn\":\"arn:aws:events:us-east-1:[your-aws-id]:rule/HelloWorld-Trigger\"}}}]}"
}

All looks good so we’re done!

Written by Mark Needham

April 5th, 2017 at 11:49 pm

Posted in Software Development

Tagged with , ,

AWS Lambda: Encrypted environment variables

with one comment

Continuing on from my post showing how to create a ‘Hello World’ AWS lambda function I wanted to pass encrypted environment variables to my function.

The following function takes in both an encrypted and unencrypted variable and prints them out.

Don’t print out encrypted variables in a real function, this is just so we can see the example working!

import boto3
import os
 
from base64 import b64decode
 
def lambda_handler(event, context):
    encrypted = os.environ['ENCRYPTED_VALUE']
    decrypted = boto3.client('kms').decrypt(CiphertextBlob=b64decode(encrypted))['Plaintext']
 
    # Don't print out your decrypted value in a real function! This is just to show how it works.
    print("Decrypted value:", decrypted)
 
    plain_text = os.environ["PLAIN_TEXT_VALUE"]
    print("Plain text:", plain_text)

Now we’ll zip up our function into HelloWorldEncrypted.zip, ready to send to AWS.

zip HelloWorldEncrypted.zip HelloWorldEncrypted.py

Now it’s time to upload our function to AWS and create the associated environment variables.

If you’re using a Python editor then you’ll need to install boto3 locally to keep the editor happy but you don’t need to include boto3 in the code you send to AWS Lambda – it comes pre-installed.

Now we write the following code to automate the creation of our Lambda function:

import boto3
from base64 import b64encode
 
fn_name = "HelloWorldEncrypted"
kms_key = "arn:aws:kms:[aws-zone]:[your-aws-id]:key/[your-kms-key-id]"
fn_role = 'arn:aws:iam::[your-aws-id]:role/lambda_basic_execution'
 
lambda_client = boto3.client('lambda')
kms_client = boto3.client('kms')
 
encrypt_me = "abcdefg"
encrypted = b64encode(kms_client.encrypt(Plaintext=encrypt_me, KeyId=kms_key)["CiphertextBlob"])
 
plain_text = 'hijklmno'
 
lambda_client.create_function(
        FunctionName=fn_name,
        Runtime='python2.7',
        Role=fn_role,
        Handler="{0}.lambda_handler".format(fn_name),
        Code={ 'ZipFile': open("{0}.zip".format(fn_name), 'rb').read(),},
        Environment={
            'Variables': {
                'ENCRYPTED_VALUE': encrypted,
                'PLAIN_TEXT_VALUE': plain_text,
            }
        },
        KMSKeyArn=kms_key
)

The tricky bit for me here was figuring out that I needed to pass the value that I wanted to base 64 encode the output of the value encrypted by the KMS client. The KMS client relies on a KMS key that we need to setup. We can see a list of all our KMS keys by running the following command:

$ aws kms list-keys

The format of these keys is arn:aws:kms:[zone]:[account-id]:key/[key-id].

Now let’s try executing our Lambda function from the AWS console:

$ python CreateHelloWorldEncrypted.py

Let’s check it got created:

$ aws lambda list-functions --query "Functions[*].FunctionName"
[
    "HelloWorldEncrypted", 
]

And now let’s execute the function:

$ aws lambda invoke --function-name HelloWorldEncrypted --invocation-type RequestResponse --log-type Tail /tmp/out | jq ".LogResult"
"U1RBUlQgUmVxdWVzdElkOiA5YmNlM2E1MC0xODMwLTExZTctYjFlNi1hZjQxZDYzMzYxZDkgVmVyc2lvbjogJExBVEVTVAooJ0RlY3J5cHRlZCB2YWx1ZTonLCAnYWJjZGVmZycpCignUGxhaW4gdGV4dDonLCAnaGlqa2xtbm8nKQpFTkQgUmVxdWVzdElkOiA5YmNlM2E1MC0xODMwLTExZTctYjFlNi1hZjQxZDYzMzYxZDkKUkVQT1JUIFJlcXVlc3RJZDogOWJjZTNhNTAtMTgzMC0xMWU3LWIxZTYtYWY0MWQ2MzM2MWQ5CUR1cmF0aW9uOiAzNjAuMDQgbXMJQmlsbGVkIER1cmF0aW9uOiA0MDAgbXMgCU1lbW9yeSBTaXplOiAxMjggTUIJTWF4IE1lbW9yeSBVc2VkOiAyNCBNQgkK"

That’s a bit hard to read, some decoding is needed:

$ echo "U1RBUlQgUmVxdWVzdElkOiA5YmNlM2E1MC0xODMwLTExZTctYjFlNi1hZjQxZDYzMzYxZDkgVmVyc2lvbjogJExBVEVTVAooJ0RlY3J5cHRlZCB2YWx1ZTonLCAnYWJjZGVmZycpCignUGxhaW4gdGV4dDonLCAnaGlqa2xtbm8nKQpFTkQgUmVxdWVzdElkOiA5YmNlM2E1MC0xODMwLTExZTctYjFlNi1hZjQxZDYzMzYxZDkKUkVQT1JUIFJlcXVlc3RJZDogOWJjZTNhNTAtMTgzMC0xMWU3LWIxZTYtYWY0MWQ2MzM2MWQ5CUR1cmF0aW9uOiAzNjAuMDQgbXMJQmlsbGVkIER1cmF0aW9uOiA0MDAgbXMgCU1lbW9yeSBTaXplOiAxMjggTUIJTWF4IE1lbW9yeSBVc2VkOiAyNCBNQgkK" | base64 --decode
START RequestId: 9bce3a50-1830-11e7-b1e6-af41d63361d9 Version: $LATEST
('Decrypted value:', 'abcdefg')
('Plain text:', 'hijklmno')
END RequestId: 9bce3a50-1830-11e7-b1e6-af41d63361d9
REPORT RequestId: 9bce3a50-1830-11e7-b1e6-af41d63361d9	Duration: 360.04 ms	Billed Duration: 400 ms 	Memory Size: 128 MB	Max Memory Used: 24 MB

And it worked, hoorah!

Written by Mark Needham

April 3rd, 2017 at 5:49 am

Posted in Software Development

Tagged with ,

C#’s Lambda ForEach: Only on Lists?

with 8 comments

One of my favourite things introduced into C# recently is the new ForEach method which can be applied to (apparently only!) lists.

Last week we had a situation where we wanted to make use of the ForEach method on an IDictionary which we were using to store a collection of Selenium clients.

1
IDictionary<string, ISelenium> seleniumClients = new Dictionary<string, ISelenium>();

We wanted to write a piece of code to exit all of the clients when our tests had completed. We thought the following would do the trick:

1
seleniumClients.Values.ForEach(client => client.Stop());

The problem is that code doesn’t actually compile!

‘seleniumClients.Values’ returns an ICollection which extends IEnumerable so we thought ForEach should be available.

We eventually got around the problem by putting the collection into a list and then applying the ForEach method but it seems like there should be a better way to do this.

1
new List<ISelenium>(seleniumClients.Values).ForEach(client => client.Stop());

Is there?

Written by Mark Needham

December 15th, 2008 at 11:52 pm

Posted in .NET

Tagged with , , ,

Lambda in C#: Conciseness v Readability

with 4 comments

One of the things I really disliked when I first came across C# 3.0 code was lambda functions.

At the time I remember speaking to my Tech Lead and expressing the opinion that they were making the code harder to understand and were valuing conciseness over readability.

After a week of reading about the new C# features and understanding how they worked the code was now more readable to me and a lot of the boiler plate code that I had come to expect was no longer necessary.

My favourite example of the power of lambda is when we want to iterate through a collection of items and apply the same operation to every item in the collection.

In normal C# we might do this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
public class Foo
{
    private String bar;
    private String baz;
 
    public Foo(String bar, String baz)
    {
        this.bar = bar;
        this.baz = baz;
    }
 
    public override string ToString()
    {
        return string.Format("{0} - {1}", bar, baz);
    }
 
}
1
2
3
4
5
6
7
8
9
var foos = new List<Foo>();
foos.Add(new Foo("bar1", "baz1"));
foos.Add(new Foo("bar2", "baz2"));
 
var fooString = new List<String>();
foreach (var foo in foos)
{
    fooString.Add(foo.ToString());
}

Using the power of C# 3.0 we can change that last for each statement to read something like this:

1
var fooString = foos.Select(f => f.ToString());

This is much more concise but I think the judgement on its readability depends on one’s understanding of the language feature.

One idea I am considering trying is using methods which describe more clearly what the lambda function is doing. This is an idea I came across from Kris Kemper’s post about using similar Ruby language features.

In the example I gave perhaps wrapping the foo.Select(…) in a method called ‘ConvertToStringRepresentation()’ might make it more readable – it’s clearly up for debate though.

When I was learning how lambda worked I found it useful to be able to do the comparison of how you would write code without it and being able to compare that with how you could do it with lambda. I also found having some understanding of how Ruby blocks worked made it easier as well.

Clearly having powerful language features means that a language is much easier to abuse but I think if used sensibly then readability and conciseness need not be mutually exclusive.

Written by Mark Needham

November 24th, 2008 at 11:41 pm

Posted in .NET

Tagged with , ,