Mark Needham

Thoughts on Software Development

Archive for the ‘serverless’ tag

Serverless: Building a mini producer/consumer data pipeline with AWS SNS

without comments

I wanted to create a little data pipeline with Serverless whose main use would be to run once a day, call an API, and load that data into a database.

It’s mostly used to pull in recent data from that API, but I also wanted to be able to invoke it manually and specify a date range.

I created the following pair of lambdas that communicate with each other via an SNS topic.

The code

serverless.yml

service: marks-blog

frameworkVersion: ">=1.2.0 <2.0.0"

provider:
  name: aws
  runtime: python3.6
  timeout: 180
  iamRoleStatements:
    - Effect: 'Allow'
      Action:
        - "sns:Publish"
      Resource:
        - ${self:custom.BlogTopic}

custom:
  BlogTopic:
    Fn::Join:
      - ":"
      - - arn
        - aws
        - sns
        - Ref: AWS::Region
        - Ref: AWS::AccountId
        - marks-blog-topic

functions:
  message-consumer:
    name: MessageConsumer
    handler: handler.consumer
    events:
      - sns:
          topicName: marks-blog-topic
          displayName: Topic to process events
  message-producer:
    name: MessageProducer
    handler: handler.producer
    events:
      - schedule: rate(1 day)

handler.py

import boto3
import json
import datetime
from datetime import timezone
 
def producer(event, context):
    sns = boto3.client('sns')
 
    context_parts = context.invoked_function_arn.split(':')
    topic_name = "marks-blog-topic"
    topic_arn = "arn:aws:sns:{region}:{account_id}:{topic}".format(
        region=context_parts[3], account_id=context_parts[4], topic=topic_name)
 
    now = datetime.datetime.now(timezone.utc)
    start_date = (now - datetime.timedelta(days=1)).strftime("%Y-%m-%d")
    end_date = now.strftime("%Y-%m-%d")
 
    params = {"startDate": start_date, "endDate": end_date, "tags": ["neo4j"]}
 
    sns.publish(TopicArn= topic_arn, Message= json.dumps(params))
 
 
def consumer(event, context):
    for record in event["Records"]:
        message = json.loads(record["Sns"]["Message"])
 
        start_date = message["startDate"]
        end_date = message["endDate"]
        tags = message["tags"]
 
        print("start_date: " + start_date)
        print("end_date: " + end_date)
        print("tags: " + str(tags))

Trying it out

We can simulate a message being received locally by executing the following command:

$ serverless invoke local \
    --function message-consumer \
    --data '{"Records":[{"Sns": {"Message":"{\"tags\": [\"neo4j\"], \"startDate\": \"2017-09-25\", \"endDate\": \"2017-09-29\"  }"}}]}'
 
start_date: 2017-09-25
end_date: 2017-09-29
tags: ['neo4j']
null

That seems to work fine. What about if we invoke the message-producer on AWS?

$ serverless invoke --function message-producer
 
null

Did the consumer received the message?

$ serverless logs --function message-consumer
 
START RequestId: 0ef5be87-a5b1-11e7-a905-f1387e68c65f Version: $LATEST
start_date: 2017-09-29
end_date: 2017-09-30
tags: ['neo4j']
END RequestId: 0ef5be87-a5b1-11e7-a905-f1387e68c65f
REPORT RequestId: 0ef5be87-a5b1-11e7-a905-f1387e68c65f	Duration: 0.46 ms	Billed Duration: 100 ms 	Memory Size: 1024 MB	Max Memory Used: 32 MB

Looks like it! We can also invoke the consumer directly on AWS:

$ serverless invoke \
    --function message-consumer \
    --data '{"Records":[{"Sns": {"Message":"{\"tags\": [\"neo4j\"], \"startDate\": \"2017-09-25\", \"endDate\": \"2017-09-26\"  }"}}]}'
 
null

And now if we check the consumer’s logs we’ll see both messages:

$ serverless logs --function message-consumer
 
START RequestId: 0ef5be87-a5b1-11e7-a905-f1387e68c65f Version: $LATEST
start_date: 2017-09-29
end_date: 2017-09-30
tags: ['neo4j']
END RequestId: 0ef5be87-a5b1-11e7-a905-f1387e68c65f
REPORT RequestId: 0ef5be87-a5b1-11e7-a905-f1387e68c65f	Duration: 0.46 ms	Billed Duration: 100 ms 	Memory Size: 1024 MB	Max Memory Used: 32 MB	
 
START RequestId: 4cb42bc9-a5b1-11e7-affb-99fa6b4dc3ed Version: $LATEST
start_date: 2017-09-25
end_date: 2017-09-26
tags: ['neo4j']
END RequestId: 4cb42bc9-a5b1-11e7-affb-99fa6b4dc3ed
REPORT RequestId: 4cb42bc9-a5b1-11e7-affb-99fa6b4dc3ed	Duration: 16.46 ms	Billed Duration: 100 ms 	Memory Size: 1024 MB	Max Memory Used: 32 MB

Success!

Written by Mark Needham

September 30th, 2017 at 7:51 am

Serverless: S3 – S3BucketPermissions – Action does not apply to any resource(s) in statement

without comments

I’ve been playing around with S3 buckets with Serverless, and recently wrote the following code to create an S3 bucket and put a file into that bucket:

const AWS = require("aws-sdk");
 
let regionParams = { 'region': 'us-east-1' }
let s3 = new AWS.S3(regionParams);
 
let s3BucketName = "marks-blog-bucket";
 
console.log("Creating bucket: " + s3BucketName);
let bucketParams = { Bucket: s3BucketName, ACL: "public-read" };
 
s3.createBucket(bucketParams).promise()
  .then(console.log)
  .catch(console.error);
 
var putObjectParams = {
    Body: "<html><body><h1>Hello blog!</h1></body></html>",
    Bucket: s3BucketName,
    Key: "blog.html"
   };
 
s3.putObject(putObjectParams).promise()
  .then(console.log)
  .catch(console.error);

When I tried to cURL the file I got a permission denied exception:

$ curl --head --silent https://s3.amazonaws.com/marks-blog-bucket/blog.html
HTTP/1.1 403 Forbidden
x-amz-request-id: 512FE36798C0BE4D
x-amz-id-2: O1ELGMJ0jjs11WCrNiVNF2z2ssRgtO4+M4H2QQB5025HjIpC54VId0eKZryYeBYN8Pvb8GsolTQ=
Content-Type: application/xml
Transfer-Encoding: chunked
Date: Fri, 29 Sep 2017 05:42:03 GMT
Server: AmazonS3

I wrote the following code to try and have Serverless create a bucket policy that would make all files in my bucket publicly accessible:

serverless.yml

service: marks-blog

frameworkVersion: ">=1.2.0 <2.0.0"

provider:
  name: aws
  runtime: python3.6
  timeout: 180

resources:
  Resources:
    S3BucketPermissions:
      Type: AWS::S3::BucketPolicy
      Properties:
        Bucket: marks-blog-bucket
        PolicyDocument:
          Statement:
            - Principal: "*"
              Action:
                - s3:GetObject
              Effect: Allow
              Sid: "AddPerm"
              Resource: arn:aws:s3:::marks-blog-bucket
 
...

Let’s try to deploy it:

./node_modules/serverless/bin/serverless deploy
Serverless: Packaging service...
Serverless: Excluding development dependencies...
Serverless: Uploading CloudFormation file to S3...
Serverless: Uploading artifacts...
Serverless: Uploading service .zip file to S3 (1.3 KB)...
Serverless: Validating template...
Serverless: Updating Stack...
Serverless: Checking Stack update progress...
........
Serverless: Operation failed!
 
  Serverless Error ---------------------------------------
 
  An error occurred: S3BucketPermissions - Action does not apply to any resource(s) in statement.

D’oh! That didn’t do what I expected.

I learnt that this message means:

Some services do not let you specify actions for individual resources; instead, any actions that you list in the Action or NotAction element apply to all resources in that service. In these cases, you use the wildcard * in the Resource element.

To fix it we need to use the wildcard * to indicate that the s3:GetObject permission should apply to all values in the bucket rather than to the bucket itself.

Take 2!

serverless.yml

service: marks-blog

frameworkVersion: ">=1.2.0 <2.0.0"

provider:
  name: aws
  runtime: python3.6
  timeout: 180

resources:
  Resources:
    S3BucketPermissions:
      Type: AWS::S3::BucketPolicy
      Properties:
        Bucket: marks-blog-bucket
        PolicyDocument:
          Statement:
            - Principal: "*"
              Action:
                - s3:GetObject
              Effect: Allow
              Sid: "AddPerm"
              Resource: arn:aws:s3:::marks-blog-bucket/*
 
...

Let’s deploy again and try to access the file:

$ curl --head --silent https://s3.amazonaws.com/marks-blog-bucket/blog.html
HTTP/1.1 200 OK
x-amz-id-2: uGwsLLoFHf+slXADGYkqW0bLfQ7EPG/kqzV3l2k7SMex4NlMEpNsNN/cIC9INLPohDtVFwUAa90=
x-amz-request-id: 7869E21760CD50F1
Date: Fri, 29 Sep 2017 06:05:11 GMT
Last-Modified: Fri, 29 Sep 2017 06:01:33 GMT
ETag: "57bac87219812c2f9a581943da34cfde"
Accept-Ranges: bytes
Content-Type: application/octet-stream
Content-Length: 46
Server: AmazonS3

Success! And if we check in the AWS console we can see that the bucket policy has been applied to our bucket:

2017 09 29 07 06 13

Written by Mark Needham

September 29th, 2017 at 6:09 am

Posted in Software Development

Tagged with , ,

Serverless: AWS HTTP Gateway – 502 Bad Gateway

without comments

In my continued work with Serverless and AWS Lambda I ran into a problem when trying to call a HTTP gateway.

My project looked like this:

serverless.yaml

service: http-gateway
 
frameworkVersion: ">=1.2.0 <2.0.0"
 
provider:
  name: aws
  runtime: python3.6
  timeout: 180
 
functions:
  no-op:
      name: NoOp
      handler: handler.noop
      events:
        - http: POST noOp

handler.py

def noop(event, context):
    return "hello"

Let’s deploy to AWS:

$ serverless  deploy
Serverless: Packaging service...
Serverless: Excluding development dependencies...
Serverless: Uploading CloudFormation file to S3...
Serverless: Uploading artifacts...
Serverless: Uploading service .zip file to S3 (179 B)...
Serverless: Validating template...
Serverless: Updating Stack...
Serverless: Checking Stack update progress...
..............
Serverless: Stack update finished...
Service Information
service: http-gateway
stage: dev
region: us-east-1
api keys:
  None
endpoints:
  POST - https://29nb5rmmd0.execute-api.us-east-1.amazonaws.com/dev/noOp
functions:
  no-op: http-gateway-dev-no-op

And now we’ll try and call it using cURL:

$ curl -X POST https://29nb5rmmd0.execute-api.us-east-1.amazonaws.com/dev/noOp
{"message": "Internal server error"}

That didn’t work so well, what do the logs have to say?

$ serverless  logs --function no-op	
START RequestId: 64ab69b0-7d8f-11e7-9db5-13b228cd4cb6 Version: $LATEST
END RequestId: 64ab69b0-7d8f-11e7-9db5-13b228cd4cb6
REPORT RequestId: 64ab69b0-7d8f-11e7-9db5-13b228cd4cb6	Duration: 0.27 ms	Billed Duration: 100 ms 	Memory Size: 1024 MB	Max Memory Used: 21 MB

So the function is completely fine. It turns out I’m not very good at reading the manual and should have been returning a map instead of a string:

API Gateway expects to see a json map with keys “body”, “headers”, and “statusCode”.

Let’s update our handler function and re-deploy.

def noop(event, context):
    return {
        "body": "hello",
        "headers": {},
        "statusCode": 200
        }

Now we’re ready to try the endpoint again:

$ curl -X POST https://29nb5rmmd0.execute-api.us-east-1.amazonaws.com/dev/noOp
hello

Much better!

Written by Mark Needham

August 11th, 2017 at 4:01 pm

Serverless: Python – virtualenv – { “errorMessage”: “Unable to import module ‘handler'” }

without comments

I’ve been using the Serverless library to deploy and run some Python functions on AWS lambda recently and was initially confused about how to handle my dependencies.

I tend to create a new virtualenv for each of my project so let’s get that setup first:

Prerequisites

$ npm install serverless
$ virtualenv -p python3 a
$ . a/bin/activate

Now let’s create our Serverless project. I’m going to install the requests library so that I can use it in my function.

My Serverless project

serverless.yaml

service: python-starter-template

frameworkVersion: ">=1.2.0 <2.0.0"

provider:
  name: aws
  runtime: python3.6
  timeout: 180

functions:
  starter-function:
      name: Starter
      handler: handler.starter

handler.py

import requests
 
def starter(event, context):
    print("event:", event, "context:", context)
    r = requests.get("http://www.google.com")
    print(r.status_code)
$ pip install requests

Ok, we’re now ready to try out the function. A nice feature of Serverless is that it lets us try out functions locally before we deploy them onto one of the Cloud providers:

$ ./node_modules/serverless/bin/serverless invoke local --function starter-function
event: {} context: <__main__.FakeLambdaContext object at 0x10bea9a20>
200
null

So far so good. Next we’ll deploy our function to AWS. I’m assuming you’ve already got your credentials setup but if not you can follow the tutorial on the Serverless page.

$ ./node_modules/serverless/bin/serverless deploy
Serverless: Packaging service...
Serverless: Excluding development dependencies...
Serverless: Uploading CloudFormation file to S3...
Serverless: Uploading artifacts...
Serverless: Uploading service .zip file to S3 (26.48 MB)...
Serverless: Validating template...
Serverless: Updating Stack...
Serverless: Checking Stack update progress...
.........
Serverless: Stack update finished...
Service Information
service: python-starter-template
stage: dev
region: us-east-1
api keys:
  None
endpoints:
  None
functions:
  starter-function: python-starter-template-dev-starter-function

Now let’s invoke our function:

$ ./node_modules/serverless/bin/serverless invoke --function starter-function
{
    "errorMessage": "Unable to import module 'handler'"
}
 
  Error --------------------------------------------------
 
  Invoked function failed
 
     For debugging logs, run again after setting the "SLS_DEBUG=*" environment variable.
 
  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Forums:        forum.serverless.com
     Chat:          gitter.im/serverless/serverless
 
  Your Environment Information -----------------------------
     OS:                     darwin
     Node Version:           6.7.0
     Serverless Version:     1.19.0

Hmmm, that’s odd – I wonder why it can’t import our handler module? We can call the logs function to check. The logs are usually a few seconds behind so we’ll have to be a bit patient if we don’t see them immediately.

$ ./node_modules/serverless/bin/serverless logs  --function starter-function
START RequestId: 735efa84-7ad0-11e7-a4ef-d5baf0b46552 Version: $LATEST
Unable to import module 'handler': No module named 'requests'
 
END RequestId: 735efa84-7ad0-11e7-a4ef-d5baf0b46552
REPORT RequestId: 735efa84-7ad0-11e7-a4ef-d5baf0b46552	Duration: 0.42 ms	Billed Duration: 100 ms 	Memory Size: 1024 MB	Max Memory Used: 22 MB

That explains it – the requests module wasn’t imported.

If we look in .serverless/python-starter-template.zip

we can see that the requests module is hidden inside the a directory and the instance of Python that runs on Lambda doesn’t know where to find it.

I’m sure there are other ways of solving this but the easiest one I found is a Serverless plugin called serverless-python-requirements.

So how does this plugin work?

A Serverless v1.x plugin to automatically bundle dependencies from requirements.txt and make them available in your PYTHONPATH.

Doesn’t sound too tricky – we can use pip freeze to get our list of requirements and write them into a file. Let’s rework serverless.yaml to make use of the plugin:

My Serverless project using serverless-python-requirements

$ npm install --save serverless-python-requirements
$ pip freeze > requirements.txt
$ cat requirements.txt 
certifi==2017.7.27.1
chardet==3.0.4
idna==2.5
requests==2.18.3
urllib3==1.22

serverless.yaml

service: python-starter-template

frameworkVersion: ">=1.2.0 <2.0.0"

provider:
  name: aws
  runtime: python3.6
  timeout: 180

plugins:
  - serverless-python-requirements

functions:
  starter-function:
      name: Starter
      handler: handler.starter

package:
  exclude:
    - a/** # virtualenv

We have two changes from before:

  • We added the serverless-python-requirements plugin
  • We excluded the a directory since we don’t need it

Let’s deploy again and run the function:

$ ./node_modules/serverless/bin/serverless deploy
Serverless: Parsing Python requirements.txt
Serverless: Installing required Python packages for runtime python3.6...
Serverless: Linking required Python packages...
Serverless: Packaging service...
Serverless: Excluding development dependencies...
Serverless: Unlinking required Python packages...
Serverless: Uploading CloudFormation file to S3...
Serverless: Uploading artifacts...
Serverless: Uploading service .zip file to S3 (14.39 MB)...
Serverless: Validating template...
Serverless: Updating Stack...
Serverless: Checking Stack update progress...
.........
Serverless: Stack update finished...
Service Information
service: python-starter-template
stage: dev
region: us-east-1
api keys:
  None
endpoints:
  None
functions:
  starter-function: python-starter-template-dev-starter-function
$ ./node_modules/serverless/bin/serverless invoke --function starter-function
null

Looks good. Let’s check the logs:

$ ./node_modules/serverless/bin/serverless logs --function starter-function
START RequestId: 61e8eda7-7ad4-11e7-8914-03b8a7793a24 Version: $LATEST
event: {} context: <__main__.LambdaContext object at 0x7f568b105f28>
200
END RequestId: 61e8eda7-7ad4-11e7-8914-03b8a7793a24
REPORT RequestId: 61e8eda7-7ad4-11e7-8914-03b8a7793a24	Duration: 55.55 ms	Billed Duration: 100 ms 	Memory Size: 1024 MB	Max Memory Used: 29 M

All good here as well so we’re done!

Written by Mark Needham

August 6th, 2017 at 7:03 pm

Posted in Software Development

Tagged with ,