Fork me on GitHub
#aws
<
2016-01-17
>
cfleming07:01:44

@ricardo: Sounds great. Here’s what I’m planning to test. I’m going to set up a bunch of different lambda functions, all running the same code. I’ll call them at different intervals: every 30 sec, 1 min, 5 min, 15 min and see how that affects instance startup time. I’m also going to add some metrics to the lambda function itself to see if I can see how that affects the first access to an AWS service, since that’s my current problem (not instance startup).

cfleming07:01:00

I’ll do that in both Java and JS.

cfleming07:01:26

Any other variables? Assigned memory, perhaps? Any suggestions for more data people would like to see while I’m at it?

cfleming07:01:05

I’ll be driving all this from the API gateway for now - later I’m going to test other invocation methods to see if that affects things.

ricardo08:01:50

@cfleming: Assigned memory is definitely a factor. On the same low-memory-use operation, it’s made a significant difference on warm-up time for a Clojure lambda, since AWS also increases CPU “proportionally” (unclear what those proportions are).

ricardo08:01:39

So even if start up time is not your problem, it may have an effect on processing the response.

cfleming08:01:37

@ricardo: Ok, I’ll test that too. I’m going to use straight Java and JS to test JVM vs Node without complicating things with Clojure’s startup time.

cfleming09:01:24

@ricardo: FYI, this is the sort of thing I’m seeing at the moment for the licence generation:

cfleming09:01:30

Intervals: 139, 7159, 0, 975, 0, 786, 71, 445, 145, 2840
Total: 12564
END RequestId: 91ed667e-bce7-11e5-9160-0f34ca00b376
REPORT RequestId: 91ed667e-bce7-11e5-9160-0f34ca00b376	Duration: 12658.36 ms	Billed Duration: 12700 ms 	Memory Size: 512 MB	Max Memory Used: 122 MB	

cfleming09:01:17

Those intervals are in ms, arbitrarily chosen as “significant” parts of the processing.

cfleming09:01:37

The 7159 is the code that reads a single record from DynamoDB after making the initial connection to it.

cfleming09:01:35

stamp[1] = System.nanoTime();

      AmazonDynamoDBClient dynamoClient = new AmazonDynamoDBClient();

      dynamoClient.setRegion(Region.getRegion(Regions.US_EAST_1));
      DynamoDB dynamoDB = new DynamoDB(dynamoClient);

      Table orderTable = dynamoDB.getTable("LicenceOrder");

      Item item = orderTable.getItem("orderId", request.orderId);
      if (item != null) {
        logger.log("Duplicate request for order ID " + request.orderId);
        return "ok";
      } else {
        stamp[2] = System.nanoTime();

cfleming09:01:15

975 is initialising the KMS client and decrypting a short block

cfleming09:01:12

71 and 145 are writing one data element each to DynamoDB

cfleming09:01:36

The 2840 is sending an email using Mailgun

cfleming09:01:47

Notice that even with a ridiculously bad response time, the difference between the entire time in the actual function and the total request duration is only about 100ms.

cfleming09:01:57

i.e. spinup time doesn’t seem to be a major factor.

cfleming09:01:16

I can’t find the corresponding event in CloudWatch for the API gateway log - CloudWatch is hilariously bad.

ricardo10:01:54

Man, no kidding. It might as well not exist.

ricardo10:01:23

Most of my reactions are “um… that’s a nice graph. What’s it showing again?"

ricardo10:01:05

Are you using API Gateway? Or directly invoking the lambda?

cfleming20:01:19

@ricardo: API Gateway, so I have the problem that it has an enforced 10sec timeout.

cfleming20:01:06

Probably 50% or more of my calls time out, so I just made it idempotent so people didn’t get two licences with every order, and the payment processor just retries the webhook until it works.

cfleming20:01:12

It’s pretty awful, actually.

clojuregeek23:01:35

do you guys know of a library for clojurescript to access ec2? and s3?

bhagany23:01:05

@clojuregeek: I use the official javascript SDK, no wrapper