@ricardo: @alandipert: @ragge: Here are some more numbers. I ran the tests again, with a 1.5GB JVM. The CPU speed is proportional to the RAM, so that's as fast as you can make them. It made a pretty big difference:

1 min

  Interval 1   Interval 2   Interval 3      Total   Duration        RTT
        1825         1098           27       2950       3083       6036
          29          114           38        181        182        205
           7           73           42        122        122        146
        2209         1038           81       3328       3456       6070
           7           67           30        104        104        144
           7           65           31        103        104        139
          13           89           24        126        126        161
          61           61           27        149        149        217
           7           45           21         73         73         92
           7           75           37        119        119        149

5 min

  Interval 1   Interval 2   Interval 3      Total   Duration        RTT
        1710         1218          100       3028       3102       4944
          39           60           27        126        126        253
           7           63           30        100        101        219
        1590         1039           53       2682       2763       3937
          72           97           24        193        192        332
           6           76           34        116        116        264
           6           60           22         88         88        207
           8           68           24        100         99        241
           7           47           30         84         85        236
           9           61           37        107        107        246

15 min

  Interval 1   Interval 2   Interval 3      Total   Duration        RTT
        2406         1281           43       3730       3775       6164
        1995          867           69       2931       3049       5186
        1697         1086           38       2821       2864       4386
          22          116           14        152        152        287
        4340         1671           55       6066       6181      14309
           6           69           32        107        108        220
          43           78           26        147        147        305
           7           85           13        105        104        242
           6           53           51        110        110        274
        2065          947           53       3065       3165       5082

Much better numbers, but still pretty large RTTs given that it's not actually doing much. This time I ran the client on an m4.xlarge instance to get the fastest networking they offer for vanilla EC2, to see if that made any difference to the network RTT - not much. We're still looking at ~120ms minimum for a lambda call, looks like.

I then ran the results on Node - these numbers are with a 128MB Lambda. The intervals are a bit different here because of differences in the way the API works - there's no getTable, so presumably that's implicitly within Interval 2 instead of 1:

1 min

  Interval 1   Interval 2   Interval 3      Total   Duration        RTT
        1906          703          220       2829       2830       3909
        2442          744          297       3483       3484       4795
           1          236          136        373        374        396
           0          103          123        226        241        264
           0          262          126        388        400        459
           0           80          147        227        249        289
           0           65          149        214        214        259
           1          100          139        240        257        308
           0           64          102        166        184        201
           1           79          140        220        234        287

5 min

  Interval 1   Interval 2   Interval 3      Total   Duration        RTT
        1799          804          277       2880       2897       3702
          17          332           99        448        449        568
           0           80          151        231        244        347
        1323          700          258       2281       2389       3603
           1          240          128        369        368        530
           0           80          143        223        229        363
           1          139           90        230        230        361
           0           61          120        181        186        338
           1          120          139        260        262        429
           1           81          118        200        208        334

15 min

  Interval 1   Interval 2   Interval 3      Total   Duration        RTT
        1623          820          299       2742       2796       3492
        1501          720          240       2461       2498       2754
           1          240          159        400        405        534
           1          240          120        361        367        490
           0          151          129        280        293        464
           0           94          107        201        201        362
        2081          761          277       3119       3125       4536
           1           85          114        200        215        382
        1943          740          239       2922       2942       4085
           1          241          129        371        388        531

These are dramatically better than the 512MB Lambda, and are even better than the 1.5GB Lambda, when things are cold. This isn't just instance spinup time, which is much less for Node than the JVM, as expected, but also just the cost of accessing the first AWS service. The only thing I can think of is that the JVM takes a big hit for classloading - this isn't enough code to even hit the JIT, I'm guessing. This costs 8% of the 1.5GB Lambda to run.

Here's Node on the 1.5GB Lambda:

1 min

  Interval 1   Interval 2   Interval 3      Total   Duration        RTT
         162           89           47        298        299        979
         181           78           53        312        313        438
           0           48           37         85         86        126
           0           33           31         64         64         97
           1           26           33         60         60         90
           1           36           39         76         76        107
           0           27           39         66         66        103
           1           30           31         62         62         88
           1           38           31         70         70        123
           0           25           37         62         62        121

5 min

  Interval 1   Interval 2   Interval 3      Total   Duration        RTT
         131           75           41        247        248       1065
           1           43           34         78         78        171
           0           30           34         64         64        168
         235           80           43        358        359        838
           0           37           45         82         82        215
           1           36           39         76         76        322
           1           34           39         74         73        188
           1           32           29         62         62        369
           1           28           35         64         64        280

15 min

  Interval 1   Interval 2   Interval 3      Total   Duration        RTT
         411           63           36        510        511       2222
         140           74           40        254        255        903
           1           38           30         69         69        202

I wanted to shut down the large instance before going to bed so it doesn't have all the data, but you can see where this is going - it's way faster than any of the JVM options. Basically, unless you need to use a JVM API, or you're going to be running a long CPU-intensive process, I'd use Node for pretty much everything.

I should really write this up in a blog, since the vengeful Slack god will have eaten it by this time tomorrow.

@cfleming: thanks again for sharing this, very useful

@cfleming: i think publishing the results in some form would be helpful, I could get our AWS account manager (or whatever it's called) to take a look and see if he can explain your findings

@cfleming: they've been quite responsive with queries like this before

@ragge: Ok, I’ll try to do that soon.

Thanks again for sharing @cfleming very interesting