Fork me on GitHub
#aws
<
2019-10-17
>
kenny17:10:16

If I'm using a shared http-client with Cognitect's aws-api, is it ok to create many client records without calling stop?

ghadi17:10:56

yeah if you're making many aws-api clients, calling stop will bork you

ghadi17:10:09

we're working on this problem actively

ghadi17:10:47

we're trying to slowly divorce ourselves from the current http-client, and have a BYO model

kenny17:10:18

Sorry, so it's ok to do as I said?

ghadi17:10:36

if you are sharing http-clients, you should avoid calling stop

kenny17:10:09

Got it. With that BYO model, seems like this lib could be easily used in CLJS too.

ghadi17:10:30

slow your roll 😉

ghadi17:10:04

there are other concerns for CLJS, like signing routines

ghadi17:10:14

porting to CLJS has not been fully evaluated

kenny17:10:41

Mostly interested in use in AWS Lambda.

ghadi17:10:43

there is a ticket if you'd like to register your interest or a motivating case

ghadi17:10:53

Cool, I suspected so. Is that because of startup time?

ghadi17:10:25

what would be an acceptable startup time for CLJ lambda?

kenny17:10:18

For this particular app, under 10s is ok. Cold starts result in 20-30s execution time with production deps.

kenny17:10:08

Almost everything is written in vanilla clojure so getting it to work in Node would be somewhat easy.

ghadi17:10:32

yeah we can get there, my goal is under 2s for clojure + core.async + aws-api

ghadi17:10:58

can't vouch for external deps, most are pigs

ghadi17:10:39

might need a custom lambda runtime + JDK11

kenny17:10:49

2s would be fantastic. 10s is really pushing the limit for us but would likely be acceptable for a while.

ghadi17:10:06

what are the deps you're pulling in, if you can share?

kenny17:10:52

Pretty much everything in there is either cljc or has a JS client available. Largest missing piece is Datomic. Hoping they publish a protocol definition for the client API so we can have JS clients.

ghadi17:10:25

that is a ton

kenny17:10:44

Yep, tis' the world we live in haha.

ghadi17:10:34

the #1 thing you can do for startup time is to load less code

ghadi17:10:51

datomic in a lambda is super weird

ghadi17:10:13

if you need to connect to Datomic, consider running in an Ion -- free lambda

kenny17:10:20

I actually don't think any one of those deps could be removed unfortunately. The only alternative is removing deps. Removing deps means writing the code ourselves which is sometimes hard to justify the engineering time.

ghadi17:10:51

I just want to paint a realistic image that it is impossible to get fast startup time with bloated deps

kenny17:10:23

Ions are too restrictive. There's no great way to instrument code in them. You're stuck with CW logs (unfortunately same applies with Lambda though). We have several extremely expensive, somewhat long-running ops that would almost certainly have a large impact on the Datomic instances.

ghadi17:10:23

long-running ops -> not a good use case for lambdas either impact on datomic instances -> use a query group

ghadi17:10:36

can you elaborate on code instrumentation?

ghadi17:10:34

using honeycomb?

kenny17:10:20

Long running = 5-10 min highly CPU intensive task. Yes

ghadi17:10:32

seems really appropriate for an Ion, IMHO

ghadi17:10:40

you can use honeycomb within an Ion

ghadi17:10:19

Query Groups allow you to isolate the CPU intensive part so that it's not running on the Primary Group

ghadi17:10:52

but you get the benefit of any queries happening locally instead of across the wire

kenny21:10:08

@ghadi Sorry, had to step out for a while. Couple things there: - We're actually using Datadog APM right now, not honeycomb, but I assumed the usage was similar -- you need to start the jvm process with a java agent. I don't think Ions allow you to change the JVM args. - Running in an Ion requires you to run infrastructure 24/7. A huge benefit to running in Lambda is you only pay for execution time. - The Ion deployment model does not fit in very well with existing infrastructure tools (e.g., Terraform, Pulumi, AWS CDK, etc).

ghadi21:10:29

honeycomb doesn’t use an agent, but agents are possible with Datomic Zion’s

ghadi21:10:35

But I get you, there’s a ton of crap to consider

kenny22:10:06

Oh they are? I didn't realize you could use Java agents with Ions.