Each retraction performed like this (with a list-form :db/retract) translates directly into a single datom, so you would need to create a :db/retract for each child you want to retract

Got it. Thank you so much marshall

the underscore syntax is only valid in pull

in this case you would want to retract from the parent entity, so something like: [:db/retract [:app/id "app_1"] :app/children [:component/id "Component 1"]]

Many thanks @marshall!

[(missing? $ ?customer :customer/address)] should do that I think.

that works thank you 🙂

I'm aware of three different strategies to deal with timeouts: First, there is a timeout option on the query object (see https://docs.datomic.com/on-prem/clojure/index.html#datomic.api/query) Second, I have found that using the async api can alleviate the problem (i think async helps when the result set is very large but I'm not certain) Third, switching to the index api rather than the query api is a last resort.

@genekim You can pass a :timeout to the query if you think it’s an issue of it just being a long-running job https://docs.datomic.com/client-api/datomic.client.api.html

https://docs.datomic.com/client-api/datomic.client.api.html#var-q you’ll need to use the arity-1 version of the q function

I believe the default timeout is 60s

Ooh!!! Promising, @marshall @mark340! I’m giving that a shot! Thx! I’ll keep you posted.

:thumbsup:

Thank you!!! People kept suggesting that to me, but I can’t say I actually understood why until your comment. That’s awesome!!!

We have some lambdas doing that in production. The connection overhead is far overshadowed by the Clojure runtime boot up time and the VPC ENI provisioning time.

AWS is rolling out a fix for the ENI problem but the Clojure startup time is still a barrier for any customer-facing use cases (we just use lambda for background processing right now)

Thanks for the info. Couple other questions. Are you AOT'ing your code? About how many lambdas are running in parallel?

Datomic itself seems to have no issues that we’ve seen so far.

Yup we are AOTing the code, however, bootstrapping the Clojure runtime on a lambda incurs a ~1.9s hit for cold starts. We’ve tested this in Java lambdas that only depend on Clojure and just import the clojure.core ns.

At most we’ve had ~8 lambdas in parallel hitting datomic.

That's totally fine for our application. Anything under 10s is probably ok actually. What is your actual startup time with all your production deps?

Hmm, we will have a significantly higher level of parallelism. Should be somewhat easy to test, I suppose.

Because of the VPC issue, it takes about 18s. However, we’ve tested in other availability zones that have the VPC fix and that goes down to ~8s.

Is the VPC issue only a problem when sources outside the VPC need to contact the Lambda?

Its an issue when you need to run a lambda in a VPC (which you need to for datomic).

The fix is likely only a few weeks away for us east 1. Can follow it here https://aws.amazon.com/blogs/compute/announcing-improved-vpc-networking-for-aws-lambda-functions/

Gotcha, ok. Are you guys using deps.edn?

Another option that we’ve considered is just exposing datomic over API gateway using ions w/ http direct to expose a REST-like interface for datomic operations and securing it with an IAM authorizer.

Yup we use deps for all our dependency managment.

Interesting idea. What do you use to AOT?

We have an internal library that we’ve soft open-sourced (use at your own risk): https://github.com/StediInc/lambda . Its pretty basic, just uses clojure.core/compile and juxt.pack

We use lambda for pretty much everything so it has some extras in there for making it easy to manage multiple lambda entrypoints in a service.

Nice, will write it down to take a look at when we're ready to investigate more. Why'd you guys write your own lib instead of using one of the existing AWS Lambda clj libs?

None of them worked quite the way we wanted. We use quite a bit of middleware internally and wanted an interface more conducive to the pattern established by ring. Additionally, we do quite a bit with cloudformation and we have a library to make connecting lambdas to that easier. We wanted better control of the output artifacts so we could line that up better.

Only other one I was aware of was https://github.com/uswitch/lambada

Yep, that was the one I'm familiar with.

Looking at the org, I see cdk-clj. We use Pulumi at our org for deployment. It sounds like cdk-clj leverages jsii to work with a TypeScript API from the JVM. Is that correct? I'd be very interested in exploring that with Pulumi.

It does. However, the library has to explicitly be built with a jsii configuration.

See https://github.com/aws/aws-cdk for an example

Pretty sure it was built specifically to make CDK cross-language compatible. I’m not aware of any other projects using jsii.

I'm not familiar with jsii. Does that mean the libraries need to be actually written in a jsii compatible way? Or is there declarative spec files the library needs to have?

I believe so, although this is stretching my knowledge at this point since we just did it to target CDK. The source library has to be in typescript and has some restrictions: https://github.com/aws/jsii/blob/master/docs/typescript-restrictions.md.

Our implementation requires the jsii bundle manifests that get produced in the JVM target config. No idea how hard it would be to build an existing project with jsii.

Looks like Pulumi is in typescript so it might work.

Not sure. Pulumi supports a few different languages now but I don't think they use jsii.

Not a high priority for us but definitely super interesting stuff. I'll bring it up with the Pulumi team to see if they have any input. Thanks for all this info - been super useful.

You guys don't happen to use any native libraries in your Lambdas do you @U0BKC8NCU?

We don’t, no. I’ve dabbled with them on side projects though.

Ok. One of the services we're interested in running on Lambda needs some native libs. Seems like other people have used native libs on the jvm on Lambda.

You should be able to build the native lib on a docker image like https://hub.docker.com/r/lambci/lambda/ and then package it into a lambda layer https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html

If you have a pre-built binary thats compatible you can just put that on a lambda layer directly and ignore the docker step.

https://hub.docker.com/_/amazonlinux is what you’d want to build on, not the ci image I posted above.

That seems like a better path than manually extracting the native libs from the classpath. Thanks! What do you guys use for logging in Lambda? I've used CloudWatch logs in the past and can hardly stand it after using Datadog logs for a while.

i.e. Do you guys ship the CW Lambda logs to somewhere else? Do you have some other log viewer?

We use cloudwatch logs for lambda logging but we’ve been moving to using traces instead with Amazon Xray and then just linking the logs from Xray.

We also post all the cloudwatch logs to an elasticsearch instance. Although haven’t really used that since we’ve gotten Xray setup properly.

Do you have some wrapper around the Xray api? We use opentracing right now and have a defn wrapper that can be used to instrument certain functions.

Yeah we just have a lightweight wrapper around the Xray java API. Nothing fancy. We are leveraging the annotation and metadata features pretty heavily, not sure how those interplay with opentracing. We’re pretty much all-in on AWS so we have tried to use their supplied libraries to get as much as we can out of their services.

Specifically this https://docs.aws.amazon.com/xray/latest/devguide/xray-sdk-java.html. They will auto instrument calls to all AWS services (which we use a lot of) and to anything using the apache http client.

We started using the cognitect aws-api so we probably will miss out on the auto-instrumentation 😞

We did too although we are reverting back to the aws sdk. We like the interface for the aws-api lib by cognitect but its missing a lot of goodies. Luckily, the aws sdk v2 exposes a data class interface that allows for the same aws-api payloads to be marshalled into request classes. We are working on an internal lib that is basically a 1-1 replacement with the cognitect aws-api library.

Oh very cool!