Clojurians
#datomic
<
2018-08-14
>

This page is not created by, affiliated with, or supported by Slack Technologies, Inc.

steveb8n00:08:56

@stuarthalloway any chance you could comment on this discussion about Ion Lambdas? https://clojurians.slack.com/archives/CC0A7PUHF/p1533611391000066

stuarthalloway00:08:23

@steveb8n what is the performance requirement?

steveb8n00:08:10

it’s arbitrary i.e. I’m determining what is acceptable but multiple 15 second waits is not gonna work

steveb8n00:08:37

hence I’m considering options to work-around it but would prefer to avoid extra complexity

stuarthalloway00:08:52

who is calling the lambdas? events don't care...

steveb8n00:08:13

api-gateway calls from SPA client

steveb8n00:08:28

it’s fine for mutations but not for reads

steveb8n00:08:42

“fine” because optimistic update

stuarthalloway00:08:04

api-gateway has independent timeout and retry capability, so why tie yourself to lambda?

steveb8n00:08:12

clarifying to ensure we are talking about same thing: re-frame client, xhr request to API-GW which invokes Ion Lambda.

stuarthalloway00:08:41

so if API gateway has a 5 second timeout you will never see a 15 second delay from lambda

steveb8n00:08:43

15 sec wait on first invocation for every GW endpoint

stuarthalloway00:08:07

how many endpoints are we talking about? how often are they invoked?

steveb8n00:08:20

true that but that’s a failure. worse than slow

stuarthalloway00:08:32

how many 9s are you promising?

steveb8n00:08:06

The Ion architecture seems to suggest 1 lambda per read or write fn so many for a SPA app

steveb8n00:08:24

#9's not set at this stage but this is not downtime but perf

stuarthalloway00:08:26

1. Lambdas will become less latent

stuarthalloway00:08:16

2. There are other options for wiring API Gateway that don't go through Lambda at all, and to the extent that #1 is a problem for applications we can investigate and implement them

stuarthalloway00:08:18

API gateway timeout is not (by itself) downtime, because API gateway should automate retry as well

steveb8n00:08:15

interesting. it’s been suggested by others that the cold-start is likely due to ENI init for Lambdas in VPCs which is a well known problem. on that basis, not using Lambdas would help

stuarthalloway00:08:20

I would like to see an actual app, under actual load, with API gateway timeout and retry configured, that has a problem meeting a particular SLA. I say this not in the "I don't believe you" sense, but in the "particular example we can use to make Datomic better" sense

steveb8n00:08:28

full disclosure - I’m giving the e.g. of API-GW but in my case I’m using AppSync and not API-GW. same end result

stuarthalloway00:08:50

I know less about AppSync, does it have timeout/retry config?

steveb8n00:08:56

ok that’s a good target for me (or others) to aim for

steveb8n00:08:21

Once I can show a better example, I’ll record the UX and pass it along

steveb8n00:08:48

not sure about AppSync features at that level yet. will dig deeper

stuarthalloway00:08:11

I understand this could be a problem, I just want to grab it hard by the specifics and not some generalized concern. We are committed to doing the best thing possible under AWS.

steveb8n00:08:50

but FWIW graphql as a service without code is great. Fits really well with Ions as well. Hence I’m really happy with this stack

steveb8n00:08:57

worth saying :slightly_smiling_face:

stuarthalloway00:08:46

Good to hear, and when you have the concrete perf problem in hand we will jump on it!

steveb8n00:08:39

I appreciate your position also. Others are interested also. What form would be best to present the perf profile to you/team? is a video of client good enough?

stuarthalloway00:08:42

even just some prose about the objectives and numbers here for starters

steveb8n00:08:49

last point: my arbitrary requirement is that 15 secs is too long for users but 2-3 secs is fine. now that Clojure (tools.deps) starts in 2-3 secs, this would be “arbitrarially” acceptable for my use case

steveb8n01:08:31

ok. I’ll see if I can get a few others on board to form up a description of stack/UX/requirements

steveb8n01:08:17

not sure we will be able to produce “prose” but will do our best

stuarthalloway01:08:32

not looking for more than the kind of prose you are producing now

steveb8n01:08:00

great. thanks. we’ll get on it

stuarthalloway01:08:15

thank you for using Datomic!

steveb8n01:08:02

it’s been a pleasure so far. looking forward to more fun with it

ivan.yaroslavov07:08:40

What is the proper way to restart Datomic Cloud? All of a sudden it stopped working for us and simply restarting the EC2 instance did not work

alex43813:08:15

Also curious if there’s a best practice. When DynamoDB throttles, the transactor dies and does NOT come back. At that point I stop the entire CloudFormation and start it again. Now I err on the side of too much DynamoDB capacity, unfortunately.

stuarthalloway15:08:51

@ Datomic Cloud does not have transactors. Are you talking about On-Prem?

alex43815:08:21

Oops I got confused. It’s cloud from my perspective but it’s not Datomic Cloud. My comment above isn’t relevant, then.

stuarthalloway15:08:26

@ if you run transactors in an ASG (as our template does) you should have HA in On-Prem also: https://docs.datomic.com/on-prem/ha.html

stuarthalloway15:08:01

also @ Cloud puts much less pressure on DynamoDB, so that would be less of (or not) a problem there

octo22111:08:44

can I confirm that retracting a :db.type/ref attribute of an entity that's isComponent true will only retract the entity's immediate sub-components, not recursively their sub-components ?

stuarthalloway12:08:34

@ retract is primitive and does exactly what you tell it. retractEntity works per https://docs.datomic.com/cloud/transactions/transaction-functions.html#sec-1-1, "components of the given entity are also recursively retracted."

octo22113:08:00

ah so :db/retract will not recursively retract the component's subtree

octo22111:08:07

(or retracting the entity itself)

octo22111:08:05

- in Cloud - I just tried and I get only one-level of retraction which is what I'd expect and what I want

andreas86213:08:12

Is there any sort of counter or time for a given entity that is updated whenever there is a transaction with that entity?

andreas86213:08:43

Not for a certain attribute, but for every attribute used with the entity

andreas86213:08:11

something like (last-modified eid)

marshall13:08:25

@alqvist Not ‘built-in’. You can use a combination of query and the log to determine that or you can track it explicity yourself

andreas86213:08:53

@marshall Ok, thanks. Probably will add another attribute since I am unsure of the performance implications

markbastian14:08:06

I have an interesting issue. I am connecting to my local datomic transactor using this connection string: "<postgresql://localhost:5432/datomic?user=...&password=...>". When I uberjar the application and run it, I get this exception: java.sql.SQLException: No suitable driver. Any ideas? For my edification, where is the driver specified for the peer in the first place (i.e. why does it even work in the REPL)?

ghadi14:08:58

need to add the PG drivers to your app

ivan.yaroslavov14:08:03

After Datomic Cloud just stopped we checked the logs to see what went wrong; we did not find any alerts. The log just ends suddenly, with normal messages regarding Datomic's compute stack. What should we look for?

stuarthalloway15:08:28

How do you distinguish stopped from working?

ivan.yaroslavov08:08:53

@stuarthalloway our application could no longer connect to Datomic - we started seeing

clojure.lang.ExceptionInfo: Connect Timeout
    cognitect.anomalies/category: :cognitect.anomalies/unavailable
     cognitect.anomalies/message: "Connect Timeout"

ivan.yaroslavov08:08:06

We saw the same error both on our dev machines and also in the logs of a deployed version (uses the same dev database). The load is fairly low as we are in development and there are only several users. It has been working for months before that.

stuarthalloway16:08:31

are you connecting through the socks proxy?

ivan.yaroslavov17:08:45

for simplicity we have just exposed the port and we are connecting directly

ivan.yaroslavov13:08:12

@stuarthalloway so what would be the difference in the logs in that case?

markbastian14:08:58

This is in my pom:

<dependency>
      <groupId>org.postgresql</groupId>
      <artifactId>postgresql</artifactId>
      <version>42.1.1</version>
    </dependency>
And both the transactor and peer work fine if the peer is a REPL.

markbastian14:08:05

So, the inclusion of the jar in the project should be sufficient, though? If that is the case I'll do some poking around in my uberjar to make sure everything was included.

ghadi14:08:52

yeah, should be. java -cp your-uberjar.jar clojure.main -e org.postgresql.Driver <-- does that error out?

markbastian14:08:26

Nope, it produces "org.postgresql.Driver".

ghadi14:08:05

that's the expected result

ghadi14:08:21

you'd get a resolution error otherwise. So the driver is in your uberjar

markbastian14:08:33

If multiple sql driver implementations are on the classpath will that cause any issues (e.g. postgres and the ms driver)?

ghadi14:08:25

unlikely, but there may be interactions with ServiceLoaders (a JVM mechanism): What does this return:

unzip -l your-uberjar.jar  | grep 'META-INF/services'

markbastian14:08:05

These are the entries in META-INF/services: com.fasterxml.jackson.core.JsonFactory io.undertow.protocols.alpn.ALPNProvider io.undertow.predicate.PredicateBuilder io.undertow.attribute.ExchangeAttributeBuilder io.undertow.server.handlers.builder.HandlerBuilder io.undertow.client.ClientProvider org.xnio.XnioProvider javax.servlet.ServletContainerInitializer io.undertow.servlet.ServletExtension javax.websocket.ContainerProvider io.undertow.websockets.jsr.WebsocketClientSslProvider javax.websocket.server.ServerEndpointConfig$Configurator java.sql.Driver javax.validation.spi.ValidationProvider org.apache.commons.logging.LogFactory javax.json.spi.JsonProvider com.fasterxml.jackson.core.ObjectCodec

markbastian14:08:06

The java.sql.Driver's contents are "org.h2.Driver"

markbastian14:08:10

Should I be seeing the postgres classname in the java.sql.Driver file?

ghadi14:08:52

I am not 100% sure if how Datomic loads drivers. I'll defer to someone more knowledgeable.

markbastian14:08:10

Interesting, when I build the uberjar with lein vs. mvn I get the other drivers in my java.sql.Driver file. I'm guessing it's my mvn assembly configuration that's the problem. I'll poke on that. Thanks!

markbastian15:08:41

Closing the loop on this, it looks like it's the way the maven assembly plugin merges (or doesn't merge) the services file. The solution is to do something along these lines https://maven.apache.org/plugins/maven-assembly-plugin/examples/single/using-container-descriptor-handlers.html or use the shade plugin. Thanks again for the help.

okocim15:08:41

has anyone experienced a StackOverflowError while trying to redirect the (cast/…) functionality? I thought I followed the guide, but I’m seeing the following error when I try to call cast/dev in the repl:

StackOverflowError 
	clojure.lang.RT.seq (RT.java:530)
	clojure.core/seq--5124 (core.clj:137)
	clojure.core/drop/step--5646 (core.clj:2919)
	clojure.core/drop/fn--5649 (core.clj:2924)
	clojure.lang.LazySeq.sval (LazySeq.java:40)
	clojure.lang.LazySeq.seq (LazySeq.java:49)
	clojure.lang.RT.seq (RT.java:528)
	clojure.core/seq--5124 (core.clj:137)
	clojure.core/take/fn--5630 (core.clj:2876)
	clojure.lang.LazySeq.sval (LazySeq.java:40)
	clojure.lang.LazySeq.seq (LazySeq.java:49)
	clojure.lang.RT.seq (RT.java:528)

cap10morgan16:08:25

With Datomic on-prem, is it possible to connect to a DynamoDB table owned by a different AWS account? If so, what does the URI look like for that?

joshkh17:08:25

Maybe someone can help me stretch the datalog part of my brain? I'm trying to write a generic query that returns a particular attribute called :global/uuid for any referenced entities that are directly related. And for silly reasons I need to act exactly like a normal pull:

({:global/uuid #uuid"d7becc50-b43a-4873-805d-890a612174ea",
  :person/email "gNJf4D4OJv6I4oc",
  ; Example A:
  :person/manages [#:db{:id 1697645953286222}]})
  
Except that {:db/id} is replaced with an entity's :global/uuid:
({:global/uuid #uuid"d7becc50-b43a-4873-805d-890a612174ea",
  :person/email "gNJf4D4OJv6I4oc",
  ; Example B:
  :person/manages [#:global{:uuid #uuid"e9c98197-9cc0-45fd-8dce-59356d780dbd"}]})
I could easily get the value back if the reference was a component, but that's not the behaviour I want in the schema. I could use two pulls, but that returns :global/uuid in its own collection and I want it to be in the value of the :person/manages key. I could use Map Specifications, but the query needs to be generic. So wild card support like: (pull ?a [* {* {:global/uuid}}]) which I don't think exists.

okocim19:08:10

What part of the query are you trying to make generic here, the pull spec, or the conditions?

joshkh21:08:42

in my case every entity in datomic has a :global/uuid, and i want to always return the uuids for entities referenced by whichever entity is being pulled.

joshkh21:08:30

i managed to do it using spec and a function to build queries from that spec but it feels like an abomination

joshkh21:08:57

(assoc :find (vector (list 'pull '?root (conj without-refs map-spec)))) where map-spec is a map specification built from a collection of attributes defined in a spec somewhere

captaingrover22:08:31

i have all my data in datomic but i want to use google's natural language service. does anyone have tips for migrate data from datomic to google cloud sql (or any other sql db)?

lockdown-23:08:05

develop the representation of your data for a RDBM and perform an ETL