Have any stack traces in any logs or operational changes preceding the exception ?

No functional operational changes, but the load of transactions can spike and these exceptions correlate with the spike. I am retrying the retryable errors at the same point in the code, so it's possible these fault anomalies are already in a retry. I'll see if I can dig up a log.

Check ddb throttling again too. Did you consider going to on demand provisioning?

I switched to on-demand the day you suggested it. I'll check throttling...

@U0CJ19XAM: no throttling since I changed to on-demand.

Also, not sure if this is relevant, but these failing transactions all use CAS and almost none of my other transactions do.

Problem happened today transacting ~30 simple transactions in a row (retraction of one datom each). After only eleven transactions, the exception was thrown:

{
 :cause "Cannot invoke \"datomic.core.impl.Database.entid(Object)\" because \"this.db\" is null"
 :data {:datomic.client-spi/context-id "a6fa60e6-2c82-4138-975c-ff2e91ad0b3f", :cognitect.anomalies/category :cognitect.anomalies/fault, :datomic.client-spi/exception java.lang.NullPointerException, :datomic.client-spi/root-exception java.lang.NullPointerException, :cognitect.anomalies/message "Cannot invoke \"datomic.core.impl.Database.entid(Object)\" because \"this.db\" is null", :dbs [{:database-id "e684b9c0-af66-450f-b29b-da7e6f080575", :t 3374093, :next-t 3374094, :history false}]}
 :via
 [{:type clojure.lang.ExceptionInfo
   :message "Cannot invoke \"datomic.core.impl.Database.entid(Object)\" because \"this.db\" is null"
   :data {:datomic.client-spi/context-id "a6fa60e6-2c82-4138-975c-ff2e91ad0b3f", :cognitect.anomalies/category :cognitect.anomalies/fault, :datomic.client-spi/exception java.lang.NullPointerException, :datomic.client-spi/root-exception java.lang.NullPointerException, :cognitect.anomalies/message "Cannot invoke \"datomic.core.impl.Database.entid(Object)\" because \"this.db\" is null", :dbs [{:database-id "e684b9c0-af66-450f-b29b-da7e6f080575", :t 3374093, :next-t 3374094, :history false}]}
   :at [datomic.client.api.async$ares invokeStatic "async.clj" 58]}]
 :trace
 [[datomic.client.api.async$ares invokeStatic "async.clj" 58]
  [datomic.client.api.async$ares invoke "async.clj" 54]
  [datomic.client.api.sync$eval77610$fn__77615 invoke "sync.clj" 119]
  [datomic.client.api.protocols$fn__13360$G__13336__13367 invoke "protocols.clj" 72]
  [datomic.client.api$transact invokeStatic "api.clj" 227]
  [datomic.client.api$transact invoke "api.clj" 188]
  [user$transact invokeStatic "user.clj" 153]
  [user$transact invoke "user.clj" 150]
  [user.events$unassign_BANG_ invokeStatic "events.clj" 111]
  [user.events$unassign_BANG_ invoke "events.clj" 105]
  [clojure.core$partial$fn__5908 invoke "core.clj" 2641]
  [clojure.core$run_BANG_$fn__8880 invoke "core.clj" 7783]
  [clojure.lang.ArrayChunk reduce "ArrayChunk.java" 63]
  [clojure.core.protocols$fn__8244 invokeStatic "protocols.clj" 136]
  [clojure.core.protocols$fn__8244 invoke "protocols.clj" 124]
  [clojure.core.protocols$fn__8204$G__8199__8213 invoke "protocols.clj" 19]
  [clojure.core.protocols$seq_reduce invokeStatic "protocols.clj" 31]
  [clojure.core.protocols$fn__8236 invokeStatic "protocols.clj" 75]
  [clojure.core.protocols$fn__8236 invoke "protocols.clj" 75]
  [clojure.core.protocols$fn__8178$G__8173__8191 invoke "protocols.clj" 13]
  [clojure.core$reduce invokeStatic "core.clj" 6886]
  [clojure.core$run_BANG_ invokeStatic "core.clj" 7778]
  [clojure.core$run_BANG_ invoke "core.clj" 7778]
  [user$eval78458 invokeStatic "NO_SOURCE_FILE" 792]
  [user$eval78458 invoke "NO_SOURCE_FILE" 792]
  [clojure.lang.Compiler eval "Compiler.java" 7194]
  [clojure.lang.Compiler eval "Compiler.java" 7149]
  [clojure.core$eval invokeStatic "core.clj" 3215]
  [clojure.core$eval invoke "core.clj" 3211]
  [nrepl.middleware.interruptible_eval$evaluate$fn__63885$fn__63886 invoke "interruptible_eval.clj" 87]
  [clojure.lang.AFn applyToHelper "AFn.java" 152]
  [clojure.lang.AFn applyTo "AFn.java" 144]
  [clojure.core$apply invokeStatic "core.clj" 667]
  [clojure.core$with_bindings_STAR_ invokeStatic "core.clj" 1990]
  [clojure.core$with_bindings_STAR_ doInvoke "core.clj" 1990]
  [clojure.lang.RestFn invoke "RestFn.java" 425]
  [nrepl.middleware.interruptible_eval$evaluate$fn__63885 invoke "interruptible_eval.clj" 87]
  [clojure.main$repl$read_eval_print__9206$fn__9209 invoke "main.clj" 437]
  [clojure.main$repl$read_eval_print__9206 invoke "main.clj" 437]
  [clojure.main$repl$fn__9215 invoke "main.clj" 458]
  [clojure.main$repl invokeStatic "main.clj" 458]
  [clojure.main$repl doInvoke "main.clj" 368]
  [clojure.lang.RestFn invoke "RestFn.java" 1523]
  [nrepl.middleware.interruptible_eval$evaluate invokeStatic "interruptible_eval.clj" 84]
  [nrepl.middleware.interruptible_eval$evaluate invoke "interruptible_eval.clj" 56]
  [nrepl.middleware.interruptible_eval$interruptible_eval$fn__63918$fn__63922 invoke "interruptible_eval.clj" 152]
  [clojure.lang.AFn run "AFn.java" 22]
  [nrepl.middleware.session$session_exec$main_loop__63988$fn__63992 invoke "session.clj" 218]
  [nrepl.middleware.session$session_exec$main_loop__63988 invoke "session.clj" 217]
  [clojure.lang.AFn run "AFn.java" 22]
  [java.lang.Thread run "Thread.java" 1583]]}

So this was from your laptop over the client api? Did it go through api-gateway client api? How many compute nodes do you have? Do you see a server side exception?

Two primary compute nodes. From my laptop with :server-type :cloud (earlier failures were :server-type :ion from ion).

Looking for server exception...

No server exception.

After a retry, the remaining 23 transactions or so completed successfully.

Can you open a case with support for this? Whatever is going on here should be impossible.

I already did. https://support.cognitect.com/hc/requests/4136

Thanks

np

Very very general advice, shared by the “clojure way” in general I think: your system should always have access to the most concrete, least abstract encoding possible, and add abstractions on top “from the outside” via adaptor functions, multimethods, rules, etc

just because of the inherent dangers of abstracting early, and clojure’s ability to abstract “after the fact” via adding more stuff to open maps, or using situationally-specific dispatch

I think in general that applies to datomic data modeling, but you do also have to worry about query performance a bit more

e.g. if you model deposit and withdrawal separately, you can always get :entry/status from some other layer on top. Doing the opposite is generally trickier

but maybe entry really is the “bottom” of your domain, and keeping the distinction will be a struggle

Thanks that's really helpful. I was starting to feel like "the clojure/datomic way" way would be to focus less on "types" like deposit/withdrawal and more on the attributes you need for what you're doing. But the way you put it makes a lot of sense. I think this might have been getting too fancy with it. I'm definitely not confident in "entries" being at the bottom

Clojure concentrates its “typing” energy into the attributes and their values, leaving their particular assembly together into an “object” more open and fluid (which is a big part of what allows “from the outside” abstraction later).

However the mainstream understanding of “type” is “the box with the specific fields on it”, so “clojure/datomic doesn’t focus on types” is true for that definition

So I don’t think “this entity is a withdrawal” is wrong at all, and that implies something about what fields may be available on it.

but doesn’t restrict the entity from being something else or having other fields on it that aren’t withdrawal-specific

an entity can be many different kinds of things at once, and that’s the point 🙂

Makes sense!

Quick answers: • There is no built-in hook for starting a system in an Ion. My preferred workaround is to use a delay that is dereferenced in every ion entry point (http-direct and lambdas). This allows either a lambda or an incoming request to initialize the system (but with a performance penalty for the first caller). (@U1QJACBUM!) • To the best of my knowledge, only one http-direct function can be defined. A router is an obvious way to multiplex HTTP requests. For others (websockets) there are proof-of-concepts out there, but nothing official or even idiomatic AFAICT.

Thanks so much @U0698L2BU!