Fork me on GitHub
#datomic
<
2022-02-10
>
favila00:02:56

Does the datomic-pro distribution really need all the aws-java-sdk- service libs in its lib/ directory? That’s over 200mb of jars, and the distribution has been getting fatter and fatter with each release (now 414MB!)

Adam Lewis12:02:50

pro peer pulls in about 28MB worth of jars, and only ec2 and core from AWS, so I'm guessing no (granted, the storage APIs are all provided dependencies, so I should add DynamoDB). I know there is more stuff in the full distro (including a complete PrestoSQL / Trino distribution -- the biggest size jump when it was added), but I can't imagine the AWS Chime SDK is needed

karol.adamiec12:02:07

you would think that cognitect aws api could replace this, but it does not support AWS_CA_BUNDLE 🙃 and this is widespread enough infra pattern that it would harm Datomic i think... Needs to get patched, but not much going on other than hanging ticket: https://github.com/cognitect-labs/aws-api/issues/127

JohnJ15:02:44

I guess everything is being pulled because is more convenient and because various of those are used for cloud

JohnJ16:02:05

I find it more annoying that the peer has a hard dependency on memcache-asg-java-client though, which pulls a very old aws-java-sdk-ec2

JohnJ16:02:48

there's also a hard dependency on h2

cl_j08:02:20

anybody knows why (d/pull db [:node/some-attr :db/id] eid) pulls the expected attribute values from the entity, but (d/pull db [:db/id] eid) pulls everything from the entity?

kenny15:02:16

I've encountered this too. It seems like a bug. We do have an open support ticket on this. Maybe we should create an ask.datomic so others can vote on this too.

favila17:02:57

another :db/id-related oddity is that it can’t be renamed (attr-spec is ignored)

kenny17:02:54

Yep. We have an open support ticket on that one too 😅

kenny17:02:53

Can we alter the JVM launch params for Datomic Cloud to include -XX:-OmitStackTraceInFastThrow? We're getting a "java.lang.NullPointerException with empty message" back from a Datomic transaction issuing a call to a custom transaction function. There's a bug in that function, and a line number would be very helpful to debug.

kenny19:02:21

Looks like you could be naughty and modify the CF template at Parameters.Datomic.defaults.JvmFlags

kenny19:02:16

Alternatively, it seems like you could ignore "Leave blank unless directed by Datomic Support." and modify OverrideSettings to be set to "export JVM_FLAGS="${JVM_FLAGS} -XX:-OmitStackTraceInFastThrow"

Joe Lane20:02:59

Why don't you just use dev-local or cast within the tx-fn to investigate this problem @U083D6HK9?

kenny20:02:35

This DB is 4b datoms and would take weeks to import with dev-local. Casting would work. I'd prefer having NPE stacktraces upfront though. We don't use NPEs for control flow so disabling that flag will have little to no perf impact.

kenny20:02:08

Maybe Datomic does though and this could have adverse effect on Datomic then?

Joe Lane20:02:34

That's a big presumption about perf and you modifying things like that puts your system squarely in Undefined Behavior territory.

kenny21:02:11

:man-shrugging::skin-tone-3: Test & measure

kenny21:02:15

We already run in prod with that flag, so it'd mostly be a matter of Datomic perf impact. Seems straightforward to measure from the CW dashboard.

kenny22:02:02

Out of curiosity, have you encountered a situation where setting that flag had an adverse impact on perf? If so, what was the area? I've seen some compilers use exceptions for control flow and would imagine this flag causing poor perf for them.

jaret18:02:14

We haven't encountered a situation where setting the flag had an impact on performance, but we also rarely ever have anyone set flags. This is to my knowledge in support.

kenny18:02:34

👌:skin-tone-2:

neilprosser17:02:23

@U083D6HK9 Hopefully you don't mind me digging this thread up but did you have any luck getting the stack trace to appear. I've tried the JVM_FLAGS suggested above and have transactions which are failing with a NullPointerException before hitting any of my transaction functions.

Benjamin19:02:10

(d/with
  (get-db)
  {:tx-data
   [{:bot.discord/user-id "foouser"}]})
Hi, what do I do wrong? Is arg-map not the same as for transact?

kenny19:02:55

lgtm. What's the problem?

Benjamin19:02:14

1. Unhandled java.lang.ClassCastException
   class datomic.core.db.Datum cannot be cast to class
   java.lang.Number (datomic.core.db.Datum is in unnamed module of
   loader 'app'; java.lang.Number is in module java.base of loader
   'bootstrap')

                    tx.clj:  397  datomic.dev-local.tx/datom-lookup-valfn
                    tx.clj:  397  datomic.dev-local.tx/datom-lookup-valfn
             local_log.clj:   56  datomic.dev-local.local-log.LocalLog/valAt
                   RT.java:  760  clojure.lang.RT/get
               btindex.clj:  281  datomic.dev-local.btindex.BTIndex/cons
                   RT.java:  677  clojure.lang.RT/conj
                  core.clj:   87  clojure.core/conj
                  core.clj:   84  clojure.core/conj
                    db.clj: 2322  datomic.core.db.Db/addData
                    db.clj: 3353  datomic.core.db/add-ensured-data
                    db.clj: 3351  datomic.core.db/add-ensured-data
                    db.clj: 3370  datomic.core.db/with-tx
                    db.clj: 3357  datomic.core.db/with-tx
                    db.clj: 2164  datomic.core.db.Db/with
              local_db.clj:   67  datomic.core.local-db/fn
              local_db.clj:   24  datomic.core.local-db/fn
             protocols.clj:  126  datomic.client.api.protocols/fn/G
                   api.clj:  363  datomic.client.api/with
                   api.clj:  353  datomic.client.api/with
                      REPL:  116  user/eval22254
                      REPL:  116  user/eval22254
             Compiler.java: 7181  clojure.lang.Compiler/eval
             Compiler.java: 7136  clojure.lang.Compiler/eval
                  core.clj: 3202  clojure.core/eval
                  core.clj: 3198  clojure.core/eval
    interruptible_eval.clj:   87  nrepl.middleware.interruptible-eval/evaluate/fn/fn
                  AFn.java:  152  clojure.lang.AFn/applyToHelper
                  AFn.java:  144  clojure.lang.AFn/applyTo
                  core.clj:  667  clojure.core/apply
                  core.clj: 1977  clojure.core/with-bindings*
                  core.clj: 1977  clojure.core/with-bindings*
               RestFn.java:  425  clojure.lang.RestFn/invoke
    interruptible_eval.clj:   87  nrepl.middleware.interruptible-eval/evaluate/fn
                  main.clj:  437  clojure.main/repl/read-eval-print/fn
                  main.clj:  437  clojure.main/repl/read-eval-print
                  main.clj:  458  clojure.main/repl/fn
                  main.clj:  458  clojure.main/repl
                  main.clj:  368  clojure.main/repl
               RestFn.java: 1523  clojure.lang.RestFn/invoke
    interruptible_eval.clj:   84  nrepl.middleware.interruptible-eval/evaluate
    interruptible_eval.clj:   56  nrepl.middleware.interruptible-eval/evaluate
    interruptible_eval.clj:  152  nrepl.middleware.interruptible-eval/interruptible-eval/fn/fn
                  AFn.java:   22  clojure.lang.AFn/run
               session.clj:  218  nrepl.middleware.session/session-exec/main-loop/fn
               session.clj:  217  nrepl.middleware.session/session-exec/main-loop
                  AFn.java:   22  clojure.lang.AFn/run
               Thread.java:  833  java.lang.Thread/run

Benjamin19:02:28

it's dev-local. With divert-system

kenny19:02:20

Does (get-db) return a db you got from calling d/with-db on a conn?

Benjamin19:02:19

no from d/db

kenny19:02:25

>

Applies tx-data to a database returned from 'with-db' or a
>  prior call to 'with'. 
> > https://docs.datomic.com/client-api/datomic.client.api.html#var-with

kenny19:02:35

You've got to pass a with-db