Fork me on GitHub
#datomic
<
2018-10-11
>
adamfrey13:10:57

when I write a script that's supposed to run and exit and that script connects to a datomic cloud db via the client api, my script always hangs around when it's finished. And I have to kill it with Ctrl-c. The only way I've found to get it to shutdown on its own is by using (System/exit 0), which is pretty extreme. Even (shutdown-agents) doesn't do anything

adamfrey13:10:52

I wrote a test script that does nothing but connect to datomic cloud and these are the threads that exist when it hangs:

[#object[java.lang.Thread 0x794fbf0d "Thread[async-dispatch-7,5,main]"], #object[java.lang.Thread 0x504f2bcd "Thread[qtp1841195153-13,5,main]"], #object[java.lang.Thread 0x3c7e7ffd "Thread[qtp1841195153-14,5,main]"], #object[java.lang.Thread 0x18c2b4c6 "Thread[async-thread-macro-1,5,main]"], #object[java.lang.Thread 0xd5d9d92 "Thread[async-dispatch-2,5,main]"],
 #object[com.amazonaws.http.IdleConnectionReaper 0x347c7b "Thread[java-sdk-http-connection-reaper,5,main]"],
 #object[java.lang.Thread 0x372b2573 "Thread[qtp1841195153-18,5,main]"],
 #object[java.lang.Thread 0x12bb3666 "Thread[async-dispatch-5,5,main]"],
 #object[java.lang.Thread 0xaad1270 "Thread[Signal Dispatcher,9,system]"],
 #object[java.lang.Thread 0x126f428e "Thread[async-dispatch-1,5,main]"],
 #object[java.lang.Thread 0x41a372c1 "Thread[async-dispatch-6,5,main]"],
 #object[java.lang.Thread 0x45d28ab7 "Thread[qtp1841195153-15,5,main]"],
 #object[java.lang.Thread 0x3b75fdd0 "Thread[qtp1841195153-12,5,main]"],
 #object[java.lang.ref.Finalizer$FinalizerThread 0x7e64b248 "Thread[Finalizer,8,system]"],
 #object[java.lang.Thread 0x7e4f5062 "Thread[main,5,main]"],
 #object[java.lang.Thread 0x66fd9613 "Thread[qtp1841195153-19,5,main]"],
 #object[java.lang.Thread 0x461e9b31 "Thread[async-dispatch-3,5,main]"],
 #object[java.lang.Thread 0x78652c15 "Thread[qtp1841195153-17,5,main]"],
 #object[java.lang.ref.Reference$ReferenceHandler 0x6e5dc02d "Thread[Reference Handler,10,system]"],
 #object[java.lang.Thread 0x1c71d704 "Thread[async-dispatch-4,5,main]"],
 #object[java.lang.Thread 0x623e578b "Thread[clojure.core/tap-loop,5,main]"],
 #object[java.lang.Thread 0x414a3c7d "Thread[clojure.core.async.timers/timeout-daemon,5,main]"],
 #object[java.lang.Thread 0x74efc394 "Thread[async-dispatch-8,5,main]"],
 #object[java.lang.Thread 0x201a84e1 "Thread[[email protected],5,main]"],
 #object[java.lang.Thread 0x51dd74a0 "Thread[qtp1841195153-16,5,main]"]]

adamfrey13:10:43

is there any way other than System/exit to fix this?

jocrau15:10:54

I am trying to parse and import a 25GB CSV file into Datomic Cloud (prod topology with two i3.large instances). I get “clojure.lang.ExceptionInfo: Busy indexing”. Before I start to implement a retry strategy on the client side, what are the dials and knobs to improve the indexing performance? (I already set :db/noHistory to “true” on all my attributes)

kvlt16:10:23

IT seems that I cannot bind nil to a var in datomic, resulting in this failing:

(d/q '[:find [?tx ...]
              :in ?log ?since ?til
              :where [(tx-ids ?log ?since ?til)
                      [?tx ...]]]
            (d/log conn) #inst "2018-10-11T15:53:51.974-00:00" nil)
Exception Unable to find data source: $__in__3 in: ($__in__1 $__in__2 $__in__3)  datomic.datalog/eval-rule/fn--5763 (datalog.clj:1450)
While this works:
(d/q '[:find [?tx ...]
              :in ?log ?since
              :where [(tx-ids ?log ?since nil)
                      [?tx ...]]]
            (d/log conn) #inst "2018-10-11T15:53:51.974-00:00")
[13194187931476 13194187931455 13194187931456]
Why is that?

Joe Lane16:10:29

@jocrau If you look at https://github.com/Datomic/mbrainz-importer you can find examples of how to pipeline async transactions which should yield much higher performance of writes. Are you doing any reads when you’re importing the data or is it pure writes?

Joe Lane16:10:38

You can look at the cloudwatch dashboard to find different bottlenecks in your system. Sometimes it may be cpu, memory, or DDB allocated write-throughput units.

Joe Lane16:10:30

The dashboards are very helpful in getting started with perf. That being said, I highly recommend setting up retries on your writes. Things happen and its a good idea to program defensively. Maybe queue things up in kinesis? Thats one approach we took.

kenny17:10:10

I am getting this exception when trying to connect to a DB running the production topology:

(def cust-conn (d/connect client {:db-name "cust-db/591b632f-6c14-4807-af7b-da30929d5791"}))
clojure.lang.ExceptionInfo: Datomic Client Exception
clojure.lang.Compiler$CompilerException: clojure.lang.ExceptionInfo: Datomic Client Exception {:cognitect.anomalies/category :cognitect.anomalies/forbidden, :datomic.client/http-result {:status nil, :headers nil, :body nil}}, compiling:(form-init3645246680023467651.clj:1:16)
The strange thing is that if I try to connect to another DB, it works:
(def admin-conn (d/connect client {:db-name "admin"}))

=> #'dev.system/admin-conn
Any idea what is going on here?

kenny17:10:42

Interesting...

(d/create-database client {:db-name "foo"})
=> true
(d/create-database client {:db-name "foo/bar"})
=> true
(d/connect client {:db-name "foo"})
=> {:db-name "foo", :database-id "55e543d1-14f0-4c9a-b3a9-8fa089a730e9", :t 3, :next-t 4, :type :datomic.client/conn}
(d/connect client {:db-name "foo/bar"})
clojure.lang.ExceptionInfo: Datomic Client Exception
Are db names with a / not allowed??

jocrau17:10:48

@lanejo01 Thanks for your help. I have studied the mbrainz importer. It makes the processing CPU bound by parallelizing it by using pipeline-blocking. I have used that approach in the past but switched to Kyle’s tesser library which works nicely (and I find easier to reason about). The ratio between actual and provisioned write capacity in DynamoDB is healthy. The current problem is that the indexing (which happens asynchronously in the background afaik) can’t keep up with the transaction throughput. And I wonder whether there is a configuration option to tune this in Datomic Cloud.

Joe Lane17:10:57

got a link to tesser? what does it buy you?

jocrau17:10:54

I use it to execute composed functions in parallel.

favila17:10:57

are you using tesser in such a way that it propagates backpressure?

favila17:10:12

I don't know for sure that datomic client acts the same way, but with the datomic peer api as long as you deref your transact somewhere you won't get exceptions. Your application may slow to nothing, but eventually the transactor will catch up

favila17:10:31

tesser is designed for cpu-level parallelism, but transacting with high throughput requires io pipelining with a bounded depth and blocking to receive backpressure

favila17:10:08

doesn't mean you can't use tesser but there has to be some care about how it is using the transactor

jocrau18:10:19

The call to the synchronous transact blocks, but in case of a transactor being busy indexing returns an ex-info map right away. The behavior differs from the on-prem client library which returns a future.

jocrau18:10:56

One way to handle that is to retry the failed transaction. On the other end, I am still trying to reduce the fails by increasing the indexing performance.

favila18:10:51

looks like that is by design

favila18:10:38

looks like they want you to do something like this:

jocrau17:10:41

I found https://github.com/BrunoBonacci/safely. It seems to be a great tool to implement a retry strategy.

favila18:10:40

thank you for that link!

grzm21:10:54

Any issues running Clojure 1.10.0-RC1 on Datomic Cloud?

csm21:10:55

I just set up a new datomic cloud instance, can start the socks proxy, but get ExceptionInfo com.amazonaws.services.s3.AmazonS3Client.beforeClientExecution(Lcom/amazonaws/AmazonWebServiceRequest;)Lcom/amazonaws/AmazonWebServiceRequest; clojure.core/ex-info on trying list-databases or create-database

csm22:10:06

aha, had the wrong version of aws-java-sdk-core from another dependency

luchini22:10:12

The EC2 instances of my query groups started shutting down and terminating non-stop recently. It seems that the culprit is this:

luchini22:10:26

#############################################################
/dev/fd/11: line 1: /sbin/plymouthd: No such file or directory
initctl: Event failed

luchini22:10:49

Anyone else with this problem?

jaret13:10:35

@U4L16CHT9 I might know what’s going on here. Could you copy out the lines above the “#” break line? The entire block delimited by the “#” break line rows.

luchini17:10:58

These are the lines I get before the #:

Calculating memory settings
No cache configured
/opt/datomic/export-environment: line 31: {:retry: command not found
#############################################################
DATOMIC EXITING PREMATURELY
Error on or near line 31; exiting with status 1
Environment: 
S3_VALS_PATH=primary-storagef7f305e7-35z8d8t26waz-s3datomic-xxjwrytensid/primary/datomic/vals
DATOMIC_INDEX_GROUP=primary
DATOMIC_TX_GROUP=primary
TERM=linux
DATOMIC_APPLICATION_PID_FILE=/opt/datomic/deploy/image/pids/application.pid
DATOMIC_CLUSTER_NODE=true
DDB_CATALOG_TABLE=datomic-primary-catalog
DATOMIC_CODE_DEPLOY_APPLICATION=red-robin
DATOMIC_PRODUCTION_COMPUTE=primary-Compute-N688L1Z9E3CF
JVM_FLAGS=-Dclojure.spec.skip-macros=true -XX:+UseG1GC -XX:MaxGCPauseMillis=50 -XX:MaxDirectMemorySize=256m
DATOMIC_CACHE_GROUP=primary
DISABLE_SSL=true
DATOMIC_HOSTED_ZONE_ID=Z38EPCPQ9NXY6O
DATOMIC_XMX=2582m
PATH=/sbin:/usr/sbin:/bin:/usr/bin
OS_RESERVE_MB=256
EFS_VALS_PATH=datomic/vals
S3_AUTH_PATH=primary-storagef7f305e7-35z8d8t26waz-s3datomic-xxjwrytensid
RUNLEVEL=3
runlevel=3
AWS_DEFAULT_REGION=us-east-1
PWD=/
LANGSH_SOURCED=1
DATOMIC_QUERY_GROUP=sandbox
DATOMIC_ENV_MAP=<REDACTED>
LANG=en_US.UTF-8
KMS_CMK=alias/datomic
FS_VALS_CACHE_PATH=/opt/datomic/efs-mount/datomic/vals
PREVLEVEL=N
previous=N
PCT_JVM_MEM_FOR_HEAP=70
HOST_IP=10.213.21.224
CONSOLETYPE=serial
SHLVL=2
CW_LOG_GROUP=datomic-primary
UPSTART_INSTANCE=
UPSTART_EVENTS=runlevel
EFS_DNS=
DDB_LOG_TABLE=datomic-primary
UPSTART_JOB=rc
S3_CERTS_PATH=primary-storagef7f305e7-35z8d8t26waz-s3datomic-xxjwrytensid/primary/datomic/access/certs
PCT_MEM_FOR_JVM=100
_=/bin/env
#############################################################

jaret13:10:15

Thanks! We’re working on a fix for this.