Fork me on GitHub
#datomic
<
2020-10-23
>
cmdrdats11:10:55

hi - we're trying to restore a backup of our datomic database (it's a tiny 9mb db, as a test), and it just seems to be hanging forever.. how do we go about figuring out what we're doing wrong? We've set the log level in the logback.xml for datomic to TRACE - and theres a tiny bit of logging (I'll attach that in thread), but nothing else

jaret14:10:45

@U050CLJ53 Can you share what command you're using to run backup/restore?

jaret14:10:04

If you'd like feel free to log a case to me by e-mailing <mailto:[email protected]|[email protected]> and we can dive further.

cmdrdats19:10:55

@U1QJACBUM we're doing this:

/storage/datomic/current/bin/datomic -Xmx1g -Xms1g restore-db "file:/storage/datomic/archive/restore/" "datomic:sql://...?jdbc:..."
if there's nothing we can really self diagnose at a high level, then sure, I'll send a mail on monday πŸ™‚

jaret19:10:32

Try restoring locally using :dev protocol, just to see if it's underlying storage related. But if it couldn't write to storage due to perms I would expect an error not a hang.

millettjon14:10:52

I am trying to get started with datomic in a dev-local setup. Any idea why my dev-local db is not found? I can see can see db.log and log.idx files were created.

(ns flow.db
  (:require [datomic.client.api :as d]))

(let [dir     (str (System/getProperty "user.dir") "/var/datomic")
      db-name "flow"
      client  (d/client {:server-type :dev-local
                         :storage-dir dir
                         :system      "dev"})
      _       (d/create-database client db-name)
      conn    (d/connect client {:db-name db-name})]
  conn)

;; Unhandled clojure.lang.ExceptionInfo
;; Db not found: flow
;; #:cognitect.anomalies{:category :cognitect.anomalies/not-found,
;;                       :message "Db not found: flow"}

jaret14:10:10

Can you try testing with the absolute path of dir and just execute d/client without the let? Also does the absolute path already have a "dev" system folder?

jaret14:10:44

then list-dbs on the client

jaret14:10:29

(d/list-databases client {})

millettjon14:10:17

Sure.

(def client (d/client {:server-type :dev-local
                       :storage-dir "/home/jam/src/flow/var/datomic"
                       :system      "dev"}))
(d/create-database client "flow")  ; => true
(d/list-databases client {})   ; => []

millettjon14:10:27

file system looks like this:

$ tree /home/jam/src/flow
/home/jam/src/flow
β”œβ”€β”€ deps.edn
β”œβ”€β”€ 
β”œβ”€β”€ src
β”‚Β Β  └── flow
β”‚Β Β      └── db.clj
└── var
    └── datomic
        └── dev
            β”œβ”€β”€ db.log
            └── log.idx

millettjon15:10:37

I created var/datomic. dev/ gets created by create-database fn

millettjon15:10:12

I tried setting :storage-dir in ~/.datomic/dev-local.edn and it has the same problem. Creates some files but no db found.

jaret15:10:42

What version of Dev-local are you using?

jaret15:10:52

Can you share the .datomic/dev-local.edn file in your home directory?

jaret15:10:24

Might be best to just share your deps.edn and I will try to re-create. So I can see version of client and dev-local.

millettjon15:10:51

jam@stick:~/.datomic$ cat dev-local.edn 
{:storage-dir "/home/jam/var/datomic"}

millettjon15:10:00

jam@stick:~/opt$ java -version
openjdk version "11.0.8" 2020-07-14

jaret16:10:53

@U071T1PT6 I don't see a version of client in your deps? Do you have datomic client in your .clojure/deps.edn? Could you include com.datomic/client-cloud "0.8.102"

jaret16:10:33

Also worth testing with the latest dev-local and use 0.9.225

millettjon16:10:49

Ok. I was following instructions here: https://docs.datomic.com/cloud/dev-local.html and didn't know about that additional dep. Unfortunately, adding it didn't make any difference. I will try updating to 0.9.225.

jaret17:10:26

@U071T1PT6 I can't reproduce is there anything else I could be missing? Are you starting a new repl to do this? Can you try making a new system name to confirm you are local when a new dir is made? What OS are you using on your system?

ghadi19:10:28

look at the args for create-database

ghadi19:10:58

needs to be {:db-name "flow"} not "flow"

πŸ‘ 3
jaret19:10:27

!!! Ghadi, great catch!

jaret19:10:30

That's it

ghadi19:10:00

calling it with a raw string probably shouldn't return true

jaret19:10:15

No it should not!

onetom04:10:09

this bit me a few times initially. it would be very helpful to provide better error message for this situation.

πŸ‘ 3
jaret14:10:22

Totally agree this seems to be a bug in using dev-local client and create DB. It should throw an error like in cloud for expected map. I've logged a bug and we'll look at fixing!

jaret14:10:07

;Cloud client
(d/create-database client "testing")
Execution error (ExceptionInfo) at datomic.client.impl.shared/api->client-req (shared.clj:258).
Expected a map

;Dev local client
(d/create-database client "testing")
=> true

jaret14:10:12

in either case no DB is created, but we should throw an error in both cases!

millettjon15:10:36

Thanks for all the help. Working as expected now.πŸ˜€

Drew Verlee17:10:58

What's the best way to report/ask about questions on the official datomic docs? e.g The docs for import-cloud on the docs: https://docs.datomic.com/cloud/dev-local.html#import-cloud Don't match the docstring. I assume the docstring is correct as the documentation page lists the same value for source and dest.

jaret20:10:47

I'll fix that!

jaret20:10:58

Thanks for catching it.

Drew Verlee18:10:31

@U1QJACBUM thanks for fixing it. Do you know if there is a light weight way to get datomic cloud consulting? I run into little blockers here and there and I would rather shell some money then get stuck for a day off someone can easily help me trouble shoot things.

jaret18:10:28

@U0DJ4T5U1 I am going to tag @U05120CBV on this. You can also e-mail him directly at <mailto:[email protected]|[email protected]>. He is running point on our Datomic consulting services along with a few other folks at Cognitect. I'll bring this up with him on Monday if you two don't connect here.

dpsutton17:10:07

There’s a new ask.datomic site that probably fits the bill

πŸ‘ 9
joshkh17:10:44

should one be alarmed when periodically seeing an Unable to load index root ref <uuid> exception appear in Datomic Cloud logs? (from com.amazonaws.services.dynamodbv2.model.InternalServerErrorException )

jaret20:10:55

Do you often delete DBs? In general, you can see this error if a client or node is asking for a deleted db. The error would also correlate to an outage but these calls are retried so if you don't see this error often or repeatedly it's probably not a major issue.

jaret20:10:53

As always, it is my support-person duty to recommend upgrading to the latest Cloud release CFT and if you'd like me to look more closely at your system logs I'd be happy to poke around with a Read-Only CloudWatch account. If you want to go down that path, log me a case at <mailto:[email protected]|[email protected]> and I can take a look.

joshkh12:10:14

thanks, Jaret. we haven't noticed any performance issues related to the exception, so i was mostly just curious. but since you mentioned that it could be related to often deleting DBs, which we very rarely do, then i'll just mention it in some future support ticket. it's a low priority for us. πŸ™‚

Drew Verlee22:10:58

When i try to use dev-tools to import a cloud locally it complains that it can't use the endpoint to connect/"name or service is not known". I'm connected and can query the databases though so i'm not sure what the issue is.