Fork me on GitHub
#datomic
<
2016-08-31
>
adammiller02:08:55

Upon further thought, that’s probably abysmal performance unless you had something to filter on before evaluating the matching of the slug (or the db is extremely small).

casperc07:08:54

I am wondering what the best practice in Datomic is for doing autoincrementing sequences. Currently I have this implementation, using a db function to increment a noHistory field.

(def gen-id
  (d/function
    '{:lang "clojure"
      :params [db entity-type entity-id]
      :code (let [seq-ent (d/entity db [:sequence/name entity-type])
                  _ (cond
                      (empty? seq-ent)
                      (throw (Exception. (str "Sequence with name " entity-type " not found in database. Are the sequences initialized?")))
                      (nil? (:sequence/sequence seq-ent))
                      (throw (Exception. (str "Sequence with name " entity-type " is nil. Are the sequences initialized?"))))
                  new-value (inc (:sequence/sequence seq-ent))]
              [[:db/add (:db/id seq-ent) :sequence/sequence new-value]
               [:db/add entity-id entity-type new-value]])}))

casperc07:08:16

It can then be used like this when creating the transaction data:

`
(let [tempid (d/tempid :db.part/user)]
    [[:fns/gen-id :task/task-id tempid]
     {:db/id tempid
      :task/status "PENDING"}])
where :fns/gen-id is the function and :task/task-id is the noHistory attribute that is incremented.

casperc07:08:26

My problem is that I want to add several incremented ids in a transaction, but adding several just means that they all see the same old value of the :task/task-id attribute and increment to the same new value, meaning that all the entities get the same id.

casperc07:08:46

So I want to be able to create one transaction, that adds several entities, each of which is assigned a (different) incrementing id. Any ideas?

casperc07:08:15

Just pinging @bkamphaus for when he gets up 🙂

Alex Miller (Clojure team)07:08:05

Ben doesn’t work at Cognitect anymore, but perhaps @marshall can help

casperc07:08:31

Ah, thanks. I’ll make sure to not bug Ben then 🙂

karol.adamiec10:08:14

trying to set up transactor on AWS proves to be problematic. AutoScaling group keeps cycling the transactor. I followed the AWS setup docs and still get the same results (had similiar issue with terraform before). I see Dynamodb table, roles, scaling group and launch configuration. But it keeps cycling transactors because it claims they fail health checks. When trying to connect using shell:

datomic % uri = "datomic:
<datomic:ddb://eu-west-1/temporary/mydb>
datomic % Peer.createDatabase(uri);
// Error: // Uncaught Exception: bsh.TargetError: Method Invocation Peer.createDatabase : at Line: 2 : in file: <unknown file> : Peer .createDatabase ( uri )

Target exception: java.lang.IllegalArgumentException: :db.error/read-transactor-location-failed Could not read transactor location from storage

pesterhazy10:08:58

perhaps dynamo aws permission issues?

karol.adamiec10:08:38

well roles are ensured

karol.adamiec10:08:51

see them in iam console

pesterhazy10:08:07

I mean maybe your peer (not the transactor) can't connect to dynamodb

karol.adamiec10:08:38

i did open cidr 0.0.0.0 so i would expect it to work...

karol.adamiec10:08:20

so basically i do have dynamo table transactor. and want to connect from datomic shell from my local machine

karol.adamiec10:08:53

thing that worries me is constant cycling of transactors

karol.adamiec10:08:01

seems like they do not initialize properly

karol.adamiec10:08:24

so they do not drop information about transactor into dynamo so shell peer can not connect

karol.adamiec10:08:39

on my ect transactor instance this is my log

karol.adamiec10:08:34

user-data:   inflating: datomic-pro-0.9.5394/LICENSE-CONSOLE  
user-data:   inflating: datomic-pro-0.9.5394/datomic-pro-0.9.5394.jar  
user-data:   inflating: datomic-pro-0.9.5394/README-CONSOLE.md  
user-data: pid is  1557 
user-data: ./startup.sh: line 26: kill: (1557) - No such process
/dev/fd/11: line 1: /sbin/plymouthd: No such file or directory
initctl: Event failed
Stopping atd: [  OK  ]
Shutting down sm-client: [  OK  ]
Shutting down sendmail: [  OK  ]
Stopping crond: [  OK  ]

karol.adamiec10:08:42

it definitely is not looking good

karol.adamiec10:08:58

it spins up and then a fatal error shuts instance down

karol.adamiec10:08:14

and what is /sbin/plymouthd ?

karol.adamiec10:08:58

and in aws console in my dynamo table there no data whatsoever…. that is why peer can not connect i suppose.

karol.adamiec11:08:28

could someone take a peek at a correct transactor logs and paste what it does right after unpacking of datomic?

karol.adamiec11:08:09

as a side note it is definitely not a license issue, i just mangled the licencse key on purpose and same results followed

karol.adamiec12:08:16

same behaviour on us-west-1 region 😞

jaret13:08:14

@karol.adamiec are you sure you are not exceeding the allotted number of processes for your license? Do you have transactor logs so I can verify it is failing on a health check to heartbeat? Finally, I assume you reviewed this page: http://docs.datomic.com/aws.html

jaret13:08:51

@karol.adamiec other thoughts: Are the memory settings valid? (e.g. heap fits within instance size) Test the license key being valid by launching transactor locally (not just garble the key). are you using a supported AWS instance type? (writable file system?) per the docs supported instance types can be found in a datomic generated cloud formation template under the key “AWSInstanceType2Arch"

karol.adamiec13:08:55

heartbeat is not on

karol.adamiec13:08:22

apparently supported

karol.adamiec13:08:40

will follow on rest of hints shortly 🙂, thx

jaret13:08:52

If the transactor fails to write heartbeat at launch the docs indicated to verify storage connection information Datomic has can be used to reach the storage.

karol.adamiec13:08:45

i will clean all infra get fresh datomic zip and follow very carefully the docs. might be some detail somewhere

karol.adamiec13:08:09

second time the charm 🙂

jaret13:08:31

I think t1 Micro is not officially supported. People have done it and I am not sure what they configured to get it to work

jaret13:08:46

ok good luck! Let us know how it goes.

karol.adamiec13:08:45

i hoped to shortcut the setup with terraform template but time to go elbows deep would come anyway….

marshall13:08:58

@karol.adamiec Yes, @jaret is correct - t1.micro is not officially supported. Default settings won’t work with that instance type. i’d recommend getting it running on a larger (supported) instance type (i.e. these: http://docs.datomic.com/capacity.html#dynamodb) then you can go back and tweak configuration to get it running on the instance type that suits your needs

karol.adamiec13:08:45

thanks @marshall will do and report back

marshall14:08:54

You need to have 2 transactions

marshall14:08:00

The first defines the partition

marshall14:08:03

the second can use it

marshall14:08:23

you can’t both define and transact something to the same partition in a single txn

marshall14:08:49

ditto install/use an attribute

tengstrand14:08:24

Ok, will try that. Thanks!

tengstrand14:08:43

CompilerException java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: :db.error/not-a-db-id Invalid db/id: #db/id[:db.part/mypartition -10100001], compiling:

marshall15:08:16

Change your data tx to: `

marshall15:08:50

Alternatively, change the ident in your partition definition to:

:db/ident :db.part/mypartition 

tengstrand15:08:05

Now it works, thanks! What is most correct, to use “:mypartition” or to use “db.part/mypartition”?

marshall15:08:15

Probably the latter.

karol.adamiec15:08:01

i had success with us and eu regions for datomic. previous issues most likely due to instance type. now using m3.medium.

karol.adamiec15:08:41

what is the cheapest supported transactor instance type so one can spin dev/qa environments?

karol.adamiec16:08:11

thanks @marshall very helpful links 🙂

karol.adamiec16:08:06

anyone having tried @mrmcc3 terraform module? i would be glad for syntax for the licence. The one from email text file pasted into tf file complains about escaping. Are the licence trailing \ needed?

karol.adamiec17:08:00

hmpphhhhh. Long story short: most likely it was license encoding issue. Would be nice to get some sensible output from transactor startup on AWS in such cases!!

marshall17:08:54

@casperc regarding your incrementing function from this morning - if you’re calling the transaction function more than once (i.e. on multiple entities), the value you get from the current db (in your call to d/entity) will be the same for all those calls - the A in ACID means there is no ‘order’ within a transaction, it all happens at the same time. If you need to have dependent ordering, you could either write a transaction function that handles the multi-entity case manually (i.e. you resolve which of the entities gets what values) or break it up into multiple transactions

jgdavey18:08:06

You probably want something like this

(sort (d/q '[:find [?ident ...]
             :where
             [?e :db/ident ?ident]
             [_ :db.install/attribute ?e]]
           db))

marshall18:08:26

@teng that would list all datoms in the database. if you…. yep what @jgdavey said

jgdavey18:08:49

Conceptually, your query is like saying: “Give me all the datoms in the database, and return a set of the attribute of each”.

tengstrand18:08:58

Great way to look at the problem @jgdavey! Thanks.

jgdavey18:08:30

Yeah, an easily queryable schema comes in very handy.

tengstrand18:08:20

I’m still new to Datomic, but I really like the design and the simplicity of it!

tengstrand18:08:01

I’m a "SQL pro", but I need some more days to adjust the mindset to Datalog!

tengstrand19:08:24

Thought the order only affected the performance?!

jaret20:08:32

@teng I got the same results with both:

jaret20:08:46

(d/q '[:find ?attr ?type ?card
       :where
       [_ :db.install/attribute ?a]
       [?a :db/valueType ?t]
       [?a :db/cardinality ?c]
       [?a :db/ident ?attr]
       [?t :db/ident ?type]
       [?c :db/ident ?card]]
     db)
=>
#{[:db.alter/attribute :db.type/ref :db.cardinality/many]
  [:db.install/function :db.type/ref :db.cardinality/many]
  [:db.install/valueType :db.type/ref :db.cardinality/many]
  [:db.excise/attrs :db.type/ref :db.cardinality/many]
  [:db/ident :db.type/keyword :db.cardinality/one]
  [:db.excise/before :db.type/instant :db.cardinality/one]
  [:db/index :db.type/boolean :db.cardinality/one]
  [:db/fn :db.type/fn :db.cardinality/one]
  [:db/fulltext :db.type/boolean :db.cardinality/one]
  [:db/unique :db.type/ref :db.cardinality/one]
  [:db.excise/beforeT :db.type/long :db.cardinality/one]
  [:db.sys/partiallyIndexed :db.type/boolean :db.cardinality/one]
  [:db/isComponent :db.type/boolean :db.cardinality/one]
  [:db/lang :db.type/ref :db.cardinality/one]
  [:db.sys/reId :db.type/ref :db.cardinality/one]
  [:db.install/partition :db.type/ref :db.cardinality/many]
  [:db/txInstant :db.type/instant :db.cardinality/one]
  [:db/valueType :db.type/ref :db.cardinality/one]
  [:db/cardinality :db.type/ref :db.cardinality/one]
  [:db/excise :db.type/ref :db.cardinality/one]
  [:db/doc :db.type/string :db.cardinality/one]
  [:fressian/tag :db.type/keyword :db.cardinality/one]
  [:db/noHistory :db.type/boolean :db.cardinality/one]
  [:db.install/attribute :db.type/ref :db.cardinality/many]
  [:db/code :db.type/string :db.cardinality/one]
  [:country/name :db.type/string :db.cardinality/one]}
(d/q '[:find ?attr ?type ?card
       :where
       [?a :db/valueType ?t]
       [_ :db.install/attribute ?a]
       [?a :db/cardinality ?c]
       [?a :db/ident ?attr]
       [?t :db/ident ?type]
       [?c :db/ident ?card]]
     db)
=>
#{[:db.alter/attribute :db.type/ref :db.cardinality/many]
  [:db.install/function :db.type/ref :db.cardinality/many]
  [:db.install/valueType :db.type/ref :db.cardinality/many]
  [:db.excise/attrs :db.type/ref :db.cardinality/many]
  [:db/ident :db.type/keyword :db.cardinality/one]
  [:db.excise/before :db.type/instant :db.cardinality/one]
  [:db/index :db.type/boolean :db.cardinality/one]
  [:db/fn :db.type/fn :db.cardinality/one]
  [:db/fulltext :db.type/boolean :db.cardinality/one]
  [:db/unique :db.type/ref :db.cardinality/one]
  [:db.excise/beforeT :db.type/long :db.cardinality/one]
  [:db.sys/partiallyIndexed :db.type/boolean :db.cardinality/one]
  [:db/isComponent :db.type/boolean :db.cardinality/one]
  [:db/lang :db.type/ref :db.cardinality/one]
  [:db.sys/reId :db.type/ref :db.cardinality/one]
  [:db.install/partition :db.type/ref :db.cardinality/many]
  [:db/txInstant :db.type/instant :db.cardinality/one]
  [:db/valueType :db.type/ref :db.cardinality/one]
  [:db/cardinality :db.type/ref :db.cardinality/one]
  [:db/excise :db.type/ref :db.cardinality/one]
  [:db/doc :db.type/string :db.cardinality/one]
  [:fressian/tag :db.type/keyword :db.cardinality/one]
  [:db/noHistory :db.type/boolean :db.cardinality/one]
  [:db.install/attribute :db.type/ref :db.cardinality/many]
  [:db/code :db.type/string :db.cardinality/one]
  [:country/name :db.type/string :db.cardinality/one]}

tengstrand20:08:11

If you @jaret have look at http://www.learndatalogtoday.org/chapter/4, the tab ”2” at the bottom and then click the link ”I give up”, then <Run Qurery> works just fine, but if you change place of the first and the second where criteria, then it doesn’t work. I have the same behavior in my database.

marshall20:08:57

@teng What version of Datomic are you using?

tengstrand20:08:14

But the query should always return the same result, regardless of the order of the predicates?

marshall20:08:16

that would be my expectation. Same set of results. Can you verify that you see the same count when you run against an empty database?

marshall20:08:29

Then re-transact whatever it is you have in your DB and try again

tengstrand20:08:02

I got the same result on an empty database.

marshall20:08:05

I verified that the same thing happens on the mbrainz example database

marshall20:08:10

(different counts)

marshall20:08:16

I’ll look into the reason and get back to you

marshall21:08:12

@teng Were you running against a restore of a db of some kind? If so, I do have an explanation. Datomic 0.9.5206 included a fix for some bootstrap datoms not being in the VAET index. If you run the queries against a database created by a version older than that the result you see is expected.

marshall21:08:40

I will look into creating a new version of the mbrainz example database that doesn’t exhibit this behavior

sdegutis22:08:31

Is there anything else needed to connect a Java program to a Datomic database, besides adding com.datomic/datomic-free:0.9.5372 and doing Connection conn = Peer.connect("datomic:); in my static main function?

sdegutis22:08:59

That line throws ExceptionInitializationError because it can't load "datomic.db__init" internally. The Clojure runtime has loaded, I've verified that.