Fork me on GitHub
#datomic
<
2015-08-19
>
sdegutis00:08:34

Question: is it alright to install an attribute multiple times?

bhagany01:08:00

@sdegutis: I've done it while developing and I'm pretty sure it's idempotent. At least I didn't notice any ill effects.

bhagany01:08:40

probably wise to wait for someone more knowledgable to weigh in though

Alex Miller (Clojure team)02:08:08

generally asserting datoms that match current state does not change anything (except creating transactions)

Alex Miller (Clojure team)02:08:24

but I cannot really attest to a definitive answer, just going by what I've seen

genRaiy09:08:41

@bostonaholic: I need a general purpose way to query the tx data and the entities rather than just getting back the entity affected by the current tx (if I understand your answer!)

genRaiy09:08:12

@robert-stuttaford: I ran the query and get back an error from the program

genRaiy09:08:37

(def tx-id 13194139534372) => #'datomic-customer.core/tx-id (d/q '[:find ?e :where ?t [?e ?t]] db tx-id) IllegalArgumentException Argument ?t in :where is not a list datomic.query/validate-query (query.clj:290)

robert-stuttaford10:08:56

my shoot-from-the-hip coding-in-slack code i gave you was nonsense. here’s the right way:

robert-stuttaford10:08:11

(d/q '[:find ?e :in $ ?t :where [?e _ _ ?t]] db tx-id)

genRaiy10:08:16

thanks robert but now I get

genRaiy10:08:23

(d/q '[:find ?e :in $ ?t :where [?e ?t]] db tx-id) Exception Insufficient bindings, will cause db scan datomic.datalog/fn--6468 (datalog.clj:368)

genRaiy10:08:44

i’m actually struggling to unify to log with the db in my head - are they different query objects?

tcrayford10:08:24

they're different. The log isn't cached either (which may matter a lot for you)

genRaiy10:08:41

ok so maybe I should reiterate my understanding and what I thought I could do and you guys can tell me where I’m right / wrong

genRaiy10:08:08

I thought I could associate some data with a txn and then simply get that data back from the db

genRaiy10:08:16

so maybe I have to use two APIs instead?

tcrayford10:08:52

@raymcdermott: no you can for sure

tcrayford10:08:00

sorry, I need to scroll up a bit

tcrayford10:08:48

(to find out your original question) 😉

tcrayford10:08:14

are you trying to load all the attributes about a specific tx-id?

genRaiy10:08:59

I want to add provenance information to each update on customer records and then be able to view that provenance information when I access the customer record … end goal is that I may prefer to show the data from the customer over data from another source which was more recent though less ‘trustworthy'

genRaiy10:08:32

that sounds a bit messy but we are trying to combine a way to show updates on a per source basis

tcrayford11:08:01

seems reasonable to me. So the trick is: attributes about a transaction use the transaction id as the entity id

genRaiy11:08:04

and I thought tx data might offer that … perhaps another, more explicit design would be better

tcrayford11:08:38

oh, are you going from tx-id to entity then?

tcrayford11:08:47

or tx-id to "attributes about that tx"

genRaiy11:08:29

or the other way around - tell me which tx-ids made this entity data

tcrayford11:08:12

I think that's gonna not fly because of the fact that it'd do a full table scan 😕

genRaiy11:08:13

and from those tx-ids I can find some tx-data

robert-stuttaford11:08:29

ok. you’ll need to use tx-data then

robert-stuttaford11:08:37

the log.html link i pasted earlier

tcrayford11:08:48

yeah. You can use d/pull with a tx-id for getting tx-data, which is dope simple_smile

robert-stuttaford11:08:05

using what pattern, tom?

tcrayford11:08:19

here's a query I ran on yeller's db a while back:

tcrayford11:08:23

(clojure.pprint/pprint (d/q '[:find [(pull ?e [*]) (pull ?t [*])] :where [?e :project/name "stress1"] [?e :project/api-token _ ?t]] (-> webapp :datomic-connection d/db d/history)))

robert-stuttaford11:08:29

all the datoms caused in that tx, or merely the values on the tx itself?

tcrayford11:08:38

ah, values in the tx 😉

tcrayford11:08:51

there's no index the other way except the log

robert-stuttaford11:08:57

ok. ray wants the first one, which means talking to d/log

tcrayford11:08:06

and as mentioned: log is uncached

robert-stuttaford11:08:10

ray, be aware that reads on d/log are not cached

tcrayford11:08:27

(yeller doesn't use the log for anything for just this reason)

robert-stuttaford11:08:34

no peer cache involved, as no index involved. it’s the tx log directly

genRaiy11:08:43

it’s not clear from the link how I go from entity to tx

tcrayford11:08:03

@raymcdermott: you'd have to scan the log, looking for entries with that eid 😕

tcrayford11:08:14

I don't think this solution is gonna work too well though 😕

tcrayford11:08:26

I'd vote towards making an explicit entity for source stuff

genRaiy11:08:49

The more I explore this, the better that option sounds!

robert-stuttaford11:08:54

(d/q '[:find ?e ?a ?v ?added
       :in ?log ?tx
       :where [(tx-data ?log ?tx) [[?e ?a ?v ?tx ?added]]]]
     (d/log conn) tx-id)

robert-stuttaford11:08:22

this is if you have tx-id and you want to know what entities were affected by it

tcrayford11:08:09

for Yeller: I attach stuff to every transaction, but it's just debugging: git sha, which user account was logged in, uri/method of the http request and so on. It's super useful, but I don't think I'd use it for more than that

genRaiy11:08:15

in my case I want to go the other way … what is all the ex data for entity X

tcrayford11:08:57

yeah, so that's hard/bad/slow imo

genRaiy11:08:10

makes sense, perhaps I was just pushing the concept too hard

robert-stuttaford11:08:37

so, all the attrs on all the txes for all modifications to an entity, right?

tcrayford11:08:01

I think specifically "all the datoms that make up the present state of the entity"

genRaiy11:08:33

yes, but specifically including the tx-data that created those datoms

genRaiy11:08:43

The point is that I don’t want to abuse the feature if the performance / queries are all going to be non-idiomatic

genRaiy11:08:02

but I would like it if they were 😉

tcrayford11:08:03

so that's like (d/q '[:find (pull ?t [*]) :in $ ?eid :where [?eid _ _ ?t]] db eid) right? Do you get a full table scan warning there?

tcrayford11:08:37

it should just be: "walk up to eavt, grab datoms about EID, then walk up to eavt, grab all datoms about TX entity"

tcrayford11:08:05

at least: I think tx metadata is stored in eavt as normal entries

genRaiy11:08:16

I didn’t try that although it looks close to Robert’s suggestion earlier which does come with that warning

genRaiy11:08:25

let me give it s a try

tcrayford11:08:56

if not, you can do "get me all the tids for this entry" on top of raw eavt index access yourself

robert-stuttaford11:08:45

(d/q '[:find [(pull ?t [*]) ...] :in $ ?e :where
       [?e _ _ ?t]]
     some-db
     some-id)

robert-stuttaford11:08:51

this works for me

tcrayford11:08:19

(I saw that before the edit)

tcrayford11:08:34

oh, maybe that's ok 😉

robert-stuttaford11:08:36

prod uses TrapperKeeper with nice config management, relax simple_smile

tcrayford11:08:43

maybe I should actually steal that thinking about it

genRaiy11:08:59

(d/q '[:find (pull ?t [*]) :in $ ?eid :where [?eid ?t]] db eid) => [[{:db/id 13194139534372, :db/txInstant #inst "2015-08-15T18:25:13.844-00:00", :data/src "A random place on the Internet, spooky heh?"}]]

genRaiy11:08:08

yes tom, that worked

robert-stuttaford11:08:24

after first cold-cache call, i get 34ms for that query for my test entity with 722 txes

tcrayford11:08:36

(if 34ms is acceptable to you)

tcrayford11:08:05

yeller has a long way to go on pageload times, so I'm just teasing simple_smile

tcrayford11:08:22

(the eventual aim: every pageload is under 100ms, ideally under 50ms)

robert-stuttaford11:08:30

code perf is like playing golf. you never get it perfect

robert-stuttaford11:08:49

yeah i think you’ll be burning your java bytecode onto roms at that point

genRaiy11:08:42

so guys … that’s great stuff … any other warnings / perf issues around what we have now?

robert-stuttaford11:08:18

you should be fine

tcrayford11:08:21

I don't think I have any, iff tx attributes are in eavt

robert-stuttaford11:08:27

it’s idiomatic datalog

robert-stuttaford11:08:51

yup, everything’s in eavt simple_smile

robert-stuttaford11:08:10

ok. have fun ray, tom!

genRaiy11:08:15

nice - thanks!

sdegutis12:08:27

Can you somehow return a map from a Datomic query?

sdegutis13:08:19

Right now I'm doing :find [k v] and (into {} query-result)

tcrayford13:08:05

@sdegutis: see the pull api

tcrayford13:08:18

and/or the entity api

sdegutis13:08:06

I'm not sure the pull API can do this: I'm matching up arbitrary keys to arbitrary values in my query and turning that into a map.

tcrayford13:08:53

ah, so no to that then. I'd strongly recommend using navigation via entities for that stuff over doing it in query (imo anyway)

sdegutis13:08:12

@tcrayford: By navigation do you just mean building the map after the query is done using regular Clojure code?

tcrayford13:08:26

yeah, via mapping over entities you get by calling d/entity on the results of query

PB15:08:58

Is there a reason I can’t do this: (d/q '[:find ?moo :in $ ?account ?moo :where [?account :account/obj ?obj] (or [?obj :obj/moo ?moo] [(missing? $ ?obj :obj/moo)])] (db/db) 17592186045619 17592186046122)

sdegutis15:08:10

How can I find duplicate values of :user/email in a Datomic query?

bostonaholic15:08:33

@sdegutis:

(d/q '[:find ?u1 ?u2
       :in $ ?email
       :where
       [?u1 :user/email ?email]
       [?u2 :user/email ?email]]
     db email)

bostonaholic15:08:37

something like that should work

sdegutis15:08:29

@bostonaholic: Sorry I meant to query the whole database looking for duplicates.

bostonaholic15:08:34

made a couple edits

sdegutis15:08:34

Not for a given email.

bostonaholic15:08:06

can you take what I gave you and adjust it?

bostonaholic15:08:31

what would you need to change in that query if you weren't passing in ?email?

sdegutis15:08:38

The context is that we didn't have a uniqueness constraint on emails because of how we used to handle authentication in the past, but now we've changed that, and I'm looking to see if I can add a uniqueness constraint now, which would require having no current duplicates.

sdegutis15:08:06

My current try was this: (d/q '[:find (count ?email) . :with ?user :where [?user :user/email ?email]] (db))

bostonaholic15:08:35

so you just want the email addresses that are duplicated?

sdegutis15:08:39

But I don't think that actually works, I think it could be filtering out duplicates by the nature of how Datomic inherently uses sets.

sdegutis15:08:52

Slack sucks btw.

bostonaholic15:08:11

take my first example, how can you adjust it without passing in ?email?

sdegutis15:08:56

Hmm I wonder.

sdegutis15:08:07

I don't think it's possible.

bostonaholic15:08:30

what about something like

(d/q '[:find [?email ...]
       :where
       [?u1 :user/email ?email]
       [?u2 :user/email ?email]]
     db)

sdegutis15:08:58

Btw the :in $ is implicit, you can omit that.

bostonaholic15:08:00

if you're not familiar, the [?email ...] syntax will return a collection of emails

sdegutis15:08:17

Hmm interesting.

sdegutis15:08:25

I do vaguely recall it. Didn't remember the [] being necessary around it.

bostonaholic15:08:26

thanks, I just removed the bound ?email and forgot to remove $

sdegutis15:08:40

I'll re-read that section of the docs.

sdegutis15:08:20

@bostonaholic: I see now in the query.html page. Thanks. I'm running the query now, it'll take a few minutes to finish though.

bostonaholic15:08:47

you might also need to add the [(not (= ?u1 ?u2))] clause

sdegutis15:08:52

@bostonaholic: That fails with an error about ?u1 not being resolveable in context.

sdegutis16:08:47

@bostonaholic: btw this seemed to work with reasonable confidence: (d/q '[:find (frequencies ?email) . :with ?user :where [?user :user/email ?email]] (db))

sdegutis16:08:24

Question: is it possible to have an (or ... 0) in the :find clause, for times when you're using something like (count ?e) but the result may be nil?

sdegutis16:08:48

I mean, I know (or ... 0) literally won't work, but I'm wondering if there's something similar to that concept available.

sdegutis16:08:04

Also, is the only difference between :db.unique/value and :db.unique/identity that the latter has upsert behavior and the former doesn't?

sdegutis16:08:34

Question: What's the benefit of Pull? When you e.g. get a user by (d/entity db [:user/email ""]), you can then get anything lazily from it as if it were a nested map, like (:user/name user) or even (-> user (:user/account) (:account/balance)) and it works fine.

PB17:08:41

Is there a reason I can’t do this?

sdegutis17:08:54

Do you often find yourself mapping d/entity over [?ent ...] find results like this? (->> (d/q '[:find [?user ...] :where [?user :user/email]] db) (map (partial d/entity db)))

sdegutis17:08:35

That's a simplistic query, but the idea is that you're querying for a list of entities matching some predicates, and you want to get them as entities.

bostonaholic17:08:40

@sdegutis: I use (pull ?e [*])

bostonaholic17:08:33

[(pull ?user [*]) ...] for your example

PB17:08:41

Anyone?

sdegutis17:08:28

@bostonaholic: Oh smart. I forgot about pull expressions.

sdegutis17:08:38

We've been using a 3-year-old version of Datomic until yesterday.

sdegutis17:08:45

So all these changes are very new to me.

sdegutis17:08:06

We were also on Clojure 1.5.1 until then, now Clojure 1.7 ❤️

bostonaholic17:08:21

@petr: I'm failing to see what you're trying to accomplish with that query

PB17:08:03

@bostonaholic: I’m trying to find ?obj that have :obj/moo set to a value or nil

bostonaholic17:08:27

that will be everything

bostonaholic17:08:34

it's either set, or not

PB17:08:37

No, a specific value or nil

bostonaholic17:08:51

nil doesn't exist in datomic

bostonaholic17:08:58

you cannot set an attribute to nil

PB17:08:14

Let me rephrase. Set to a specific value or not set at all

bostonaholic17:08:02

so "find me all dogs whos names are "Fido" or not set at all?"

PB17:08:06

Correct

sdegutis17:08:12

You can use missing? tho

sdegutis17:08:24

Or whatever it's called. It's a new-ish expresison.

bostonaholic17:08:07

@petr: try unwrapping your missing? call

bostonaholic17:08:31

(or [?obj :obj/moo ?moo]
    (missing? $ ?obj :obj/moo))

PB17:08:35

@sdegutis: It complains when I try to use missing [#{?obj ?moo} #{?obj}

PB17:08:00

@bostonaholic: i get the same error

bostonaholic17:08:28

what's the error?

PB17:08:46

Assert failed: All clauses in 'or' must use same set of vars, had [#{?obj ?moo} #{?obj}] (apply = uvs) datomic.datalog/unifying-vars (datalog.clj:817)

PB17:08:12

Yeah. I’m not really sure how to do this

marshall18:08:22

@petr: If your :obj/moo is cardinality-one, you can use get-else: http://docs.datomic.com/query.html#get-else

PB18:08:44

@marshall: unfortunately it’s not

PB18:08:08

There can be multiple moos

PB18:08:32

I also don’t know that get-else would allow me to use missing?

PB18:08:55

Nor would get-some for that matter

marshall18:08:04

how about this:

(or 
  [?obj :obj/moo ?moo]
  (and
    [(missing? $ ?obj :obj/moo)]
    [(ground :nil) ?moo]))

sdegutis18:08:11

It might not. I didn't notice the get-some part.

marshall18:08:40

^ if it’s missing assign it nil, otherwise assign it the value there

PB18:08:28

@marshall: that’s a good guess. But same result

marshall18:08:52

Ah, think I had a typo. Edited.

bostonaholic18:08:15

does missing? work on :db.cardinality/many?

PB18:08:02

@bostonaholic: It does indeed

bostonaholic18:08:29

just checking. I haven't used it much so I wasn't sure

sdegutis19:08:36

What do the docs mean by the example [(ground [:a :e :i :o :u]) [?vowel ...]] ?

marshall19:08:52

@sdegutis: it binds the ?vowel variable to the values :a :e :i 😮 and :u

marshall19:08:25

yeah, emojis in code ftw

sdegutis20:08:07

I thought you could omit :db/id from a nested map in a transaction if one of the keys was a unique attribute?

sdegutis20:08:13

It's saying java.lang.IllegalArgumentException: :db.error/invalid-nested-entity Nested entity is not a component and has no :db/id

sdegutis20:08:15

Am I misreading the requirement?

bostonaholic20:08:39

I don't believe you can

bostonaholic20:08:57

if I'm not mistaken, all entities in datomic have a :db/id

sdegutis20:08:21

Except components in a transact apparently.

sdegutis20:08:33

The wording in their documentation wasn't clear, but that's effectively what it seems to mean.

sdegutis20:08:49

Whoa! You can use the /_ syntax in pull patterns!

sdegutis20:08:22

(d/pull db '[:user/_orders] [:order/id "abc123"])

sdegutis20:08:39

This returns a list of users who have that order ID!

sdegutis20:08:53

Enjoying this.

PB21:08:58

@sdegutis: You need to supply a tempid to the map if you’re transacting nested entities and the nested entity is not a component of the parent