Question: is it alright to install an attribute multiple times?

@sdegutis: I've done it while developing and I'm pretty sure it's idempotent. At least I didn't notice any ill effects.

probably wise to wait for someone more knowledgable to weigh in though

generally asserting datoms that match current state does not change anything (except creating transactions)

but I cannot really attest to a definitive answer, just going by what I've seen

Thanks.

@bostonaholic: I need a general purpose way to query the tx data and the entities rather than just getting back the entity affected by the current tx (if I understand your answer!)

@robert-stuttaford: I ran the query and get back an error from the program

(def tx-id 13194139534372) => #'datomic-customer.core/tx-id (d/q '[:find ?e :where ?t [?e ?t]] db tx-id) IllegalArgumentException Argument ?t in :where is not a list datomic.query/validate-query (query.clj:290)

sorry @raymcdermott, i’m a dork

my shoot-from-the-hip coding-in-slack code i gave you was nonsense. here’s the right way:

(d/q '[:find ?e :in $ ?t :where [?e _ _ ?t]] db tx-id)

@raymcdermott: look at http://docs.datomic.com/log.html#log-in-query too

thanks robert but now I get

(d/q '[:find ?e :in $ ?t :where [?e ?t]] db tx-id) Exception Insufficient bindings, will cause db scan datomic.datalog/fn--6468 (datalog.clj:368)

i’m actually struggling to unify to log with the db in my head - are they different query objects?

they're different. The log isn't cached either (which may matter a lot for you)

ok so maybe I should reiterate my understanding and what I thought I could do and you guys can tell me where I’m right / wrong

I thought I could associate some data with a txn and then simply get that data back from the db

so maybe I have to use two APIs instead?

@raymcdermott: no you can for sure

sorry, I need to scroll up a bit

(to find out your original question) 😉

are you trying to load all the attributes about a specific tx-id?

I want to add provenance information to each update on customer records and then be able to view that provenance information when I access the customer record … end goal is that I may prefer to show the data from the customer over data from another source which was more recent though less ‘trustworthy'

that sounds a bit messy but we are trying to combine a way to show updates on a per source basis

seems reasonable to me. So the trick is: attributes about a transaction use the transaction id as the entity id

and I thought tx data might offer that … perhaps another, more explicit design would be better

oh, are you going from tx-id to entity then?

or tx-id to "attributes about that tx"

or the other way around - tell me which tx-ids made this entity data

ack yeah

I think that's gonna not fly because of the fact that it'd do a full table scan 😕

and from those tx-ids I can find some tx-data

ok. you’ll need to use tx-data then

the log.html link i pasted earlier

yeah. You can use d/pull with a tx-id for getting tx-data, which is dope

using what pattern, tom?

here's a query I ran on yeller's db a while back:

(clojure.pprint/pprint (d/q '[:find [(pull ?e [*]) (pull ?t [*])] :where [?e :project/name "stress1"] [?e :project/api-token _ ?t]] (-> webapp :datomic-connection d/db d/history)))

all the datoms caused in that tx, or merely the values on the tx itself?

ah, values in the tx 😉

there's no index the other way except the log

ok. ray wants the first one, which means talking to d/log

yeah

and as mentioned: log is uncached

ray, be aware that reads on d/log are not cached

haha ^_^

(yeller doesn't use the log for anything for just this reason)

no peer cache involved, as no index involved. it’s the tx log directly

it’s not clear from the link how I go from entity to tx

@raymcdermott: you'd have to scan the log, looking for entries with that eid 😕

I don't think this solution is gonna work too well though 😕

I'd vote towards making an explicit entity for source stuff

The more I explore this, the better that option sounds!

(d/q '[:find ?e ?a ?v ?added
       :in ?log ?tx
       :where [(tx-data ?log ?tx) [[?e ?a ?v ?tx ?added]]]]
     (d/log conn) tx-id)

this is if you have tx-id and you want to know what entities were affected by it

for Yeller: I attach stuff to every transaction, but it's just debugging: git sha, which user account was logged in, uri/method of the http request and so on. It's super useful, but I don't think I'd use it for more than that

in my case I want to go the other way … what is all the ex data for entity X

s/ex/tx/

yeah, so that's hard/bad/slow imo

thinking

makes sense, perhaps I was just pushing the concept too hard

so, all the attrs on all the txes for all modifications to an entity, right?

yes

I think specifically "all the datoms that make up the present state of the entity"

yes, but specifically including the tx-data that created those datoms

The point is that I don’t want to abuse the feature if the performance / queries are all going to be non-idiomatic

but I would like it if they were 😉

so that's like (d/q '[:find (pull ?t [*]) :in $ ?eid :where [?eid _ _ ?t]] db eid) right? Do you get a full table scan warning there?

it should just be: "walk up to eavt, grab datoms about EID, then walk up to eavt, grab all datoms about TX entity"

at least: I think tx metadata is stored in eavt as normal entries

I didn’t try that although it looks close to Robert’s suggestion earlier which does come with that warning

let me give it s a try

if not, you can do "get me all the tids for this entry" on top of raw eavt index access yourself

(d/q '[:find [(pull ?t [*]) ...] :in $ ?e :where
       [?e _ _ ?t]]
     some-db
     some-id)

this works for me

(I saw that before the edit)

dev-mode only

oh, maybe that's ok 😉

prod uses TrapperKeeper with nice config management, relax

maybe I should actually steal that thinking about it

boom

(d/q '[:find (pull ?t [*]) :in $ ?eid :where [?eid ?t]] db eid) => [[{:db/id 13194139534372, :db/txInstant #inst "2015-08-15T18:25:13.844-00:00", :data/src "A random place on the Internet, spooky heh?"}]]

yay

yes tom, that worked

after first cold-cache call, i get 34ms for that query for my test entity with 722 txes

so, perf is fine

(if 34ms is acceptable to you)

its not seconds

yeah

and it caches

yeller has a long way to go on pageload times, so I'm just teasing

😁

(the eventual aim: every pageload is under 100ms, ideally under 50ms)

code perf is like playing golf. you never get it perfect

yeah i think you’ll be burning your java bytecode onto roms at that point

so guys … that’s great stuff … any other warnings / perf issues around what we have now?

you should be fine

I don't think I have any, iff tx attributes are in eavt

it’s idiomatic datalog

yup, everything’s in eavt

ok. have fun ray, tom!

nice - thanks!

Can you somehow return a map from a Datomic query?

Right now I'm doing :find [k v] and (into {} query-result)

@sdegutis: see the pull api

and/or the entity api

I'm not sure the pull API can do this: I'm matching up arbitrary keys to arbitrary values in my query and turning that into a map.

ah, so no to that then. I'd strongly recommend using navigation via entities for that stuff over doing it in query (imo anyway)

@tcrayford: By navigation do you just mean building the map after the query is done using regular Clojure code?

yeah, via mapping over entities you get by calling d/entity on the results of query

Is there a reason I can’t do this: (d/q '[:find ?moo :in $ ?account ?moo :where [?account :account/obj ?obj] (or [?obj :obj/moo ?moo] [(missing? $ ?obj :obj/moo)])] (db/db) 17592186045619 17592186046122)

How can I find duplicate values of :user/email in a Datomic query?

@sdegutis:

(d/q '[:find ?u1 ?u2
       :in $ ?email
       :where
       [?u1 :user/email ?email]
       [?u2 :user/email ?email]]
     db email)

something like that should work

Hmm.

@bostonaholic: Sorry I meant to query the whole database looking for duplicates.

made a couple edits

Not for a given email.

can you take what I gave you and adjust it?

what would you need to change in that query if you weren't passing in ?email?

The context is that we didn't have a uniqueness constraint on emails because of how we used to handle authentication in the past, but now we've changed that, and I'm looking to see if I can add a uniqueness constraint now, which would require having no current duplicates.

been there

My current try was this: (d/q '[:find (count ?email) . :with ?user :where [?user :user/email ?email]] (db))

so you just want the email addresses that are duplicated?

But I don't think that actually works, I think it could be filtering out duplicates by the nature of how Datomic inherently uses sets.

Yep.

Slack sucks btw.

take my first example, how can you adjust it without passing in ?email?

Hmm I wonder.

I don't think it's possible.

what about something like

(d/q '[:find [?email ...]
       :where
       [?u1 :user/email ?email]
       [?u2 :user/email ?email]]
     db)

Btw the :in $ is implicit, you can omit that.

if you're not familiar, the [?email ...] syntax will return a collection of emails

Hmm interesting.

I do vaguely recall it. Didn't remember the [] being necessary around it.

thanks, I just removed the bound ?email and forgot to remove $

I'll re-read that section of the docs.

@bostonaholic: I see now in the query.html page. Thanks. I'm running the query now, it'll take a few minutes to finish though.

you might also need to add the [(not (= ?u1 ?u2))] clause

@bostonaholic: That fails with an error about ?u1 not being resolveable in context.

@bostonaholic: btw this seemed to work with reasonable confidence: (d/q '[:find (frequencies ?email) . :with ?user :where [?user :user/email ?email]] (db))

Question: is it possible to have an (or ... 0) in the :find clause, for times when you're using something like (count ?e) but the result may be nil?

I mean, I know (or ... 0) literally won't work, but I'm wondering if there's something similar to that concept available.

Also, is the only difference between :db.unique/value and :db.unique/identity that the latter has upsert behavior and the former doesn't?

Question: What's the benefit of Pull? When you e.g. get a user by (d/entity db [:user/email ""]), you can then get anything lazily from it as if it were a nested map, like (:user/name user) or even (-> user (:user/account) (:account/balance)) and it works fine.

Is there a reason I can’t do this?

Do you often find yourself mapping d/entity over [?ent ...] find results like this? (->> (d/q '[:find [?user ...] :where [?user :user/email]] db) (map (partial d/entity db)))

That's a simplistic query, but the idea is that you're querying for a list of entities matching some predicates, and you want to get them as entities.

@sdegutis: I use (pull ?e [*])

[(pull ?user [*]) ...] for your example

Anyone?

@bostonaholic: Oh smart. I forgot about pull expressions.

We've been using a 3-year-old version of Datomic until yesterday.

whoa

So all these changes are very new to me.

We were also on Clojure 1.5.1 until then, now Clojure 1.7 ❤️

@petr: I'm failing to see what you're trying to accomplish with that query

@sdegutis: good for you!

@bostonaholic: I’m trying to find ?obj that have :obj/moo set to a value or nil

that will be everything

it's either set, or not

No, a specific value or nil

nil doesn't exist in datomic

you cannot set an attribute to nil

Let me rephrase. Set to a specific value or not set at all

ah, ok

so "find me all dogs whos names are "Fido" or not set at all?"

Correct

You can use missing? tho

Or whatever it's called. It's a new-ish expresison.

@petr: try unwrapping your missing? call

(or [?obj :obj/moo ?moo]
    (missing? $ ?obj :obj/moo))

@sdegutis: It complains when I try to use missing [#{?obj ?moo} #{?obj}

@bostonaholic: i get the same error

what's the error?

Assert failed: All clauses in 'or' must use same set of vars, had [#{?obj ?moo} #{?obj}] (apply = uvs) datomic.datalog/unifying-vars (datalog.clj:817)

hm

Yeah. I’m not really sure how to do this

@petr: If your :obj/moo is cardinality-one, you can use get-else: http://docs.datomic.com/query.html#get-else

@marshall: unfortunately it’s not

There can be multiple moos

I also don’t know that get-else would allow me to use missing?

Nor would get-some for that matter

how about this:

(or 
  [?obj :obj/moo ?moo]
  (and
    [(missing? $ ?obj :obj/moo)]
    [(ground :nil) ?moo]))

It might not. I didn't notice the get-some part.

^ if it’s missing assign it nil, otherwise assign it the value there

@marshall: that’s a good guess. But same result

Ah, think I had a typo. Edited.

does missing? work on :db.cardinality/many?

@bostonaholic: It does indeed

just checking. I haven't used it much so I wasn't sure

What do the docs mean by the example [(ground [:a :e :i :o :u]) [?vowel ...]] ?

@sdegutis: it binds the ?vowel variable to the values :a :e :i 😮 and :u

😮

lol

@sdegutis: http://docs.datomic.com/query.html#collection-binding

yeah, emojis in code ftw

okay

I thought you could omit :db/id from a nested map in a transaction if one of the keys was a unique attribute?

It's saying java.lang.IllegalArgumentException: :db.error/invalid-nested-entity Nested entity is not a component and has no :db/id

Am I misreading the requirement?

I don't believe you can

if I'm not mistaken, all entities in datomic have a :db/id

Except components in a transact apparently.

The wording in their documentation wasn't clear, but that's effectively what it seems to mean.

Whoa! You can use the /_ syntax in pull patterns!

(d/pull db '[:user/_orders] [:order/id "abc123"])

This returns a list of users who have that order ID!

Enjoying this.

@sdegutis: You need to supply a tempid to the map if you’re transacting nested entities and the nested entity is not a component of the parent