Fork me on GitHub
#datomic
<
2016-02-28
>
casperc09:02:02

@val_waeselynck: I think the problem with that implementation is that it only looks at the parent entity and not all the entities in the pull for determining the tx’es to look at.

casperc09:02:59

E.g. my pull might look something like this:

[* {:building/_property-ref [*}]
where I am pulling on the property entity and using a reverse lookup from the building entity to pull that as well. I want to have all the versions of the whole pull (meaning in this case also the tx’es that changes building entity).

casperc09:02:32

My thinking is that I need to parse the pull expression and follow the lookups and reverse lookups in order to get all the tx’es in the “tree” structure being pulled.

val_waeselynck17:02:00

@casperc: 😕 not sure how I would go about this then

val_waeselynck17:02:52

the thing is, as you pull in other entities, these may add other versions of the database as well

val_waeselynck17:02:05

I don't know how the pull clauses behave relative to the history db

casperc17:02:14

@val_waeselynck: Yeah, I would need to follow the refs in all the “versions” of the entities, so it might end up being alot. For my usecase, mostly the refs will be constant (but obviously, I still need to write the code to take that into account)

casperc17:02:14

But if I can find the tx’es, it’s pretty much doing what you are doing by pulling once for each tx

val_waeselynck17:02:45

I think I would split the pull clause into a set of paths, then programmatically build a query which gets the txs of the terminal datoms of the paths on a history db

val_waeselynck17:02:08

@casperc: actually that seems pretty doable

casperc17:02:53

Ah interesting, you mean paths in terms of the actual entities that the pull hits through history?

val_waeselynck17:02:04

not entities, datoms

val_waeselynck17:02:50

well the paths would just be lists of datalog clauses actually

casperc17:02:49

I am not totally sure I understand your thinking, but I am thinking that each lookup, e.g. :building/_property in the pull would be a query in the history db

val_waeselynck17:02:20

[:a {:b [:c]}] -> #{([?e1 :a _ ?tx]) ([?e1 :b _ ?tx]) ([?e1 :b ?e2] [?e2 :c _ ?tx])}

val_waeselynck17:02:18

then the query you build would be

val_waeselynck17:02:40

let me put that in a snippet

val_waeselynck18:02:45

@casperc: voila, you'd build this query programmatically and run it on the history, this should yield the correct txes

casperc18:02:27

@val_waeselynck: Ha, that is quite awesome actually simple_smile

val_waeselynck18:02:56

oh and for the * clauses all you'd have to do is put an _ in attribute position

casperc18:02:56

@val_waeselynck: I get the principle and I agree that an approach like this is the way to go, I think I need some time to understand it though tbh simple_smile

casperc18:02:30

I don’t know 'rules-to-match’ for instance, so need to look that up simple_smile

val_waeselynck18:02:41

oh that's nothing

val_waeselynck18:02:57

just a placeholder for whatever logic you use to find the root entity

casperc18:02:29

Makes sense simple_smile

casperc18:02:38

@val_waeselynck: But again, quite awesome. Mind if I ping you for a sanity check when I have a working implementation?

val_waeselynck18:02:56

sure, I'd gladly do it right now if I had the time, let me know

casperc18:02:16

Thanks, I appreciate it.

casperc18:02:41

I am considering making a library for datomic common patterns, since I keep coming up against problems that I suspect other people are having (or have had) as well.

val_waeselynck18:02:35

@casperc: yeah that would be useful

val_waeselynck18:02:03

what's more, this could be a data library, i.e a library of common attributes definitions or transaction functions

casperc18:02:37

Yep for sure. Plenty of transaction functions are being written over and over again i am sure. And it is a real barrier for entry in using Datomic, when common usecases aren’t available as a library.

val_waeselynck18:02:04

@casperc, having read the docs a bit, I think you can't easily build the set of paths directly from the pull clause because of the way pull works wrt component attributes, recursive attributes etc.

val_waeselynck18:02:56

So I think the safest course of action would be: 1. run your pull clause on the history db 2. derive the set of paths from the result 3. build the query which finds the set of txes 4. then run you pull clause on each of the databases of the txes Given the time steps 3 and 4 will take I don't think this adds a lot of overhead

casperc18:02:15

@val_waeselynck: I think you are right about component entities. I guess in principle it would be possible to query the schema to still construct the query programmatically as you originally proposed (just adding in the ones that are components from the schema).

val_waeselynck18:02:11

@casperc: that could lead to an infinity of paths

val_waeselynck18:02:43

(components attributes could form a cycle)

casperc18:02:25

@val_waeselynck: I never considered doing the pull against the history db. Does that return all the datums that the pull matches throughout the history?

casperc18:02:38

@val_waeselynck: Hmm, you may be right, but then wouldn’t the pull do that if performed normally as well?

val_waeselynck18:02:45

@casperc: never tried, but I guess it's as if all your attributes had a cardinality many

val_waeselynck18:02:14

@casperc: no, because datoms might form a cycle doesn't mean they do

val_waeselynck18:02:49

actually, forget what I said about cycles. All I mean is that there's no limit to the length of a path that a pull clause could yield, because of components and recursive rules

val_waeselynck18:02:24

@casperc: gotta go now, have fun!

casperc18:02:41

@val_waeselynck: See ya, and thanks for the input simple_smile

val_waeselynck20:02:57

@casperc: (seems I cannot get my head off this idea). Instead of deriving the paths from the results of step 1, just derived the datoms (i.e a set of [entity attribute value]) and run a query to find the txes of this datoms.

casperc21:02:21

@val_waeselynck: Hehe, not complaining here, and I think it is an interesting problem as well.

casperc21:02:20

@val_waeselynck: I actually tried doing a pull query against a history db and it is not supported, so unless I am missing something, I don’t think that would work.

casperc21:02:06

(ffirst (d/q '[:find (pull ?e [*])
                                 :where [?e :building/id]]
                               (d/history (d/db conn))))
IllegalStateException Can't pull from history  datomic.pull/pull* (pull.clj:291)

casperc21:02:42

@val_waeselynck: My current thinking is to use the entity api to traverse the pull expression. Each entity/lookup will be traversed in all its versions, to find all the txes.

casperc21:02:52

Of course implementation is still pending. I hope to have time for it tomorrow.

val_waeselynck22:02:55

@casperc: aaaarg, yeah I was afraid it would not be supported. FWIW if I wanted to be exhaustive my approach would probably be to use the indexes directly to get the equivalent of this "pull history" query.

casperc22:02:18

@val_waeselynck: That could be an option. One reason I want to use the entity api though, is to get the same semantics for components that pull has