Fork me on GitHub

@val_waeselynck: I think the problem with that implementation is that it only looks at the parent entity and not all the entities in the pull for determining the tx’es to look at.


E.g. my pull might look something like this:

[* {:building/_property-ref [*}]
where I am pulling on the property entity and using a reverse lookup from the building entity to pull that as well. I want to have all the versions of the whole pull (meaning in this case also the tx’es that changes building entity).


My thinking is that I need to parse the pull expression and follow the lookups and reverse lookups in order to get all the tx’es in the “tree” structure being pulled.


@casperc: 😕 not sure how I would go about this then


the thing is, as you pull in other entities, these may add other versions of the database as well


I don't know how the pull clauses behave relative to the history db


@val_waeselynck: Yeah, I would need to follow the refs in all the “versions” of the entities, so it might end up being alot. For my usecase, mostly the refs will be constant (but obviously, I still need to write the code to take that into account)


But if I can find the tx’es, it’s pretty much doing what you are doing by pulling once for each tx


I think I would split the pull clause into a set of paths, then programmatically build a query which gets the txs of the terminal datoms of the paths on a history db


@casperc: actually that seems pretty doable


Ah interesting, you mean paths in terms of the actual entities that the pull hits through history?


not entities, datoms


well the paths would just be lists of datalog clauses actually


I am not totally sure I understand your thinking, but I am thinking that each lookup, e.g. :building/_property in the pull would be a query in the history db


[:a {:b [:c]}] -> #{([?e1 :a _ ?tx]) ([?e1 :b _ ?tx]) ([?e1 :b ?e2] [?e2 :c _ ?tx])}


then the query you build would be


let me put that in a snippet


@casperc: voila, you'd build this query programmatically and run it on the history, this should yield the correct txes


@val_waeselynck: Ha, that is quite awesome actually simple_smile


oh and for the * clauses all you'd have to do is put an _ in attribute position


@val_waeselynck: I get the principle and I agree that an approach like this is the way to go, I think I need some time to understand it though tbh simple_smile


I don’t know 'rules-to-match’ for instance, so need to look that up simple_smile


oh that's nothing


just a placeholder for whatever logic you use to find the root entity


Makes sense simple_smile


@val_waeselynck: But again, quite awesome. Mind if I ping you for a sanity check when I have a working implementation?


sure, I'd gladly do it right now if I had the time, let me know


Thanks, I appreciate it.


I am considering making a library for datomic common patterns, since I keep coming up against problems that I suspect other people are having (or have had) as well.


@casperc: yeah that would be useful


what's more, this could be a data library, i.e a library of common attributes definitions or transaction functions


Yep for sure. Plenty of transaction functions are being written over and over again i am sure. And it is a real barrier for entry in using Datomic, when common usecases aren’t available as a library.


@casperc, having read the docs a bit, I think you can't easily build the set of paths directly from the pull clause because of the way pull works wrt component attributes, recursive attributes etc.


So I think the safest course of action would be: 1. run your pull clause on the history db 2. derive the set of paths from the result 3. build the query which finds the set of txes 4. then run you pull clause on each of the databases of the txes Given the time steps 3 and 4 will take I don't think this adds a lot of overhead


@val_waeselynck: I think you are right about component entities. I guess in principle it would be possible to query the schema to still construct the query programmatically as you originally proposed (just adding in the ones that are components from the schema).


@casperc: that could lead to an infinity of paths


(components attributes could form a cycle)


@val_waeselynck: I never considered doing the pull against the history db. Does that return all the datums that the pull matches throughout the history?


@val_waeselynck: Hmm, you may be right, but then wouldn’t the pull do that if performed normally as well?


@casperc: never tried, but I guess it's as if all your attributes had a cardinality many


@casperc: no, because datoms might form a cycle doesn't mean they do


actually, forget what I said about cycles. All I mean is that there's no limit to the length of a path that a pull clause could yield, because of components and recursive rules


@casperc: gotta go now, have fun!


@val_waeselynck: See ya, and thanks for the input simple_smile


@casperc: (seems I cannot get my head off this idea). Instead of deriving the paths from the results of step 1, just derived the datoms (i.e a set of [entity attribute value]) and run a query to find the txes of this datoms.


@val_waeselynck: Hehe, not complaining here, and I think it is an interesting problem as well.


@val_waeselynck: I actually tried doing a pull query against a history db and it is not supported, so unless I am missing something, I don’t think that would work.


(ffirst (d/q '[:find (pull ?e [*])
                                 :where [?e :building/id]]
                               (d/history (d/db conn))))
IllegalStateException Can't pull from history  datomic.pull/pull* (pull.clj:291)


@val_waeselynck: My current thinking is to use the entity api to traverse the pull expression. Each entity/lookup will be traversed in all its versions, to find all the txes.


Of course implementation is still pending. I hope to have time for it tomorrow.


@casperc: aaaarg, yeah I was afraid it would not be supported. FWIW if I wanted to be exhaustive my approach would probably be to use the indexes directly to get the equivalent of this "pull history" query.


@val_waeselynck: That could be an option. One reason I want to use the entity api though, is to get the same semantics for components that pull has