i’m curious how folks are handling joining/merging data from systems that aren’t quite as, ahem, time-friendly, as datomic

graphql endpoints for example tend to just give you some inconsistent interleaving of whatever backend services happen to get queried at the time - which seems like the best you can do for distributed systems w/o shared clocks (like the transaction id number) - and it also seems like you’re just out of luck for merging db history in many cases

i realize i’m rambling a bit, but wondering if any of the folks here have spent time thinking about this

Now I’m curious how you would do it even with only datomic databases/transactors. Using the asOf filter with the request time for each? This still leaves you with inconsistencies, when the data is not transacted atomically over both databases/systems. But how does datomic help with this? It looks like you still need the same techniques you mentioned (whether you have datomic or not).

@bbloom, when I encounter such a problem, I stare it in a face and then walk away

"ignoring the problem" usually works

is there a way to convert an Entity to a map recursively, but only include the data that's in the entity cache?

(other than using the pull api)

(into {} entity)

oh well that's not recursive but you can extend that pretty easily

if you need a quick and dirty way, (-> entity pr-str read-string) may work 🙂

🙂

@stijn In reflection i can see the (.cache ..) method

@rauh: that's exactly what I would need!

it is probably not a public api then

Well it's not "java-private" 🙂

Having re-read some stuff about transaction processing and database isolation, I think I can better formulate my question from above ^^ but also formulate my il-conceived desires

Datomic’s entity API provides snapshot isolation, which is awesome. However, most GraphQL resolvers provide repeatable read committed isolation in that it lets you read from disparate backend services and memoizes all of the results.

I almost want an entity API that offers that behavior inherently, such that i can effectively join up data from multiple sources with the nice object-oriented-ish navigatable entity graph

So my question then is: Has anybody here tried anything like that? Curious to see solutions.

apologies if this is a FAQ but is there more in-depth documentation for how indexes work in Datomic past what is here? http://docs.datomic.com/indexes.html

I’d also love to have something like explain for PostgreSQL for datomic queries, but guessing that’s just a dream

@ddellacosta query explain: https://github.com/dwhjames/datomic-q-explain

@ddellacosta datomic internals: http://tonsky.me/blog/unofficial-guide-to-datomic-internals/

I saw that but it’s kinda old, and seems like one dude’s conception of what is actually going on in a query…is it actually a good thing?

will poke at the second one, thanks

that unofficial guide already seems better than the official docs

q-explain is a very well educated guess. It doesn't always match up because of bugs in datomic datalog, but there is very little query optimization that goes on

but why they don’t include this kind of info in the official docs makes no sense

@favila I see…so some of this arises naturally based on how datalog works, is that right? That is helpful to understand if so

in the sense of, queries can only really work in certain ways

@ddellacosta We know for a fact that datomic does not reorder where clauses, does not have any index statistics, etc. So mostly what the query plan can do just follows what indexes are available

I think he made a good effort to discern which datomic actually does

gotcha, thanks for the context…I’m assuming you know that because of how datomic datalog works (insofar as it is documented publicly) and based on other statements Cognitect folks have made…?

and I remember he got the query grammar docs corrected in a few places during is work, so it's not a half-assed effort

(that said, I don't use it)

yeah to be clear I didn’t think it was half-assed or anything, it’s just barely documented and old-ish, and figuring out its worth is, in and of itself, a bit of an effort

but will check it out further

google groups, talks, etc

same sources as for the internals guide

k

thanks @favila, all of this is super helpful, much appreciated

q-explain looks like it is useful to give someone who is unfamiliar with datalog some idea of how a query will perform

well, it could be helpful to me then because I feel like I’m cargo culting it much of the time, even if I can generally make it do what I want at this point

or who is unfamiliar with the data in their db (e.g. selectivity of attributes, cardinality of values, etc)

but performance is still a bit of a mystery to me at times

compared to how I used to construct SQL queries in PostgreSQL

anyways, I digress

thanks again

sql optimizers are generally much, much more complicated and aggressive than datomic

gotcha

even that I didn’t realize

glad to help