Fork me on GitHub
#datomic
<
2017-02-22
>
bbloom00:02:47

i’m curious how folks are handling joining/merging data from systems that aren’t quite as, ahem, time-friendly, as datomic

bbloom00:02:24

graphql endpoints for example tend to just give you some inconsistent interleaving of whatever backend services happen to get queried at the time - which seems like the best you can do for distributed systems w/o shared clocks (like the transaction id number) - and it also seems like you’re just out of luck for merging db history in many cases

bbloom00:02:48

i realize i’m rambling a bit, but wondering if any of the folks here have spent time thinking about this

nottmey09:02:43

Now I’m curious how you would do it even with only datomic databases/transactors. Using the asOf filter with the request time for each? This still leaves you with inconsistencies, when the data is not transacted atomically over both databases/systems. But how does datomic help with this? It looks like you still need the same techniques you mentioned (whether you have datomic or not).

pesterhazy11:02:15

@bbloom, when I encounter such a problem, I stare it in a face and then walk away

pesterhazy11:02:47

"ignoring the problem" usually works

stijn14:02:02

is there a way to convert an Entity to a map recursively, but only include the data that's in the entity cache?

stijn14:02:35

(other than using the pull api)

pesterhazy14:02:21

(into {} entity)

pesterhazy14:02:43

oh well that's not recursive but you can extend that pretty easily

pesterhazy14:02:13

if you need a quick and dirty way, (-> entity pr-str read-string) may work 🙂

rauh14:02:58

@stijn In reflection i can see the (.cache ..) method

stijn14:02:45

@rauh: that's exactly what I would need!

stijn14:02:55

it is probably not a public api then

rauh14:02:12

Well it's not "java-private" 🙂

bbloom19:02:02

Having re-read some stuff about transaction processing and database isolation, I think I can better formulate my question from above ^^ but also formulate my il-conceived desires

bbloom19:02:02

Datomic’s entity API provides snapshot isolation, which is awesome. However, most GraphQL resolvers provide repeatable read committed isolation in that it lets you read from disparate backend services and memoizes all of the results.

bbloom19:02:01

I almost want an entity API that offers that behavior inherently, such that i can effectively join up data from multiple sources with the nice object-oriented-ish navigatable entity graph

bbloom19:02:26

So my question then is: Has anybody here tried anything like that? Curious to see solutions.

ddellacosta19:02:33

apologies if this is a FAQ but is there more in-depth documentation for how indexes work in Datomic past what is here? http://docs.datomic.com/indexes.html

ddellacosta19:02:59

I’d also love to have something like explain for PostgreSQL for datomic queries, but guessing that’s just a dream

ddellacosta19:02:49

I saw that but it’s kinda old, and seems like one dude’s conception of what is actually going on in a query…is it actually a good thing?

ddellacosta19:02:05

will poke at the second one, thanks

ddellacosta19:02:29

that unofficial guide already seems better than the official docs

favila19:02:47

q-explain is a very well educated guess. It doesn't always match up because of bugs in datomic datalog, but there is very little query optimization that goes on

ddellacosta19:02:50

but why they don’t include this kind of info in the official docs makes no sense

ddellacosta19:02:16

@favila I see…so some of this arises naturally based on how datalog works, is that right? That is helpful to understand if so

ddellacosta19:02:32

in the sense of, queries can only really work in certain ways

favila19:02:16

@ddellacosta We know for a fact that datomic does not reorder where clauses, does not have any index statistics, etc. So mostly what the query plan can do just follows what indexes are available

favila19:02:54

I think he made a good effort to discern which datomic actually does

ddellacosta19:02:22

gotcha, thanks for the context…I’m assuming you know that because of how datomic datalog works (insofar as it is documented publicly) and based on other statements Cognitect folks have made…?

favila19:02:32

and I remember he got the query grammar docs corrected in a few places during is work, so it's not a half-assed effort

favila19:02:54

(that said, I don't use it)

ddellacosta19:02:09

yeah to be clear I didn’t think it was half-assed or anything, it’s just barely documented and old-ish, and figuring out its worth is, in and of itself, a bit of an effort

ddellacosta19:02:14

but will check it out further

favila19:02:23

google groups, talks, etc

favila19:02:33

same sources as for the internals guide

ddellacosta19:02:44

thanks @favila, all of this is super helpful, much appreciated

favila19:02:06

q-explain looks like it is useful to give someone who is unfamiliar with datalog some idea of how a query will perform

ddellacosta19:02:43

well, it could be helpful to me then because I feel like I’m cargo culting it much of the time, even if I can generally make it do what I want at this point

favila19:02:43

or who is unfamiliar with the data in their db (e.g. selectivity of attributes, cardinality of values, etc)

ddellacosta19:02:50

but performance is still a bit of a mystery to me at times

ddellacosta19:02:10

compared to how I used to construct SQL queries in PostgreSQL

ddellacosta19:02:13

anyways, I digress

favila19:02:46

sql optimizers are generally much, much more complicated and aggressive than datomic

ddellacosta19:02:05

even that I didn’t realize

favila19:02:42

glad to help