Fork me on GitHub
#datomic
<
2016-05-19
>
arthur.boyer05:05:10

I’m trying to work out which partitions the various attributes in the database belong to. I can get a list of partitions:

(defn get-partitions [db]
  (d/q '[:find ?ident
         :where [:db.part/db :db.install/partition ?p]
         [?p :db/ident ?ident]]
       db))
And I can get all the attributes:
(d/q '[:find ?attr ?name
       :where
       [_ :db.install/attribute ?attr]
       [?attr :db/ident ?name]]
     db)
But I can’t seem to find how they’re connected.

eploko11:05:13

Hey all. I’m quite new to Datomic and have an issue trying to query the database in the Console. I have a dev transactor and the console app running and can access the Console UI on http://localhost:9000/browse, but… whenever I try to click on the database selection dropdown in the UI, or pretty much do anything else, the only thing that happens is a browser refresh and I’m back to the initial state. The browser console shows this:

0.js:6027 XHR finished loading: POST "…Fbrowse&v-wn=browse-1380604278-0.20254203379239133&v-wsver=7.1.10&v-uiId=8".qy @ 0.js:6027Hec @ 0.js:6116afc @ 0.js:6068Dec @ 0.js:6274Gec @ 0.js:5870Ifc @ 0.js:6374Xk @ 0.js:5587Nk @ 0.js:4640tk @ 0.js:4415sk @ 0.js:5616(anonymous function) @ 0.js:4646
Navigated to 
and nothing else. Any ideas what can be the cause of this and how this can be fixed? I’m running datomic-pro 0.9.5359 and the corresponding Console that comes with the distribution.

eploko11:05:09

Also, if that ever matters, I’m on OS X 10.11.4 and run Chrome 50.0.2661.102. I’ve tried to access the console in Safari, but it behaves in exactly the same way.

eploko11:05:13

Ha. Apparently the console that comes with 0.9.5359 is broken somehow. I’ve just downloaded the separate Console distribution from http://my.datomic.com and it works fine. 🙂

Dustin Getz15:05:57

Whena datomic peer does a query like [:find (count ?r :where [complicated]], and the jvm peer cache is cold, how does datomic know what pages of entities to load from storage?

Lambda/Sierra17:05:06

@dustingetz: The query engine evaluates the datalog clauses in order, fetching index segments from storage as needed.

Lambda/Sierra17:05:28

It knows which index segments to fetch based on which parts of the datalog clause are constants or bound variables.

adamfrey18:05:22

Using (d/tx-range (d/log conn) nil nil) gives me a lazy iterator over the log of transactions from past to present. But is there anyway to get reverse time access to transactions? e.g. Find the 10 most recent transactions?

Ben Kamphaus18:05:41

@adamfrey: you’ll want to figure out from the basis-t of the latest db what a reasonable most recent bound to use to pass as a first argument there (not super simple because there’s not a one to one relationship, though maybe simpler with t->tx that way), then do usual clojure sequence manipulation to get the last X values.

jeff.terrell18:05:49

@bkamphaus: Not really sure whether I or you are missing something here 🙂, but wouldn't your approach only give what's happened since a point in time, rather than iterating backwards through time?

Ben Kamphaus18:05:22

@jeff.terrell: I think you’re not inferring enough from the “figure out from” part of my sentence. To be fair, it’s a long sentence.

Ben Kamphaus18:05:48

but yes, something like (subtract 1000 from the latest basis-t) as a dumb heuristic to only have a small collection to do seq manipulation on.

Ben Kamphaus18:05:31

there’s isn’t a direct otherwise (last 10), it’s get something you know will be small and do the last 10 of that.

Ben Kamphaus18:05:41

There is the tx report queue if you want to hold onto the last few things in a live situation

jeff.terrell18:05:54

OK, that makes sense. Thanks for explaining. That would solve the problem, but a bit awkwardly (I should perhaps mention that @adamfrey and I are trying to solve the same problem). Is there no way to get a lazy sequence from Datomic (whether transactions, datoms, or whatever) that's sorted by time, most recent first? Even if it were fairly low-level, I imagine that would be easier to work with in the end.

Ben Kamphaus18:05:45

nothing in api for it at present

jeff.terrell18:05:21

@bkamphaus: Fair enough. I'm not sure whether y'all discuss such things—and please forgive the question if you don't—but do y'all have any plans to add a feature like that? It sure would make our problem (ultimately, paging through most recent things in a webapp) easier, and I imagine our problem is not very unusual, especially for Datomic-backed webapps.

Ben Kamphaus18:05:11

can’t pass on specific plans, but I’ll note the feature request. For most recently updated entities, etc. though note that log is still ordered by all transactions.

Ben Kamphaus18:05:44

entities are ordered monotonically by transaction time by their id, but_only by partition_ (i.e. :db.part/user :db.part/mydomain), so can’t infer anything if they span partitions.

jeff.terrell18:05:34

@bkamphaus: Thanks, I appreciate that. Good to know about the partitioning. I expect that will be more helpful than harmful for us, but definitely worth noting, thanks. I'm not sure what you mean that the log is only the last few transactions. I think the log does go back to the first transaction, right?

Ben Kamphaus18:05:03

just caught me editing

Ben Kamphaus18:05:27

brain to fingers message garble. I mean that it was ordered by all transactions, so not constrained by “last five of these types of entities"

jeff.terrell18:05:23

Right, OK, that makes sense. Thanks again for explaining all that.

adamfrey18:05:24

@bkamphaus: so to start receiving the log at a given time for a given partition I have to use entid-at right?

adamfrey18:05:11

as opposed to a bare Date being passed to tx-range

adamfrey18:05:13

wait, that might be wrong. when you said "entities are ordered monotonically by transaction time by their id, but_only by partition_” where were you talking entities being ordered?

Ben Kamphaus18:05:53

@adamfrey: entid-ad just simulates/fabricates a new entity id reachable from a db -- entities are ordered in datoms/seek-datoms :eavt

adamfrey18:05:42

yes I see that now. but could explain this a little further, please: https://clojurians.slack.com/archives/datomic/p1463683184000677

Ben Kamphaus18:05:09

you’ll have to tell me what’s not clear before I know what to explain differently 🙂

Ben Kamphaus18:05:53

or maybe you’re feeling in catch 22 territory

Ben Kamphaus18:05:44

Hmm, entities are created in partitions. For each entity in each partition, they are ordered by increasing entity id in a way that matches the time in which they were transacted.

Ben Kamphaus18:05:06

this is also the sort order you access with datoms or seek-datoms specifying :eavt as the index.

Ben Kamphaus18:05:15

But if you install entities to multiple partitions, they are only ordered by their entity ids in relation to the other entity id’s in the same partition (partition is part of the entity id)

adamfrey18:05:27

ok. But we can always infer transaction time moving forward from the order of the transaction Log right? Partitions aren’t relevant to the Log ordering?

adamfrey19:05:23

or am I missing something because transactions are also entities?

Ben Kamphaus19:05:59

partitions aren’t relevant to the log ordering.

Ben Kamphaus19:05:15

I’m just saying the transaction ordering doesn’t constraint by entity type

adamfrey19:05:37

yes, that makes sense

Ben Kamphaus19:05:41

i.e. “I want to see the last users who bought this product” is not the last 10 facts in the db typically, but something that must be constrained by “users” entities first.

adamfrey19:05:08

right. Thanks for your help!