datomic 2023-01-05 | Slack Archive

nottmey13:01:39

I use this query to pull out a list of my apps entities (specified by attributes returned by (keys schema)). It’s quite nice, because I directly know the count and can split it into pages via offset/limit, via a single call.

(d/qseq {:query '[:find (pull ?e [*])
                  :in $ [?as ...]
                  :where [?e ?as]]
         :args  [db (keys schema)]})

But it returns the entities in increasing order (by id). How do I tell the query resolver to start with the highest number? (effectively reversing the result, without loading the whole dataset)

✅ 2

favila13:01:50

Query results are in arbitrary order and are eager (realize the whole set)

💡 4

favila13:01:06

Only the pull is deferred

💡 2

favila13:01:50

So if you want them in a certain order, sort in Clojure then pull, and don’t use qseq+pull

nottmey14:01:56

ah wow, thanks. what’s a the better way to bulk pull than this?

(d/q '[:find (pull ?es [*])
       :in $ [?es ...]]
     db
     recently-updated-entities)

it seems quite fast (10x faster than pmap+pull), but feels awkward to use q for it

favila14:01:53

On cloud/client that’s the only option

👌 2

favila14:01:00

on on-prem, there’s d/pull-many

favila14:01:45

The speed you see is from avoiding round-trips and blocking; you could recover it by pipelining

favila14:01:29

Although I see you tried pmap already

favila14:01:42

kinda surprised at a 10x difference

nottmey14:01:28

Ok, I’m on cloud. I can stick to the q version, which funnily enough needs resorting afterwards again ^^ Yes, on my local machine q (with consumption) is 30ms and the lines below are 400ms, but maybe that’s not how to use pmap ^^

(->> recently-updated-entities
     (pmap #(d/pull db '[*] %))
     doall)

favila14:01:26

pmap is for cpu not io workloads, but on an idle system it’s a quick and dirty way to pipeline even io tasks

👌 2

favila14:01:12

so that looks right to me and I expected it to be faster

👌 2

favila14:01:28

you can include the sort key with your followup query to make things easier

nottmey14:01:05

ah yes, good idea

favila14:01:21

(->> (d/q '[:find (pull ?es [*]) ?i
       :in $ [[?i ?es]]
     db
     (into [] (map-indexed vector) recently-updated-entities)
    (sort-by peek)
    (mapv first))

👍 2

favila14:01:37

for e.g.

pieterbreed14:01:25

I've just run into the "Loading database" issue with datomic cloud. Is there any way to prevent the ion instance from receiving requests (lambda, http) until the database has finished loading?

cch120:01:05

I’m almost certainly misunderstanding the question, because I thought ion deploys were supposed to ensure that DBs were https://docs.datomic.com/cloud/ions/ions-reference.html#deploy.

2023-01-05

Channels