Fork me on GitHub
Slack Archive

This page is not created by, affiliated with, or supported by Slack Technologies, Inc.

2019-10-03

Channels

#datomic
<
2019-10-03
>
dmarjenburgh08:10:34

I’m trying to get items that have :item/id and :item/uploadedAt attributes (among others) and want to get the latest top 100 results. I get all ids and timestamps with a query first, sort the list and take 100. Then I want to do a second query to pull the required attributes for those entity ids. First I did: 

(mapv (fn [id] (d/pull db pattern id)) ids)
To my surprise, this was actually very slow. So I replaced it with: 
(d/q {:query '[:find (pull ?ids pattern)
               :in $ [?ids ...] pattern]
      :args [db ids pull-pattern]})
which is much faster. The question is, is this (still) the best way to do this? And why is the first method much slower?

favila12:10:49

Is this client or peer api? (Looks like client?)

favila12:10:32

Your mapv is 100 blocking pull requests with work done serially. Your query is one request with work done potentially in parallel

favila12:10:18

Include a sorting index with your query input

favila12:10:30

(into [] (map-indexed vector) sorted-eids)

favila12:10:09

Then in query you can :find (pull ?e [*]) ?i in $ [[?i ?e]]

favila12:10:21

Then (->> result (sort-by peek) (mapv first)) the result to get back in order

dmarjenburgh13:10:28

Yes it’s client (cloud). Thanks, so it’s still two sorts regardless.

dmarjenburgh08:10:50

Ofcourse, I lost the sort-ordering again in the second case…

marshall14:10:35

@hadilsabbagh18 AWS has fixed the permissions on the AMI - you should be able to launch the latest now

hadils14:10:14

@marshall still doesn't work with the same error...

hadils14:10:27

Yes. Correct.

marshall14:10:45

i see what’s going on

tyler14:10:21

Is there a way to install custom dependencies on the datomic ions EC2 instances? We are starting to use AWS xray for tracing but this requires a daemon process to be running for EC2 and I can’t see a straightforward way of installing this without hacking the cloudformation yaml for ions (which is not recommended per the docs).

➕ 4
stuarthalloway17:10:00

If you can control the daemon from a Java lib then you can add that Java lib to your ion, but atm there is no supported path for deps other than Java libs.

tyler22:10:00

Unfortunately it doesn’t look like there is one. Might have to move the logic to lambda if there’s no pathway there.

stuarthalloway13:10:29

We are looking into installing xray on the nodes.

👍 4
hadils15:10:12

@marshall it seems to be working now! Thanks, Marshall!

marshall15:10:20

@hadilsabbagh18 yep - should be good now

Msr Tim16:10:13

(d/q {:query '[:find (pull ?e pattern)
               :in $ ?name pattern
               :where [?e :artist/name ?name]]
      :args [db "The Beatles" [:artist/startYear :artist/endYear]]})

Msr Tim16:10:31

is there a reason I can't pass history db to that query

favila16:10:25

reason is ” Can’t pull from history” (in the exception)

favila16:10:38

pulling from a history database doesn’t make sense

favila16:10:50

(Unrelated, maybe pseudocode in your example, but pull patterns must be literal)

Msr Tim16:10:32

thank you sir. that makes sense.

Msr Tim16:10:26

Unhandled clojure.lang.ExceptionInfo
   Can't pull from history
   {:datomic.client-spi/request-id "c4080ae6-32f4-414b-8359-6a4c0128b2ef",
    :cognitect.anomalies/category :cognitect.anomalies/conflict,
    :cognitect.anomalies/message "Can't pull from history",
    :dbs
    [{:database-id "956cc673-373e-4c39-8504-86f51b4cb11f",
      :t 10,
      :next-t 11,
      :history false}]}