Fork me on GitHub
#datomic
<
2020-02-19
>
Joe Lane00:02:49

FWIW I think the most complete library that is close to what you're asking for is https://github.com/denistakeda/re-posh

Joe Lane00:02:09

And all the libraries it depends on.

lilactown00:02:37

we have a lot of microservices that are currently exposed at various endpoints. I would like to be be able to query our system, via Datalog, to get a response that contains data from multiple endpoints. E.g. if there’s a books and authors microservice, I’d be able to write a query on the client-side:

'[:find ?title ?author
  :where
  [?e :book/title ?title]
  [?e :book/author ?author-id]
  [_ :author/name ?author]]
and the service would query across the books and authors microservices to resolve the facts I want

Joe Lane00:02:24

Ahh, pathom is probably the closes thing to that 🙂

lilactown00:02:42

right, I know of pathom but it isn’t datalog, it’s more akin to pull syntax

Joe Lane00:02:57

Ha, thats what I was just typing.

Joe Lane00:02:26

If you're going to make something to do this, I'd probably build it on top of pathom since it compiles indexes.

Joe Lane00:02:54

I'm not familiar with anything that that will do it for you.

Joe Lane00:02:09

Datascript is (obviously) datalog implemented in the browser. Another interesting one is built into https://github.com/arachne-framework/factui , which builds an impressive datalog on top of clara rules (in the browser!).

Joe Lane00:02:54

To be clear, what we are talking about now is pretty far away from the initial question of: > has anyone used datascript as a client-side cache for datomic? Nothing wrong with that, per se, it just sounds like datalog to query across n-services is a very different (more general) problem than client-side datomic cache.

lilactown00:02:36

yes, the next step of the idea is that I would like to handle caching on the client of these queries so that it doesn’t have to send the request for datums that have already been requested from the microservices.

lilactown00:02:53

and my thinking was: what if my datalog-query-service responded with all of the datums that were requested in order to resolve the query, and then the client transacted those to a local datascript db?

lilactown01:02:56

Does my question make more sense, now?

lilactown01:02:58

I am looking for experience reports of using datascript to cache queries for a service that already uses datalog (datomic)

pithyless06:02:44

The closest thing I'm aware of is https://github.com/replikativ/datahike and the replikativ stack as an alternative trying to build a JS/JVM distributed data stack. But, I would also argue that your books and authors example sounds like you want to build a js-based datomic peer (that will fetch and cache datoms and do joins locally):

'[:find ?title ?author
  :in $book-db $author-db
  :where
  [$book-db ?e :book/title ?title]
  [$book-db ?e :book/author ?author-id]
  [$author-db ?a :author/id ?author-id]
  [$author-db ?a :author/name ?author]]

lilactown14:02:14

a client with caching is fairly similar to a datomic peer, I suppose!

asier13:02:57

Hi, we have a memory issue (system crashes) because we get many records (over a million) and then sort by time.

asier13:02:02

This is the schema:

asier13:02:09

{:db/id #db/id[:db.part/db]
  :db/ident :lock/activities
  :db/valueType :db.type/ref
  :db/cardinality :db.cardinality/many
  :db/isComponent true
  :db.install/_attribute :db.part/db}


 {:db/id #db/id[:db.part/db]
  :db/ident :activity/author
  :db/valueType :db.type/ref
  :db/cardinality :db.cardinality/one
  :db.install/_attribute :db.part/db}
 
 {:db/id #db/id[:db.part/db]
  :db/ident :activity/at
  :db/valueType :db.type/instant
  :db/cardinality :db.cardinality/one
  :db/index true
  :db.install/_attribute :db.part/db}

asier13:02:22

and the code that crashes is this:

asier13:02:39

(sort-by :at 
 (mapv #(get-activity-data %) (:lock/activities lock)))

asier13:02:47

Is there a simpler way to get activities of a lock sorted?

Joe Lane13:02:20

@asier does it crash if you just call (mapv #(get-activity-data %) (:lock/activities lock)) without sorting?

Joe Lane13:02:18

So, the issue isn't sorting then, maybe the issue is eagerly realizing a million+ entities in memory at once?

favila13:02:23

Do you need all results here? It seems like you simply can’t fit all activities in memory; what are you willing to give up?

asier13:02:59

we just need the newest 100 activities, but we don't know how to do it.

Joe Lane13:02:40

Can you use the :avet index?

favila13:02:55

Would a time bound instead of a count bound be acceptable?

favila13:02:33

If not, I think you need to rearrange your schema a bit so you can make a composite attr sorted how you want

asier13:02:06

Thanks both - I'll investigate further

favila13:02:29

(Or, could you throw more ram at it)

asier13:02:01

that's an option, indeed

asier14:02:48

With this code we don't need to increase the memory:

asier14:02:07

(sort #(compare (:activity/at %2)
                (:activity/at %1))
                            (:lock/activities (or (d/entity db [:lock/id lock-id])
                                                  (d/entity db [:lock/serial-number lock-id])))

favila14:02:45

how is this different from your get-activity-data? (Which I now realize you never showed us)

asier14:02:56

(defn get-activity-data
  "Gets the attributes from entity"
  [activity]
  (merge
    {:at (to-long (:activity/at activity))
     :name (-> activity 
             :activity/kind 
             :activity-kind/name)
     :desc (-> activity 
             :activity/kind 
             :activity-kind/desc)
     :status (-> activity 
               :activity/kind 
               :activity-kind/status)
     :image (->> activity 
              :activity/kind 
              :activity-kind/image
              (str "assets/"))}
    (if (:activity/author activity)
      {:author (some-> activity :activity/author :user/username)}
      {:author nil})))

favila16:02:07

Ah, I see, you were building full result sets, and you couldn’t fit that in memory. but you can fit the things you sort by

favila16:02:21

consider using a query instead of the entity API?

favila16:02:50

e.g.

(->> (d/q '[:find ?activity ?at
            :in $ ?lock
            :where
            [?lock :lock/activities ?activity]
            [?activity :activity/at ?at]]
       db lock-eid)
     (sort-by peek)
     (into []
           (comp
            (map first)
            (take 100))))

favila16:02:20

still realizes everything you sort on, but should be lighter than using entity sets

asier13:02:15

I take note - thanks!

asier14:02:29

old code - from 2015 or so

Brian16:02:02

Hello! I'm trying to upgrade my system for the first time. I'm running a solo topology in Datomic Cloud. I've selected my root stack and used the formation template https://s3.amazonaws.com/datomic-cloud-1/cft/589-8846/datomic-storage-589-8846.json which I found from https://docs.datomic.com/cloud/releases.html however the stack update has failed with the status reason of Export with name XXX-MountTargetSecurityGroup is already exported by stack XXX-StorageXXX-XXX . I have never updated my system since I created it in August. Any guidance would be appreciated!

Brian16:02:07

I was using the upgrading.html page but I was not using a split stack system. Let me try splitting the stacks and see if the problem persists after that. Thanks @joe.lane

Brian16:02:20

Where should I go from here? My system is now down with the ec2 instance terminated so some part of that delete worked

Joe Lane17:02:04

I'd open a ticket at this point, sorry I can't be more helpful ATM.

Brian17:02:19

No problem. Thanks for the help!

Brian17:02:03

I resolved the above issue by navigating to the ENIs page and deleting the ENIs manually

Joe Lane18:02:03

@brian.rogers Did you then split the stack and upgrade?

Brian18:02:21

Currently in the process of doing so!

Brian18:02:25

Have not yet finished

Joe Lane18:02:28

Great to hear 👍

tatut06:02:44

does it need some configuration? I'm getting Error building classpath. Could not find artifact com.datomic:tools.ops:jar:0.10.81 in central () when trying to run datomic command

maxt08:02:28

I get the same

maxt08:02:23

If I add the datomic cloude s3 repo to deps I can get it to work

:mvn/repos {"datomic-cloud" {:url ""}}

marshall15:02:17

It should have been on Maven central; we are looking again - it should show up there soon if we need to re-release

timcreasy16:02:41

Looks like it’s up now 👍 I had been hitting this same issue.