FWIW I think the most complete library that is close to what you're asking for is https://github.com/denistakeda/re-posh

And all the libraries it depends on.

we have a lot of microservices that are currently exposed at various endpoints. I would like to be be able to query our system, via Datalog, to get a response that contains data from multiple endpoints. E.g. if there’s a books and authors microservice, I’d be able to write a query on the client-side:

'[:find ?title ?author
  :where
  [?e :book/title ?title]
  [?e :book/author ?author-id]
  [_ :author/name ?author]]

Ahh, pathom is probably the closes thing to that 🙂

right, I know of pathom but it isn’t datalog, it’s more akin to pull syntax

Ha, thats what I was just typing.

If you're going to make something to do this, I'd probably build it on top of pathom since it compiles indexes.

I'm not familiar with anything that that will do it for you.

Datascript is (obviously) datalog implemented in the browser. Another interesting one is built into https://github.com/arachne-framework/factui , which builds an impressive datalog on top of clara rules (in the browser!).

To be clear, what we are talking about now is pretty far away from the initial question of: > has anyone used datascript as a client-side cache for datomic? Nothing wrong with that, per se, it just sounds like datalog to query across n-services is a very different (more general) problem than client-side datomic cache.

yes, the next step of the idea is that I would like to handle caching on the client of these queries so that it doesn’t have to send the request for datums that have already been requested from the microservices.

and my thinking was: what if my datalog-query-service responded with all of the datums that were requested in order to resolve the query, and then the client transacted those to a local datascript db?

Does my question make more sense, now?

I am looking for experience reports of using datascript to cache queries for a service that already uses datalog (datomic)

The closest thing I'm aware of is https://github.com/replikativ/datahike and the replikativ stack as an alternative trying to build a JS/JVM distributed data stack. But, I would also argue that your books and authors example sounds like you want to build a js-based datomic peer (that will fetch and cache datoms and do joins locally):

'[:find ?title ?author
  :in $book-db $author-db
  :where
  [$book-db ?e :book/title ?title]
  [$book-db ?e :book/author ?author-id]
  [$author-db ?a :author/id ?author-id]
  [$author-db ?a :author/name ?author]]

Hi, we have a memory issue (system crashes) because we get many records (over a million) and then sort by time.

This is the schema:

{:db/id #db/id[:db.part/db]
  :db/ident :lock/activities
  :db/valueType :db.type/ref
  :db/cardinality :db.cardinality/many
  :db/isComponent true
  :db.install/_attribute :db.part/db}


 {:db/id #db/id[:db.part/db]
  :db/ident :activity/author
  :db/valueType :db.type/ref
  :db/cardinality :db.cardinality/one
  :db.install/_attribute :db.part/db}
 
 {:db/id #db/id[:db.part/db]
  :db/ident :activity/at
  :db/valueType :db.type/instant
  :db/cardinality :db.cardinality/one
  :db/index true
  :db.install/_attribute :db.part/db}

and the code that crashes is this:

(sort-by :at 
 (mapv #(get-activity-data %) (:lock/activities lock)))

Is there a simpler way to get activities of a lock sorted?

@asier does it crash if you just call (mapv #(get-activity-data %) (:lock/activities lock)) without sorting?

yes

So, the issue isn't sorting then, maybe the issue is eagerly realizing a million+ entities in memory at once?

Do you need all results here? It seems like you simply can’t fit all activities in memory; what are you willing to give up?

we just need the newest 100 activities, but we don't know how to do it.

Can you use the :avet index?

Would a time bound instead of a count bound be acceptable?

If not, I think you need to rearrange your schema a bit so you can make a composite attr sorted how you want

Thanks both - I'll investigate further

(Or, could you throw more ram at it)

that's an option, indeed

With this code we don't need to increase the memory:

(sort #(compare (:activity/at %2)
                (:activity/at %1))
                            (:lock/activities (or (d/entity db [:lock/id lock-id])
                                                  (d/entity db [:lock/serial-number lock-id])))

how is this different from your get-activity-data? (Which I now realize you never showed us)

(defn get-activity-data
  "Gets the attributes from entity"
  [activity]
  (merge
    {:at (to-long (:activity/at activity))
     :name (-> activity 
             :activity/kind 
             :activity-kind/name)
     :desc (-> activity 
             :activity/kind 
             :activity-kind/desc)
     :status (-> activity 
               :activity/kind 
               :activity-kind/status)
     :image (->> activity 
              :activity/kind 
              :activity-kind/image
              (str "assets/"))}
    (if (:activity/author activity)
      {:author (some-> activity :activity/author :user/username)}
      {:author nil})))

old code - from 2015 or so

Hello! I'm trying to upgrade my system for the first time. I'm running a solo topology in Datomic Cloud. I've selected my root stack and used the formation template https://s3.amazonaws.com/datomic-cloud-1/cft/589-8846/datomic-storage-589-8846.json which I found from https://docs.datomic.com/cloud/releases.html however the stack update has failed with the status reason of Export with name XXX-MountTargetSecurityGroup is already exported by stack XXX-StorageXXX-XXX . I have never updated my system since I created it in August. Any guidance would be appreciated!

Have you looked at https://docs.datomic.com/cloud/operation/upgrading.html and https://docs.datomic.com/cloud/operation/split-stacks.html ?

I was using the upgrading.html page but I was not using a split stack system. Let me try splitting the stacks and see if the problem persists after that. Thanks @joe.lane

Where should I go from here? My system is now down with the ec2 instance terminated so some part of that delete worked

I'd open a ticket at this point, sorry I can't be more helpful ATM.

No problem. Thanks for the help!

I resolved the above issue by navigating to the ENIs page and deleting the ENIs manually

@brian.rogers Did you then split the stack and upgrade?

Currently in the process of doing so!

Have not yet finished

Great to hear 👍

ANN: Datomic CLI Tools 0.10.81 now available: https://forum.datomic.com/t/datomic-cli-0-10-81-now-available/1363 Check out the video overview: https://docs.datomic.com/cloud/livetutorial/clitools.html