Fork me on GitHub
#datomic
<
2022-09-19
>
Dustin Getz21:09:20

has anyone implemented the entity api for datomic clients under the presumption that it will run in an ion and be fast? or even datafy/nav

seepel21:09:44

I'm working on an app that I plan to deploy to an Ion, and I do have the presumption that it will be fast. I'm curious why you ask, do you expect it to not be fast?

Drew Verlee02:09:27

I think... The Entity api is given an id/key give me the value. Ions in aws run on top of aws dynamodb which is a key value store. I assume the id lookup would be constant time. But im probably missing more then half the story. In general, The faster you make it, the more space you have to use and it also helps to know ahead of time you were going to get asked. Basically, fast answers are ones that are usually almost the question themselves. (Tiny rant over)

seepel02:09:50

Ah, so the network round trip might be too slow to do something like the following?

(map #(db/pull db '[*] %) dbids)

Drew Verlee02:09:17

Im not sure my understanding of the entity api is strong enough to answer dustins question. But get id, should be constant time. But the "entity api" might include a lot more.

Dustin Getz10:09:28

Yeah i think (d/entity db e) is not much more than (d/datoms db {:index :eavt, :components [e]}) wrapped in datafy/nav – the problem is if the client is remote then each nav is a round trip with no way to batch them, which the ions model solves by colocating the app process/classpath with the index

Drew Verlee12:09:30

Dustin, you said "ions model" but did you mean "on prem" ? I thought the ion client wasn't collocated with the index cache anymore. (It's been a while since I thought about the internals)

Daniel Jomphe14:09:31

DynamoDB isn't pertinent to queries in Datomic Cloud. It's only used by the transactor to guarantee serialization of the transactions (enforce ACID properties of transactions).

Daniel Jomphe14:09:52

I might be wrong in some details... Once a tx passes through DynamoDB, the transactor updates indexes in S3 and EFS, and notifies query groups of the availability of the tx's novelty in e.g. index incremental updates. Query groups (to serve DB queries, and in which our Ions can be deployed) can then read the novelty in their EFS distributed filesystem, and copy that to the Valcache, a SSD-based cache that plays the role of e.g. Memcache for a single instance server of Datomic Cloud (thus, a hot cache based on this instance's query habits). This SSD-backed Valcache is an optional step for users who pay for a specific category of server instances in their query group. When our Ions in a query group query the DB, the updated indexes required to serve the query are loaded from Valcache-SSD if available, or EFS otherwise, into the server RAM as the hottest and most performant cache an Ion can use. They will stay there in RAM until they are swapped out by more useful stuff later (based on future queries and size of RAM available). In this context, Dustin's question seems to be: Who, relying on an Ion's expected RAM-locality of hot caches, decided to implement either an entity or datafy/nav api on top of Datomic Cloud's Client API? What would it take to make it work?

👀 2
👍 3
Dustin Getz14:09:24

"I thought the ion client wasn't collocated with the index cache anymore" I dont know anything about this, this is my first time using cloud

Dustin Getz14:09:32

I am not using onprem – onprem has the real entity api so we'd just use that

Daniel Jomphe14:09:16

The Ion is our code added to a Datomic Cloud server instance. It has direct access to the instance's Java Heap. An Ion doesn't make requests through the network, except of course when the Datomic Cloud server instance has no local caches and will download them to satisfy a query. This is all transparent to the Ion.

Daniel Jomphe20:09:26

Hi @U1QJACBUM, pinging one of you cognitects, hoping you might bring some of your perspective to the OP's question! 🙏:skin-tone-3: I'm very interested in using Hyperfiddle in an Ion context, and I know many of us are too.

Dustin Getz22:09:25

hyperfiddle/photon will work (assuming websocket or other transport); my q is just about a demo datomic browser app that i want entity walking for

Joe Lane23:09:29

When using hyperfiddle/photon with onprem do you access the server side entity api over a wire from the browser?

Daniel Jomphe15:09:17

@U0CJ19XAM, photon does relays the browser's need for DB entity api navigation, but the DB calls are made strictly from the backend. So nothing special for you to consider about the browser. Just Clojure and Datomic Cloud. So what is sought by Photon in Datomic Cloud Ions, is "simply" some kind of entity api or datafy/nav in the context of an Ion's clojure process.

Dustin Getz15:09:42

I think it's easy enough to build, and then Onprem and Ion users will get a fast db explorer app, Client (remote) users will get a slower app but it probably won't matter – and our abstraction is streaming, so slow information will just stream in late without slowing down the UX

mario-star 1
Dustin Getz15:09:17

When I said "browser" I meant in the REBL sense

Daniel Jomphe14:09:52

I might be wrong in some details... Once a tx passes through DynamoDB, the transactor updates indexes in S3 and EFS, and notifies query groups of the availability of the tx's novelty in e.g. index incremental updates. Query groups (to serve DB queries, and in which our Ions can be deployed) can then read the novelty in their EFS distributed filesystem, and copy that to the Valcache, a SSD-based cache that plays the role of e.g. Memcache for a single instance server of Datomic Cloud (thus, a hot cache based on this instance's query habits). This SSD-backed Valcache is an optional step for users who pay for a specific category of server instances in their query group. When our Ions in a query group query the DB, the updated indexes required to serve the query are loaded from Valcache-SSD if available, or EFS otherwise, into the server RAM as the hottest and most performant cache an Ion can use. They will stay there in RAM until they are swapped out by more useful stuff later (based on future queries and size of RAM available). In this context, Dustin's question seems to be: Who, relying on an Ion's expected RAM-locality of hot caches, decided to implement either an entity or datafy/nav api on top of Datomic Cloud's Client API? What would it take to make it work?

👀 2
👍 3