so what should I do to enable a transactor to sustainably keep up with this load?

@cap10morgan it depends on the load, can you describe it in some detail?

laujensen [2:25 PM]
I have an immutant app, which has many handlers responding to web requests. These pull out some data, generate some html and store some stuff in datomic. When it hands off to the browser, nothing is left hanging in memory. And still, once per day, sometimes twice, the entire system crashes due to the GC spinning out of control. What am I looking for?


noisesmith [2:27 PM]
> When it hands off to the browser, nothing is left hanging in memory.
do you have proof of this?
[2:29 PM]
doesn't datomic implicitly use a local cache so that you'd have to make new dbs and let the old be gc'd or else the other facts are sticking around in memory?

Is the peer library hoarding data?

Ive just run a profiler. As soon as the app is taken into use, datomic hoards 2gb of char[]. How do I control this?

hi @laujensen — by Datomic do you mean peer, transactor, client, peer server, … ?

peer

@stuarthalloway

This is what happens to the heap when I trigger the first few handlers @stuarthalloway, GC does not diminish it, and the heapdump has 2GBs og char[] from Datomic

lemme try it locally

(with my own data -- back in a minute)

how much JVM RAM are you allowing the peer?

The App and Peer have 3Gb

Same for Transactor

Would lowering the objectCacheMax reduce performance but guarantee lower memory footprint?

I was about to say almost that

My OCM is 512m

I just ran a peer on a small database here, confirming it doesn’t hold nearly that much memory

so you have three variable knobs that relate to the size of your db

1. objectCacheMax (like you said)

This is not a small db (relatively), its 25 websites, each holding about 20 pages, each having quite a bit of html and css stored in the db

2. memoryIndexMax (unindexed log tail held in all peers)

3. index nodes held in memory (implied by your usage)

HTML and CSS do not make good datoms, better to content address them and store in S3

That would be better in terms of datomic, but we're rendering webpages in about 260msecs now and I'd hate to see that drop. Which reaching into s3 would do

But do you have a way for me to calculate a good objectMax for 3Gb total heap ?

core.memoize - put it on S3 but memoize it in memory

or core.cached or guava — don’t think memoize evicts things

or even better use cloudfront and avoid dynamic processing

if your HTML or CSS have occasional changes halfway down the page and you save the history, can be particularly nasty for Datomic indexes

root and branch nodes must store a prefix big enough to distinguish two datoms

The HTML/CSS are frequently updated, as any change in the editor updates the datoms

if those datoms are 50K HTML blobs that change say at the 25K mark, then you will blow up the dir and root nodes

Then Im pretty sure thats happening routinely

this will get worse quickly as you add data

and I am tallying another implicit vote for blob support in Datomic 🙂

This will be problematic to resolve then, as I need to store html/css outside datomic, and implement my own version control, for which Im relying on tx history now

store the hash in Datomic and you still get version control

the only thing you don’t get is garbage collection

of S3 or wherever

Yeah hashes would work

Thanks for your input, very valuable! I'll start by reducing the OCM and then work out a file-storage solution

@laujensen thanks for your feedback! definitely thinking about how to automate this in Datomic

@stuarthalloway what’s the backup/restore story for Cloud? could I e.g. restore an on-prem db?

@robert-stuttaford still in development. There is more subtlety with Cloud being encrypted at rest

gotcha

teaser: Cloud’s support for the time model is superior to On-Prem in such a way that On-Prem data is insufficient to directly populate a Cloud database

@stuarthalloway thanks! And just an FYI, we would have switched to cloud ASAP if there was a good import story.

is it the plan to be able to restore from one to the other?

Ideally, just providing a datomic-uri during the first setup, and having it import everything

ah 🙂 now i’m super curious about how the time model is superior 🙂

We'd also want a backup-db option, but thats prio #2 🙂

@laujensen hear you loud and clear. The import will need to have knobs to control how semantic differences are handled

are these semantic differences still in flux, @stuarthalloway? i’ve read the words you’ve shared thus far - curious what else has changed!

@stuarthalloway seems datomic-client doesn’t support [:find [?e ...] :in or [:find ?e . :in or [:find [?e] :in ? [:find ?e works fine. com.datomic/client-pro "0.8.14". "Only find-rel elements are allowed in client find-spec, see "

@robert-stuttaford that's correct. Clients only support the relation find

That's true of all clients (for peer server and for cloud)

gosh. might want to make that clear in the docs, @marshall 🙂 curious why that is?

https://docs.datomic.com/cloud/query/query-data-reference.html#arg-grammar

right - i meant more that it was surprising that client doesn’t support this, and then i couldn’t find explicit mention of this difference in the docs, leading me to think i was doing something wrong

Newbie question about this, just to help cement my own understanding: would you work around that limitation of the client query model by using a pull as your find-rel and then extracting your tuple or scalar therefrom?

@chris_johnson the pull result would still be wrapped in a vec, so even then, you still have to use first or similar to get at the value per result

datomic.api/squuid has no counterpart in Datomic Client. what’s the reasoning for that? it’s the only thing i was still using from Peer in a project i converted. happily i still have Datascript in the project, and could use its impl instead — however, i’m curious - what should folks typically do here?

also amusing, needing this pattern to deal with the lack of d/entid / d/ident:

(defn ea->v [db e a]
  (get (d/pull db [a] e) a))

i must say, the client api feels a whole heck of a lot simpler and therefore approachable, so i’m feeling empathy for removing all these conveniences