Clojurians Log v2
datomic
Is it possible to apply a rule optionally? e.g. I have an ancestor rule that is serving me well, but I also want the results for the query sans this rule.
Am I forced to execute two separate queries and create a union of the results?
Currently I do this:
(set/union
(d/q (conj query '(ancestor ?msItem ?e)) db manuscript-ancestor-rule)
(d/q query db []))
Or-join (which is just a rule you don’t name) where one of the branches uses identity to convey input to output unchanged. See this https://clojurians.slack.com/archives/C03RZMDSH/p1685543111246059?thread_ts=1685541923.981759&channel=C03RZMDSH&message_ts=1685543111.246059, which is probably similar to your ancestor rule.
We’re seeing occasional OOM kills in kubernetes with our valcache enabled peers and starting to suspect direct memory allocation could be related. Is there any recommendation for setting MaxDirectMemorySize when using valcache for pro (not cloud), it looks like it may be being set in the datomic cloud defaults
@joe.lane just following up here on the OOM problem, basically we’re seeing our valcache enabled peers hitting OOM limits in kubernetes for what appears to be off-heap allocation areas. We ran an experiment this week to run a smaller than normal heap size to give the pod plenty of headroom (heap is around 6gb, pod has 28gb). Datadog memory tracking is showing us here that heap is stable, but the actual RSS of the pod grows overtime
in other experiments, we’ve observed a large OS level file page cache, which is not surprising due to valcache usage
-Ddatomic.valcacheMaxGb=64
And do you get different results when you disable valcache on peers? Could you run the same analysis on the transactor? What Datomic version are you running?
we also have direct allocation profiling enabled, and we can see that valcache (and redis) are high contributors there
com.datomic/peer {:mvn/version "1.0.7277"}we will try disabling valcache entirely today
What version of core.async are you using in your peers? Can you find it (may be pulled in transitively)
checking
clojure -Stree | grep async
. org.clojure/core.async 1.5.648
X org.clojure/core.async 1.5.648 :superseded
. org.clojure/core.async 1.6.681
X org.clojure/core.async 1.5.648 :older-version
X org.clojure/core.async 1.5.648 :older-version
X org.clojure/core.async 1.5.648 :older-version
X org.clojure/core.async 1.5.648 :older-version
X org.clojure/core.async 1.5.648 :older-version
. org.clojure/core.async 1.6.681 :newer-versionOn mobile now, can’t parse. which is being used?
looks like we don’t pin core.async dep ourselves, but likely its coming transitively
looks like org.clojure/core.async 1.6.681
looks like we also have com.cognitect.aws/api providing it, but it should be being superseded
Could you upgrade to 1.8.741
yes, we can try that
Please also a/b test the transactor and valcache on/off in peers
will do
Was it the jvm that oomed or the Linux OOMkiller?
not the jvm, kubernetes/linux is identifying that the container memory usage is breaching the defined limit
which is why this is peculiar to us, because heap remains fixed at a reasonable size, and there’s plenty of headroom for gc, compressed code etc. Its the “other” category that’s growing unlimited, hence why we’re investigating off-heap allocation/fs area
just rolled out a change to disable valcache, monitoring today
FWIW, we run valcache in peers in k8s at Nu and afaik we don’t have this problem.
that’s great to know
I recognize a recommended value is probably highly specific to the application and not generalizable, but simply knowing if this is commonly restricted at all might help
Not generally, no, there isn’t a recommended setting. What version are you running and do you have any diagnostics information? Is there a reason you suspect MaxDirectMemorySize matters here?
let me confirm a few more things this week and respond