This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2022-05-18
Channels
- # announcements (2)
- # asami (20)
- # aws (4)
- # babashka (35)
- # beginners (47)
- # calva (65)
- # cider (19)
- # clj-kondo (63)
- # clojure (177)
- # clojure-austin (2)
- # clojure-europe (27)
- # clojure-nl (1)
- # clojure-uk (4)
- # clojurescript (13)
- # community-development (5)
- # conjure (5)
- # css (2)
- # data-oriented-programming (9)
- # datalevin (13)
- # datascript (15)
- # datomic (4)
- # devcards (6)
- # duct (4)
- # emacs (8)
- # funcool (1)
- # gratitude (2)
- # helix (3)
- # hyperfiddle (3)
- # introduce-yourself (1)
- # jobs (4)
- # jobs-discuss (26)
- # lambdaisland (2)
- # lsp (20)
- # malli (2)
- # meander (2)
- # mid-cities-meetup (5)
- # missionary (15)
- # music (4)
- # off-topic (37)
- # reagent (3)
- # reitit (2)
- # releases (2)
- # ring (18)
- # shadow-cljs (70)
- # specter (4)
- # sql (20)
- # timbre (3)
- # tools-build (43)
- # tools-deps (11)
- # vim (29)
- # xtdb (61)
My nodes stop indexing tx_events
after some time for an unknown reason. How do I debug this?
Caused by: java.lang.OutOfMemoryError: Java heap space
I think it’s an OutOfMemoryError
I had a similar problem running in ECS because container envs lie about the available memory... so I had to specify both max heap and direct memory size
but oom is notorious for causing weird issues as the JVM doesn't even guarantee that it can be caught by an exception handler
I'm running on 8 gigabytes of memory and settings -Xmx4G -XX:MaxDirectMemorySize=2G
for the jvm and haven't had problems after that, but ymmv... hope this helps at all
^ I added some notes on these memory configs to the 1.21 docs: https://docs.xtdb.com/storage/rocksdb/#_project_dependency
https://github.com/facebook/rocksdb/issues/4112#issuecomment-623147508 > you should set MALLOC_ARENA_MAX=2 and check > --Do you mind to share the additional tuning of RocksDB configs? > I can't 😩
it should, but for some reason couldn't... I guessed that it was because the containerized environment lied about how much actual physical memory there was, but can't be sure
but my spesific problem went away after specifying the max direct memory size explicitly
my node runs in the same process as some other stuff, so I’m not certain if xtdb is the cause or just the symptom yet
If I use xtdb to model a todo list with subitems:
- A
-- a
-- b
-- c
lowercase subitems a b c can be dragged to re-order. How should I model this?
I think I could do something this:
{:xt/id :A
:list ["a" "b" "c"]}
and https://docs.xtdb.com/language-reference/datalog-queries/#_maps_and_vectors_in_data says a vector will be broken down to individual components, so I believe a, b c will not have an order.
Should I have a additional attribute ":order" in identity :A, like string "a,b,c" to reflect the order of this list. (so I can show this in the UI), or there's a better modeling ideas?I suppose that sentence in the docs is structured a bit strangely. But the remainder of the sentence is the operative part: > XTDB breaks down vectors into individual components so the query engine is able see all elements on the base level. The "breaking down" here is referring to the fact that the contents of vectors are indexed, even though they are a form of nested data. (Other nested data in XTDB is not indexed.) Vectors are ordered. You don't need to track order elsewhere.
you likely want pull
here - it'll give you back the vector in the order in the original document
(with-open [node (xt/start-node {})]
(xt/submit-tx node [[::xt/put {:xt/id :parent, :children [:child2 :child1 :child3]}]
[::xt/put {:xt/id :child1, :idx 1}]
[::xt/put {:xt/id :child2, :idx 2}]
[::xt/put {:xt/id :child3, :idx 3}]])
(xt/sync node)
(xt/q (xt/db node)
'{:find [pid (pull pid [{:children [:xt/id :idx]}])]
:where [[pid :children]]}))
;; => #{[:parent {:children [{:idx 2, :xt/id :child2}
;; {:idx 1, :xt/id :child1}
;; {:idx 3, :xt/id :child3}]}]}
so, if you're ok to put the whole list of ids every time, I'd go that way; otherwise, @U051V5LLP has a good shout in the other thread
@U050V1N74 I find (xt/entity (xt/db node) :parent)
can let me get the :childern list, then what's the difference between using pull
?
Is that because using xt/entity
will return the whole document, which is inefficient?
It's more that pull can also traverse the graph to pull back data from related docs - e.g. in my example above, I pulled back the children's documents too
It took me some time to read the pull section on xtdb and it starts to make a lot sense now haha. Thanks for your help! Really appreciate it.
you may want to to put the relationship on the children {:xt/id :a :parent :A}
if you're not having to deal with pagination then performing the sort in memory on the server or client would probably be easiest
Do you mean I don't have to maintain a list in parent "A", only maintaining parent attribute in the child is sufficient. But the order is defined by users dragging subitems around. So the sorting logic might not like sorting alphabetically. This way, is it still necessary to have a list in the parent's attribute.
exactly, you wouldn't need to store the list of children on the parent if you add the parent pointer to the children. I think that would work - one idea is to add on each child an attribute to store the sort order:
{:xt/id :a
:parent :A
:parent-A-sort 2}
;; ^ the idea here would be thinking about a potential use case where this item is stored in multiple collections, and it is sorted
;; differently in each
another idea is using a nested map:
{:xt/id :a
:parent :A
:sorting {:A 2}}
;; ^ keys are parent, value is order to use
for the single collection case you can do something like:
;; or just with one parent support:
{:xt/id :a
:parent :A
:parent-sort 2}
and then either on the client or the server (sort-by :parent-sort children)
sure thing! would be interested to hear how things work out when you come up with something that works for you
1.21.0 is officially out of beta. 🙂 https://twitter.com/xtdb_com/status/1526978958687973377 https://github.com/xtdb/xtdb/releases/tag/1.21.0 https://discuss.xtdb.com/t/ann-1-21-0s-out/77 As always, a huge thank you to everyone in the community who helped test the betas and harden this release. ♥️
btw, https://docs.xtdb.com/guides/quickstart/ seems to refer to 1.20.0 version still
Our Cloudfront cache hasn't been invalidated since May 11th (somehow), but it also would appear the old docs were the last ones to be released. We had a bit of a snag with the new (post-crux-to-xtdb-rename) docs release instructions being duplicated in two places, with some stale instructions in both sets. This is probably a symptom of that. I'm rebuilding them now.
Should be fixed now. Thanks for the heads-up, @U11SJ6Q0K!
xt/db
throws an exception if it hasn't indexed a given tx, but what if the initial indexing is still going on? Does the db know that there are newer transactions and block? Or throw? Or just make a snapshot out of whatever the node had indexed at the moment?
> just make a snapshot out of whatever the node had indexed at the moment? unless you pass in the submit-tx receipt or an explicit tx-time, pretty much this
you can also use xt/await-tx
to wait until the node has indexed the transaction you're interested in, and then call xt/db
Yeah, that's what we do, though not for the initial indexing but for whatever was put in
And related to previous question and the announcement, how can the caller know when the node has finished indexing? Something in https://docs.xtdb.com/clients/clojure/#_status?
you probably want to look at latest-submitted-tx
and latest-completed-tx
to come up with your "finished" logic (which can possibly mean a few different things)
Ah, so I can run await on latest submitted, and then the "initial" catching up is done (to say nothing of the new data entered while indexing
Might be actually a good idea to put that in our db creation helpers. Not perfect, but a good start to make sure that when the node starts to get real traffic, the first await won't be surprisingly long
I am trying to configure xtdb to connect to a local psql database. Inside my deps.edn
com.xtdb/xtdb-core {:mvn/version "1.20.0"}
com.xtdb/xtdb-sql {:mvn/version "1.21.0"}
com.xtdb/xtdb-jdbc {:mvn/version "1.21.0"}
org.postgresql/postgresql {:mvn/version "42.2.18"}
My node is defined as such:
(xt/start-node {:xtdb.jdbc/connection-pool {:dialect {:xtdb/module 'xtdb.jdbc.psql/->dialect}
:pool-opts {:maximumPoolSize 100}
:db-spec {:host "moo"
:dbname "cmooo"
:user "cmoo"
:password "moo"}}
:xtdb/tx-log {:xtdb/module 'xtdb.jdbc/->tx-log
:connection-pool :xtdb.jdbc/connection-pool}
:xtdb/document-store {:xtdb/module 'xtdb.jdbc/->document-store
:connection-pool :xtdb.jdbc/connection-pool}})
However it's throwing the error
2. Unhandled clojure.lang.Compiler$CompilerException
Error compiling NO_SOURCE_FILE at (8:13)
#:clojure.error{:phase :execution,
:line 8,
:column 13,
:source "NO_SOURCE_FILE"}
Compiler.java: 3711 clojure.lang.Compiler$InvokeExpr/eval
Compiler.java: 457 clojure.lang.Compiler$DefExpr/eval
Compiler.java: 7186 clojure.lang.Compiler/eval
Compiler.java: 7136 clojure.lang.Compiler/eval
core.clj: 3202 clojure.core/eval
core.clj: 3198 clojure.core/eval
interruptible_eval.clj: 87 nrepl.middleware.interruptible-eval/evaluate/fn/fn
AFn.java: 152 clojure.lang.AFn/applyToHelper
AFn.java: 144 clojure.lang.AFn/applyTo
core.clj: 667 clojure.core/apply
core.clj: 1977 clojure.core/with-bindings*
core.clj: 1977 clojure.core/with-bindings*
RestFn.java: 425 clojure.lang.RestFn/invoke
interruptible_eval.clj: 87 nrepl.middleware.interruptible-eval/evaluate/fn
main.clj: 437 clojure.main/repl/read-eval-print/fn
main.clj: 437 clojure.main/repl/read-eval-print
main.clj: 458 clojure.main/repl/fn
main.clj: 458 clojure.main/repl
main.clj: 368 clojure.main/repl
RestFn.java: 1523 clojure.lang.RestFn/invoke
interruptible_eval.clj: 84 nrepl.middleware.interruptible-eval/evaluate
interruptible_eval.clj: 56 nrepl.middleware.interruptible-eval/evaluate
interruptible_eval.clj: 152 nrepl.middleware.interruptible-eval/interruptible-eval/fn/fn
AFn.java: 22 clojure.lang.AFn/run
session.clj: 218 nrepl.middleware.session/session-exec/main-loop/fn
session.clj: 217 nrepl.middleware.session/session-exec/main-loop
AFn.java: 22 clojure.lang.AFn/run
Thread.java: 833 java.lang.Thread/run
1. Caused by clojure.lang.ExceptionInfo
Error locating module
{:module xtdb.jdbc/->document-store}
Things to try:
• do this in a regular REPL, removing CIDER/nREPL middleware from the equation
• restart your REPL (did you perhaps update deps.edn
after starting your current REPL?)
Interesting. If I just open a repl in my terminal... It shows this:
Execution error at xtdb.system.ModuleRef/fn (system.clj:107).
No such var: xio/conform-tx-log-entry
And maybe *e
will give you more of an insight after that (showing the full exception/stacktrace)?
I just noticed in your deps.edn
, you have a mix of 1.20.0
and 1.21.0
-- is that a copy'n'paste error?
Glad I could help! 🙂
That seems like it fits my rule ! - if something takes longer than ~15mins to solve with no clues or incremental information gleaned along the way - the answer will be a single character change.
btw, https://docs.xtdb.com/guides/quickstart/ seems to refer to 1.20.0 version still
Our Cloudfront cache hasn't been invalidated since May 11th (somehow), but it also would appear the old docs were the last ones to be released. We had a bit of a snag with the new (post-crux-to-xtdb-rename) docs release instructions being duplicated in two places, with some stale instructions in both sets. This is probably a symptom of that. I'm rebuilding them now.