Fork me on GitHub
#xtdb
<
2022-05-18
>
flefik08:05:49

My nodes stop indexing tx_events after some time for an unknown reason. How do I debug this?

tatut08:05:49

and logs don't provide any clues?

flefik08:05:11

Caused by: java.lang.OutOfMemoryError: Java heap space I think it’s an OutOfMemoryError

flefik08:05:30

but it occurs elsewhere in the application

tatut08:05:14

I had a similar problem running in ECS because container envs lie about the available memory... so I had to specify both max heap and direct memory size

tatut08:05:33

but oom is notorious for causing weird issues as the JVM doesn't even guarantee that it can be caught by an exception handler

tatut08:05:45

I'm running on 8 gigabytes of memory and settings -Xmx4G -XX:MaxDirectMemorySize=2G for the jvm and haven't had problems after that, but ymmv... hope this helps at all

refset09:05:30

^ I added some notes on these memory configs to the 1.21 docs: https://docs.xtdb.com/storage/rocksdb/#_project_dependency

Hukka10:05:36

Uf, I thought that the heap size would set a bound for the direct buffers

Hukka10:05:44

https://github.com/facebook/rocksdb/issues/4112#issuecomment-623147508 > you should set MALLOC_ARENA_MAX=2 and check > --Do you mind to share the additional tuning of RocksDB configs? > I can't 😩

tatut10:05:14

it should, but for some reason couldn't... I guessed that it was because the containerized environment lied about how much actual physical memory there was, but can't be sure

tatut10:05:46

but my spesific problem went away after specifying the max direct memory size explicitly

Hukka10:05:07

30 years of Java, still tuning memory params

truestory 1
hidethepain 2
tatut10:05:46

memory and logging params... things you will be forever tuning

Hukka10:05:48

I just copy and paste the same logback.xml around 🤞

flefik11:05:02

Thanks guys, I’m going to see if any of this helps

🤞 1
flefik11:05:17

trying to get a heap dump now to see where the problem is actually occuring

flefik11:05:38

my node runs in the same process as some other stuff, so I’m not certain if xtdb is the cause or just the symptom yet

stagmoose15:05:01

If I use xtdb to model a todo list with subitems:

- A
-- a
-- b
-- c
lowercase subitems a b c can be dragged to re-order. How should I model this? I think I could do something this:
{:xt/id :A
 :list ["a" "b" "c"]}
and https://docs.xtdb.com/language-reference/datalog-queries/#_maps_and_vectors_in_data says a vector will be broken down to individual components, so I believe a, b c will not have an order. Should I have a additional attribute ":order" in identity :A, like string "a,b,c" to reflect the order of this list. (so I can show this in the UI), or there's a better modeling ideas?

Steven Deobald17:05:27

I suppose that sentence in the docs is structured a bit strangely. But the remainder of the sentence is the operative part: > XTDB breaks down vectors into individual components so the query engine is able see all elements on the base level. The "breaking down" here is referring to the fact that the contents of vectors are indexed, even though they are a form of nested data. (Other nested data in XTDB is not indexed.) Vectors are ordered. You don't need to track order elsewhere.

🙏 1
jarohen17:05:03

you likely want pull here - it'll give you back the vector in the order in the original document

jarohen17:05:09

should even work with nested pull specifications, too

jarohen17:05:28

(with-open [node (xt/start-node {})]
  (xt/submit-tx node [[::xt/put {:xt/id :parent, :children [:child2 :child1 :child3]}]
                      [::xt/put {:xt/id :child1, :idx 1}]
                      [::xt/put {:xt/id :child2, :idx 2}]
                      [::xt/put {:xt/id :child3, :idx 3}]])
  (xt/sync node)
  (xt/q (xt/db node)
        '{:find [pid (pull pid [{:children [:xt/id :idx]}])]
          :where [[pid :children]]}))

;; => #{[:parent {:children [{:idx 2, :xt/id :child2}
;;                           {:idx 1, :xt/id :child1}
;;                           {:idx 3, :xt/id :child3}]}]}

jarohen17:05:55

so, if you're ok to put the whole list of ids every time, I'd go that way; otherwise, @U051V5LLP has a good shout in the other thread

👍 1
stagmoose14:05:27

@U050V1N74 I find (xt/entity (xt/db node) :parent) can let me get the :childern list, then what's the difference between using pull ? Is that because using xt/entity will return the whole document, which is inefficient?

jarohen14:05:58

It's more that pull can also traverse the graph to pull back data from related docs - e.g. in my example above, I pulled back the children's documents too

🙏 1
stagmoose15:05:46

It took me some time to read the pull section on xtdb and it starts to make a lot sense now haha. Thanks for your help! Really appreciate it.

🙏 1
dvingo16:05:13

you may want to to put the relationship on the children {:xt/id :a :parent :A} if you're not having to deal with pagination then performing the sort in memory on the server or client would probably be easiest

stagmoose14:05:16

Do you mean I don't have to maintain a list in parent "A", only maintaining parent attribute in the child is sufficient. But the order is defined by users dragging subitems around. So the sorting logic might not like sorting alphabetically. This way, is it still necessary to have a list in the parent's attribute.

stagmoose14:05:33

Or I can have :parent :idx in the child's attribute, this way seems to work.

dvingo14:05:23

exactly, you wouldn't need to store the list of children on the parent if you add the parent pointer to the children. I think that would work - one idea is to add on each child an attribute to store the sort order:

{:xt/id :a
 :parent :A
 :parent-A-sort 2}
;; ^ the idea here would be thinking about a potential use case where this item is stored in multiple collections, and it is sorted 
;; differently in each
another idea is using a nested map:
{:xt/id :a
 :parent :A
 :sorting {:A 2}}
;; ^ keys are parent, value is order to use
for the single collection case you can do something like:
;; or just with one parent support:
{:xt/id :a
 :parent :A
 :parent-sort 2}
and then either on the client or the server (sort-by :parent-sort children)

👌 1
stagmoose15:05:44

Wow totally a new perspective for me to think on modeling data! Really helps!!

dvingo18:05:09

sure thing! would be interested to hear how things work out when you come up with something that works for you

dvingo16:05:31

that way you can easily change how you sort

Steven Deobald17:05:51

1.21.0 is officially out of beta. 🙂 https://twitter.com/xtdb_com/status/1526978958687973377 https://github.com/xtdb/xtdb/releases/tag/1.21.0 https://discuss.xtdb.com/t/ann-1-21-0s-out/77 As always, a huge thank you to everyone in the community who helped test the betas and harden this release. ♥️ gratitude

❤️ 1
Steven Deobald18:05:13

Our Cloudfront cache hasn't been invalidated since May 11th (somehow), but it also would appear the old docs were the last ones to be released. We had a bit of a snag with the new (post-crux-to-xtdb-rename) docs release instructions being duplicated in two places, with some stale instructions in both sets. facepalm This is probably a symptom of that. I'm rebuilding them now.

Steven Deobald18:05:27

Should be fixed now. Thanks for the heads-up, @U11SJ6Q0K!

Hukka17:05:42

xt/db throws an exception if it hasn't indexed a given tx, but what if the initial indexing is still going on? Does the db know that there are newer transactions and block? Or throw? Or just make a snapshot out of whatever the node had indexed at the moment?

refset22:05:43

> just make a snapshot out of whatever the node had indexed at the moment? unless you pass in the submit-tx receipt or an explicit tx-time, pretty much this

jarohen08:05:54

you can also use xt/await-tx to wait until the node has indexed the transaction you're interested in, and then call xt/db

Hukka08:05:05

Yeah, that's what we do, though not for the initial indexing but for whatever was put in

Hukka17:05:47

And related to previous question and the announcement, how can the caller know when the node has finished indexing? Something in https://docs.xtdb.com/clients/clojure/#_status?

refset22:05:21

you probably want to look at latest-submitted-tx and latest-completed-tx to come up with your "finished" logic (which can possibly mean a few different things)

jarohen08:05:47

xt/await-tx is probably your friend here too 🙂

Hukka08:05:06

Ah, so I can run await on latest submitted, and then the "initial" catching up is done (to say nothing of the new data entered while indexing

👍 1
Hukka09:05:41

Might be actually a good idea to put that in our db creation helpers. Not perfect, but a good start to make sure that when the node starts to get real traffic, the first await won't be surprisingly long

PB20:05:52

I am trying to configure xtdb to connect to a local psql database. Inside my deps.edn

com.xtdb/xtdb-core {:mvn/version "1.20.0"}
           com.xtdb/xtdb-sql {:mvn/version "1.21.0"}
           com.xtdb/xtdb-jdbc {:mvn/version "1.21.0"}
           org.postgresql/postgresql {:mvn/version "42.2.18"}
My node is defined as such:
(xt/start-node {:xtdb.jdbc/connection-pool {:dialect {:xtdb/module 'xtdb.jdbc.psql/->dialect}
                                              :pool-opts {:maximumPoolSize 100}
                                              :db-spec {:host "moo"
                                                        :dbname "cmooo"
                                                        :user "cmoo"
                                                        :password "moo"}}

                  :xtdb/tx-log {:xtdb/module 'xtdb.jdbc/->tx-log
                                :connection-pool :xtdb.jdbc/connection-pool}
                  :xtdb/document-store {:xtdb/module 'xtdb.jdbc/->document-store
                                        :connection-pool :xtdb.jdbc/connection-pool}})
However it's throwing the error
2. Unhandled clojure.lang.Compiler$CompilerException
   Error compiling NO_SOURCE_FILE at (8:13)
   #:clojure.error{:phase :execution,
                   :line 8,
                   :column 13,
                   :source "NO_SOURCE_FILE"}
             Compiler.java: 3711  clojure.lang.Compiler$InvokeExpr/eval
             Compiler.java:  457  clojure.lang.Compiler$DefExpr/eval
             Compiler.java: 7186  clojure.lang.Compiler/eval
             Compiler.java: 7136  clojure.lang.Compiler/eval
                  core.clj: 3202  clojure.core/eval
                  core.clj: 3198  clojure.core/eval
    interruptible_eval.clj:   87  nrepl.middleware.interruptible-eval/evaluate/fn/fn
                  AFn.java:  152  clojure.lang.AFn/applyToHelper
                  AFn.java:  144  clojure.lang.AFn/applyTo
                  core.clj:  667  clojure.core/apply
                  core.clj: 1977  clojure.core/with-bindings*
                  core.clj: 1977  clojure.core/with-bindings*
               RestFn.java:  425  clojure.lang.RestFn/invoke
    interruptible_eval.clj:   87  nrepl.middleware.interruptible-eval/evaluate/fn
                  main.clj:  437  clojure.main/repl/read-eval-print/fn
                  main.clj:  437  clojure.main/repl/read-eval-print
                  main.clj:  458  clojure.main/repl/fn
                  main.clj:  458  clojure.main/repl
                  main.clj:  368  clojure.main/repl
               RestFn.java: 1523  clojure.lang.RestFn/invoke
    interruptible_eval.clj:   84  nrepl.middleware.interruptible-eval/evaluate
    interruptible_eval.clj:   56  nrepl.middleware.interruptible-eval/evaluate
    interruptible_eval.clj:  152  nrepl.middleware.interruptible-eval/interruptible-eval/fn/fn
                  AFn.java:   22  clojure.lang.AFn/run
               session.clj:  218  nrepl.middleware.session/session-exec/main-loop/fn
               session.clj:  217  nrepl.middleware.session/session-exec/main-loop
                  AFn.java:   22  clojure.lang.AFn/run
               Thread.java:  833  java.lang.Thread/run

1. Caused by clojure.lang.ExceptionInfo
   Error locating module
   {:module xtdb.jdbc/->document-store}

seancorfield20:05:51

Things to try: • do this in a regular REPL, removing CIDER/nREPL middleware from the equation • restart your REPL (did you perhaps update deps.edn after starting your current REPL?)

PB21:05:15

Interesting. If I just open a repl in my terminal... It shows this:

Execution error at xtdb.system.ModuleRef/fn (system.clj:107).
No such var: xio/conform-tx-log-entry

seancorfield21:05:38

And maybe *e will give you more of an insight after that (showing the full exception/stacktrace)?

PB21:05:06

That's all it's giving me

PB21:05:20

Looking at the xio namespace, that function doesnt' seem to be defined

seancorfield21:05:25

I just noticed in your deps.edn, you have a mix of 1.20.0 and 1.21.0 -- is that a copy'n'paste error?

PB21:05:27

I'm wondering if it's a versioning thing

PB21:05:00

it migh bery well be

PB21:05:27

That was it, I feel silly

1
PB21:05:31

Thank you @U04V70XH6

2
seancorfield21:05:10

Glad I could help! 🙂

dvingo22:05:48

That seems like it fits my rule ! - if something takes longer than ~15mins to solve with no clues or incremental information gleaned along the way - the answer will be a single character change.

1
😎 1
Steven Deobald18:05:13

Our Cloudfront cache hasn't been invalidated since May 11th (somehow), but it also would appear the old docs were the last ones to be released. We had a bit of a snag with the new (post-crux-to-xtdb-rename) docs release instructions being duplicated in two places, with some stale instructions in both sets. facepalm This is probably a symptom of that. I'm rebuilding them now.