Clojurians Log v2

@jarohen perhaps this will help reproduce the issue I hosted my node on DO droplet (2vCPU, 4GB RAM). I was trying to ingest 1M rows using your example above, but with progressively bigger maps (to simulate more realistic scenarios) Your example above (with only xt/id in the map) works OK, and is consistently done in about ~50s. Some helpers to generate a random map:

(defn random-string
  "Generates a random string of a given length."
  [length]
  (let [chars (concat
               (range 48 58) ; 0-9
               (range 65 91) ; A-Z
               (range 97 123) ; a-z
               )]
    (->> (repeatedly length #(char (rand-nth chars)))
         (apply str))))

(defn random-keyword [length] (keyword (random-string length)))

(defn generate-random-map
  "Generates a random map with a number of keys `count` and using string values.
  between 0 and 50 characters long. The keys are random keywords."
  [count]
  (into
   {}
   (repeatedly
    count
    #(hash-map (random-keyword 8) (random-string (rand-int 50))))))

;; here's an example what this returns
  (repeatedly 5 #(generate-random-map 3))
  ;; => ({:WoJ2yLJb "oMKETcImHQkz",
  ;;      :bRs1z1T3 "YZgyufaiFEKY7U",
  ;;      :LcXk0JBm "UQrCwAOJx2R55O"}
  ;;     {:najb5kOt "bRUFZcPqvB7npTs6Pt0",
  ;;      :OYN20KTX "zSgtU",
  ;;      :QCBAXT6T "DzTkVndh9QQ5uDLJTHb7FKTWTaqiaYUwwZ4AD"}
  ;;     {:vTPV75Pn "n",
  ;;      :8RZfI3KL "WOMhQ3j24C0CSUfVHj",
  ;;      :zgTIMWlF "pAbq3MAeh0VZe9raYZXQAyHmjB9"}
  ;;     {:GQsOoHfH "zWnfckRgy69WRi3zkd6hfOyevafW",
  ;;      :x9E2x0hL "ri1GvcnE8zJcYVDclo3",
  ;;      :MOOe1XoB "X6wV1VxWmNoJTa77zn26N2Kc7ks5Alr"}
  ;;     {:4yebo9an "C4jBXtApJQ9JGBcjtNRt",
  ;;      :1Onl1G3U "mPWb2zo8ci5IWIn65lgB7UIJ5h",
  ;;      :dCBiTGlB "8cGFNAGmmXdRHElTI2TVLrgU4MytlzE3ewVUx"})

(dotimes [n 1000]
    (xt/submit-tx
     node
     [(into [:put-docs :table1]
            (for [m (range 1000)]
              (assoc (generate-random-map 1) :xt/id (str n "_" m))))]))

thanks @alen - we'll take a look at what's taking up the memory here, see whether there's anything lingering. could you confirm the heap settings on that JVM? looking like 1GB direct memory, did you explicitly set -Xmx?

-XX:+PrintFlagsFinal will get the defaults that it used, if you didn't specify it

we do have a requirement that the indexer will need to keep at least 100k rows in direct memory (along with all of the surrounding metadata) in addition to everything else the node's doing, so there will be a natural minimum that XT needs to run - but I'd hope it would cope with less than 1GB

I haven't set up anything explicitly on the hosted node, I just ran:

docker run --pull=always -tip 6543:3000 -v ./xtdb-data:/var/lib/xtdb 

once it gets above its minimum it should be fairly static - we then take up anything in excess for caches

I assumed it would use as much as possible, which is why I used 4GB droplet. I can try with 8 or 16 if you think that might help it?

depends on the JVM and JVM version (and I never remember them off-hand) but it's common for the JVM to only take up a fraction of the available memory unless Xmx is set

(we need to provide a way for users to specify memory params to the Docker container - atm the standalone one is quite primitive)

but yes, more RAM would definitely help, even if it's not a good long-term solution 😅

I see, do you have instructions how to run the XTDB binary directly, without docker?

not currently - the Docker image contains an uberjar, could use docker cp to get that out and run java -jar

ah, although you could also override the entrypoint in the Docker container, that'd probably be easier

uberjar is at /usr/local/lib/xtdb/xtdb-standalone.jar in the container

also could run against the underlying JVM docker image to get its settings, something like docker run --rm -ti eclipse-temurin:21 java -XX:+PrintFlagsFinal -version | grep HeapSize

managed to run it with

java -XX:+PrintFlagsFinal -Xmx3g -jar xtdb-standalone.jar

ah, ok, try appending -m xtdb.main?

will run on localhost:3000

(there's an arguably unnecessary port mapping in the Docker standalone instructions - -p 6543:3000)

> ah, ok, try appending -m xtdb.main? getting somewhere, just forgot the NIO opens... 😄

sorry 🤦‍♂️ the ones we use are https://github.com/xtdb/xtdb/blob/main/docker/standalone/Dockerfile

java -XX:+PrintFlagsFinal -Xmx3g --add-opens=java.base/java.nio=ALL-UNNAMED -Dio.netty.tryReflectionSetAccessible=true -jar xtdb-standalone.jar -m xtdb.main

sweet - I've got to shoot off for a bit I'm afraid - will be back 11-11:30 UK. good luck 🙂

thanks a lot for your help! 🙇

no trouble - and thanks for your perseverance 🙂

I'll pick it up tomorrow, but the above doesn't seem to expose port 3000 somehow 😕 I tried to execute what XTDB Dockerfile specifies (more or less)

java -Xmx3g --add-opens=java.base/java.nio=ALL-UNNAMED -Dio.netty.tryReflectionSetAccessible=true -Dclojure.main.report=stderr -cp xtdb-standalone.jar clojure.main -m xtdb.main
Starting XTDB 2.x (pre-alpha) ...
09:24:17 | INFO  xtdb.cli | Node started

curl localhost:3000/status
curl: (7) Failed to connect to localhost port 3000 after 0 ms: Couldn't connect to server

hey @alen per the Dockerfile https://github.com/xtdb/xtdb/blob/main/docker/standalone/local_xtdb.edn edn config file needs to be available: https://github.com/xtdb/xtdb/blob/32604a9948ea7c9a5808c8f40b480edad49b619a/docker/standalone/Dockerfile#L34 - it specifies via config how to set up the HTTP server (the HTTP server isn't created by default)

ahhh yes, this makes sense! I was reading the source code and wondering where :http-server options were 😅

probably a time to open a new thread, but it might be related to some of these recent changes 🙈 I've created xdtb.edn file like @taylor.jeremydavid suggested, and have explicitly limited JVM to 4GB:

java -Xmx4g --add-opens=java.base/java.nio=ALL-UNNAMED -Dio.netty.tryReflectionSetAccessible=true -Dclojure.main.report=stderr -cp xtdb-standalone.jar clojure.main -m xtdb.main

Starting XTDB 2.x (pre-alpha) ...
11:27:06 | INFO  xtdb.metrics | Metrics server started on port:  8080
11:27:07 | INFO  xtdb.server | HTTP server started on port:  3000
11:27:07 | INFO  xtdb.pgwire | PGWire server started on port: 5432
11:27:07 | INFO  xtdb.cli | Node started

(dotimes [n 1000]
    (xt/submit-tx
     node
     [(into [:put-docs :table1]
            (for [m (range 1000)]
              (assoc (generate-random-map 1) :xt/id (str n "_" m))))]))

1. Caused by java.io.EOFException
   EOF reached while reading

   Http1AsyncReceiver.java:  601  jdk.internal.net.http.Http1AsyncReceiver$Http1TubeSubscriber/onComplete
           SocketTube.java:  648  jdk.internal.net.http.SocketTube$InternalReadPublisher$ReadSubscription/signalCompletion
           SocketTube.java:  853  jdk.internal.net.http.SocketTube$InternalReadPublisher$InternalReadSubscription/read
           SocketTube.java:  181  jdk.internal.net.http.SocketTube$SocketFlowTask/run
  SequentialScheduler.java:  207  jdk.internal.net.http.common.SequentialScheduler$SchedulableTask/run
  SequentialScheduler.java:  280  jdk.internal.net.http.common.SequentialScheduler/runOrSchedule
  SequentialScheduler.java:  233  jdk.internal.net.http.common.SequentialScheduler/runOrSchedule
           SocketTube.java:  782  jdk.internal.net.http.SocketTube$InternalReadPublisher$InternalReadSubscription/signalReadable
           SocketTube.java:  965  jdk.internal.net.http.SocketTube$InternalReadPublisher$ReadEvent/signalEvent
           SocketTube.java:  253  jdk.internal.net.http.SocketTube$SocketFlowEvent/handle
       HttpClientImpl.java: 1476  jdk.internal.net.http.HttpClientImpl$SelectorManager/handleEvent
       HttpClientImpl.java: 1421  jdk.internal.net.http.HttpClientImpl$SelectorManager/lambda$run$3
            ArrayList.java: 1597  java.util.ArrayList/forEach
       HttpClientImpl.java: 1421  jdk.internal.net.http.HttpClientImpl$SelectorManager/run

Killed

this looks like what'd happen if you got an OOM in the container, I suspect

I've managed to repro the above case locally, fwiw - in this case, if we're generating the attribute keyword every time, XT will essentially have a table with 1 million columns, which it hasn't been designed for 😅 if you change generate-random-map to the following, to simulate the documents having a relatively consistent key-set it goes through fine:

(defn generate-random-map
  "Generates a random map with a keys `ks` and using string values between 0 and 50 characters long.
   The keys have an 80% chance of being present."
  [ks]
  (into {}
        (for [k ks
              :when (> 0.8 (rand))]
          [k (random-string (rand-int 50))])))

(let [ks (repeatedly 10 #(random-keyword 10))]
  (dotimes [n 1000]
    (when (zero? (mod n 100))
      (prn n))

    (xt/submit-tx node [(into [:put-docs :table1]
                              (for [m (range 1000)]
                                (assoc (generate-random-map ks) :xt/id (str n "_" m))))])))

I've passed it via a profiler - surprisingly, random-string was taking up 98% of the overall runtime 😮 I extracted the chars out into a constant, which dropped random-string by ~6x:

(def chars
  (->> (concat (range 48 58) ; 0-9
               (range 65 91) ; A-Z
               (range 97 123) ; a-z
               )
       (mapv char)))

(defn random-string
  "Generates a random string of a given length."
  [length]
  (->> (repeatedly length #(rand-nth chars))
       (apply str)))

conscious we're now a fair way down the rabbit hole, worth coming back up to the top and seeing where we're at?

This is a really good insight about unique columns, and a mistake in my implementation. I actually don’t need that at all, and in fact will have 30-50 columns at most. This is the first thing I’ll try tomorrow and report back here, thanks a lot for trying it out! What kind of profiler did you use, I’d like to try the same :) I’m not a seasoned Clojure programmer (you can probably tell hehe), I’m kind of learning it as I go, choosing the database first, and then the language that makes most sense to work with it 🙂

personally, YourKit, but only really because that's the one I've always used and so I know where the buttons are 😅 IME they're mostly equivalent. like IDEs - one might be 'better' but if you're familiar with something else you'll be way more productive in that one

I used jconsole a bit, but will give this a try! Thanks!

oh, and price too - if it's a personal project then some do cheaper personal licenses

I used xtc/start-client to create the node (connecting to local Docker instance)

hey @alen, sorry to hear this 🤔 have you got a repro you could share? I'm doing the following, and it's returning instantly, as you'd expect:

(with-open [node (xtn/start-node)]
  ;; submit 1M docs
  (dotimes [n 1000]
    (xt/submit-tx node [(into [:put-docs :table] (for [m (range 1000)]
                                                   {:xt/id (str n "_" m)}))]))

  (xt/q node "SELECT COUNT(*) FROM table2"))

oh, it looks like your latest-completed-tx is a fair way behind your latest-submitted-tx - XT will wait for that to catch up. to confirm, can try passing {:tx-timeout #xt.time/duration "PT2S"} as the query opts - if this is the cause, you'll get a timeout exception

are you getting any exceptions in your log, perhaps?

hmm, I do see a bunch of exceptions now, ~30min after the last submitted tx

'clojure.lang.ExceptionInfo: Ingestion stopped: Failure allocating buffer. {}'

looks like it went OOM

org.apache.arrow.memory.OutOfMemoryException: Failure allocating buffer.
...
Caused by: io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 4194304 byte(s) of direct memory (used: 3401580544, max: 3401580544)
...
15:56:03 | DEBUG xtdb.server | response error (class java.util.concurrent.ExecutionException): 'clojure.lang.ExceptionInfo: Ingestion stopped: Failure allocating buffer. {}'

ah, ok.

what's your setup, btw? i.e. choice of tx-log, object-store (in-memory/local, maybe?), are you running this at a REPL, or maybe a separate XT process?

cause looks genuine - got ~3GB of off-heap memory allocated and looks like it's run out, so looks like our memory management isn't getting it right somewhere

• Running node locally w/ docker

docker run --pull=always -tip 6543:3000 -v ./xtdb-data:/var/lib/xtdb ghcr.io/xtdb/xtdb-standalone-ea

so I'm guessing tx log and object store are all default values, right? Object store should be FS, I'm not sure about tx-log

yep, both FS in the Docker image 👍

would it be relatively straightforward/not business sensitive to post a repro?

not business sensitive, but I don't have a clear repro since I was trying different things and I'm not sure what ran at which point 🙈 I've wiped my local volume now and started again, I'll try to see whether I can repeat the same thing 😅

heh, nw - cheers 🙂 🙏

I've also noticed that when the compactor runs, queries won't get through. For example, I've used your code to submit 1000 documents, and immediately after got about 3.5 min of compactor doing it's thing, during which the xt/q was blocking. Is this to be expected?

it shouldn't do, at least, compactor should be completely in the background, and not a dependency of the query

depending on the shape of your documents, we do have a known issue where the metadata creation takes a disproportionately large time for large documents, it might be that...?

(talking 100s-1000s of KVs in a single doc)

no, this is a simple example you provided, with just an id in the map. Perhaps CIDER or Emacs might be doing something here, I'll have to try with REPL in the terminal 🤔

compactor and indexer may be starving the query of threads, but that's only if you've got <=2 threads on your container

it shouldn't be limited unless explicitly set (which I didn't) AFAIK, but you actually raise a valid point, maybe my laptop is doing other things, reducing resources for XTDB?

Interestingly enough i tried it but failed. Retried again with exact code you shared - failed again. I had to add

:repositories [["public-github" {:url ""
                                   :checksum :warn}]
                 #_["private-github" {:url "" :protocol :ssh}]]

I have with lein-git-down which allows you to use git deps from leiningen

try adding these to project.clj

:plugins [[reifyhealth/lein-git-down "0.4.0"]]
  :git-down {xtdb-inspector {:coordinates tatut/xtdb-inspector}}
  :middleware [lein-git-down.plugin/inject-properties]