I'm trying to make a simple persistence abstraction for my polylith project. When trying to implement it for XTDB v2 and testing it with an in-memory in-process node, execute-tx hangs for ever when running with clojure -M:poly test :dev Running the development environment and trying execute-tx
in the REPL works as intended.
(def registry
{:person
[:map {::entity/name :person}
[:first-name :string]
[:last-name :string]
[:age :int]]})
(use-fixtures :once (with-registry registry))
(use-fixtures :each (with-system
(component/system-map
:config {:xtdb {}}
:persistence (component/using
(fresh-xtdb-persistence)
[:config]))))
(deftest fetch--test
(let [{:keys [persistence]} *system*
{:keys [node]} persistence
person-id (random-uuid)
person (-> :person
(gen-entity)
(assoc :xt/id person-id))]
; TODO: Fix blocking forever
(xt/execute-tx node [[:put-docs :person person]])
(testing "Inserted random person should be fetch-able"
(is (= person
(-> persistence
(persistence/fetch :person person-id)))))))
:poly {:jvm-opts [
; Required for in-process XTDB
"--add-opens=java.base/java.nio=ALL-UNNAMED"
"-Dio.netty.tryReflectionSetAccessible=true"]
:main-opts ["-m" "polylith.clj.core.poly-cli.core"]
:extra-deps {polylith/clj-poly {:mvn/version "0.2.19"}
polylith-kaocha/test-runner
{:git/url ""
:git/tag "v0.8.4"
:git/sha "f096de8"
:deps/root "projects/test-runner"}}}
Tried using both the default polylith runner and https://github.com/imrekoszo/polylith-kaocha test runner.thanks both π I've put the call out on our internal Slack, will let you know what we find
yep, confirmed green here on Linux, hanging on Apple silicon clojure-spin
we'll try to minimise it further - @seancorfield: anything you'd recommend we try taking out of the loop as far as polylith's concerned?
we've forked it to https://github.com/xtdb/xtdb-poly-test-hang, added instructions to the readme. on Mac, we're now getting it hanging with the poly cmd, passing with test-runner
have a cause: https://github.com/xtdb/xtdb/issues/3454
but I'll be honest, at this point I have zero clue why the combination of Apple + Polylith would make the slightest bit of difference that wouldn't also affect Apple + Cognitect runner or Linux + Polylith runner - it's a pure function
Saw the bug report -- that sure is a weird one. Especially since it seems to work directly from the REPL.
The built-in poly test runner tries to use classloader manipulation to isolate testing (but isn't entirely successful, which is why I had to create the external test runner for use at work). The external test runner constructs the java command-line directly to run a completely separate process, so there may be subtle differences in classpath/JVM options, or maybe it's picking up a different java command on Apple Silicon?
Might be interesting to have the repro test dump out (System/getProperties) to look for anomalies.
@tomi.hukkalainen_slac Normal repl development works with an in-memory node. Doesn't work when running from polylith test runner(s). Don't know if plain clojure test runner or kaocha would work.
@jarohen Is there someone on the team that has apple silicon and can reproduce?
we've not managed to repro it so far I'm afraid - will raise an issue for it to give it more visibility and see if anyone else in the community can
@pafitisnick is there any way to get a more minimal repro? any components you can inline, for example?
I can try and make a leaner reproduction and get back to you
Did you try running the test in the aforementioned repository?
I did, I've not managed to get hold of someone with a Mac as yet (we're mostly Linux at JUXT)
could you also share the command you run for the test and the directory you run it from? I managed to hack together a REPL with the right deps and source-path but would prefer to run it as you are
Its clojure -A:poly test :dev from root but i think i pushed some changes were the hanging test is commented out,now.
thanks - I'm now getting:
ERROR in (fetch--test) (core.cljc:136)
Uncaught exception, not in assertion.
expected: nil
actual: clojure.lang.ExceptionInfo: :malli.core/invalid-schema
{:type :malli.core/invalid-schema, :message :malli.core/invalid-schema, :data {:schema :person, :form :person}}
at malli.core$_exception.invokeStatic (core.cljc:136)
malli.core$_exception.invoke (core.cljc:136)
malli.core$_fail_BANG_.invokeStatic (core.cljc:140)
malli.core$_fail_BANG_.invoke (core.cljc:138)
malli.core$_lookup_BANG_.invokeStatic (core.cljc:272)
malli.core$_lookup_BANG_.invoke (core.cljc:268)
malli.core$schema.invokeStatic (core.cljc:2059)
malli.core$schema.invoke (core.cljc:2037)
malli.generator$generator.invokeStatic (generator.cljc:541)
malli.generator$generator.invoke (generator.cljc:534)
malli.generator$generate.invokeStatic (generator.cljc:547)
malli.generator$generate.invoke (generator.cljc:543)
malli.generator$generate.invokeStatic (generator.cljc:545)
malli.generator$generate.invoke (generator.cljc:543)
streamcraft.entity.core.MalliEntityRegistry.generate (core.cljc:56)
streamcraft.utils.test.persistence$persistence_fetch_tests.invokeStatic (persistence.cljc:11)
streamcraft.utils.test.persistence$persistence_fetch_tests.invoke (persistence.cljc:7)
streamcraft.persistence_xtdb.core_test$fn__76621.invokeStatic (core_test.cljc:28)
which I suspect is unrelated. could you minimise the test so that we can investigate further?@jarohen I have reverted the changes so that it hangs again now. I will also try to make a minimal reproduction tonight.
@jarohen Here's a repository with a minimalistic reproduction. On my machine it still hangs https://github.com/conjurernix/xtdb-poly-test-hang
You should add the repro steps to the README...
It throws an exception for me:
(~/clojure/xtdb-poly-test-hang)-(!2023)-> clojure -A:poly test :dev
Downloading: polylith/clj-poly/0.2.19/clj-poly-0.2.19.pom from clojars
Downloading: polylith/clj-poly/0.2.19/clj-poly-0.2.19.jar from clojars
WARNING: Use of :main-opts with -A is deprecated. Use -M instead.
WARNING: Implicit use of clojure.main with options is deprecated, use -M
point_up::skin-tone-2 You said -A above but -M is what the CLI wants -- do you get this message and ignore it? If not, what does clojure -version show for you?
Projects to run tests from: development
Running tests for the development project using test runner: Polylith built-in clojure.test runner...
Running tests from the development project, including 1 brick: persistence-xtdb
Testing testing-xtdb-poly.persistence-xtdb.interface-test
Starting node
SLF4J(W): No SLF4J providers were found.
SLF4J(W): Defaulting to no-operation (NOP) logger implementation
SLF4J(W): See for further details.
Transacting person
Person transacted
ERROR in (dummy-test) (error.clj:11)
this hangs indefinitely
expected: (= person (xt/q node (quote (from :person [* #:xt{:id $id}])) {:id id}))
actual: xtdb.IllegalArgumentException: Unknown symbol: '?id'
{:symbol ?id, :xtdb.error/error-key :xtdb.expression/unknown-symbol}
at xtdb.error$illegal_arg.invokeStatic (error.clj:11)
xtdb.error$illegal_arg.invoke (error.clj:3)
xtdb.error$illegal_arg.invokeStatic (error.clj:7)
xtdb.error$illegal_arg.invoke (error.clj:3)
xtdb.expression$form__GT_expr.invokeStatic (expression.clj:40)
xtdb.expression$form__GT_expr.invoke (expression.clj:33)
xtdb.expression$eval11487$fn__11489$fn__11493.invoke (expression.clj:147)
clojure.core$mapv$fn__8535.invoke (core.clj:6980)
clojure.lang.PersistentList.reduce (PersistentList.java:144)
clojure.core$reduce.invokeStatic (core.clj:6886)
clojure.core$mapv.invokeStatic (core.clj:6971)
clojure.core$mapv.invoke (core.clj:6971)
xtdb.expression$eval11487$fn__11489.invoke (expression.clj:147)
clojure.lang.MultiFn.invoke (MultiFn.java:234)
xtdb.expression$form__GT_expr.invokeStatic (expression.clj:57)
xtdb.expression$form__GT_expr.invoke (expression.clj:33)
xtdb.operator.scan$eval19000$fn$reify__19005$iter__19052__19056$fn__19057.invoke (scan.clj:464)
clojure.lang.LazySeq.sval (LazySeq.java:42)
clojure.lang.LazySeq.seq (LazySeq.java:51)
clojure.lang.RT.seq (RT.java:535)
clojure.core$seq__5467.invokeStatic (core.clj:139)
clojure.core.protocols$seq_reduce.invokeStatic (protocols.clj:24)
clojure.core.protocols$fn__8236.invokeStatic (protocols.clj:75)
clojure.core.protocols/fn (protocols.clj:75)
clojure.core.protocols$fn__8178$G__8173__8191.invoke (protocols.clj:13)
clojure.core$reduce.invokeStatic (core.clj:6887)
clojure.core$into.invokeStatic (core.clj:6959)
clojure.core$into.invoke (core.clj:6951)
xtdb.operator.scan$eval19000$fn$reify__19005.emitScan (scan.clj:466)
xtdb.operator.scan$eval19160$fn__19162.invoke (scan.clj:530)
clojure.lang.MultiFn.invoke (MultiFn.java:234)
xtdb.query$prepare_ra$reify$reify__25045.apply (query.clj:150)
java.util.concurrent.ConcurrentHashMap.computeIfAbsent (ConcurrentHashMap.java:1708)
xtdb.query$prepare_ra$reify__25039.bind (query.clj:138)
xtdb.node.impl$then_execute_prepared_query.invokeStatic (impl.clj:55)
xtdb.node.impl$then_execute_prepared_query.invoke (impl.clj:54)
xtdb.node.impl.Node.open_xtql_query (impl.clj:121)
xtdb.api$q.invokeStatic (api.clj:111)
xtdb.api$q.invoke (api.clj:61)
testing_xtdb_poly.persistence_xtdb.interface_test$fn__2010.invokeStatic (interface_test.clj:15)
testing_xtdb_poly.persistence_xtdb.interface_test/fn (interface_test.clj:6) I wondered if I had an old snapshot so I tried to force an update and now I get
Error building classpath. Could not find artifact com.xtdb:xtdb-api:jar:2.0.0-SNAPSHOT in central ( )Ah, you're missing
:mvn/repos {"ossrh-snapshots" {:url ""}}
in deps.ednNope, even with the very latest 2.0.0 SNAPSHOT, I still get
ERROR in (dummy-test) (error.clj:11)
this hangs indefinitely
expected: (= person (xt/q node (quote (from :person [* #:xt{:id $id}])) {:id id}))
actual: xtdb.IllegalArgumentException: Unknown symbol: '?id'
{:symbol ?id, :xtdb.error/error-key :xtdb.expression/unknown-symbol}
at xtdb.error$illegal_arg.invokeStatic (error.clj:11)@seancorfield ah yes I'll add the mvn repo
Even with that added, the example still won't run -- see the exception I reported.
ye well I guess it doesn't matter because in my case it hangs at the transaction
Even if you do `
(is (= 1 1))
it doesn't matter, updated the repo with that.Person transacted never prints for me
OK, just want to set expectations when the JUXT folks try to run your repro and get an exception / test failure.
hmm π€ I'm afraid I don't know enough about polylith to know what might be different.
are you able to run jstack or equivalent on the offending process to see where it's getting stuck?
I'll check around. Will also try @seancorfield external test runner for polylith see what's going on
This is the output of jstack . TBH don't really know how to read it.
thanks π it's what every thread in the JVM is currently doing - I usually look out for main and (in our case) anything containing xtdb
in this case, looks like we've got main waiting on awaiting a transaction and our transaction subscription thread waiting on receiving a transaction π€ does this happen every time with Polylith, and never without? I'd expect if there were a potential deadlock here that it'd be intermittent both with and without - we've not seen one without in any of our test runs and benchmarks, at least
@pafitisnick you are starting a fresh XTDB node before each test runs -- are you correctly shutting down the node in you fixture after each test runs? @jarohen If there were multiple in-memory XTDB nodes started up in a single process, could that cause this sort of deadlock?
@seancorfield It's the only test that I have so it's the only time the fixture runs. Also my component looks like this
(defrecord XtdbPersistence [config node]
component/Lifecycle
(start [this]
(let [{:keys [xtdb]} config]
(-> this
(assoc :node (xtn/start-node xtdb)))))
(stop [this]
(when node
(.close node))
(-> this
(assoc :node nil)))@jarohen It happens every time with poly test which is the command that runs the component tests using it's own test runner (I tried it with other poly pluggable test runners, same result). Hasn't happened when running the dev environment normally
Is this project up on GH where we can try it out? If not, can you cut things down and publish a version on GH that exhibits the problem and I'll take a look.
I edited workspace.edn to use the external test runner, ran clojure -M:poly test :dev and got this failure:
FAIL in (fetch--test) (core_test.cljc:35)
Inserted random person should be fetch-able
expected: (= person (-> persistence (persistence/fetch :person person-id)))
actual: (not (= {:first-name "", :last-name "24xf", :age 211, :xt/id #uuid "6d4a85b8-7946-4b0c-84f6-e05edb14ba38"} [{:id #uuid "6d4a85b8-7946-4b0c-84f6-e05edb14ba38", :age 211, :last-name "24xf", :first-name "", :xt/id #uuid "6d4a85b8-7946-4b0c-84f6-e05edb14ba38"}]))
but it completed successfully (i.e., did not hang).I changed the test to this and it all passed (and did not hang):
; TODO: Fix blocking forever
(xt/execute-tx node [[:put-docs :person person]])
(testing "Inserted random person should be fetch-able"
(is (= [(assoc person :id person-id)] ; add :id and wrap in a vector
(-> persistence
(persistence/fetch :person person-id)))))))(the projects don't seem to be set up in a way that Polylith can run tests from them)
If it helps, I'm using this JDK:
sean@sean-win11-desk:~/oss/streamcraft$ $JAVA_CMD -version
openjdk version "21.0.1" 2023-10-17 LTS
OpenJDK Runtime Environment Temurin-21.0.1+12 (build 21.0.1+12-LTS)
OpenJDK 64-Bit Server VM Temurin-21.0.1+12 (build 21.0.1+12-LTS, mixed mode, sharing)@seancorfield projects are not really taken care of to be honest yet, they are just there. I'm using
java -version
openjdk version "21.0.2" 2024-01-16
OpenJDK Runtime Environment (build 21.0.2+13-58)
OpenJDK 64-Bit Server VM (build 21.0.2+13-58, mixed mode, sharing)
Let me investigate why it hangs on my machineOn my machine it still hangs even when switching to
java -version
openjdk version "22.0.1" 2024-04-16
OpenJDK Runtime Environment Temurin-22.0.1+8 (build 22.0.1+8)
OpenJDK 64-Bit Server VM Temurin-22.0.1+8 (build 22.0.1+8, mixed mode)
or
java -version
openjdk version "21.0.3" 2024-04-16 LTS
OpenJDK Runtime Environment Temurin-21.0.3+9 (build 21.0.3+9-LTS)
OpenJDK 64-Bit Server VM Temurin-21.0.3+9 (build 21.0.3+9-LTS, mixed mode)What is your dev environment? I'm running this on Ubuntu (on WSL2).
macos apple silicon
Ah, I bet that's your issue...
Are there any known issues with JVM not running right on apple silicon?
or polylith/xtdb for that matter?
I've seen some people report strange behavior on Apple silicon but haven't paid much attention since I switched to Windows a few years back (after three decades on Mac systems).
But I'd say it has to be environmental, and you're on Apple silicon while I expect nearly everyone using XTDB / Arrow is on x64...
I see
If you do figure something out, like same code working on x64, or in a virtual machine on arm mac, and so on, please let us know. Most of the people in our team are using macs, so it would definitely be a problem if xtdb cannot run locally on them.
Thanks for all the help @jarohen and @seancorfield. Hopefully this can be resolved soon.
I'm trying to write some data to XTDB2, here's what I'm doing
(defn with-id [n]
{:xt/id (random-uuid) :n n})
(defn write-it! [node table]
(let [data (map with-id (range 8))
partitioned (partition 2 data)
result (atom [])]
(doseq [part partitioned]
(a/go
(let [txop (into [:put-docs (keyword table)] part)]
(swap! result conj txop) ;; helper to collect txops for later inspection
(xt/execute-tx node txop))))
result))
(def node (xtc/start-client ""))
(def r
(write-it! node :helloworld))
Executing above gives me a couple of exceptions
Exception in thread "async-dispatch-104" Exception in thread "async-dispatch-103" Exception in thread "async-dispatch-105" IllegalArgumentException(key=:xtql/malformed-tx-op, message=Illegal argument: 'xtql/malformed-tx-op', data={:tx-op :put-docs}, cause=null)
Exception in thread "async-dispatch-102" IllegalArgumentException(key=:xtql/malformed-tx-op, message=Illegal argument: 'xtql/malformed-tx-op', data={:tx-op :put-docs}, cause=null)
However, when I inspect txops I tried to submit, I can't see anything wrong π€
[[:put-docs
:helloworld
{:xt/id #uuid "b2e829f8-55aa-453e-bf31-e59d3bb08700", :n 0}
{:xt/id #uuid "ede76494-b79b-4b3d-924c-a24fe69a16c5", :n 1}]
[:put-docs
:helloworld
{:xt/id #uuid "85bbae6e-0c07-4f15-9fc3-31f093aff9d7", :n 4}
{:xt/id #uuid "8a98c1f8-8de8-4f69-a212-6b6de452758f", :n 5}]
[:put-docs
:helloworld
{:xt/id #uuid "2a343f7d-6e7a-4892-a9f9-c68417711199", :n 2}
{:xt/id #uuid "20141feb-903c-4052-8a98-f9952ff4dcd2", :n 3}]
[:put-docs
:helloworld
{:xt/id #uuid "d53d5632-6989-44cd-834c-9c290fce45e5", :n 6}
{:xt/id #uuid "12543471-b6c1-4857-a66d-064b10555508", :n 7}]]
and in fact I can submit @r without issues
(xt/execute-tx node @r)
I'm probably doing something silly, but I've been at it for a long time and would appreciate another pair of eyes πIt looks like each individual execute-tx is passed [:put-docs ..] but that should be [[:put-docs ..]] I think?
Trying xtdbv2 following docker quick start and get json decode error. Did I miss some step(s)?
after
1. docker run --pull=always -v ~/work/xtdb-prototype/xtdb-data:/var/lib/xtdb -tip 6543:3000
2. follow python interactive insert a people record and select from people. Got exception of below. Longer details in thread to avoid spamming top level.
Caused by: kotlinx.serialization.json.internal.JsonDecodingException: Unexpected JSON token at offset 2: Encountered an unknown key 'query' at path: $
Use 'ignoreUnknownKeys = true' in 'Json {}' builder to ignore unknown keys.
JSON input: {"query": {"sql": "SELECT * FROM people"}, "queryOpts": {"txTimeout": "PT1S", "keyFn": "SNAKE_CASE_STRING"}}hey @xuelin thanks for the report - I'll verify these steps this morning
Thank you so much!
accidently, I'm trying it today (according the latest docs) and it still doesn't work
xtdb-> SELECT people.info.contact[2].tel FROM people WHERE people.name = 'fred';
No columns returned.
xtdb-> SELECT people.info.contact FROM people WHERE people.name = 'fred';
No columns returned.
xtdb-> SELECT FROM people WHERE people.name = 'fred';
| info |
|------------------------------------------------------------------------------|
| {'contact': [{'loc': 'home', 'tel': '123'}, {'loc': 'work', 'tel': '456'}]} |
Hi @brdloush apologies you hit this also before we pushed a fix out, so the docs simply haven't been updated here yet π
We're actually switching over to using psql as the main entry point very soon so I was just fixing this issue on https://github.com/xtdb/xtdb/pull/3496/files#diff-acd5f203807a937e389a5e7a6d807f37797577297f48284d8ca0b447d973b650R75 other branch earlier today, as it happens!
No problem. With v2 being discussed more and more, I thought it's finally time to read and try the docs π
thanks for the link. This query seems to work
xtdb-> SELECT ().contact[2].tel FROM people WHERE people.name = 'fred';
| tel |
|--------|
| '456' |
Of course, it raises questions at this point - such as "why does need to be wrapped in parens, why isthe contact field "indexed from 1" etc..> why does need to be wrapped in parens
answered here https://github.com/xtdb/xtdb/issues/3436#issuecomment-2206253209
> why is the contact field "indexed from 1"
SQL is 1-indexed based, officially speaking π¬
python session:
curl -s -O && python xtsql.py --url=
XTDB v2.0 -- SQL Console
Copyright Β© 2021-2024, JUXT LTD.
Type 'help' for commands
xtdb-> INSERT INTO people (_id, name) VALUES (6, 'fred');
| systemTime | txId |
|--------------------------------|-------|
| '2024-06-23T19:56:07.660193Z' | 1309 |
xtdb-> SELECT * FROM people;
Error decoding JSON!
xtdb->
main process output:
Starting XTDB 2.x (pre-alpha) @ dev-SNAPSHOT @ 1533cc0 ...
19:55:04 | DEBUG xtdb.metrics | Metrics server started on port: 8080
19:55:04 | INFO xtdb.server | HTTP server started on port: 3000
19:55:04 | INFO xtdb.pgwire | PGWire server started on port: 5432
19:55:04 | INFO xtdb.cli | Node started
19:56:15 | DEBUG xtdb.server | response error (class clojure.lang.ExceptionInfo): 'Malformed application/json request.'
clojure.lang.ExceptionInfo: Malformed application/json request.
at muuntaja.core$fail_on_request_decode_exception.invokeStatic(core.clj:132)
at muuntaja.core$fail_on_request_decode_exception.invoke(core.clj:126)
at muuntaja.core$create$_decode_request_body__12371.invoke(core.clj:424)
at muuntaja.core$create$reify__12393.format_request(core.clj:476)
at muuntaja.interceptor$format_request_interceptor$fn__17081.invoke(interceptor.clj:96)
at sieppari.core$_try.invokeStatic(core.cljc:20)
at sieppari.core$_try.invoke(core.cljc:17)
at sieppari.core$enter.invokeStatic(core.cljc:62)
at sieppari.core$enter.invoke(core.cljc:49)
at sieppari.core$execute$fn__17755.invoke(core.cljc:125)
at sieppari.core$execute.invokeStatic(core.cljc:123)
at sieppari.core$execute.invoke(core.cljc:117)
at reitit.interceptor.sieppari$reify__17764.execute(sieppari.clj:18)
at reitit.http$ring_handler$fn__16822.invoke(http.cljc:165)
at clojure.lang.AFn.applyToHelper(AFn.java:160)
at clojure.lang.AFn.applyTo(AFn.java:144)
at clojure.lang.AFunction$1.doInvoke(AFunction.java:31)
at clojure.lang.RestFn.invoke(RestFn.java:436)
at ring.adapter.jetty9.handlers.async$_doHandle.invokeStatic(async.clj:34)
at ring.adapter.jetty9.handlers.async$_doHandle.invoke(async.clj:22)
at ring.adapter.jetty9.handlers.AsyncProxyHandler.doHandle(Unknown Source)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1381)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484)
at ring.adapter.jetty9.handlers.AsyncProxyHandler.doScope(Unknown Source)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1303)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
at org.eclipse.jetty.server.Server.handle(Server.java:563)
at org.eclipse.jetty.server.HttpChannel$RequestDispatchable.dispatch(HttpChannel.java:1598)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:753)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:501)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:287)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100)
at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:421)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:390)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:277)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:193)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: clojure.lang.ExceptionInfo: Malformed application/json in :muuntaja/decode
at muuntaja.core$on_exception.invokeStatic(core.clj:284)
at muuntaja.core$on_exception.invoke(core.clj:267)
at clojure.core$partial$fn__5908.invoke(core.clj:2643)
at muuntaja.core$create_coder$decode__12321.invoke(core.clj:322)
at muuntaja.core$create$_decode_request_body__12371.invoke(core.clj:422)
... 43 common frames omitted
Caused by: xtdb.IllegalArgumentException: Error decoding JSON!
at xtdb.JsonSerde.decode(JsonSerde.kt:248)
at xtdb.server$json_query_decoder$reify__18036.decode(server.clj:242)
at muuntaja.core$create_coder$decode__12321.invoke(core.clj:320)
... 44 common frames omitted
Caused by: kotlinx.serialization.json.internal.JsonDecodingException: Unexpected JSON token at offset 2: Encountered an unknown key 'query' at path: $
Use 'ignoreUnknownKeys = true' in 'Json {}' builder to ignore unknown keys.
JSON input: {"query": {"sql": "SELECT * FROM people"}, "queryOpts": {"txTimeout": "PT1S", "keyFn": "SNAKE_CASE_STRING"}}
at kotlinx.serialization.json.internal.JsonExceptionsKt.JsonDecodingException(JsonExceptions.kt:24)
at kotlinx.serialization.json.internal.JsonExceptionsKt.JsonDecodingException(JsonExceptions.kt:32)
at kotlinx.serialization.json.internal.AbstractJsonLexer.fail(AbstractJsonLexer.kt:598)
at kotlinx.serialization.json.internal.AbstractJsonLexer.failOnUnknownKey(AbstractJsonLexer.kt:593)
at kotlinx.serialization.json.internal.StreamingJsonDecoder.handleUnknown(StreamingJsonDecoder.kt:257)
at kotlinx.serialization.json.internal.StreamingJsonDecoder.decodeObjectIndex(StreamingJsonDecoder.kt:243)
at kotlinx.serialization.json.internal.StreamingJsonDecoder.decodeElementIndex(StreamingJsonDecoder.kt:178)
at xtdb.api.query.QueryRequest$$serializer.deserialize(QueryRequest.kt:8)
at xtdb.api.query.QueryRequest$$serializer.deserialize(QueryRequest.kt:8)
at kotlinx.serialization.json.internal.StreamingJsonDecoder.decodeSerializableValue(StreamingJsonDecoder.kt:69)
at kotlinx.serialization.json.internal.JsonStreamsKt.decodeByReader(JsonStreams.kt:43)
at kotlinx.serialization.json.JvmStreamsKt.decodeFromStream(JvmStreams.kt:61)
at xtdb.JsonSerde.decode(JsonSerde.kt:240)
... 46 common frames omitted
Suppressed: java.io.IOException: mark/reset not supported
at java.base/java.io.InputStream.reset(InputStream.java:741)
at xtdb.JsonSerde.decode(JsonSerde.kt:243)
... 46 common frames omittedDigged into source a bit, this diff seems fixed it:
--- a/api/src/main/kotlin/xtdb/api/query/QueryRequest.kt
+++ b/api/src/main/kotlin/xtdb/api/query/QueryRequest.kt
@@ -7,6 +7,6 @@ import kotlinx.serialization.Serializable
*/
@Serializable
data class QueryRequest(
- @JvmField val sql: String,
+ @JvmField val query: QueryRequestQuery,
@JvmField val queryOpts: QueryOptions? = null
)
diff --git a/api/src/main/kotlin/xtdb/api/query/QueryRequestQuery.kt b/api/src/main/kotlin/xtdb/api/query/QueryRequestQuery.kt
index e69de29bb..9cc641cf5 100644
--- a/api/src/main/kotlin/xtdb/api/query/QueryRequestQuery.kt
+++ b/api/src/main/kotlin/xtdb/api/query/QueryRequestQuery.kt
@@ -0,0 +1,11 @@
+package xtdb.api.query
+
+import kotlinx.serialization.Serializable
+
+/**
+ * @suppress
+ */
+@Serializable
+data class QueryRequestQuery(
+ @JvmField val sql: String,
+)
diff --git a/http-server/src/main/clojure/xtdb/server.clj b/http-server/src/main/clojure/xtdb/server.clj
index df0d89e96..3465dde4a 100644
--- a/http-server/src/main/clojure/xtdb/server.clj
+++ b/http-server/src/main/clojure/xtdb/server.clj
@@ -33,7 +33,7 @@
(xtdb JsonSerde)
(xtdb.api HttpServer$Factory IXtdb TransactionKey Xtdb$Config)
xtdb.api.module.XtdbModule
- (xtdb.api.query Basis IKeyFn Query QueryRequest SqlQuery XtqlQuery)
+ (xtdb.api.query Basis IKeyFn Query QueryRequest QueryRequestQuery SqlQuery XtqlQuery)
(xtdb.api.tx TxOp TxOptions TxRequest)))
(defn decoder [_options]
@@ -241,7 +241,7 @@
(with-open [^InputStream data data]
(let [^QueryRequest query-request (JsonSerde/decode data QueryRequest)]
(-> (into {} (.queryOpts query-request))
- (assoc :query (.sql query-request))))))))
+ (assoc :query (.sql (.query query-request)))))))))Below query returned No columns returned though:
xtdb-> SELECT * FROM people;
| _id | info | likes | name |
|-----|------------------------------------------------------------------------------|-----------------------------------------|---------|
| 6 | {'contact': [{'loc': 'home', 'tel': '123'}, {'loc': 'work', 'tel': '456'}]} | NULL | 'fred' |
| 9 | NULL | ['fishing', 3.14, {'dynamic': 'data'}] | 'bob' |
xtdb-> SELECT people.info.contact[2].tel FROM people
.. WHERE people.name = 'fred';
No columns returned.