Clojurians Log v2

• I’ve cloned the repo so I’m running from tip of main (`9a19326c577490f9a7ec4bb1f2f7088ad3b5ab63`) • I ran docker/bin/build-standalone-image.sh • I started a container using that image • I ran xtsql (the Python script) I then sent this statement, which seemed to work:

xtdb-> INSERT INTO people (_id, name) VALUES (6, 'fred');

| systemTime                     | txId  |
|--------------------------------|-------|
| '2024-07-04T16:05:11.962697Z'  | 1309  |

xtdb-> select * from people;
500 Server Error
clojure.lang.ExceptionInfo: Ingestion stopped: Cannot invoke "java.lang.Class.getName()" because the return value of "clojure.lang.IFn.invoke(Object)" is null {}

16:17:54 | DEBUG xtdb.server | response error (class java.util.concurrent.ExecutionException): 'clojure.lang.ExceptionInfo: Ingestion stopped: Cannot invoke "java.lang.Class.getName()" because the return value of "clojure.lang.IFn.invoke(Object)" is null {}'
java.util.concurrent.ExecutionException: clojure.lang.ExceptionInfo: Ingestion stopped: Cannot invoke "java.lang.Class.getName()" because the return value of "clojure.lang.IFn.invoke(Object)" is null {}
	at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
	at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
	at clojure.core$deref_future.invokeStatic(core.clj:2317)
	at clojure.core$deref.invokeStatic(core.clj:2337)
	at clojure.core$deref.invoke(core.clj:2323)
	at xtdb.indexer.Indexer.awaitTx(indexer.clj:567)
	at xtdb.node.impl.Node.prepareQuery(impl.clj:132)
	at xtdb.node.impl.Node.open_sql_query(impl.clj:114)
	at xtdb.server$eval18037$fn__18038$fn__18040.invoke(server.clj:268)
	at sieppari.interceptor$eval17547$fn__17548$fn__17549.invoke(interceptor.cljc:33)
	at sieppari.core$_try.invokeStatic(core.cljc:20)
	at sieppari.core$_try.invoke(core.cljc:17)
	at sieppari.core$enter.invokeStatic(core.cljc:62)
	at sieppari.core$enter.invoke(core.cljc:49)
	at sieppari.core$execute$fn__17753.invoke(core.cljc:125)
	at sieppari.core$execute.invokeStatic(core.cljc:123)
	at sieppari.core$execute.invoke(core.cljc:117)
	at reitit.interceptor.sieppari$reify__17762.execute(sieppari.clj:18)
	at reitit.http$ring_handler$fn__16820.invoke(http.cljc:165)
	at clojure.lang.AFn.applyToHelper(AFn.java:160)
	at clojure.lang.AFn.applyTo(AFn.java:144)
	at clojure.lang.AFunction$1.doInvoke(AFunction.java:31)
	at clojure.lang.RestFn.invoke(RestFn.java:436)
	at ring.adapter.jetty9.handlers.async$_doHandle.invokeStatic(async.clj:34)
	at ring.adapter.jetty9.handlers.async$_doHandle.invoke(async.clj:22)
	at ring.adapter.jetty9.handlers.AsyncProxyHandler.doHandle(Unknown Source)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1381)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484)
	at ring.adapter.jetty9.handlers.AsyncProxyHandler.doScope(Unknown Source)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1303)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
	at org.eclipse.jetty.server.Server.handle(Server.java:563)
	at org.eclipse.jetty.server.HttpChannel$RequestDispatchable.dispatch(HttpChannel.java:1598)
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:753)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:501)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:287)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100)
	at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:421)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:390)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:277)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:199)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: clojure.lang.ExceptionInfo: Ingestion stopped: Cannot invoke "java.lang.Class.getName()" because the return value of "clojure.lang.IFn.invoke(Object)" is null
	at xtdb.await$__GT_ingester_ex.invokeStatic(await.clj:18)
	at xtdb.await$__GT_ingester_ex.invoke(await.clj:17)
	at xtdb.await$await_tx_async$fn__23993.invoke(await.clj:31)
	at xtdb.await$await_tx_async.invokeStatic(await.clj:26)
	at xtdb.await$await_tx_async.invoke(await.clj:22)
	at xtdb.indexer.Indexer.awaitTx(indexer.clj:678)
	... 46 common frames omitted
Caused by: java.lang.NullPointerException: Cannot invoke "java.lang.Class.getName()" because the return value of "clojure.lang.IFn.invoke(Object)" is null
	at xtdb.trie$__GT_iid.invokeStatic(trie.clj:41)
	at xtdb.trie$__GT_iid.invoke(trie.clj:33)
	at xtdb.indexer$$reify__32668.indexOp(indexer.clj:327)
	at xtdb.indexer$query_indexer$eval_query$reify__32723.accept(indexer.clj:399)
	at xtdb.operator.project.ProjectCursor$reify__27625.accept(project.clj:130)
	at xtdb.operator.rename.RenameCursor$reify__28434.accept(rename.clj:39)
	at xtdb.operator.table.TableCursor$fn__28775.invoke(table.clj:37)
	at xtdb.operator.table.TableCursor.tryAdvance(table.clj:35)
	at xtdb.operator.rename.RenameCursor.tryAdvance(rename.clj:29)
	at xtdb.operator.project.ProjectCursor.tryAdvance(project.clj:116)
	at xtdb.query$wrap_cursor$reify__32484.tryAdvance(query.clj:78)
	at java.base/java.util.Spliterator.forEachRemaining(Spliterator.java:332)
	at xtdb.indexer$query_indexer$eval_query__32722.invoke(indexer.clj:396)
	at xtdb.indexer$wrap_sql_args$fn__32729.invoke(indexer.clj:419)
	at xtdb.indexer$foreach_arg_row.invokeStatic(indexer.clj:403)
	at xtdb.indexer$foreach_arg_row.invokePrim(indexer.clj)
	at xtdb.indexer$$reify__32734.indexOp(indexer.clj:401)
	at xtdb.indexer.Indexer$fn__32784$index_tx_ops__32787$fn__32800.invoke(indexer.clj:622)
	at clojure.lang.AFn.call(AFn.java:18)
	at io.micrometer.core.instrument.AbstractTimer.recordCallable(AbstractTimer.java:175)
	at xtdb.indexer.Indexer$fn__32784$index_tx_ops__32787.invoke(indexer.clj:619)
	at xtdb.indexer.Indexer$fn__32784$fn__32808.invoke(indexer.clj:633)
	at xtdb.indexer.Indexer$fn__32784.invoke(indexer.clj:632)
	at xtdb.indexer.Indexer.indexTx(indexer.clj:596)
	at xtdb.log.watcher$watch_log_BANG_$reify__33854.acceptRecord(watcher.clj:53)
	at xtdb.log$tx_handler$fn__11137.invoke(log.clj:37)
	at clojure.core.protocols$fn__8249.invokeStatic(protocols.clj:168)
	at clojure.core.protocols$fn__8249.invoke(protocols.clj:124)
	at clojure.core.protocols$fn__8204$G__8199__8213.invoke(protocols.clj:19)
	at clojure.core.protocols$seq_reduce.invokeStatic(protocols.clj:31)
	at clojure.core.protocols$fn__8236.invokeStatic(protocols.clj:75)
	at clojure.core.protocols$fn__8236.invoke(protocols.clj:75)
	at clojure.core.protocols$fn__8178$G__8173__8191.invoke(protocols.clj:13)
	at clojure.core$reduce.invokeStatic(core.clj:6886)
	at clojure.core$reduce.invoke(core.clj:6868)
	at xtdb.log.NotifyingSubscriberHandler$fn__11170.invoke(log.clj:96)
	at clojure.lang.AFn.run(AFn.java:22)
	... 1 common frames omitted

Hey @aviflax I don't recognise that error, we'll take a look. Thanks for sharing the full stack trace. Did you have success using the published image at least?

Thanks Jeremy! That sounds great. > Thanks for sharing the full stack trace. My pleasure. > Did you have success using the published image at least? I don’t think so; I did try it at first and had a problem, which is when I checked how recent the image was and it seemed pretty old. That’s when I thought I’d try building the image from source with the latest code. (I probably should have just written a Dockerfile that would download the latest snapshot JAR from somewhere (?) and run it when started, but, I dunno, this is what I did.) In a few minutes I’ll try switching back to the publicly available image from and I’ll let you know what happens.

Interesting… I switched back, and did a fresh pull, created a new container, etc — and now the simple statements are indeed working. However, now I’m running into my original problem again, which is that my slightly more complex statements are not working. And not only that — they seem to be putting the DB into a failure state! This is my insert statement: insert into partners (_id, name, default_location_id, partner_types) values ("89355da0-5841-e4e0-fd7f-328877cb603e", "Aptera", "2fda6433-609e-cb05-27e0-37cf5ef213f8", ["vendor", "customer"]); that seems to work; it returns:

| systemTime                     | txId  |
|--------------------------------|-------|
| '2024-07-04T17:20:53.314482Z'  | 1173  |

500 Server Error
clojure.lang.ExceptionInfo: Ingestion stopped: Cannot invoke "java.lang.Class.getName()" because the return value of "clojure.lang.IFn.invoke(Object)" is null {}

Thanks for the confirmations. Your latest issue looks to be your use of double-quotes - in XTDB we follow the SQL spec very strictly here (e.g. unlike MySQL), I think you want to use single quotes across that insert statement instead

i.e. insert into partners (_id, name, default_location_id, partner_types) values ('89355da0-5841-e4e0-fd7f-328877cb603e', 'Aptera', '2fda6433-609e-cb05-27e0-37cf5ef213f8', ['vendor', 'customer']);

In general though "ingestion stopped" without any kind of rollback is bad UX (you have no choice but to start with a fresh db - apologies!) and we are still working through the error handling flows on this front

That all makes a ton of sense. Thank you!!!

BTW, quick follow-up… I think part of why I ended up using double-quotes accidentally in my SQL statement is because in my application code, my db client, I have a JSON object, and really all I want to do is save that JSON object to the db. So it’d be really convenient if there were an API that accepted an id and a JSON object as a document. In lieu of that, I’m building a SQL statement with a StringBuilder, which is… well, let’s say, at the very least it’s inelegant. My impression is that, at least for now, XTDB has no such API. Unless… maybe I’m missing something?

That's right, unfortunately there's no "here's some arbitrary JSON" SQL function...yet. Would supplying the JSON as a string input to a function be acceptable for you?

> That's right, unfortunately there's no "here's some arbitrary JSON" SQL function...yet. OK, thanks, that’s what I thought but the confirmation is helpful. > Would supplying the JSON as a string input to a function be acceptable for you? Yes, absolutely. In a way, I guess that’s kinda what the function I’m writing does.

BTW I’m currently writing Kotlin code to use the (v2) Web API directly because I couldn’t get the Kotlin driver library to work. (I’m also new to Kotlin 😅 but I know Java so I thought “how hard could it be”). It would be incredibly helpful if you’d publish some full, working examples of using your driver libraries somewhere. I did a bunch of searches in multiple places (GitHub, Discourse, Slack, the docs, the repo, the Web) but couldn’t find any.

Oh interesting! So, if you want to be at the real bleeding edge with Kotlin and XT2, you could try using our pgwire interface. We're working closely with one of our Design Partner users on this currently, and they're using https://github.com/cashapp/sqldelight (in case that approach is of interest!)

Thanks! I’ll take a look.

@taylor.jeremydavid sorry to keep bothering you but… I’m not quite getting this working yet. Can you see anything wrong with this statement?

insert into partners (_id, category_refs, name, default_location_id, partner_group_refs, partner_types)
values ('876a7d0e-f087-4dd0-9d2a-699999925e8b', {}, '\u001a\u0011', '4baf6d8f-f99b-3300-98c4-25d571373ae1', [{'id':'876a7d0e-f087-4dd0-9d2a-699999925e8b'}], ['vendor','customer']);

@aviflax FYI, a helpful debugging tool can be querying the _txs table. Transactions might be submitted but not necessarily have been committed successfully as in this case.

Aha that’s great help — thank you!

In this case, the error in _txs isn't so helpful. But to fix it just make the key in the map a symbol not a string like so: {id: ...}

Thanks! I thought I tried that, but I will double check.