2024-06-18 xtdb | Clojure Slack Archive

xtdb 2024-06-18

jarohen 2024-06-18T12:41:53.355349Z

Hey all - we've released another snapshot of the XT2 early access: 🚀 • SQL: built-in columns now accessible through _id, _valid_from etc (rather than xt$id, xt$valid_from) - they remain :xt/id in Clojure. • XTQL: we've removed support for XTQL from the Java/Kotlin and JSON API. Openly, this didn't fulfil the promises we wanted it to, it wasn't nearly as elegant to use from any other language as it is using Clojure's s-exprs, and the feedback we received from outside the Clojure community was largely that people would much prefer to use existing knowledge and familiar tooling - fair enough! • Pgwire: ◦ As part of our ongoing schema inference work, columns that only contain one data-type are now returned as such - e.g. a column containing only integers will now be returned as integers. ◦ We've added the ability to start the server on any available port - pass it :port 0 and then call node.getPgPort()/`(pgw/pg-port node)` to get the assigned port. Useful for test fixtures where you're creating a new in-memory XTDB node for every test. • ... as well as a good handful of bugfixes. As always, let us know if you run into any issues or have any questions! Cheers, XT Team

💯 1

🎉 8

jarohen 2024-06-19T08:49:30.355469Z

Old-school Datalog (i.e. not Datomic's dialect) is all about relations - Datomic's is in some ways closer to SPARQL - so we'd likely take inspiration from this. The from in XTQL would be consistent within a Datalog :where clause, for example, in place of the triples.

jarohen 2024-06-19T08:51:32.760319Z

more generally, this has definitely been one of the more difficult, ongoing design decisions with XT2 - where to position it on the spectrum of XT1++ to SQL-only relational database

jarohen 2024-06-19T08:52:26.594319Z

XT1++ would no doubt have been an easier migration, more incremental, but also a big restriction in the improvements we could make

jarohen 2024-06-19T08:53:20.539419Z

I don't think we'd have gotten anything like the SQL support we have, to open XTDB up to a much wider audience, nor the indexing/query performance without a more substantial change

jarohen 2024-06-19T08:59:04.992969Z

but OTOH we also really don't want to lose the essence of XTDB, the core concepts, the feedback loop DX that we all know and love from Clojure

jarohen 2024-06-19T08:59:17.819899Z

'threading the needle', we affectionately call it 🙂 hopefully we've found a good spot though

tatut 2024-06-19T09:07:21.301869Z

those who like Datomic style query language, will probably want to use that… although the feature set is not exactly the same of course

➕ 1

jarohen 2024-06-19T09:16:47.865579Z

we've perhaps got a little scope to move on that particular spectrum, too - when we were targeting XTQL at a wider audience, we made certain decisions on what the language should look like; now that that's scaled back, the context and aims are sufficiently different that we'd make different decisions. i.e. it probably makes more sense now to optimise for reducing the learning curve for people already familiar with EDN Datalog rather than needing to balance the learning curve for newer, non-Clojure users

tatut 2024-06-19T09:27:51.406219Z

but there are things that compile to SQL, like https://github.com/EvgSkv/logica so I don’t see that being table based means you couldn’t have datomicish query language, perhaps some additional definitions would be needed

jarohen 2024-06-19T09:29:29.713779Z

we did consider Logica and similar things - unfortunately as far as we could tell some of the SQL queries they generated, while obviously valid and correct SQL, would have been non-trivial to optimise without control over the generated SQL

jarohen 2024-06-19T09:31:20.308159Z

but yes, agreed in that I don't think there's anything (apart from time, of course!) in either having a SQL query engine nor underlying tables that would have prevented us offering something akin to EDN Datalog

🤞 1

tatut 2024-06-19T09:37:03.098009Z

so there’s hope? (that someone will make that, in the fullness of time)

jarohen 2024-06-19T09:40:10.802319Z

ah man, you know I can't give a definitive answer to that one 😅

tatut 2024-06-19T09:40:48.013929Z

a 3rd party could also make it, not trying to extract any promises from you 😄

😄 1

seancorfield 2024-06-18T16:34:02.708439Z

> we've removed support for XTQL from the Java/Kotlin and JSON API I was having a discussion the other day about preferring SQL (with HoneySQL) over XTQL and mentioned that I expected non-Clojure clients to never use XTQL so this is nice confirmation for me. It begs the question: are you really intending to keep XTQL up to date with the SQL API, just for the Clojure folks who use it (rather than them also switching to SQL)?

lispyclouds 2024-06-18T16:34:15.778829Z

I just tried this out by pulling: docker run --rm -itp 3000:3000 and running these:

(def node (xtc/start-client ""))

(xt/submit-tx node [[:put-docs :people
                     {:xt/id "1" :name "Bob" :loc "London"}
                     {:xt/id "2" :name "Alice" :loc "Paris"}]])

(xt/q node '(from :people [{:name "Bob"} *]))

causes

response error (class xtdb.IllegalArgumentException): 'Illegal argument: 'unknown-query-type''
xtdb.IllegalArgumentException: Illegal argument: 'unknown-query-type'
        at xtdb.error$illegal_arg.invokeStatic(error.clj:11)
        at xtdb.error$illegal_arg.invoke(error.clj:3)
        at xtdb.error$illegal_arg.invokeStatic(error.clj:7)
        at xtdb.error$illegal_arg.invoke(error.clj:3)
        at xtdb.server$eval18049$fn__18050$fn__18052.invoke(server.clj:276)
        at sieppari.interceptor$eval17559$fn__17560$fn__17561.invoke(interceptor.cljc:33)
        at sieppari.core$_try.invokeStatic(core.cljc:20)
        at sieppari.core$_try.invoke(core.cljc:17)
        at sieppari.core$enter.invokeStatic(core.cljc:62)
        at sieppari.core$enter.invoke(core.cljc:49)
        at sieppari.core$execute$fn__17765.invoke(core.cljc:125)
        at sieppari.core$execute.invokeStatic(core.cljc:123)
        at sieppari.core$execute.invoke(core.cljc:117)
        at reitit.interceptor.sieppari$reify__17774.execute(sieppari.clj:18)
        at reitit.http$ring_handler$fn__16832.invoke(http.cljc:165)
        at clojure.lang.AFn.applyToHelper(AFn.java:160)
        at clojure.lang.AFn.applyTo(AFn.java:144)
        at clojure.lang.AFunction$1.doInvoke(AFunction.java:31)
        at clojure.lang.RestFn.invoke(RestFn.java:436)
        at ring.adapter.jetty9.handlers.async$_doHandle.invokeStatic(async.clj:34)
        at ring.adapter.jetty9.handlers.async$_doHandle.invoke(async.clj:22)
        at ring.adapter.jetty9.handlers.AsyncProxyHandler.doHandle(Unknown Source)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1381)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484)
        at ring.adapter.jetty9.handlers.AsyncProxyHandler.doScope(Unknown Source)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1303)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
        at org.eclipse.jetty.server.Server.handle(Server.java:563)
        at org.eclipse.jetty.server.HttpChannel$RequestDispatchable.dispatch(HttpChannel.java:1598)
        at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:753)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:501)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:287)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100)
        at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
        at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:421)
        at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:390)
        at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:277)
        at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:193)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
        at java.base/java.lang.Thread.run(Thread.java:1583)
16:32:11 | WARN  o.eclipse.jetty.server.HttpChannel | /query
java.lang.IllegalStateException: COMPLETED
        at org.eclipse.jetty.server.HttpChannelState.sendError(HttpChannelState.java:911)
        at org.eclipse.jetty.server.Response.sendError(Response.java:498)
        at ring.adapter.jetty9.handlers.async$_doHandle$fn__692.invoke(async.clj:42)
        at sieppari.core$execute$fn__17765.invoke(core.cljc:128)
        at sieppari.core$execute.invokeStatic(core.cljc:123)
        at sieppari.core$execute.invoke(core.cljc:117)
        at reitit.interceptor.sieppari$reify__17774.execute(sieppari.clj:18)
        at reitit.http$ring_handler$fn__16832.invoke(http.cljc:165)
        at clojure.lang.AFn.applyToHelper(AFn.java:160)
        at clojure.lang.AFn.applyTo(AFn.java:144)
        at clojure.lang.AFunction$1.doInvoke(AFunction.java:31)
        at clojure.lang.RestFn.invoke(RestFn.java:436)
        at ring.adapter.jetty9.handlers.async$_doHandle.invokeStatic(async.clj:34)
        at ring.adapter.jetty9.handlers.async$_doHandle.invoke(async.clj:22)
        at ring.adapter.jetty9.handlers.AsyncProxyHandler.doHandle(Unknown Source)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1381)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484)
        at ring.adapter.jetty9.handlers.AsyncProxyHandler.doScope(Unknown Source)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1303)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
        at org.eclipse.jetty.server.Server.handle(Server.java:563)
        at org.eclipse.jetty.server.HttpChannel$RequestDispatchable.dispatch(HttpChannel.java:1598)
        at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:753)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:501)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:287)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100)
        at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
        at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:421)
        at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:390)
        at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:277)
        at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:193)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
        at java.base/java.lang.Thread.run(Thread.java:1583)
16:32:41 | WARN  o.e.j.util.thread.QueuedThreadPool | Job failed
java.lang.IllegalStateException: ABORTED
        at org.eclipse.jetty.server.HttpChannelState.sendError(HttpChannelState.java:911)
        at org.eclipse.jetty.server.HttpChannelState.nextAction(HttpChannelState.java:493)
        at org.eclipse.jetty.server.HttpChannelState.unhandle(HttpChannelState.java:418)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:650)
        at org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:461)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
        at java.base/java.lang.Thread.run(Thread.java:1583)

lispyclouds 2024-06-18T16:34:27.479109Z

is this something new on this one?

lispyclouds 2024-06-18T16:40:07.452479Z

im also seeing this when it idles for a bit

16:32:41 | WARN  o.e.j.util.thread.QueuedThreadPool | Job failed
java.lang.IllegalStateException: ABORTED
        at org.eclipse.jetty.server.HttpChannelState.sendError(HttpChannelState.java:911)
        at org.eclipse.jetty.server.HttpChannelState.nextAction(HttpChannelState.java:493)
        at org.eclipse.jetty.server.HttpChannelState.unhandle(HttpChannelState.java:418)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:650)
        at org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:461)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
        at java.base/java.lang.Thread.run(Thread.java:1583)

lispyclouds 2024-06-18T16:47:54.192279Z

@seancorfield the example here still shows some more love for XTQL? 😅 https://docs.xtdb.com/guides/xtql-walkthrough.html#_pull

lispyclouds 2024-06-18T19:30:38.648419Z

made sure all the deps in m2 were latest and that wasnt the case. all good now.

👌 1

🙏 1

jarohen 2024-06-18T19:47:33.390639Z

It begs the question: are you really intending to keep XTQL up to date with the SQL API, just for the Clojure folks who use it (rather than them also switching to SQL)?

@seancorfield: we can - the abstraction layer between the two is relatively high level, both SQL and XTQL compile reasonably quickly into our logical plan, and everything below that is common code. if anything (and probably no surprise) the SQL takes a lot more analysing to get to the same point 🙂 but honestly, it'll depend on the uptake and feedback - if the collective response to XTQL from the Clojure community is take-it-or-leave-it, then yeah, it doesn't makes sense for us to maintain it; if it gets people excited that's great; if people'd prefer less of a breaking change from v1, something more familiar, less of a learning curve, we can consider that too - we're still in early access, at the end of the day. I obviously like XTQL (I'm biased 🙂) but we're also being pragmatic, yep. great question, thanks 😄

Stef Coetzee 2024-06-18T23:45:01.732209Z

> I obviously like XTQL (I'm biased 🙂) +1! 😇

tatut 2024-06-19T06:01:03.513739Z

will datalog be supported? “upgrading” a big app from v1 would be difficult without

seancorfield 2024-06-19T06:29:34.422439Z

@tatut given that v2 is table-based, datalog has no support for that... I don't know what they have in mind for migration tools

tatut 2024-06-19T06:30:31.461389Z

yes, I expect it would be more of a rewrite than a migration

lispyclouds 2024-06-19T06:32:54.002129Z

I’m in the process of moving a moderately large thing to XTQL and it does feel almost a rewrite. Although some v1 queries have shrunk in moving parts.

Clojurians Log v2

xtdb 2024-06-18