Clojurians Log v2

My personal opinion: it was a really bad idea to encourage qualified keys when SQL was one of the goals... It's a giant pain to work with in HoneySQL and next.jdbc. As the maintainer of those libraries, that's probably my #1 complaint about XTDB 2 😞

In both HoneySQL and next.jdbc, :foo/bar means table foo, column bar, but XTDB SQL via the pg-wire driver doesn't support table name on columns (which has precedence: Oracle doesn't support it, MS SQL Server requires certain options passed to queries to support it, and PG does an extra SQL query behind the scenes in order to support it).

Table foo, column bar doesn't translate well to inserting records / document. I was just wondering why I can't roundtrip a namespaced keyword, could be a apache arrow restriction.

Well, XT2 has "tables" so...

My personal opinion: it was a really bad idea to encourage qualified keys when SQL was one of the goals... It's a giant pain to work with in HoneySQL and next.jdbc.

anyway, enough of me being nostalgic 🙂 @avocabio could you clarify how you're inserting/querying those docs (i.e. through which API), and what you're expecting?

I’m using sql with pg object conversion copied from the xtdb.sente (or something ns called something like that I’m on mobile) I was expecting to be able to roundtrip clj map with namespaced keyword or even keep the same casing, atm I insert kebab case and get snake out.

if you're using it through the Clojure API, then you can add {:key-fn <:kebab-case-keyword|:snake-case-keyword>} to the options; if you're going through JDBC this'll be trickier

If you're using HoneySQL, the quoting mechanism on SQL entities assumes that :foo/bar should become "foo"."bar" -- i.e., table.column -- and if you use next.jdbc.sql, those "friendly" functions assume they can just call name on keywords to get column names, even if you provide :table-fn and/or :column-fn. Recent versions now have a :name-fn that lets you override that (nope, removed), but for HoneySQL, the closest approach (which will not work for you) is the :nrql dialect which turns off splitting so :foo/bar would become "foo/bar" (except NRQL uses backticks for stropping, like MySQL, and I don't think you can not override that). That's all "inbound", going _to SQL. For "outbound" in next.jdbc, you can control this with :builder-fn -- XT2 doesn't provide table names at the JDBC level (yet?), so you can't get qualified names back from next.jdbc, which means you could write a :builder-fn that maps $ to / in column names to get "qualified keywords" back in result sets -- but that wouldn't affect plan so you'd still have :foo$bar or "foo$bar" inside your reducing fns, since they side-step the whole overhead of a builder in the first place. So, yeah, "trickier"...

heh, yeah, 'trickier' was probably an understatement 😅 I'm working on https://github.com/xtdb/xtdb/issues/3844 today, hopefully that should make things a little easier 🤞

see https://github.com/xtdb/xtdb/pull/3885 and https://github.com/xtdb/xtdb/pull/3886, now on main 🙂

thanks @seancorfield for the chat last week that led to xtdb.next.jdbc, hope it's roughly what you pictured 🙂

note that James also included an example in the tests that demonstrates a viable way of handling qualified keywords by transforming them before handing them to HoneySQL (using our ->sql-col helper, that converts e.g. :user/first-name into :user$first_name): https://github.com/xtdb/xtdb/pull/3886/files#diff-1a8ccbf1b5d5b6dd1cb8a2c0200c29a783994c1c1e59b78d8aa82e0e6f3f6003R1856-R1876 I believe this means you can now roundtrip qualified keywords fine, but within SQL you will still always need to work with this $-normalized form

Very nice! And a lot less code than I expected 🙂 Still lots of stuff to work on in HoneySQL for XT's SQL -- and maybe a :xtdb dialect that would automatically use your new column handler function (dynamically, so it doesn't need a dependency directly, but would require users to add the XTDB JDBC module) (nope, dialects don't really work like that). I guess next.jdbc could add "xtdb" as a :dbtype and map it to whatever JDBC driver class you introduced to support jdbc:xtdb:... URLs?

Is there a way to start using the new feature now, or do I need to wait for a release ?

I don't think they've released new SNAPSHOTs with these features yet.

All good I manage to copy paste the new ns.

pushing 2.0.0-SNAPSHOT now 🚀

don't think there's any changes that affect the Docker images, so beta3 should still be fine, but it'll definitely be in nightly/edge

> whatever JDBC driver class you introduced to support jdbc:xtdb:... it's xtdb.jdbc.Driver, but should all work through the JDBC DriverManager service loader - I got it working with :jdbcUrl, at least 🙂

Right, but with :dbtype rather than a JDBC URL, next.jdbc constructs the URL and uses the class name for preloading the driver, as I recall. And it is a lot nicer to use a db-spec of {:dbtype "xtdb"} and have it fill in the host/port/etc 🙂

do you need to pre-load the driver much now with the service-loader - I thought all of that was supposed to make it Just Work™?

(admittedly I don't recall which version of Java that came in)

Dunno. It mattered for some drivers more than others, as I recall. It may still matter for some drivers... as an edge case... I prefer to assume at least some drivers are still badly-behaved...

I tried to restart my repl and nuke my .m2 but i still don't have the new ns xtdb.next.jdbc available while I'm on com.xtdb/xtdb-api {:mvn/version "2.0.0-SNAPSHOT"}

@avocabio have you got the Maven Central snapshots repo enabled?

{:mvn/repos {"ossrh-snapshots"
             {:url ""}}}

lein

:repositories [["ossrh-snapshots" ""]]

{:paths ["src" "resources"]
 :deps {org.postgresql/postgresql {:mvn/version "42.7.4"}
        com.xtdb/xtdb-api {:mvn/version "2.0.0-SNAPSHOT"}
        com.xtdb/xtdb-jdbc {:mvn/version "1.24.4"}}
 :mvn/repos {"ossrh-snapshots" {:url ""}}
 :aliases {:test {:extra-paths ["test"]
                  :extra-deps {}}
           :xtdb {:jvm-opts ["--add-opens=java.base/java.nio=ALL-UNNAMED"
                             "-Dio.netty.tryReflectionSetAccessible=true"]}}}

com.xtdb/xtdb-jdbc {:mvn/version "2.0.0-SNAPSHOT"} too

Thanks mate I used the latest I https://mvnrepository.com/artifact/com.xtdb/xtdb-jdbc but this bump fixed it !