Clojurians Log v2

I could easily query each referencing doc type individually but would like to know if there is a way to do this within a single query?

It’s been a while since I’ve written a v1 query, but my instinct is something like (or [e :settings/root-id id] [e :api-key/root-id id] [e :some-other/root-id id])

Ah, indeed. Optional matches over same logic variable.

Thanks, that works for my simple needs!

hey @seancorfield 👋 I've tried running these against 8.4, I'm getting one failure: database-metadata-datafy-tests - XT is missing :SQLKeywords - but can no longer see the error above - could you give that another shot?

next.jdbc.datafy-test
  database-metadata-datafy-tests

database metadata datafication
Expected: (= reference-keys (set/intersection reference-keys data))
Actual: false
Evaluated arguments: #{...}
Only in first argument:
#{:SQLKeywords}
Only in second argument:
nil
The same in both:
#{... everything else ...}

in next/jdbc/datafy_test.clj:115

@jarohen will update next.jdbc to beta8.4 (api) and try again. Why has SQLKeywords disappeared from the metadata between beta8.1 and 8.3?

I don't think I've come across SQLKeywords tbh 🙂 any idea where that would have been before it disappeared?

think we bumped pgjdbc in that time, so maybe Postgres has removed it...?

but that was only a single minor version bump

It would be a getSQLKeywords() method on the DatabaseMetadata - and it used to either return nil or something, but now throws an exception?

we did put a delegate in for the PgConnection, possibly in that time window - but I didn't come across SQLKeywords at the time, so I'd guess it was one of the ones that was delegated by default

aha, so if we delegate through to the pgjdbc connection impl that then makes a query of the backend - I wonder whether that query regressed

https://docs.oracle.com/en/java/javase/21/docs/api/java.sql/java/sql/DatabaseMetaData.html#getSQLKeywords()

here's pgjdbc's impl: https://github.com/pgjdbc/pgjdbc/blob/master/pgjdbc/src/main/java/org/postgresql/jdbc/PgDatabaseMetaData.java#L288

I'm seeing an NPE:

xtdb-1       | Caused by: java.lang.NullPointerException: Cannot invoke "java.lang.Character.charValue()" because "x" is null
xtdb-1       |  at clojure.lang.RT.intCast(RT.java:1241)
xtdb-1       |  at $fn__27079.invokeStatic(io.clj:227)
xtdb-1       |  at $fn__27079.invoke(io.clj:224)
xtdb-1       |  at $io_record$write_record__27103.invoke(io.clj:296)
xtdb-1       |  at $io_list$fn__27086$fn__27087.invoke(io.clj:245)
xtdb-1       |  at clojure.core$run_BANG_$fn__8922.invoke(core.clj:7907)
xtdb-1       |  at clojure.lang.PersistentVector.reduce(PersistentVector.java:418)
xtdb-1       |  at clojure.core$reduce.invokeStatic(core.clj:6964)
xtdb-1       |  at clojure.core$run_BANG_.invokeStatic(core.clj:7902)
xtdb-1       |  at clojure.core$run_BANG_.invoke(core.clj:7902)
xtdb-1       |  at $io_list$fn__27086.invoke(io.clj:245)
xtdb-1       |  at $io_record$write_record__27103.invoke(io.clj:296)
xtdb-1       |  at xtdb.pgwire.io.SocketFrontend.send_client_msg_BANG_(io.clj:131)
xtdb-1       |  at $cmd_write_msg.invokeStatic(io.clj:79)
xtdb-1       |  at $cmd_write_msg.invoke(io.clj:73)
xtdb-1       |  at xtdb.pgwire$cmd_send_row_description.invokeStatic(pgwire.clj:520)
xtdb-1       |  at xtdb.pgwire$cmd_send_row_description.invoke(pgwire.clj:501)
xtdb-1       |  at xtdb.pgwire$cmd_describe_canned_response.invokeStatic(pgwire.clj:539)
xtdb-1       |  at xtdb.pgwire$cmd_describe_canned_response.invoke(pgwire.clj:537)
xtdb-1       |  at xtdb.pgwire$eval27726$fn__27729.invoke(pgwire.clj:558)
xtdb-1       |  at clojure.lang.MultiFn.invoke(MultiFn.java:234)
xtdb-1       |  at xtdb.pgwire$handle_msg.invokeStatic(pgwire.clj:1392)

this on 8.4 still?

Yes. I just updated to 8.4 so I can repro what you're seeing.

ah, gotcha, repro'd here too

(t/deftest test-database-metadata
  (with-open [conn (jdbc-conn)]
    (.getSQLKeywords (.getMetaData conn))))

right, let's see what's going on...

Detail: ["~#xt/error",["~:fault","~:xtdb.error/unknown",null,["^ "]]]
   :at [org.postgresql.core.v3.QueryExecutorImpl receiveErrorResponse QueryExecutorImpl.java 2733]}]
 :trace
 [[org.postgresql.core.v3.QueryExecutorImpl receiveErrorResponse QueryExecutorImpl.java 2733]
  [org.postgresql.core.v3.QueryExecutorImpl processResults QueryExecutorImpl.java 2420]
  [org.postgresql.core.v3.QueryExecutorImpl execute QueryExecutorImpl.java 372]
  [org.postgresql.jdbc.PgStatement executeInternal PgStatement.java 517]
  [org.postgresql.jdbc.PgStatement execute PgStatement.java 434]
  [org.postgresql.jdbc.PgStatement executeWithFlags PgStatement.java 356]
  [org.postgresql.jdbc.PgStatement executeCachedSql PgStatement.java 341]
  [org.postgresql.jdbc.PgStatement executeWithFlags PgStatement.java 317]
  [org.postgresql.jdbc.PgStatement executeQuery PgStatement.java 266]
  [org.postgresql.jdbc.PgDatabaseMetaData getSQLKeywords PgDatabaseMetaData.java 354]
  [jdk.internal.reflect.DirectMethodHandleAccessor invoke DirectMethodHandleAccessor.java 104]

I suspect that's not terribly helpful 🙂

cool, think I have a fix - try docker tag edge?

Just a sec...

f96046d?

(and I assume this bug is also in the newly-released beta9?)

Yup, that passes locally! getSQLKeywords() no longer throws.

sweet 🙌 and yep, still affects beta9

it'll be in the dated nightly as of ~2200Z tonight - nightly-20250604

I'll wait for beta10 -- I've been burned by relying on latest etc in CI for next.jdbc 😛

heh, yeah, understandable 🙂

(although that dated nightly is immutable)

OK, next.jdbc CI updated to depend on that nightly for now. It all passes.

Test failure:

Testing against xtdb
      √ Anonymous test case

next.jdbc.datafy-test
  database-metadata-datafy-tests
    Anonymous test case:

database metadata datafication
Expected: (= reference-keys (set/intersection reference-keys data))

Only in first argument:
#{:SQLKeywords}
Only in second argument:
nil

xtdb:
    image: 
    pull_policy: always
    ports:
      - "5432:5432"

Obvs, I could change the test to allow :SQLKeywords to be optional but that seems... hacky...

That test, for all databases, starts here: https://github.com/seancorfield/next-jdbc/blob/develop/test/next/jdbc/datafy_test.clj#L49

The test output shows that calling the getter for SQLKeywords throws an exception in CI (only):

xtdb :db-meta    (:maxLogicalLobSize :SQLKeywords/exception)

Here's more information about the exception:

SQLKeywords exception: #error {
 :cause ERROR: Cannot invoke "java.lang.Character.charValue()" because "x" is null
  Detail: ["~#xt/error",["~:fault","~:xtdb.error/unknown","Cannot invoke \"java.lang.Character.charValue()\" because \"x\" is null",["^ "]]]
 :via
 [{:type java.lang.reflect.InvocationTargetException
   :message nil
   :at [jdk.internal.reflect.DirectMethodHandleAccessor invoke DirectMethodHandleAccessor.java 118]}
  {:type org.postgresql.util.PSQLException
   :message ERROR: Cannot invoke "java.lang.Character.charValue()" because "x" is null
  Detail: ["~#xt/error",["~:fault","~:xtdb.error/unknown","Cannot invoke \"java.lang.Character.charValue()\" because \"x\" is null",["^ "]]]
   :at [org.postgresql.core.v3.QueryExecutorImpl receiveErrorResponse QueryExecutorImpl.java 2733]}]
 :trace
 [[org.postgresql.core.v3.QueryExecutorImpl receiveErrorResponse QueryExecutorImpl.java 2733]
  [org.postgresql.core.v3.QueryExecutorImpl processResults QueryExecutorImpl.java 2420]
  [org.postgresql.core.v3.QueryExecutorImpl execute QueryExecutorImpl.java 372]
  [org.postgresql.jdbc.PgStatement executeInternal PgStatement.java 517]
  [org.postgresql.jdbc.PgStatement execute PgStatement.java 434]
  [org.postgresql.jdbc.PgStatement executeWithFlags PgStatement.java 356]
  [org.postgresql.jdbc.PgStatement executeCachedSql PgStatement.java 341]
  [org.postgresql.jdbc.PgStatement executeWithFlags PgStatement.java 317]
  [org.postgresql.jdbc.PgStatement executeQuery PgStatement.java 266]
  [org.postgresql.jdbc.PgDatabaseMetaData getSQLKeywords PgDatabaseMetaData.java 354]
  [jdk.internal.reflect.DirectMethodHandleAccessor invoke DirectMethodHandleAccessor.java 103]
  [java.lang.reflect.Method invoke Method.java 580]
  [clojure.java.data$make_shallow_getter_fn$fn__6772 invoke data.clj 105]

that may be an 8.3 error 🤔

I tried xtdb-api 2.0.0-beta8.3 since I saw there was a core release but that api version is not on Maven.

right, let's get that up onto Maven 🙂

the stacktrace there looks like an 8.3 stacktrace

(and why does it only fail in CI, not locally?)

pass. potential for a version difference?

could you run with XTDB_LOGGING_LEVEL_PGWIRE=debug on CI? that'll give us the server-side stacktrace rather than the client-side one

just pushing that up to CI

Hmm, that seemed to produce the same stacktrace:

database-metadata-datafy-tests
  xtdb :db-meta    (:maxLogicalLobSize :SQLKeywords/exception)
SQLKeywords exception: #error {
 :cause ERROR: Cannot invoke "java.lang.Character.charValue()" because "x" is null
  Detail: ["~#xt/error",["~:fault","~:xtdb.error/unknown","Cannot invoke \"java.lang.Character.charValue()\" because \"x\" is null",["^ "]]]
 :via
 [{:type java.lang.reflect.InvocationTargetException
   :message nil
   :at [jdk.internal.reflect.DirectMethodHandleAccessor invoke DirectMethodHandleAccessor.java 118]}
  {:type org.postgresql.util.PSQLException
   :message ERROR: Cannot invoke "java.lang.Character.charValue()" because "x" is null
  Detail: ["~#xt/error",["~:fault","~:xtdb.error/unknown","Cannot invoke \"java.lang.Character.charValue()\" because \"x\" is null",["^ "]]]
   :at [org.postgresql.core.v3.QueryExecutorImpl receiveErrorResponse QueryExecutorImpl.java 2733]}]
 :trace
 [[org.postgresql.core.v3.QueryExecutorImpl receiveErrorResponse QueryExecutorImpl.java 2733]
  [org.postgresql.core.v3.QueryExecutorImpl processResults QueryExecutorImpl.java 2420]
  [org.postgresql.core.v3.QueryExecutorImpl execute QueryExecutorImpl.java 372]
  [org.postgresql.jdbc.PgStatement executeInternal PgStatement.java 517]
  [org.postgresql.jdbc.PgStatement execute PgStatement.java 434]
  [org.postgresql.jdbc.PgStatement executeWithFlags PgStatement.java 356]
  [org.postgresql.jdbc.PgStatement executeCachedSql PgStatement.java 341]
  [org.postgresql.jdbc.PgStatement executeWithFlags PgStatement.java 317]
  [org.postgresql.jdbc.PgStatement executeQuery PgStatement.java 266]
  [org.postgresql.jdbc.PgDatabaseMetaData getSQLKeywords PgDatabaseMetaData.java 354]
  [jdk.internal.reflect.DirectMethodHandleAccessor invoke DirectMethodHandleAccessor.java 103]
  [java.lang.reflect.Method invoke Method.java 580]
  [clojure.java.data$make_shallow_getter_fn$fn__6772 invoke data.clj 105]

Here's the CI command details:

clojure -T:build ci :snapshot true
  shell: /usr/bin/bash -e {0}
  env:
    JAVA_HOME: /opt/hostedtoolcache/Java_Temurin-Hotspot_jdk/21.0.7-6/x64
    JAVA_HOME_21_X64: /opt/hostedtoolcache/Java_Temurin-Hotspot_jdk/21.0.7-6/x64
    CLOJURE_INSTALL_DIR: /tmp/usr/local/opt/ClojureTools
    MYSQL_ROOT_PASSWORD: testing
    NEXT_JDBC_TEST_MYSQL: yes
    NEXT_JDBC_TEST_XTDB: yes
    XTDB_LOGGING_LEVEL_PGWIRE: debug
    NEXT_JDBC_TEST_MSSQL: yes
    MSSQL_SA_PASSWORD: Str0ngP4ssw0rd

I pinned the Docker image to 2.0.0-beta8.1 (to match the xtdb-api version) and it all passed on CI, so I'll make an effort to keep them in lock step in future.