Clojurians Log v2

@taylor.jeremydavid Numeric up to 256 bit (unsigned) is critical to our needs. How long could it take to get this in v2? This quarter? this year?

Hi @jon151 just a quick message to say that we've now implemented DECIMAL support, see https://discuss.xtdb.com/t/numeric-precision-of-xt2/540/3

Hi @jon151 our priorities can always flex - right now this wouldn't be on our critical path ahead of GA ...but it could be. I'll DM to discuss options

Hey @me1214 'bigint' (which is big from the legacy SQL perspective 🙂) uses Apache Arrow's 64-bit int representation under the hood. Neither Arrow nor v2 currently handles anything larger natively, although we could add an extension type in future. For now it would probably be best to store such token amounts using VARBINARY

Thanks. I just checked arrow docs. Looks like it supports unsigned 64 bit as well. Can we have access to that?

Arrow also lists a decimal256 type. https://arrow.apache.org/docs/cpp/api/datatype.html

We have support for NUMERIC on our backlog, which I think would be sufficient (🤔) - also discussed recently on a recent forum post: https://discuss.xtdb.com/t/numeric-precision-of-xt2/540

@stefcoetzee think I've https://github.com/xtdb/xtdb/issues/4018#issuecomment-2593043778 - when you get a mo, would you mind confirming it works your end?

Thanks for taking a look at this @jarohen. Nuked .m2, followed by clj -P for a clean repo with the following deps.edn:

{:deps  {org.clojure/clojure                     {:mvn/version "1.12.0"}
         ch.qos.logback/logback-classic          {:mvn/version "1.5.15"}
         #_#_org.jetbrains.kotlin/kotlin-reflect     {:mvn/version "2.1.0"}
         #_#_org.jetbrains.kotlin/kotlin-stdlib-jdk8 {:mvn/version "2.1.0"}
         com.xtdb/xtdb-core                      {:mvn/version "2.0.0-beta5"}
         com.xtdb/xtdb-http-server               {:mvn/version "2.0.0-beta5"}}

 :aliases {:xtdb {:jvm-opts ["--add-opens=java.base/java.nio=ALL-UNNAMED"
                             "-Dio.netty.tryReflectionSetAccessible=true"]}}}

I.e. still run into the following at the end:

Error building classpath. Could not find artifact org.jetbrains.kotlin:kotlin-reflect:jar:unspecified in central ()

yep, we won't be able to fix it on beta5 I'm afraid, it's an immutable release - could you bump the XT deps to 2.0.0-SNAPSHOT (including the snapshot repo if need be)?

Ah, of course!facepalm My mistake. Confirmed to be working with the following deps.edn:

{:deps  {org.clojure/clojure                     {:mvn/version "1.12.0"}
         ch.qos.logback/logback-classic          {:mvn/version "1.5.15"}
         #_#_org.jetbrains.kotlin/kotlin-reflect     {:mvn/version "2.1.0"}
         #_#_org.jetbrains.kotlin/kotlin-stdlib-jdk8 {:mvn/version "2.1.0"}
         com.xtdb/xtdb-core                      {:mvn/version "2.0.0-SNAPSHOT"}
         com.xtdb/xtdb-http-server               {:mvn/version "2.0.0-SNAPSHOT"}}

 :mvn/repos {"ossrh-snapshots" {:url ""}}

 :aliases {:xtdb {:jvm-opts ["--add-opens=java.base/java.nio=ALL-UNNAMED"
                             "-Dio.netty.tryReflectionSetAccessible=true"]}}}

Thanks for the release, @jarohen & @taylor.jeremydavid! When updating a server repo I have running locally (not in Docker) to beta5, I had to add two dependencies:

{:deps  {org.clojure/clojure                     {:mvn/version "1.12.0"}
         ch.qos.logback/logback-classic          {:mvn/version "1.5.15"}
         org.jetbrains.kotlin/kotlin-reflect     {:mvn/version "2.1.0"} ; added for beta5
         org.jetbrains.kotlin/kotlin-stdlib-jdk8 {:mvn/version "2.1.0"} ; added for beta5
         com.xtdb/xtdb-core                      {:mvn/version "2.0.0-beta5"}
         com.xtdb/xtdb-http-server               {:mvn/version "2.0.0-beta5"}}

 :aliases {:xtdb {:jvm-opts ["--add-opens=java.base/java.nio=ALL-UNNAMED"
                             "-Dio.netty.tryReflectionSetAccessible=true"]}}}

Error building classpath. Could not find artifact org.jetbrains.kotlin:kotlin-reflect:jar:unspecified in central ()

Error building classpath. Could not find artifact org.jetbrains.kotlin:kotlin-stdlib-jdk8:jar:unspecified in central ()

Hey @stefcoetzee 👋 that's strange, I don't recall having to add that ourselves, I wouldn't have thought anything would work at least without stdlib, and I'm not aware we depend on reflect 🤔 thanks for letting us know - I'm OoO today but will get the team to look into it 🙏 https://github.com/xtdb/xtdb/issues/4018

-- standard SQL, calculate frequency distribution using sub-queries
SELECT order_count, COUNT(*) AS freq
FROM (SELECT customer, COUNT(*) AS order_count
      FROM orders
      GROUP BY customer) counts
GROUP BY order_count
ORDER BY order_count DESC

-- XT SQL
FROM orders
SELECT customer, COUNT(*) AS order_count
SELECT order_count, COUNT(*) AS freq
ORDER BY order_count DESC

-- filtering by the result of a window function
SELECT _id
FROM (SELECT _id, ROW_NUMBER () OVER (PARTITION BY v ORDER BY _id) row_num
      FROM foo) subq
WHERE row_num = 0

-- XT
FROM foo
SELECT _id, ROW_NUMBER () OVER (PARTITION BY v ORDER BY _id) row_num
WHERE row_num = 0 -- N.B. WHERE applied after SELECT
SELECT _id

to reiterate, standard SQL still works 🙂 see the https://docs.xtdb.com/reference/main/sql/queries.html#_query_term for more details

on a vaguely related note: are navigation window functions supported?

@markaddleman not just yet I'm afraid, but definitely on the backlog - any you particularly use?

all of them 🙂 : first_value, last_value, lead and lag. there’s no rush on my part - i’m just curious. i don’t think xtdb can be realistic for my use case until there’s something like tx-listen so i can compute indices in real time.

> i don’t think xtdb can be realistic for my use case until there’s something like tx-listen so i can compute indices in real time. interested in this one too - indeed, only this morning we were chatting to someone about incremental view maintenance. obviously a much bigger project 🙂

I'll be running through the above changes in an Office Hours session on Thursday at 1600 UTC - for anyone who's curious and wants to say hello (or ask more questions about our numeric type support!) https://discuss.xtdb.com/t/office-hours-thursday-16th-at-1600-utc/551

So, when you’re ready to chat IVM, hit me up! I have a wish list 🙂