Fork me on GitHub
#xtdb
<
2023-12-19
>
refset11:12:56

Hi folks, just a quick reminder that James and I are running this presentation on v2 and live REPL session in a few hours' time (in case you were interested to join previously and now find yourself available!) :) https://clojurians.slack.com/archives/CG3AM2F7V/p1702647335864769

👍 1
Marius14:12:57

Hey there, I’m running the docker image from

and when trying to make a simple put like
(with-open [node (xtc/start-client "")]
              (xt/submit-tx node
                            (xt/put :companies {:company/name "Test" :xt/id 1})))
I get error Malformed request. The server log states
14:30:08 | WARN  xtdb.server | response error
clojure.lang.ExceptionInfo: Request coercion failed: #reitit.coercion.CoercionError{:spec #Spec{:form (clojure.spec.alpha/keys :req-un [:xtdb.server/tx-ops] :opt-un [:xtdb.server/opts]), :type :map, :leaf? false}, :problems {:clojure.spec.alpha/problems ({:path [:tx-ops], :pred clojure.core/vector?, :val #xt.tx/put {:table-name :companies, :doc {:company/name "Test", :xt/id 1}, :valid-from nil, :valid-to nil}, :via [:xtdb.server/tx-ops], :in [:tx-ops]}), :clojure.spec.alpha/spec #Spec{:form (clojure.spec.alpha/keys :req-un [:xtdb.server/tx-ops] :opt-un [:xtdb.server/opts]), :type :map, :leaf? false}, :clojure.spec.alpha/value {:opts {}, :tx-ops #xt.tx/put {:table-name :companies, :doc {:company/name "Test", :xt/id 1}, :valid-from nil, :valid-to nil}}}}
What did I do wrong?

refset14:12:30

Hey @U02G3DBJ2SY I think the (xt/put ...) needs wrapping in a [ ] (indicated by :pred clojure.core/vector? in that exception)

Marius15:12:32

Ah ok, thank you! I failed at understanding the spec problem.

🙏 1
refset15:12:57

well, it's not the easiest thing to read 😅

refset15:12:23

we could probably try adding https://github.com/bhb/expound or similar into the mix - would anyone here recommend that lib? or is there other good prior art we should look at?

Marius15:12:20

Assuming the audience of XTDB are not only Clojure but also Java dev’s, then I think more human-readable messages are better.

💯 1
jarohen15:12:31

Yep, admittedly, this is very much an early error message 😳 We're currently working on our own hand-rolled HTTP req parsers for XTQL, which will have more knowledge of the language, and hence more ability to give better error messages 🙂

Marius15:12:51

Another noob question if you don’t mind:

(with-open [node (xtc/start-client "")]
  (xt/submit-tx node
                [(xt/put :companies {:company/name "Test" :xt/id 1})])
  (xt/q node '(from :companies [:company/name])))
returns
[{}]
but I would expect
[{:company/name "Test}]
Any idea why the doc is empty?

jarohen15:12:28

try '(from :companies [company/name]) (i.e. without the : on company/name)?

jarohen15:12:34

(we need a better error message for this one, too)

Marius15:12:20

Oooh… yeah, that nasty colon! Thank you. I totally expected that to be a keyword.

refset18:12:03

This bit me the other day also 🙂 if you want to track the resolution you can subscribe to https://github.com/xtdb/xtdb/issues/3041

mmer15:12:43

With the XTQL, I am slightly puzzled by the threaded forms using the from function. If you build up a set of functions that in effect make up the parts of the query, does this rely on a lazy model to know what to pass down the threaded functions. It looks rather strange when compared with a sub query as in that case the sub query probably fires first and then the outer query, but in the threaded forms I have seen it does not seem to apply that way, or am I missing something?

refset18:12:56

Hey @U4C3ZU6KX sorry this question got missed in the excitement yesterday, I just answered it here https://discuss.xtdb.com/t/is-the-xtql-pipeline-operator-lazy/312 🙂

mmer09:12:35

Thanks.

👌 1
jarohen15:12:52

Just doing final sound checks, and then we'll be kicking off - see you soon!

🚀 2
p14n09:12:45

Thanks for the session, really enjoyed it. I'm particularly interested in the change(?) in storage backed to s3 as doc store - is there anything like ADRs that explain the move?

jarohen10:12:58

Nothing public, I'm afraid - IIRC it was part of the original idea for v2 even before we started working on it. If I were to write a retroactive ADR, the decision would fall out of the decision to go to columnar data, and then Apache Arrow - small files don't make as much sense in this world, so blob stores were a more natural home for this data (not to mention significantly cheaper)

👍 1
Marius17:12:46

A document containing a map with string key

(with-open [node (xtc/start-client "")]
    (xt/submit-tx node
                  [(xt/put :companies {:company/name {:foo {"123" :bar}} :xt/id 1})])
    (xt/q node '(from :companies [company/name])))
causes
15:44:24 | WARN  xtdb.server | response error
java.lang.UnsupportedOperationException: Arrow maps currently not supported
	at xtdb.vector.writer$eval11557$fn__11558.invoke(writer.clj:716)
	at xtdb.vector.writer$eval11002$fn__11027$G__10991__11032.invoke(writer.clj:36)
	at xtdb.vector.writer$eval11557$fn__11562.invoke(writer.clj:741)
	at xtdb.vector.writer$eval11002$fn__11014$G__10993__11021.invoke(writer.clj:36)
	at xtdb.vector.writer$eval11557$fn__11562.invoke(writer.clj:742)
Is this something on the short-term or rather long-term roadmap?

seancorfield17:12:00

ISTR in one of their talks they said they're thinking about how to support this (so I get the idea it's on the roadmap but non-trivial).

FiVo17:12:11

Depends a bit of what you want to do.

FiVo17:12:39

Arrow (our underlying storage) explicitly differentiates between 'structs' and 'mapsThis is the important bit. We are currently supporting the "structs". I suspect we allow for strings pretty soon so that the Java folks can also submit documents. Just be aware that then there is no way to differentiate between keywords and strings (in key position) on the way out (to Arrow it's the same).

FiVo17:12:48

If you need that differentiation than you need to wait for the Arrow maps support which will likely be a bit further down the road.

Marius19:12:15

Ok, thanks for the detailed explanation! In that case I think I can change my data model use keywords instead.

👍 1