xtdb 2022-10-10 | Slack Archive

m.q.warnock16:10:20

I could ask the libpython-clj folks, but they've already helped me a lot, and this might be anything from a general Clojure question to something XTDB specific, so I'll start here: tech.v3.tensor can be serialized with nippy, as long as the tech.v3.datatype.nippy namespace is loaded. This works calling freeze/thaw manually, and storing in XTDB in between, but my understanding is that XTDB uses nippy already, and that I should be able to get it to serialize/deserialize that type automatically, which would be lovely. Or maybe I'm (also?) looking for a general sort of middleware kind of thing, so that I can convert it (and move to the jvm heap) only if/when it's being written to the db, but copy it back to the native heap as it's deserialized.

m.q.warnock16:10:54

I did try requiring that namespace in a couple places in my code, including first thing in the main file, but I'm also not requiring xtdb directly, as I'm using biff.

malcolmsparks17:10:38

The datatype ns most likely contains extend-protocol for defining those nippy conversions, so as long as you require it, it should just work. Maybe I'm not understanding your question, so please feel free to expand.

m.q.warnock17:10:07

Require it anywhere? It doesn't work, so I figured it was trickier than that.

m.q.warnock17:10:57

(I get a nippy error I was getting before I ever required that namespace anywhere- will recreate it if it's helpful)

malcolmsparks17:10:35

Yes that would help. I'm trying to find some code snippet which will explain more, but you can use Clojure's extend-protocol with Nippy to provide custom serialisation logic. Unfortunately I have to dash off to an appt but will be back later.

m.q.warnock17:10:40

[qtp1660789756-62] ERROR com.biffweb.impl.middleware - Exception while handling request
xtdb.IllegalArgumentException: invalid tx-op: n-elems
        at xtdb.error$illegal_arg.invokeStatic(error.clj:12)
        at xtdb.error$illegal_arg.invoke(error.clj:3)
        at xtdb.tx.conform$conform_tx_op.invokeStatic(conform.clj:108)
        at xtdb.tx.conform$conform_tx_op.invoke(conform.clj:99)
        at xtdb.tx.conform$conform_tx_op.invokeStatic(conform.clj:100)
        at xtdb.tx.conform$conform_tx_op.invoke(conform.clj:99)
        at clojure.core$mapv$fn__8535.invoke(core.clj:6979)
        at clojure.lang.PersistentVector.reduce(PersistentVector.java:343)
        at clojure.core$reduce.invokeStatic(core.clj:6885)
        at clojure.core$mapv.invokeStatic(core.clj:6970)
        at clojure.core$mapv.invoke(core.clj:6970)
        at xtdb.node.XtdbNode.submit_tx_async(node.clj:215)
        at xtdb.node.XtdbNode.submit_tx_async(node.clj:206)
        at xtdb.node.XtdbNode.submit_tx(node.clj:223)
        at com.biffweb.impl.xtdb$submit_tx.invokeStatic(xtdb.clj:246)
        at com.biffweb.impl.xtdb$submit_tx.invoke(xtdb.clj:238)
        at com.biffweb$submit_tx.invokeStatic(biffweb.clj:532)
        at com.biffweb$submit_tx.invoke(biffweb.clj:526)
        at software.sacred.feat.importer$import_generative_art.invokeStatic(importer.clj:218)

m.q.warnock17:10:15

er- what the heck- sorry; that's not the stacktrace I was thinking of

m.q.warnock17:10:23

that's biff, though it is the lack of a manual freeze that causes it - I could have sworn I was seeing a nippy-related trace the last time I tried this particular combination. I'll investigate further before I bug you again

m.q.warnock17:10:55

oh; the "Caused by" threw me off - here it is starting there:

Caused by: java.lang.NoSuchFieldException: n-elems
        at java.base/java.lang.Class.getField(Class.java:2117)
        at juxt.clojars_mirrors.nippy.v3v1v1.taoensso.nippy$eval17826$fn__17827$fn__17828$fn__17829.invoke(nippy.clj:1209)
[main] INFO com.biffweb.impl.util - starting: com.biffweb$use_jetty@48aafd6f
51 1948 1922 1876 1920 1875 1853 1924 1862 1909 1790 1882 1912 1841 1882 1691 1848 1840 1886 1797 1684 1722 1787 1688 1680 1716 1641 1760 1
        at juxt.clojars_mirrors.nippy.v3v1v1.taoensso.nippy$eval17826$fn__17827$fn__17828.invoke(nippy.clj:1207)
        at clojure.lang.PersistentVector.reduce(PersistentVector.java:343)
        at clojure.core$reduce.invokeStatic(core.clj:6885)
        at clojure.core$reduce.invoke(core.clj:6868)
        at juxt.clojars_mirrors.nippy.v3v1v1.taoensso.nippy$eval17826$fn__17827.invoke(nippy.clj:1207)
        at juxt.clojars_mirrors.nippy.v3v1v1.taoensso.nippy$eval17510$fn__17511$G__17501__17518.invoke(nippy.clj:567)
        at juxt.clojars_mirrors.nippy.v3v1v1.taoensso.nippy$eval17826$fn__17827$fn__17828$fn__17829.invoke(nippy.clj:1211)
        at juxt.clojars_mirrors.nippy.v3v1v1.taoensso.nippy$eval17826$fn__17827$fn__17828.invoke(nippy.clj:1207)
        at clojure.lang.PersistentVector.reduce(PersistentVector.java:343)
        at clojure.core$reduce.invokeStatic(core.clj:6885)
        at clojure.core$reduce.invoke(core.clj:6868)
        at juxt.clojars_mirrors.nippy.v3v1v1.taoensso.nippy$eval17826$fn__17827.invoke(nippy.clj:1207)
        at juxt.clojars_mirrors.nippy.v3v1v1.taoensso.nippy$eval17510$fn__17511$G__17501__17518.invoke(nippy.clj:567)
        at juxt.clojars_mirrors.nippy.v3v1v1.taoensso.nippy$eval17557$fn__17558.invoke(nippy.clj:576)
        at juxt.clojars_mirrors.nippy.v3v1v1.taoensso.nippy$eval17538$fn__17539$G__17529__17546.invoke(nippy.clj:568)
        at juxt.clojars_mirrors.nippy.v3v1v1.taoensso.nippy$write_vec$fn__17603$fn__17604.invoke(nippy.clj:751)
        at juxt.clojars_mirrors.nippy.v3v1v1.taoensso.nippy$write_vec$fn__17603.invoke(nippy.clj:751)
        at clojure.lang.PersistentVector.reduce(PersistentVector.java:343)
        at clojure.core$reduce.invokeStatic(core.clj:6885)
        at clojure.core$reduce.invoke(core.clj:6868)
        at juxt.clojars_mirrors.nippy.v3v1v1.taoensso.nippy$write_vec.invokeStatic(nippy.clj:751)
        at juxt.clojars_mirrors.nippy.v3v1v1.taoensso.nippy$write_vec.invoke(nippy.clj:729)
        at juxt.clojars_mirrors.nippy.v3v1v1.taoensso.nippy$eval17798$fn__17799.invoke(nippy.clj:1170)
        at juxt.clojars_mirrors.nippy.v3v1v1.taoensso.nippy$eval17510$fn__17511$G__17501__17518.invoke(nippy.clj:567)
        at juxt.clojars_mirrors.nippy.v3v1v1.taoensso.nippy$eval17557$fn__17558.invoke(nippy.clj:576)
        at juxt.clojars_mirrors.nippy.v3v1v1.taoensso.nippy$eval17538$fn__17539$G__17529__17546.invoke(nippy.clj:568)
        at juxt.clojars_mirrors.nippy.v3v1v1.taoensso.nippy$write_kvs$fn__17608$fn__17609.invoke(nippy.clj:761)
        at juxt.clojars_mirrors.nippy.v3v1v1.taoensso.nippy$write_kvs$fn__17608.invoke(nippy.clj:758)
        at xtdb.codec$$reify__18738.kvreduce(codec.clj:148)
        at clojure.core$fn__8525.invokeStatic(core.clj:6908)
        at clojure.core$fn__8525.invoke(core.clj:6888)
        at clojure.core.protocols$fn__8257$G__8252__8266.invoke(protocols.clj:175)
        at clojure.core$reduce_kv.invokeStatic(core.clj:6919)
        at clojure.core$reduce_kv.invoke(core.clj:6910)
        at juxt.clojars_mirrors.nippy.v3v1v1.taoensso.nippy$write_kvs.invokeStatic(nippy.clj:758)
        at juxt.clojars_mirrors.nippy.v3v1v1.taoensso.nippy$write_kvs.invoke(nippy.clj:753)
        at clojure.lang.Var.invoke(Var.java:393)
        at xtdb.codec$freeze_map.invokeStatic(codec.clj:166)
        at xtdb.codec$freeze_map.invoke(codec.clj:160)
        at xtdb.codec$eval18784$fn__18785.invoke(codec.clj:188)
        at juxt.clojars_mirrors.nippy.v3v1v1.taoensso.nippy$eval17510$fn__17511$G__17501__17518.invoke(nippy.clj:567)
        at juxt.clojars_mirrors.nippy.v3v1v1.taoensso.nippy$eval17557$fn__17558.invoke(nippy.clj:576)
        at juxt.clojars_mirrors.nippy.v3v1v1.taoensso.nippy$eval17538$fn__17539$G__17529__17546.invoke(nippy.clj:568)
        at juxt.clojars_mirrors.nippy.v3v1v1.taoensso.nippy$write_kvs$fn__17608$fn__17609.invoke(nippy.clj:761)
        at juxt.clojars_mirrors.nippy.v3v1v1.taoensso.nippy$write_kvs$fn__17608.invoke(nippy.clj:758)
        at xtdb.codec$$reify__18738.kvreduce(codec.clj:148)
        at clojure.core$fn__8525.invokeStatic(core.clj:6908)
        at clojure.core$fn__8525.invoke(core.clj:6888)
        at clojure.core.protocols$fn__8257$G__8252__8266.invoke(protocols.clj:175)
        at clojure.core$reduce_kv.invokeStatic(core.clj:6919)
        at clojure.core$reduce_kv.invoke(core.clj:6910)
        at juxt.clojars_mirrors.nippy.v3v1v1.taoensso.nippy$write_kvs.invokeStatic(nippy.clj:758)
        at juxt.clojars_mirrors.nippy.v3v1v1.taoensso.nippy$write_kvs.invoke(nippy.clj:753)
        at clojure.lang.Var.invoke(Var.java:393)
        at xtdb.codec$freeze_map.invokeStatic(codec.clj:166)
        at xtdb.codec$freeze_map.invoke(codec.clj:160)
        at xtdb.codec$eval18784$fn__18785.invoke(codec.clj:188)
        at juxt.clojars_mirrors.nippy.v3v1v1.taoensso.nippy$eval17510$fn__17511$G__17501__17518.invoke(nippy.clj:567)
        at juxt.clojars_mirrors.nippy.v3v1v1.taoensso.nippy$eval17557$fn__17558.invoke(nippy.clj:576)
        at juxt.clojars_mirrors.nippy.v3v1v1.taoensso.nippy$eval17538$fn__17539$G__17529__17546.invoke(nippy.clj:568)
        at juxt.clojars_mirrors.nippy.v3v1v1.taoensso.nippy$fast_freeze$fn__17871.invoke(nippy.clj:1319)
        at juxt.clojars_mirrors.nippy.v3v1v1.taoensso.nippy$fast_freeze.invokeStatic(nippy.clj:1319)
        at juxt.clojars_mirrors.nippy.v3v1v1.taoensso.nippy$fast_freeze.invoke(nippy.clj:1306)
        at xtdb.codec$eval18951$fn__18952$fn__18953.invoke(codec.clj:614)
        at xtdb.codec$eval18951$fn__18952.invoke(codec.clj:613)
        at xtdb.codec$eval18682$fn__18683$G__18673__18690.invoke(codec.clj:76)
        at xtdb.codec$eval19041$fn__19042.invoke(codec.clj:682)
        at xtdb.codec$eval18645$fn__18657$G__18636__18662.invoke(codec.clj:72)
        at xtdb.codec$eval19041$fn__19044.invoke(codec.clj:685)
        at xtdb.codec$eval18645$fn__18646$G__18634__18651.invoke(codec.clj:72)
        at xtdb.codec$hash_doc.invokeStatic(codec.clj:402)
        at xtdb.codec$hash_doc.invoke(codec.clj:401)
        at xtdb.tx.conform$eval62952$fn__62954.invoke(conform.clj:41)
        at clojure.lang.MultiFn.invoke(MultiFn.java:234)
        at xtdb.tx.conform$conform_tx_op.invokeStatic(conform.clj:106)

refset02:10:03

I've never tried it before, but I think you could extend XT's nippy codec similar to the following examples: https://github.com/xtdb/xtdb/blob/1a361a485e64066f5ad7a174b1c2df8bcba31f1e/core/src/xtdb/codec.clj#L173-L188 and https://github.com/xtdb/xtdb/blob/1a361a485e64066f5ad7a174b1c2df8bcba31f1e/core/src/xtdb/codec.clj#L814-L823 However, you can also store the serialized byte-array, which is ultimately less brittle/intrusive, and would aid portability should you want to switch to something else (other than XT) later down the road

👍 1

2022-10-10

Channels