datomic 2022-04-12 | Slack Archive

dazld07:04:06

I had a bit of a weird one while developing an on-prem transactor function - perhaps it’s my lack of experience with them, but it was surprising. The function has a set as part of an argument map, and all worked fine on a forked local connection, or on an in memory db. However, when deployed to the transactor we started getting odd errors about trying to treat a set as a string. It was a classic error (below), but we couldn’t spot the error in our logic, and the tests were green.

Execution error (ClassCastException) at dev/eval118048$fn (form-init6861183615777452247.clj:1).
class java.util.HashSet cannot be cast to class java.lang.CharSequence (java.util.HashSet and java.lang.CharSequence are in module java.base of loader 'bootstrap')

The missing piece was figuring out that the types changed (perhaps due to de/serialization), such that coll? for example, returned false on something that originally had been a clojure.lang.PersistentHashSet - it had become a java.util.HashSet . Adding a section to the tx function docs explaining what happens to data provided to the function would have helped. It was quite some debugging to figure it out, although the d/cancel api helped to bisect exactly which bit of code was blowing up.

favila12:04:35

PersistentVector can also become arraylist

👍 2

favila12:04:26

And this is indeed because of fressian serialization. It doesn’t have to be this way but these are the handlers they chose (maybe for performance)

Ivar Refsdal14:04:20

I've also been bitten by this

Ivar Refsdal14:04:50

I'm always processing arguments like this when writing transaction functions:

...
  (:import (java.util HashSet List)
...

(defn to-clojure-types [m]
  (walk/prewalk
    (fn [e]
      (cond (instance? HashSet e)
            (into #{} e)

            (and (instance? List e) (not (vector? e)))
            (vec e)

            :else e))
    m))

👌 1

favila14:04:07

It’s surprising but I’ve rarely found that it matters (which is why it’s annoying when it bites!)

favila14:04:41

e.g. queries return HashSet/ArrayList at the top.

favila14:04:13

clojure interop with j.u.Collections is really good

dazld17:04:47

in theory.. yes, but at least coll? returns false for java.util.HashSet

dazld17:04:16

map etc work fine, so perhaps it’s just an oversight? not sure.

dazld17:04:49

I guess the big thing was that it works differently on the transactor, compared to in memory, which was really surprising.. hard to write tests that cover this without quite some gymnastics..

Ivar Refsdal07:04:05

Running a full fledged container with datomic would catch this, right? I should look into clj-test-containers: https://github.com/javahippie/clj-test-containers

Ivar Refsdal10:05:42

A month has passed already... I "solved" this problem in the following way using https://github.com/clojure/data.fressian:

(ns com.github.ivarref.add-fressian
  (:require [clojure.data.fressian :as fress]
            [datomic.api :as d]))

(defn transact [org-transact]
  (fn [conn tx-data]
    (org-transact conn (fress/read (fress/write tx-data)))))

(defn with-fressian [f]
  (with-redefs [d/transact (transact d/transact)]
    (f)))

and then inside my tests I have something like:

(test/use-fixtures :each add-fressian/with-fressian)

While this works the problem is still how Datomic works (and differs in networked database vs. local)

frankitox17:07:02

Thanks for pointing this out @U3ZUC5M0R! You saved me a lot of frustrating debugging time 🙏

👍 2

🙌 2

Ivar Refsdal19:07:52

PS: You can test remote transactor funtcions' behavior if you want using https://github.com/sikt-no/datomic-testcontainers/. There is also https://github.com/ivarref/gen-fn for writing datomic functions that automatically de-mangles fressian types into regular Clojure ones. I've documented the differences https://github.com/ivarref/gen-fn/blob/main/test/com/github/ivarref/remote_vs_local_test.clj#L79-L93, as part of gen-fn. Disclaimer (?): I wrote both these libraries.

🚀 2

Maciej Szajna12:04:33

Hi guys! A question about @(d/transact conn ..) and subsequent (d/db conn) : is the d/db guaranteed to see the effects of the transaction (T value equal or greater that the transaction?) I am aware d/transact returns the db value immediately after transaction completion, and I use it 99% of the time, but in this particular instance it's particularly inconvenient. The question is really this: is it possible at all, through the JVM instruction reordering or any other kind of magic, that (do @(d/transact conn ..) (d/db conn)) might return a db value representing state before the transaction? Edit: it's been answered before https://stackoverflow.com/questions/47693495/datomic-on-a-peer-does-connection-db-read-your-writes-after-connection-trans

Maciej Szajna12:04:52

Oh, it's been answered before https://stackoverflow.com/questions/47693495/datomic-on-a-peer-does-connection-db-read-your-writes-after-connection-trans The answer is: yes, the guarantee is there

nottmey17:04:34

Would anyone be interested in a library/example about this? 😄

respatialized18:04:15

https://yyhh.org/blog/2021/11/t-wand-beat-lucene-in-less-than-600-lines-of-code/ This isn't Datomic, but it is an example of adding full text search to a different Datalog based DB.

nottmey18:04:07

oh nice, thanks for the tip

nottmey18:04:13

this actually sounds really promising 😳

2022-04-12

Channels