datalevin 2023-05-28 | Slack Archive

andersmurphy13:05:00

So I’m running into an error when inserting values that are :db.type/bigint I’ve managed to narrow it down to encode-bigint:

(encode-bigint (biginteger "-1"))
  ;; fv8
  (encode-bigint (biginteger "-34"))
  ;; ft4 
  (encode-bigint (biginteger "34"))
  ;; Value out of range for byte: 129
  (encode-bigint (biginteger "123456789123456789"))
  ;; Value out of range for byte: 136

  (encode-bigint (BigInteger. "-1"))
  ;; fv8
  (encode-bigint (BigInteger. "-34"))
  ;; ft4 
  (encode-bigint (BigInteger. "34"))
  ;; Value out of range for byte: 129
  (encode-bigint (BigInteger. "123456789123456789"))
  ;; Value out of range for byte: 136

So looks like the problem is with this function:

(defn encode-bigint
  [^BigInteger x]
  (let [bs (.toByteArray x)
        n  (alength bs)]
    (assert (< n 128)
            "Does not support integer beyond the range of [-2^1015, 2^1015-1]")
    (let [bs1 (byte-array (inc n))]
      (aset bs1 0 (byte (if (= (.signum x) -1)
                          (- 127 n)
                          (bit-flip n 7))))
      (System/arraycopy bs 0 bs1 1 n)
      bs1)))

Specifically on the branch when the number is positive.

(byte (bit-flip 7 n))
  
  (byte (bit-flip 7 (alength (.toByteArray (biginteger "34")))))
  ;; 129

  (byte 129)
  ;; Value out of range for byte: 129

As byte is between -128 and 127. Unfortunately, my understanding of bit encoding etc is completely non existent. So I have no idea what (bit-flip n 7) is supposed to do.

Huahai16:05:42

This is not reproducible. What’s your setup?

andersmurphy18:05:58

So tried this (direct copy from datalevin/bits.clj on github) in an empty repl session (no dependencies) Clojure 1.11.1:

(defn encode-bigint
  [^BigInteger x]
  (let [bs (.toByteArray x)
        n  (alength bs)]
    (assert (< n 128)
      "Does not support integer beyond the range of [-2^1015, 2^1015-1]")
    (let [bs1 (byte-array (inc n))]
      (aset bs1 0 (byte (if (= (.signum x) -1)
                          (- 127 n)
                          (bit-flip n 7))))
      (System/arraycopy bs 0 bs1 1 n)
      bs1)))

(encode-bigint (biginteger "34"))

;; Execution error (IllegalArgumentException) at user/encode-bigint (REPL:8).
;; Value out of range for byte: 129

Still get the error, so that means it’s most likely the fact that I’m running JDK17? I’ll give that a test.

Huahai20:05:10

jdk17 shouldn’t be a problem. I am on jdk17 too.

Huahai20:05:49

what you can do is to change byte to unchecked-byte, see if it fixes your problem

Huahai20:05:33

master branch now uses unchecked-byte, the next release will also do

andersmurphy21:05:31

Yes, changing it to unckecked-byte fixes it :thinking_face:🙏

andersmurphy14:05:59

(defn encode-bigint
    [^BigInteger x]
    (let [bs (.toByteArray x)
          n  (alength bs)]
      (assert (< n 128)
        "Does not support integer beyond the range of [-2^1015, 2^1015-1]")
      (let [bs1 (byte-array (inc n))]
        (aset bs1 0 (byte (if (= (.signum x) -1)
                            (- 127 n)
                            n)))
        (System/arraycopy bs 0 bs1 1 n)
        bs1)))

  (defn decode-bigint [bs]
    (BigInteger. ^bytes (Arrays/copyOfRange bs 1 (alength bs))))

(decode-bigint (encode-bigint (biginteger "343")))
  ;; "343"
(decode-bigint (encode-bigint (biginteger "-343")))
  ;; "-343"

So replacing (bit-flip n 7) with n seems to make it roundtrip. But haven’t tested this with the actual db and as mentioned before don’t actually know enough about any of this to know what bit-flip was/is doing.

joshcho19:05:37

Datalevin has worked well for me. Transaction functions work well, and composing them is a pleasure. One thing I noticed with transaction functions is that I cannot use any external libraries within them (I believe this is also how it is in datomic). Is this something that is an inherent property of the datalevin tx-fn implementation, or something that can be amended?

Huahai16:05:29

Have you tried with fully qualified symbols for the external functions?

joshcho22:05:42

Yeah, here is a simple test case:

(ns slix.node)

(defn testing-fn []
  "test-value")

(di/definterfn test-tx1 [db tempid]
  [{:db/id tempid :node/value (slix.node/testing-fn)}])

(di/definterfn test-tx2 [db tempid]
  [{:db/id tempid :node/value "test-value"}])

(d/transact! test-conn [{:db/ident :add-node1
              :db/fn test-tx1}])

(d/transact! test-conn [{:db/ident :add-node2
              :db/fn test-tx2}])

:add-node1 fails, but :add-node2 succeeds.

Huahai22:05:13

If you are using datalevin embedded, you don’t need to use inter-fn

joshcho22:05:00

I am getting "Can only freeze an inter-fn" when not defining test-tx2 as definterfn, just simply:

(defn testing-fn []
  "test-value")

joshcho22:05:21

How do I know if I am using embedded? I think I am because I haven't separated out the db from the app, but I am getting the "Can only freeze an inter-fn" error.

Huahai22:05:00

you can try :db.fn/call

👍 2

Huahai22:05:43

if you don’t need to persist the function in db

joshcho22:05:24

Hmm. I will try that. I have transaction functions call other transaction functions, so I might need persistence. For instance, :db.fn/call might call an external function within a transaction function definition, which won't work (given what I have seen).

Huahai22:05:20

:db.fn/call just call the function, without persistence, it does not involve serialization of the function

👍 2

Huahai22:05:31

serialized code is handled with sci, so it doesn’t have access to clojure vars, but can probably be made so

Huahai23:05:56

(defn testing-fn []  "test-value")

(deftest test-fn
  (let [dir                (u/tmp-dir (str "test-fn-" (UUID/randomUUID)))
        conn               (d/create-conn dir)
        test-tx            (fn [_ tempid]
                             [{:db/id tempid :node/value (testing-fn)}])
        {:keys [db-after]} (d/transact! conn [[:db.fn/call test-tx -1]])
        e                  (d/entity db-after 1)]
    (is (= (:node/value e) "test-value"))
    (d/close conn)
    (u/delete-files dir)))

❤️ 2

Huahai23:05:11

this test passses

joshcho23:05:45

Working for me as well, thank you!

Huahai23:05:29

You are welcome. We can probably expose a facility to allow exposing clojure vars to inter-fn, I will file an issue

👍 2

2023-05-28

Channels