Clojurians Log v2

I can run 0.9.20, but not 0.9.21

There are native dependencies that we expect. Unfortunately, we have to use whatever CI/CD build server uses.

does setting the mac osx deployment target not work? https://stackoverflow.com/questions/25352389/what-is-the-difference-between-macosx-deployment-target-and-mmacosx-version-min#25362535

Don’t know about this

that’s what I use when building native libs for jars. It should work.

I’m away from keyboard, but setting the environment variable is usually the easiest method.

https://github.com/juji-io/dtlvnative PR welcomes

I think https://github.com/juji-io/dtlvnative/commit/f2fdcb94d0420f0c4fec85eebf289c3e098ac9bd is the only change you need to make. I'm trying to test, but I don't see where macosx-arm64 is built. I see the script for macosx-x86_64 in the github workflow, but not arm64.

It is in https://github.com/juji-io/dtlvnative/blob/master/.cirrus.yml

Setting the mac osx deployment target fixes the original error, (built for macOS 15.0 which is newer than running OS), but it's still not loading due to a dependency on homebrew's libomp. I use macports which stores libraries in a different path, so the shared library fails to load.

It's possible to workaround this by messing with rpaths, but that's beyond the scope of what I have time to look into and implement.

Would you like me to file a github issue?

sure

Yes. We use nippy for serialization of Clojure data structure. As long as it is less than the key size limit.

does it compare by hasheq? I guess I was just trying to make sure that unordered maps can reliably be used as lookup keys.

No, it compare by bytes

Yes, it can be lookup keys, as two equal maps will be serialized into the same bytes

It doesn't seem like nippy guarantees that two equal maps will deserialize to the same bytes:

(let [m1 (-> {}
             (assoc :bar :foo)
             (assoc :foo :bar))
      m2 {:foo :bar
          :bar :foo}]
  {:equal? (= m1 m2)
   :same-bytes? (= (seq (nippy/freeze m1))
                  (seq (nippy/freeze m2)))})
;; {:equal? true
;;  :same-bytes? false}

No

ok, so maps can't be used reliably as lookup keys.

https://github.com/taoensso/nippy/tree/master?tab=readme-ov-file#stability-of-byte-output > It has never been an objective of Nippy to offer predictable byte output, and I'd generally recommend against depending on specific byte output. It seems like comparing bytes for keys might be a footgun.

Why would you want to use maps for keys in a database?

My plan was to use datalevin's key value db as a cache for downloading files where the key is something like:

{:git/sha (:git/sha repo)
 :repo name
 :owner owner
 :file fname}

Use a tuple

You don’t want to store those keys, it’s wasteful, isn’t it?

Using a tuple works fine in this case.

The extra disk space required for the key is negligible.

Although the fact that the key value db relies on byte equality makes me wonder if there are subtle bugs with using other types of keys across versions.

and potentially even when the key is read and written using the same version.

All key value stores do this.

presumably, some key value stores use a serialization mechanism that guarantees predictable byte output if they rely on byte equality.

It is predictable. As long as you use the same version of nippy. Cross major DB version migration needs data migration anyway

All a DB can do is to automate it

So MySQL automates migration, whereas Postgres doesn’t. Auto migration is on our roadmap.

Don’t know what you are talking about, “unpredictable byte layout” is just nonsense.

> It has never been an objective of Nippy to offer predictable byte output, and I'd generally recommend against depending on specific byte output. I believe that nippy probably does have predictable byte output for some datatypes, but it's not clear which ones (maps do not). There may be other types that also do not generally have predictable byte output > As long as you use the same version of nippy This is important to know. Basically, it means that if you're using datalevin, you should also pin your nippy dep or some transitive dependency might break your db.

Isn’t it the same for all libraries?

I don't have anything against datalevin. It's my favorite embedded db. I'm just trying to understand what the guarantees are.

The thing is, if you use Datalevin, you should prefer to use its data types, rather than just throwing random things in.

Isn’t it the same for all libraries?I don't think there's a comparable example for how a transitive dep could break a sqlite database.

That’s because you cannot store a Clojure map in SQLite

What datatypes do you suggest? How am I supposed to know which datatypes guarantee predictable byte output?

I said tuple

Supporting arbitrary data is on the roadmap, but for that, you will need to implement protocols. There is no free lunch

Is there a difference between a tuple and a clojure vector?

Of course

I see https://cljdoc.org/d/datalevin/datalevin/0.9.22/api/datalevin.built-ins?q=tuple#tuple, but I can't find any other reference to tuple in the guide part of the docs.

Tuple is a Datalevin data type. Whereas a Clojure vector is just a blob that will be serialized with nippy

You are looking at wrong thing

build-in name space is for datalog build in query function and predicate

I just searched the docs for "tuple" and that's the only thing that came up besides stuff in the changelog

Datalevin.core is where most info about the public api is at

Do you have a link to any references or docs that can help explain how to use tuples?

Looking at https://cljdoc.org/d/datalevin/datalevin/0.9.22/api/datalevin.core. I'm not sure where to start.

You can start by ctrl-f searching for “tuple” on that page.

put-buffer

transact-kv

Doc string of these should tell you everything.

what is a buffer?

bytebuffer?

Yes