Fork me on GitHub
#clojure
<
2024-02-11
>
mike_ananev08:02:09

Hi, @alexmiller! I've found compile error using 1.12.0-alpha7. Here is small repro snippet: mike@mbp03 clj -Sdeps '{:deps {org.clojure/clojure {:mvn/version "1.12.0-alpha5"} hashp/hashp {:mvn/version "0.2.2"}}}' Clojure 1.12.0-alpha5 user=> (require '[hashp.core]) nil user=> mike@mbp03 clj -Sdeps '{:deps {org.clojure/clojure {:mvn/version "1.12.0-alpha7"} hashp/hashp {:mvn/version "0.2.2"}}}' Clojure 1.12.0-alpha7 user=> (require '[hashp.core]) Syntax error compiling var at (fipp/ednize/instant.clj:11:13). Unable to resolve var: clojure.instant/thread-local-utc-timestamp-format in this context user=> mike@mbp03 java --version openjdk 21 2023-09-19 LTS OpenJDK Runtime Environment (build 21+37-LTS) OpenJDK 64-Bit Server VM (build 21+37-LTS, mixed mode, sharing)

thheller08:02:49

fipp, a dependency of hashp, used to use a hack to access a private var in clojure.instant. https://github.com/brandonbloom/fipp/commit/ca9b107788d90804dc2ce986c33421cf987e63ed

thheller08:02:10

that var has been removed, so the lib break entirely because of it doing something it shouldn't have done

thheller08:02:43

but if you add fipp/fipp {:mvn/version "0.6.26"} to the deps map above that should be fixed

👍 1
☝️ 1
thheller08:02:40

(could also be that there is a new hashp version. I didn't check that)

Alex Miller (Clojure team)14:02:00

Yes, this was necessary to get rid of the ThreadLocal

Alex Miller (Clojure team)14:02:59

Please file an issue with hashp if needed

vlaaad18:02:16

Has anyone tried making a persistent data structure resembling a db table with indexes, e.g. for expressing a coll of maps more efficiently in terms of memory (like tech.ml.dataset), but with indexes for efficient lookups by id, and, like in databases, the indexes are updated when the table changes? UPD: something that is not datascript or datomic, but more column-oriented for efficiency like tech.ml.dataset

vlaaad18:02:55

and with relations...

vlaaad18:02:59

it feels like this is something that might be very useful for managing an in-memory app state...

p-himik18:02:35

Some libs have done it for re-frame or just CLJS app state management.

vlaaad18:02:03

any pointers?

cjohansen18:02:34

Datascript comes to mind

cjohansen18:02:45

Not tables, but RDF triples

ghadi18:02:52

The Datomic AEVT index is essentially a column-stored such data structure

vlaaad18:02:00

ah, I was hoping for something other than datascript and EAVTs

p-himik18:02:33

Given the channel, that addendum should've been in the OP. :D

vlaaad18:02:38

more like, column-oriented for memory efficiency

ghadi19:02:35

datomic contains a column-oriented persistent datastructure: the AEVT index

ghadi19:02:49

not sure what exactly your criteria are

vlaaad19:02:29

I want typed value columns

ghadi19:02:41

literally exactly the AEVT index

ghadi19:02:08

in the entity-attribute-value model, sorting by the attribute first gives you a column-oriented view

ghadi19:02:23

sorting by E first gives you a row oriented view

ghadi19:02:03

(...and sorting by V first gives you a reverse lookup by value, e.g. for uniqueness checks)

vlaaad19:02:31

Do e.g. boolean value types in datomic use boolean arrays/vectors for memory efficiency?

ghadi19:02:32

what's wrong with using tech.ml.dataset?

vlaaad19:02:41

No indexes

vlaaad19:02:14

Also I got the impression that tmd is for read-only data analysis

ghadi19:02:45

i'm not certain, but I think tech.ml.dataset is a lower-level component than an index

vlaaad19:02:45

Where I'd like something to incrementally update in the running app

vlaaad19:02:13

Sort of like an SQLite db for app state, but as persistent data structures

ghadi19:02:13

i.e. you can build indexes on top of t.m.d. by sorting/partitioning your data

vlaaad19:02:31

Because SQLite does not have MVCC..

phronmophobic19:02:10

Another option in the space that people have tried is rules engines. eg. https://github.com/oakes/odoyle-rules

vlaaad19:02:52

Thanks so much @U7RJTCH6J!

👍 1
phronmophobic19:02:20

I've been looking for something similar and still think there are opportunities for new options in the space. Would love to hear back if you find a solution you like!

vlaaad21:02:00

Relic looks very interesting

vlaaad21:02:48

I feel that at some level of domain complexity and data size, using uber-map for a client app state is not enough, and what is needed is a "database"

vlaaad21:02:00

something like sqlite with in-memory db

p-himik21:02:53

Is Relic column-based?

vlaaad21:02:21

but well, relic does not seem to be what I feel is needed

vlaaad21:02:11

but sqlite is transactional and uses locking, which puts readers and writers at odds with each other

ghadi21:02:48

this thread is going nowhere because of feelings rather than criteria

🎯 3
p-himik21:02:48

> at some level of domain complexity and data size, using uber-map for a client app state is not enough Yeah, a few years ago I ended up coming to the same conclusion and evaluating all of the available options that I could find. In the end, came up with my own thing (haven't published yet) because I wasn't quite satisfied with what was available. But it's for CLJS (hence me mentioning it at the start) and not column-based either.

vlaaad21:02:03

would be cool to have something performant like sqlite/tmd, but persistent and lock-free like clojure data structures

phronmophobic21:02:17

Yea, I've been looking and haven't found an answer that feels right yet. I also think Relic is the most interesting. One of the features I want is to be able to use nested maps, which sqlite doesn't do very well.

phronmophobic21:02:05

Having a push API that can run effects when certain attributes change is also interesting.

p-himik21:02:46

FWIW, DuckDB has been making the rounds. It seems to support MVCC, there's also a library to integrate it with tech.ml.dataset. Haven't used it myself yet, no clue about anything else.

vlaaad21:02:57

hm, datomic local is apache licensed?

vlaaad21:02:15

I thought it had some EULA that prevented its redistribution...

ghadi21:02:03

All Datomics are free as in beer

vlaaad21:02:04

I evaluated datascript long time ago, but it was too slow for the kind of workloads I expected from it

1
vlaaad21:02:11

maybe I should give datomic local a try

vlaaad21:02:23

@U050ECB92 but datomic local is free as in freedom?

phronmophobic21:02:03

If you're considering datalog dbs, then I've been having fun with https://github.com/juji-io/datalevin. The database itself isn't immutable, but it's similar to sqlite in that regard.

phronmophobic21:02:14

Same with https://xtdb.com/docs/, but I've found datalevin is little more straightforward as an embedded db compared to xtdb which seems more oriented towards db as a standalone process.

Linus Ericsson16:02:39

Clojure data.int-map does not have a cljs-version yet, but I wrote this: https://clojureverse.org/t/data-modelling-with-vectors/9826/1

Linus Ericsson16:02:52

Also re Datomic use of memory - I don't know for sure, since I don't have the source code, but there is a clojure.core/vector-of which have primitive-typed vectors. I just assume this is used for the datomic indexes where applicable.

p-himik20:02:58

Clojure 1.11.1
user=> (type (hash-map))
clojure.lang.PersistentArrayMap
Huh?

p-himik20:02:51

Ah, the [] arity.

seancorfield21:02:18

(hash-set) produces a PersistentHashSet not an array set on both 1.11 and 1.12 for me...

seancorfield21:02:44

(but, yeah, (hash-map) produces an array map)

p-himik21:02:22

Oops, my bad, amended. TIL there's no array set.