Clojurians Log v2

@brownjoshua490 ah wow. Yeah maybe just my laptop, but do you have some constraints on your datoms' values to make more efficient? Not sure whether tauri would affect performance. Are queries over the million datoms performant? I am persisting with the serializable and from-serializable functions. Is that what you mean by datomic has persistance? I will try the init-db. For this do you represent the datoms as a vector of vectors? [ [1 :example/age 30 1 true] [2 :example/name "Bob" 2 true]] I don't mind the initial load but when I reopen the app, I think I want it to be fast, so might give datomic local a try.

Queries are very inefficient in datascript, we almost exclusively use d/pull and direct index access with d/datoms. For our data access patterns this is fine, there aren't actually a lot of cases where we need a query (usually index access and pull can replace it). We don't have constraints on what we store, I do think the type of data you store will have an effect on speed since you have to do a lot of comparisons. We store lots of ints (refs) and strings. What I mean re datomic persistence is that datomic writes the data to disk and reads data from disk. Datascript is in memory (unless you use the new storage API). The tradeoff is that datascript will take a long time to load initially and take up RAM but reads after it's loaded should be faster.

init-db takes a vector of datoms like this

(d/init-db
 [(d/datom 1 :block/uid "123")
  (d/datom 2 :block/uid "456")]
 schema)

IMO from watching lots of people new to datomic use queries, they are way overused and even in datomic a pull is faster

Creating DB from scratch, in order from slowest to fastest: • transact! • init-db • from-serializable Loading already created database lazily from disk: • restore-conn Last one is the fastest one because it doesn’t fetch everything on first call, instead it pulls more segments as they are needed. I’d recommend to try that one (see here for an example https://github.com/tonsky/grumpy/blob/d1c89979a25e82ce64cd04f219e3da8d85fec2ba/src/grumpy/db.clj#L43-L62), although I’m not sure if you have JVM part (it’s for JVM only)

8 seconds sounds more like it. I think this comparable to what we had at Roam Research team graph, and what lead to the creation of from-serializable

transacting this much from scratch is not a good idea for sure

at roam we have dbs with millions of datoms that load and work fairly well. I'm sort of surprised by those timings but my computer is pretty fast. For initial import if you can create datoms yourself and do init-db instead of (ds/transact! conn tx-data) that will cut the time to import down by a lot

initial load into datomic will be slower because it's also writing to disk, but next time you open it up it will be fast. Datascript also has persistance now via the storage API if that fits your semantics. Datalevin is faster than both datascript and datomic but lacks immutability

Sounds a bit too long. Which API do you use to import?

https://github.com/tonsky/datascript/releases/tag/1.2.0 this one should be fastest one

how many datoms is that?

540,405 datoms. Maybe this is too much for datascript? Using from-serializable takes 8 seconds to load in. Importing the raw data for the first time and doing the (ds/transact! conn tx-data) takes the most time around 1.5mins.

Actually got it down to 329542 datoms. It is pretty slow and don't think datascript was meant for this many datom. I am going to try spinning up datomic locally or try xtdb