Clojurians Log v2

btw, I'm not running any proprietary info with the current db. If you want me to pass along a generation script + seed data, I'd be happy to put that together. In the current shape, the data is about 80k files -> 4m datoms.

Found my problem - the konserve :file backend was killing me. I swapped to the sqlite+jdbc backend and got a 20x storage improvement

Interesting, I think the difference should not be that big, but maybe sqlite compresses. Which configurations did you exactly use in each case?

Before
{:store              {:backend :file
                      :path    (index-dir root "datahike")
                      :id      (stable-uuid (str ws-id))}
 :schema-flexibility :write
 :keep-history?      false}

After
{:store              {:backend :jdbc
                      :dbtype  "sqlite"
                      :dbname  (datahike-sqlite-store-dbname root)
                      ;; store.sqlite?busy_timeout=30000&journal_mode=WAL&synchronous=NORMAL
                      :table   "konserve"
                      :id      (stable-uuid (str ws-id))
                      :maxPoolSize     1
                      :minPoolSize     1
                      :initialPoolSize 1}
 :schema-flexibility :write
 :keep-history?      false
 :store-cache-size   100000}

This, roughly. The DB is distilled from about 70mb of json. I was getting about 20gb of konserve with about 1m datom files, but with sqlite I'm seeing about 0.57gb.

I have tested Datahike on 250 million Datoms in February (DBPedia), albeit with some optimizations that have not fully landed yet. If you only have a single branch and want to cleanup garbage on the way you can use https://github.com/replikativ/datahike/blob/1dd9a2ca7204524c93710f5730780edfc237abc1/doc/gc.md#online-garbage-collection-incremental-gc for instance. This is planned to land soonish https://github.com/replikativ/datahike/pull/836, which will drastically reduce the write amplification by ~10x (but will be experimental first to make sure it is solid). It took a bit of time to figure out how to land this hitchhiker-tree optimization properly, since the original hitchhiker-tree was not ideal. If you have a synthetic work load then I can also test it myself or take a closer look. Btw. I am also working on a Roam/notion-like functionality in the simm.is app, I am thinking about whether I should open source it or not, since I will build a commercial product on top.

lol it's in the water.

I think a convincing Clojure web framework should actually be built based on such a frontend concept, because even if you don't show it to the user necessarily, it is just a great way to organize information in general.

I'm building a closed source version pretty tightly tailored for an org I'm a part of (501c3). AI is awesome to remove cost to build the thing you actually need.

Yes, although having built quite a bit of infrastructure with AI lately I still care about maintenance cost and sharing it ultimately. It still costs time and nerves to fix bugs, for instance in the tail end. You don't want to create your own database or OS for an app for instance, although it is great to be able to deep dive and modify if needed.

Totally. I'm trying really hard to kill mantinance burden with this project. It's the first one I've done with AI from scratch, and it's easily 10x easier to maintain if that was understood as a design goal from the start

this might be vaguely interesting to you - I've decided to dump the entire default writer backend of datahike and replaced it with a custom LMDB backend, but still servicing the standard datahike query interface. I made a bunch of little changes for my use case, and I got ingest performance to be ~30x better and another 5x decrease in storage space. I'm getting worse query performance, but that's a reasonable tradeoff for me, and I also think I know how to potentially fix that.

@whilo plus_one for “Roam/notion-like functionality” I’ve been on Roam for years. I believe the granular nature of datoms and first class outline nodes are probably better abstractions than starting with markdown, like obsidian and logseq do. But the graphlike nature is somewhat buried in the somewhat clunky ui. There are affordances to create custom ui and extensions, and use api’s… but I’ve been hesitant fully dive in because of the proprietary nature of the platform (I cant even get a repl connection to the cljs runtime in the app). I want to invest in something more tech forward and hackable. Something like Simmis or Easel is where i will probably migrate to eventually. I am fascinated by the forking stuff and think all this must be an excellent foundation for not just personal notes, but multiplayer notes and CRM / CMS functionality, etc.

Can confirm @chromalchemy 🙂. The YAML frontmatter is awesome. I'm building an obsidian-adjacent tool that focuses heavily on user-defined schema and user defined features. Stuff like SCHEMA.md which is a self describing user-facing schema defintion, and FORM.md, which is just a form definition based around MD, have been super cool. I'm not planning to OSS this most likely, but I'd be happy to put out the basis of it. The only downside of my approach is that it's tuned web-first, not local-first.

For https://github.com/replikativ/datahike-saas-starter I landed a series of latency and write improvements that reduce write amplifications considerably, most importantly the diff-buf feature. If you have additional suggestions @jatkin lmk.

I'll try and do an a/b comparison. Ty