Clojurians Log v2
announcements
https://github.com/filipesilva/datomic-pro-manager#library-usage is out with support for starting Datomic from within your app:
(ns app
(:require
[datomic.api :as d]
[filipesilva.datomic-pro-manager :as dpm]))
(defn start []
(future (dpm/up))
(dpm/wait-for-up)
(d/create-database (dpm/db-uri "app"))
(def conn (d/connect (dpm/db-uri "app"))))dpm/db-uri takes a db name and returns the full Datomic URI for it.
dpm/up downloads and starts the transactor if needed, and dpm/wait-for-up blocks until it's ready.
You'll still need to add com.datomic/peer and org.xerial/sqlite-jbc (if using sqlite) to your deps.edn, get their versions from running dpm once if you're not sure.
Initial release of https://github.com/mpenet/k7 - A high-performance disk-backed queue for Clojure.
https://github.com/mpenet/k7#performance (using :flush on Apple M1, JVM 20)
• Append-only log backed by preallocated mmap'd segment files
• Zero-copy reads — payloads are read-only ByteBuffer slices into the mmap
• Single writer, multiple independent consumer groups with crash-safe cursors persisted in their own mmap'd files
• Ack / nack / seek — at-least-once delivery, redelivery on nack, arbitrary seek; adaptive batching on poll!
• Crash recovery — segments are scanned on open; the last valid CRC32C-verified frame is found automatically
• Configurable fsync strategy per queue (`:async`, :flush, :sync) and per consumer group
• Low allocation — enqueue! is zero-alloc; poll! allocates only the read-only ByteBuffer slice per message
wow, this is sick
btw, I just tested the lib, and the Qs and CGs operations still work after close (macos).
hi! No it's not really suited for super high number of queues: between fd, the commit thread (depending the fsync strategy), and memory requirement. It makes more sense to have dozens/hundreds of queues max
2. file segments are cross platform
3. we leave a single file on disk to hold cursor information per consumer group. So if you re-use consumer group ids you'd be fine, They are just a file ultimately you can just delete them
there's no cleanup on this. the point is to have them survive, but you can delete them
something I will add eventually is rolling strategies for segment files
so that you don't fill up your disk, but that's also doable outside of the lib currently
well ish. the consumer group currently holds a bit of state regarding that, but it's a minor change
to go back to 1. the number of messages in a queue is not really an issue
about close I'd have to check it, could be that it might seem it works, but doesn't fully
could be a bug also
thanks. regarding #1, is there anyway to push past hundreds of Qs max? 1. if commit threads are used per Q, are they disposed after close? i can tolerate reading ~100 max concurrently. if not, i don't mind using fsync 2. regarding the mem requirement, can I simply use smaller segments?
1. yes they are disposed of 2. yes! that's a good workaround
if you use :flush strategy there's no commit thread also
but you'll trade some durability
cool, thanks. i'd have claude run some tests and report back
awesome work. I've had several issues with various chronicle-queue-backed libs. a few questions, if you don't mind: 1. can it handle lots of queues? as in, 100k - 1 mil, each with 100k-1mil messages? 2. are the file segments cross-platform? i.e., can i expect the same set of bytes if i were to copy the files over from a mac or windows/linux. this isn't critical, I'm simply curious. 3. is there any disk cleanup regarding cursors? as in, i may want a process to always read all messages... if the cursors leave behind a bunch of bytes on the disk (whether for closed/unclosed CGs), is it a cause for concern if I may reopen the queue hundreds/thousands of times?
how does it compare to https://github.com/mpenet/tape
seems to be targeting the same niche
yep, they are very similar. k7 is inspired by chronicle-queue but it removes some of the stuff I didn't like about it. • k7 has 0 dependency (just clj and core.async), tape has tons (via chronicle-queue) • k7 is more barebones, some things are left to the user to decide (threading/locking, serialization etc) • k7 is a lot faster writing to the queue, like 5x faster on some of the benchmarks I ran (using custom serialization on tape to be fair, since k7 just deals with bytes). • tape is a bit faster to read from queues, but I think I can close that gap, but they are withing 20% of each other, it's quite minimal • on "large" payloads, k7 is a lot faster, ~2x for 64kb on roundtrips (write+read) Then k7 is ~700loc, 100% clojure (in very java'esque style), chronicle-queue is quite a bit larger :)
under the hood they are quite similar, I think the differences are just down to tradeoffs for various usecases
then I ran simple benchmarks, both chronicle-queue and k7 have tons of little details that can move the needle one way or the other, like fsync strategies & whatnot, I didn't do some deep analysis of chronicle-queue internals to do a benchmark that matches it super accurately. The underlying idea is the same, mmap'ed segment files, limiting allocs, etc... As long as one is careful not to be wasteful outside of this part the rest is just left to the OS to deal with and the differences are just in high level plumbing. Regardless I think the performance of both make them usable for almost any context.
then chronicle-queue is old/established. k7 is a few evenings of fun-experiment, one-man-show 🙂 But it works, so...
I've had an idea mulling around in the back of my mind for quite a few years now. Today, I finally got around to prototyping it: > "Eido is a declarative, EDN-based language for creating graphics. Describe the image as data, not drawing instructions." You can find it here: https://eido.leifericf.com
That's awesome! Thanks for letting me know. I wanted to create the best possible tool for generative art that I've always wanted for myself. And I honestly didn't even consider it could be useful for data visualization as well 😅 Very cool.
I've done a massive push this week to get it to “feature complete” with optimized output for plotters, mills and CNC machines. And optimized output for print, etc.
And to stabilize the API.
Now I’m doing more extensive testing, bug fixing and polishing to remove the “Beta” tag.
After that, I’ll see if it’s possible to squeeze out more performance without introducing GPU compute.
Now Eido also has some basic 3D capabilities. See https://github.com/leifericf/eido?tab=readme-ov-file#3d-gallery. Still 100% Clojure with no external dependencies, as it's all CPU-based rendering.
And now Eido also has some basic particle effects (https://github.com/leifericf/eido?tab=readme-ov-file#particle-gallery) 🙂
Maybe post repeated follow-ups in #releases so you don't keep posting in the main #announcements channel?
Sure
#announcements is intended for once-a-month cadence or just major releases, like your original post. Thanks.
I wasn't aware of #releases Thanks for letting me know! And sorry for spamming.
Hi @leif.eric.fredheim, this is so inspiring! Today we played a bit with Eido in the https://scicloj.github.io/docs/community/groups/real-world-data/ (that meets regularly every week). We tried a few examples and looked into integrating them in the https://clojurecivitas.org/ blog. The session is recorded as unlisted video and shared internally in the Zulip chat (requires login): https://clojurians.zulipchat.com/#narrow/channel/315077-real-world-data/topic/recordings/near/584751580 recordings @ 💬>