This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2022-01-17
Channels
- # announcements (24)
- # babashka (22)
- # beginners (49)
- # cider (16)
- # clj-kondo (8)
- # cljsrn (4)
- # clojure (87)
- # clojure-australia (7)
- # clojure-europe (44)
- # clojure-nl (4)
- # clojure-sweden (7)
- # clojure-uk (24)
- # clojurescript (5)
- # core-async (7)
- # cryogen (8)
- # cursive (22)
- # data-oriented-programming (2)
- # datomic (1)
- # emacs (6)
- # events (4)
- # fulcro (11)
- # google-cloud (1)
- # introduce-yourself (1)
- # java (8)
- # jobs (3)
- # lsp (10)
- # observability (1)
- # off-topic (12)
- # polylith (12)
- # re-frame (6)
- # reitit (36)
- # remote-jobs (1)
- # ring (4)
- # ring-swagger (1)
- # rum (4)
- # schema (1)
- # shadow-cljs (18)
- # sql (56)
- # tools-deps (33)
Hi all 👋, I'm pleased to announce an early alpha of relic
is now available. 🎉
relic
is an in memory database and data processing library with a closer-to-sql query dsl, constraints and incremental materialized views. I do not know if its a good, ok, or terrible idea - but I wanted something like this to exist so I could better find out.
github: https://github.com/wotbrew/relic
channel: #relic
Questions, discussion (and contributions!) massively appreciated.
This looks really interesting, great work! I know its still in alpha, so performance is probably still way worse than it could be. Disclaimers aside, how would the materialized views compare to something like re-frames subscribers? I know that replacing it with datascript is a lot slower, but this seems like more performant alternative.
I haven't published any benchmarks yet, but perf should be competitive with the competition, YMMV! Main things to be aware of would likely be memory usage and write performance if you materialize a lot of queries. Certainly a design goal is to be able to use relic with react and invalidate hooks / ratoms directly using tracked-transact. Relic is just a fancy signal graph itself so its quite natural to do this.
The upper ceiling on perf tuning for something like this is basically 'write a compiler' so lots of room at the top.
thank you for the quick reply. I’ll defiantly play around with it, looks very interesting.
@U0GE2S1NH the usage with React is also something I’d be interested in, please keep sharing how that turns out 🙂
I have a hacky demo of its use with reagent - I'd love to see somebody build a glue library that makes it a little less hacky. I don't currently spend a lot of time working on cljs ui's.
BTW, you mentioned that recursive queries aren’t implemented yet. Did you just not get around to it yet or is it going to be a really hard problem because of the signal graph cycles that would exist? just curious
I didn't get around to it. Its certainly tricky, but I think something can be done in the future.
i love that folks don't consider this to be a solved problem, and keep coming up with new and interesting approaches to solving it. nice work @U0GE2S1NH!
you say curse, i say creative playground -sung to the tune of tomato, tomahto-
@U02EMBDU2JU > ...recursive queries aren’t implemented yet... is it going to be a really hard problem The math does indeed get quite a bit trickier for this. The typical trick used in for resolving recursive queries (recursively evaluating a rule till you reach a fixed point) is more or less the same trick used to perform incremental view maintenance, so you have to generalize the iteration from a well-ordered set of indices to a partial order (in other words, you don't just iterate over either data-update or recursive-rule-update, you have to iterate over both at the same time). The solution is differential view maintenance (differential dataflow/datalog/computation, etc). Frank McSherry et al have worked this out: https://www.microsoft.com/en-us/research/wp-content/uploads/2013/01/differentialdataflow.pdf
This one is better at explaining some of the background theory: https://users.soe.ucsc.edu/~abadi/Papers/differential_revision-app.pdf
Very interesting stuff, but also requires quite a bit of additional machinery.
I read some of those papers, my brain is a bit too smooth for all the math but I believe it can be done. In my head I see a specialised data flow node that closes over the fixed point recursion and intermediate indexes. You won't just be able to recur anywhere like a madman, I imagine some kind of CTE-ish syntax
Thanks for the links again @U05100J3V, will review when it's time. Would be super happy to take notes / ideas in an issue too!
Yeah; The background is super heady stuff. This post may be helpful, since it's a bit more pragmatic, and assumes the underlying diff-dataflow machinery: https://github.com/frankmcsherry/blog/blob/master/posts/2016-06-21.md
It would be amazing if we had something akin to the timely-dataflow library; Then this wouldn't be too much work at all. But that's actually a tall order.
I'd recommend chatting with @U8YCZLZH6, who built clj-3df (https://github.com/sixthnormal/clj-3df). They left all the dataflow stuff to Rust/timely-dataflow, and wrapped as a clj api.