Fork me on GitHub
#announcements
<
2022-01-17
>
wotbrew09:01:13

Hi all 👋, I'm pleased to announce an early alpha of relic is now available. 🎉 relic is an in memory database and data processing library with a closer-to-sql query dsl, constraints and incremental materialized views. I do not know if its a good, ok, or terrible idea - but I wanted something like this to exist so I could better find out. github: https://github.com/wotbrew/relic channel: #relic Questions, discussion (and contributions!) massively appreciated.

🎉 22
❤️ 5
Lukas Domagala11:01:41

This looks really interesting, great work! I know its still in alpha, so performance is probably still way worse than it could be. Disclaimers aside, how would the materialized views compare to something like re-frames subscribers? I know that replacing it with datascript is a lot slower, but this seems like more performant alternative.

wotbrew11:01:56

I haven't published any benchmarks yet, but perf should be competitive with the competition, YMMV! Main things to be aware of would likely be memory usage and write performance if you materialize a lot of queries. Certainly a design goal is to be able to use relic with react and invalidate hooks / ratoms directly using tracked-transact. Relic is just a fancy signal graph itself so its quite natural to do this.

wotbrew11:01:38

The upper ceiling on perf tuning for something like this is basically 'write a compiler' so lots of room at the top.

❤️ 1
Lukas Domagala11:01:14

thank you for the quick reply. I’ll defiantly play around with it, looks very interesting.

martinklepsch11:01:48

@U0GE2S1NH the usage with React is also something I’d be interested in, please keep sharing how that turns out 🙂

wotbrew11:01:46

if you want to go down this rabbit hole happy to talk in #relic

wotbrew11:01:54

I have a hacky demo of its use with reagent - I'd love to see somebody build a glue library that makes it a little less hacky. I don't currently spend a lot of time working on cljs ui's.

Lukas Domagala13:01:24

BTW, you mentioned that recursive queries aren’t implemented yet. Did you just not get around to it yet or is it going to be a really hard problem because of the signal graph cycles that would exist? just curious

wotbrew13:01:37

I didn't get around to it. Its certainly tricky, but I think something can be done in the future.

robert-stuttaford16:01:58

i love that folks don't consider this to be a solved problem, and keep coming up with new and interesting approaches to solving it. nice work @U0GE2S1NH!

wotbrew16:01:37

the curse of lisp I guess 🙂

robert-stuttaford16:01:12

you say curse, i say creative playground -sung to the tune of tomato, tomahto-

😄 2
☝️ 1
metasoarous19:01:57

Nice work @U0GE2S1NH! Excited to see this!

❤️ 1
metasoarous19:01:25

@U02EMBDU2JU > ...recursive queries aren’t implemented yet... is it going to be a really hard problem The math does indeed get quite a bit trickier for this. The typical trick used in for resolving recursive queries (recursively evaluating a rule till you reach a fixed point) is more or less the same trick used to perform incremental view maintenance, so you have to generalize the iteration from a well-ordered set of indices to a partial order (in other words, you don't just iterate over either data-update or recursive-rule-update, you have to iterate over both at the same time). The solution is differential view maintenance (differential dataflow/datalog/computation, etc). Frank McSherry et al have worked this out: https://www.microsoft.com/en-us/research/wp-content/uploads/2013/01/differentialdataflow.pdf

👍 1
metasoarous19:01:57

This one is better at explaining some of the background theory: https://users.soe.ucsc.edu/~abadi/Papers/differential_revision-app.pdf

metasoarous19:01:37

Very interesting stuff, but also requires quite a bit of additional machinery.

wotbrew19:01:34

I read some of those papers, my brain is a bit too smooth for all the math but I believe it can be done. In my head I see a specialised data flow node that closes over the fixed point recursion and intermediate indexes. You won't just be able to recur anywhere like a madman, I imagine some kind of CTE-ish syntax

wotbrew19:01:59

Thanks for the links again @U05100J3V, will review when it's time. Would be super happy to take notes / ideas in an issue too!

👍 1
metasoarous19:01:43

Yeah; The background is super heady stuff. This post may be helpful, since it's a bit more pragmatic, and assumes the underlying diff-dataflow machinery: https://github.com/frankmcsherry/blog/blob/master/posts/2016-06-21.md

❤️ 1
metasoarous19:01:06

It would be amazing if we had something akin to the timely-dataflow library; Then this wouldn't be too much work at all. But that's actually a tall order.

metasoarous19:01:30

I'd recommend chatting with @U8YCZLZH6, who built clj-3df (https://github.com/sixthnormal/clj-3df). They left all the dataflow stuff to Rust/timely-dataflow, and wrapped as a clj api.

👍 1
wotbrew20:01:36

Thanks so much! I hope once the dust has settled and I figure out how good an idea it actually is, to have another pass at the internals and align it with the best-of-breed approach from timely