Thanks for sharing that link @UPH6EL9DH! 😄

Hey @UPH6EL9DH thanks for taking a look and for your feedback 🙂 For better or worse, SQL remains the only ~standard way forward for the majority of the data industry for the foreseeable future. No doubt vanilla SQL makes expressing certainly kinds of queries much harder than is strictly needed (e.g. compared to implicit Datalog joins and pull), but conversely it has a tonne of features (compared with Datalog) that makes it fairly user friendly for constructing high-level queries very quickly. Ultimately, our intention here is to implement a version of SQL that is simpler for application developers to work with (using Clojure or otherwise) than anything else that exists today, thanks to immutability/dynamism/etc.

For anyone curious about the context here ("core2"), please see the newsletter we circulated earlier: https://us13.campaign-archive.com/?u=b72ef384b5199134185cbeed8&id=1f5ad9139b (and sign-up!)

@U899JBRPF thanks for the detailed response! > our intention here is to implement a version of SQL that is simpler for application developers to work with is it too early to ask to share an example code snippet of what this may look like? 🙂 is there a suggestion for new users considering xtdb? shall we consider waiting for early releases of core2 or is this still too experimental to tell?

> is it too early ... is this still too experimental Yes and yes, sorry 😅 We will have some docs and end-to-end examples to point at within the next few weeks (in time for Strange Loop!), but this current phase is merely an incremental opening up of what has until recently been a private repo with only code and tests. It definitely remains (rather promising) research at this stage. > is there a suggestion for new users considering xtdb? For the avoidance of doubt: new XT users should absolutely embrace 1.21.0 (the latest version) as-is and be excited for the upcoming ingest performance gains in 1.22.0 (currently hovering around ~40% faster for bulk ingestion/re-indexing). At the moment the focus is on stability and performance, but we also have some near-term functional enhancements planned, e.g. re-working the checkpointing functionality a bit. If people are particularly excited about this concept of SQL-first and core2 more generally though then it would be great to chat sometime and hear about the kinds of things one might hope to achieve with XT in the future!

I definitely don’t want to go back to SQL, having embraced datalog

after a quick glance, it seems xtdb2 will be a completely different thing, so you wouldn’t be able to upgrade to it without a complete migration

hopefully that won’t be confusing having 2 different products with the same name

the timeline states growing xtdb1 market share up to 2023, will it be abandoned or continued to be supported after that? I’m a little worried… luckily it’s open source, but still

> I definitely don’t want to go back to SQL, having embraced datalog > Hey @U11SJ6Q0K that's fair, and I can strongly identify with the sentiment, FWIW. One thing to keep in mind is that Datalog is a much simpler thing to implement than SQL, so we're essentially just tackling the (by far) biggest unknowns first right now. I can confirm that withdrawing support for Datalog and XTDB 1.* is not at all on the cards for us at any point without a clear & workable migration plan for our customers and OSS user community (including many internal JUXT users also!), and absolutely not whilst we have any existing support relationships and partnerships in place. Fortunately having everything be ~immutable opens up a lot of technical possibilities, particularly also when the essential data model (bitemporal documents) is unchanged. We may well even end up simply folding all this work back into the main repository once it's more complete, which hopefully could also minimise many such concerns and confusion. I'll have a think about addressing this more clearly in the readme today! Thank you (as ever) for chiming in with your thoughts and feedback 🙂

thanks, that is encouraging to hear

Is SQL first even relevant? As both datalog and SQL map to relational algebra, why not just have a relational algebra engine with two language frontends?

> why not just have a relational algebra engine with two language frontends? Great question - in fact that's essentially how we have it all working internally already, and the edn-based relational algebra IR looks something like: https://github.com/xtdb/core2/blob/dc8edd13995717675a6a2b8f6ee3f05a8dc05f5c/modules/datasets/src/core2/tpch.clj#L95 Note that we have the seeds of an edn-Datalog frontend implementation here: https://github.com/xtdb/core2/blob/master/test/core2/datalog/datalog_test.clj / https://github.com/xtdb/core2/blob/master/core/src/core2/datalog.clj (crucially though there is no rules / recursion support yet, so not yet a true Datalog!) A big engineering unknown has been that figuring out how to build a sufficiently generic and high-performance relational algebra engine really does require understanding the essential requirements of any proposed frontends ahead of time, and SQL has much more complicated semantics (in terms of bags, 3VL, scoping etc.). Therefore whatever we might have designed with a Datalog-first approach would almost certainly be insufficient to later build SQL atop, at least without lots more hammock time and budget to potentially rework everything, and so SQL-first was decided as the lowest risk path. It's probably also worth mentioning that we actually couldn't find any suitable "standard"/off-the-shelf approaches to implementing relational algebra for SQL that we might have been able to port across (which might have saved us design time), but the closest things are perhaps https://calcite.apache.org/ (same thing that we used to implement https://github.com/xtdb/xtdb/tree/master/modules/sql) and https://substrait.io/ ...so we forged our own path with the edn IR 🙂

Regarding a sufficiently generic (and reactive) relational engine, I thought #relic achieved that already

I'm not knowledgeable enough (yet!) to comment on the specific IR differences with #relic, but @U0GE2S1NH (relic's author) actually joined the XT dev team back in March...so should be fairly well-qualified to expand on that at some point 😄

That's pretty cool

I guess that does mean no relic2 anytime soon :thinking_face:

2.0 will need some headline feature like 'quantum joins', or 'microsoft excel integration'. Hope to get back to #relic soon do not worry. Datalog will come, hash will be given a day off, and uh - I'll make it more debuggable or something.

Has any consideration been given to 'reactive queries' / materialised views for core2?

> Has any consideration been given to 'reactive queries' / materialised views for core2? Very briefly - in that it's an interesting research space but is definitely out of scope for the time being 🙂

Immutability makes integrating differential-dataflow (or equivalent) much easier though - and maybe the intent behind https://github.com/xtdb-labs/crux-dataflow will see the light of day after all

One idea: use https://zelark.github.io/nano-id-cc/.

I would say it depends on how you want to use the data, what sort of queries you need to efficiently answer

like, instead of having a list of children, you could have a child refer to its (possibly many) parents, but then children wouldn’t have order

Order is what is missing from my analysis above. Order is required and can be changeable. I've also realised that having the dynamic [parent order id] (or just [parent order]) as identifier cannot work because it's is changeable. E.g. cannot make a reliable URL from it.

And graph traversal is the most fundamental/common kind of query.