@U01PE7630AC Thanks for the idea! That would be fun to talk about. I could see a number of different motivations in that space. Eg. making a "relational" interface that can map to any relational DB. Eg. Shielding the application from how the data is persisted to avoid tight coupling with the application code. Eg. Avoiding having to write CRUD. etc.

In the last 4 years or so, for me, I've ended up working with extremely distributed systems, so all of the databases that exist are local only to each specific node in the system. To put it another way, there is no one shared data store that multiple nodes read and write to.

So that affects my opinions on databases for sure. They mainly devolve into application-level caches for my problem domains.

The only exception to that I can think of in my own work is reporting databases. I'll have one node responsible for maintaining a relational database that is hooked up to a reporting tool (like Metabase or Superset). It gets all its information from the rest of the system via messaging and its sole concern is recording things that can be reported on.

I work with extremely live and real-time systems, so most of the data in the system is the "current state" of other systems.

@U5FV4MJHG Your context sounds ideal for Erlang/OTP 😅 https://www.erlang.org/doc/man/gen_server.html, stateful https://www.erlang.org/doc/reference_manual/processes.html and https://www.erlang.org/doc/man/mnesia.html.

Yes, I use a "process" like abstraction. I ended up writing me own little framework for it. I opened sourced it, but I've changed it significantly since then and I need to update the public version.

@U5FV4MJHG Did you implement something similar to https://en.m.wikipedia.org/wiki/Actor_model?” That’s basically what Erlang/OTP is at its core. Similar to https://en.m.wikipedia.org/wiki/Akka_(toolkit) in the Scala universe.

@U01PE7630AC Yes and no. Akka and actors is what really got me into Scala about 13 years ago. What I have is more like a reactive dataflow graph, where each node is the graph is like an Erlang "process", but a node cannot hold any references to any other node. Just messages in and then out again. I need to share out my latest version and then put some material together about it.

@U5FV4MJHG Cool! So it's kind of like a https://en.wikipedia.org/wiki/Directed_acyclic_graph, where each node is kind of like a function that just transforms the data and sends it along to the next process? Sort of like a big pipe operator but with processes instead of functions. I'll have a look when you get around to sharing it!

Yup. It's a DAG pipeline.

Nice! We use that concept at work for data engineering, but through visual tools like https://www.matillion.com, https://docs.getdbt.com, and https://airflow.apache.org.