This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2021-03-21
Channels
- # announcements (2)
- # asami (8)
- # aws (4)
- # beginners (32)
- # calva (12)
- # cider (72)
- # clj-kondo (16)
- # clojure (11)
- # clojure-germany (2)
- # clojure-italy (4)
- # clojure-serbia (2)
- # clojurescript (26)
- # data-oriented-programming (8)
- # datomic (9)
- # deps-new (17)
- # eastwood (4)
- # emacs (20)
- # fulcro (18)
- # funcool (1)
- # graalvm (8)
- # leiningen (12)
- # lsp (34)
- # malli (25)
- # meander (4)
- # membrane (4)
- # off-topic (153)
- # practicalli (1)
- # releases (2)
- # remote-jobs (3)
- # rewrite-clj (77)
- # ring (5)
- # shadow-cljs (108)
- # spacemacs (12)
- # tools-deps (9)
- # vscode (11)
- # xtdb (4)
and i'm unsure if its because I am getting the address of the callback function wrong (it should be a symbol in the library) or something I am doing wrong with struct padding
There are several query DSLs (GraphQL, EQL, pull syntax, Datalog, meander, SQL, specter, etc) that are fairly generic, flexible, expressive, and independent of datastore (ie. crux, datascript, datomic, in memory). It seems like mutations/updates/transactions should be equally flexible and generic, but I'm having trouble finding many good references. The closest thing I could find is https://github.com/redplanetlabs/specter navigators. Generally, most options for updating or inserting new facts seem less expressive, especially when updating info that refers to data that already exists in the datastore. I've looked at several different options, but compared to guides covering querying, the sections covering mutations/updates/transactions seem like the "then a miracle occurs" part of the documentation: https://docs.datomic.com/cloud/transactions/transaction-processing.html https://netflix.github.io/falcor/doc/DataSource.html#set https://relay.dev/docs/guided-tour/updating-data/graphql-mutations/ https://pathom3.wsscode.com/docs/mutations https://day8.github.io/re-frame/api-builtin-effects/#db https://github.com/mpdairy/posh#transact https://opencrux.com/reference/21.02-1.15.0/transactions.html https://github.com/cljfx/cljfx#event-handling-on-steroids https://book.fulcrologic.com/#Mutations https://edn-query-language.org/eql/1.0.0/specification.html#_mutations Are there any good resources or ideas I'm missing?
Asami seems to have a slightly different take on transactions than some of its datalog peers: https://github.com/threatgrid/asami/wiki/Transactions I'm not familiar with it enough to comment on how it compares in terms of expressivity though
I think that graphql and pathom mutations are more general than updating a datastore, although they are often used for this. they can be used for any kind of side-effecting procedure.
That's true. I most interested in just the datastore update part, but having a story around combining side effects with updates (eg. the coordination between Agent's send
and STM transactions) would be neat too.
Other than specter, I can't find many resources that have much documentation/examples on how to set, update, and compose changes.
The tricky thing is most of those query dsls are based on the semantics of triple stores, and a triple store is a kind of universal encoding
Even if the references only pertained to triple stores, that would be helpful
And in that model a transaction would be a set of assertions and retractions of triples
You can't just retract a single attribute, you have to retract the whole row or nothing
Right. Assertions and retractions seems kind of low level (especially compared to the query side). In theory, it seems like it should be possible to ā¢ given a specter path, a query, and a schema ā¢ automatically translate that into a transaction that targets any of the available datastores
like you can run datalog against any datastore, or just an in memory datastructure
it should be possible to write an update description that is as expressive as specter's navigators that would update any datastore (it already works on in memory data structures)
You will need to make lots of decisions on things like the above mentioned "what if I retract a triple on a sql store?" Is it an error if you don't retract everything else in the same row too? Or does it automatically imply they all get dropped? Or is a nop?
Are there any options that work just on triple stores?
Right, but afaict, the transaction interface is much less expressive than the query interface
Transaction functions(or whatever they call it) let you run arbitrary functions on the value of the database
right. Are there any tools/libraries that help with that?
To summarize, query feels very well documented (guides, APIs, examples) and expressive. The options for Updates/transactions seem less well documented and expressive.
But tightly coupled to a concrete data structure, where a relational/tuple store query language is not
This is just off the top of my head, so it's not a great example: Given several todo lists (work, home, hobby), toggle all work todo list items created more than a year ago. (Toggle would be logical negation, even though that use case doesn't make sense. I just wanted an example that updated an attribute using a function rather than setting the value) This is just an example, but I'm looking for something that is composable similar to the way queries are composable.
But when we use data in programs it has physical relationships (as things are connected in memory etc) as well
the logical basis for query languages is first order logic, the academic treatments of datalog use first order logic directly, and sql does its best
so you might look at something like dedalus https://www2.eecs.berkeley.edu/Pubs/TechRpts/2009/EECS-2009-173.html which explicitly models time
I'm not saying specter is the right answer, but what I like about it is that
1. paths can be reused for either query or transformation
2. complex paths can be composed of simpler paths.
Nothing about specter is tied to concrete data structures (it could work with dbs as well). It's more similar to lens in that you just need to implement select
and transform
. It seems like you should be able to reuse datalog, EQL, etc. for transformation and I was hoping to find an example.
can you provide an example?
so you have a transaction function that runs a query for the todo items, then asserts and retracts them
for a lot of the abstractions you listed, they really don't have transactions, which is why their updates suck, you can't do a read and expect things to be consistent with that read when you write
maybe I'm just missing something, but I can't find any examples that reify the transaction description. I found this todomvc (which is kind of old), https://github.com/madvas/todomvc-omnext-datomic-datascript/blob/master/src/cljc/todomvc/queries.cljc#L48
1. It's doesn't even do toggle, it reads and then writes in two separate transactions. 2. It's not composed of simpler pieces
one of the things that makes query languages like datalog, EQL, SQL etc. "worth it" are the fact that you can give them to another process or machine to run. you're essentially moving the compute of the result to where the data lives. datomic and datascript IME run under an assumption that whatever you're modifying can fit in memory, and to just tell them the facts that have changed - and then transaction functions fill in the gap when that's not the case.
maybe some other corners of tech that deal with writing/updating large (i.e. won't fit in memory) data would yield some more ideas?
Do you have an example that you would recommend?
https://docs.datomic.com/on-prem/reference/database-functions.html#transaction-functions has an example, but not of toggling
it shows the transaction function takes a db as an arg, and that the result of the function is a list of assertions and retractions
yea, SQL actually does have a way to combine the query with an update. I was assuming EQL, datalog, etc would also
I know that it's theoretically possible. I was hoping to find some references/guides/anything where someone is already doing that
We do have some code at work that treats a database row as an atom, it implements IAtom and does a compare and set in the database of the row contents
You want to express something like given some formula that is true for this version of the database this other formula must true in the next version
The first formula is used to query, and find all the triples that make it true, the second formula needs to generate a set of assertions and retractions that would make it true
And then you actually need some way to talk about different versions of a database (transactions)
There's a few different types of updates with increasing complexity 1. updating a specific attribute 2. updating multiple attributes on a single entity 3. updating an entity's attribute and the attribute of another entity related through a foreign key attribute 4. Updating several entities related that may be linked through attributes. The first 2 are fairly straightforward. Starting with 3), it gets more interesting
> And then you actually need some way to talk about different versions of a database (transactions) Most databases let you just put everything in a single transaction, right?
I think the insight is that the language needed to specify the kinds of updates you'd want to do is close to the level of clojure.core
and at that point, why not ship your database with clojure.core? hence, transaction functions
I agree. Are there examples of using EQL, GraphQL, datomic, datalog where the transactions updates are constructed of simpler pieces?
ok, for applications that use those query languages, do they compose their transactions of simple pieces?
will do
IME most graphQL and EQL services run some procedure on the backend that executes some SQL, or calls another service that updates some store
which makes sense on the whole. a lot of systems are built in a CQRS style and so updating a store might look nothing like querying it beyond the facade of a service call
If queries can be generic, flexible, and composable, is it the wrong question to ask if updates/requests/mutations can be structured to be generic, flexible, and composable?
I think that we generally accept more restrictions on how we express our reads than how we want to update something
I dunno, the whole thing is fraught because many of those tools are not just pass throughs, they can transform and merge data from multiple sources
We used to create REST apis for querying that were bespoke with the same justification. It seems at least plausible that specifying updates could have more structure.
like, EQL is strictly less expressive or powerful than datalog. but it meets a lot of use cases, and that lack of power is useful in many cases
if I were using a language that allowed me to declaratively express updates, does the costs in power outweigh the benefits?
I'm sure there's a better approach than starting from scratch and building a custom solution every time. Are there any good examples for handling the update side that let you start with more than low level mutations?
Graphql isn't even relational, and has some many odd bits, I am not even sure how you could infer how to call mutations to update data returned by a query
GraphQL gives you a result. Assuming you have a schema, the original query, and can state what changes you would like, it should be possible to generate a transaction
With something like datalog, a query (a formula) can be used both to search a database for a set of facts that make it true, or (this you see mostly in prolog or core.logic) it can be used to generate the set of facts that make it true
Right. I'm not saying it helps you do that. I'm saying it would be possible to write a library that allows that
I think so, you would also need either an update function or a new value. I think you might also need a schema to make sure you include unique ids.
@U7RJTCH6J how would you specify an update like, "For each user, capitalize the first letter of every part of a name except <blacklist words>"
In theory no, because the bindings from first query should fill in any unknowns in the second
I'm imagining some syntax like:
(update user-name-query capitalize-parts)
what remote server?
I guess that's not a req. then I'll ask slightly differently, how is capitalize-parts
implemented?
regular clojure code
#(->> % split (map capitalize-if-not-blacklisted) join)
it's really being able to specify which entities/attributes that I was looking for composability
it wold be something like (update <query> (fn [<bindings from query that includes names>] <some-new-query-that includes the names as capitalize>))
I don't understand the new-query part. How does that work?
wouldn't the function just return the names capitalized
but you can reverse that and say, from this statement in first order logic, generate the data that makes it true
so the first query is run "forwards" as a query to find the data you want to make changes to, the second query is run "backwards" to generate the changes you need to make
I guess I was thinking more like:
(update user-name-query (map capitalize-parts))
I think you also need some way of validating that the result of the query matches whatever the update fn expects
I guess I'm stuck in this thinking that, most of these query langs (EQL, GraphQL, datomic datalog) exist in systems that have a server-client relationship. often, GraphQL and EQL are exposed to the internet. so there's some adversarial thinking that goes into the design as well.
almost none of these things is strictly a first order logic langauge, so you may only be able to support subsets of them
I don't need that in my design
yeah, I think the question is flawed, in the way it conflates things that are not at all alike (databases and api middleware layers)
for api middleware layers, the fact that mutations are only exposed in an adhoc way is almost a feature
the other consideration in EQL and GraphQL are that often they are often layers in front of many different stores, and it takes special care to ensure the relative correctness of what is basically a distributed transaction š (which is basically what hiredman is saying)
I agree that receiving updates from an untrusted client requires more adversarial thinking. Building mutations from scratch doesn't seem like a bonus
It would be great to have a solution that is as flexible and generic as EQL, datalog, etc. I would settle for a solution that works locally with a trusted client with datascript
like what if the way that your system reads data is from a SQL store, but the way you write is by putting a bunch of messages in kafka?
then I'm in agreement that the documentation for datascript is woefully lacking š
If there's an example for any of crux, autonormal, datomic,etc, that would be a good start as well.
you could probably do something like this in datascript:
(defn transact-q! [conn q f]
(swap! conn (fn [db] (ds/db-with db (f (ds/q q db)))))
datascript also has transaction functions
where q
is some datalog and f
is some function that takes the result of the query and returns tx-data
yea, that's more or less the plan. I was hoping there would be some prior art that I could cheat off of
autonormal you could do basically the same. autonormal doesn't have the notion of a conn
, so you can handle the transactional guarantees however you like
but yeah, pathom and GraphQL aren't really meant to be fronts to a single store. autonormal is more of a utility library for reading, it doesn't deal with the whole time thing that hiredman was talking about
what I like about pathom is that it focuses on just specifying what data you need rather than how to get it. I still think it's possible to use something that's pathom like to say what changes you want without specifying how to run them.
in addition to queries being generic and composable
I don't need a generic way to handle multiple data stores, but I do plan on implementing a way to have some state in memory with an option to have some state stored more durably.
Interesting convo. I don't have much to add but I did think of another example of a query/transform library that might be useful as another point of reference: Odin https://github.com/halgari/odin#transforming-data
I think that as long as you can actually update all the data in one transaction, you can do that. even having two atoms, in a multithreaded context you could run into inconsistencies
the idea is that the atom would generally hold incidental, ephermeral, and generally less important state
Transaction functions(or whatever they call it) let you run arbitrary functions on the value of the database
https://docs.datomic.com/on-prem/reference/database-functions.html#transaction-functions has an example, but not of toggling