Fork me on GitHub
#off-topic
<
2021-03-21
>
emccue01:03:09

has anyone tried out the new foreign linker api?

emccue01:03:20

I'm having difficulties setting a callback function in a struct

emccue01:03:05

and i'm unsure if its because I am getting the address of the callback function wrong (it should be a symbol in the library) or something I am doing wrong with struct padding

phronmophobic20:03:24

There are several query DSLs (GraphQL, EQL, pull syntax, Datalog, meander, SQL, specter, etc) that are fairly generic, flexible, expressive, and independent of datastore (ie. crux, datascript, datomic, in memory). It seems like mutations/updates/transactions should be equally flexible and generic, but I'm having trouble finding many good references. The closest thing I could find is https://github.com/redplanetlabs/specter navigators. Generally, most options for updating or inserting new facts seem less expressive, especially when updating info that refers to data that already exists in the datastore. I've looked at several different options, but compared to guides covering querying, the sections covering mutations/updates/transactions seem like the "then a miracle occurs" part of the documentation: https://docs.datomic.com/cloud/transactions/transaction-processing.html https://netflix.github.io/falcor/doc/DataSource.html#set https://relay.dev/docs/guided-tour/updating-data/graphql-mutations/ https://pathom3.wsscode.com/docs/mutations https://day8.github.io/re-frame/api-builtin-effects/#db https://github.com/mpdairy/posh#transact https://opencrux.com/reference/21.02-1.15.0/transactions.html https://github.com/cljfx/cljfx#event-handling-on-steroids https://book.fulcrologic.com/#Mutations https://edn-query-language.org/eql/1.0.0/specification.html#_mutations Are there any good resources or ideas I'm missing?

jjttjj20:03:28

Asami seems to have a slightly different take on transactions than some of its datalog peers: https://github.com/threatgrid/asami/wiki/Transactions I'm not familiar with it enough to comment on how it compares in terms of expressivity though

šŸ‘€ 3
šŸ‘ 3
lilactown21:03:05

I think that graphql and pathom mutations are more general than updating a datastore, although they are often used for this. they can be used for any kind of side-effecting procedure.

phronmophobic21:03:25

That's true. I most interested in just the datastore update part, but having a story around combining side effects with updates (eg. the coordination between Agent's send and STM transactions) would be neat too. Other than specter, I can't find many resources that have much documentation/examples on how to set, update, and compose changes.

hiredman21:03:43

The tricky thing is most of those query dsls are based on the semantics of triple stores, and a triple store is a kind of universal encoding

hiredman21:03:19

So all sql tables can be represented sets of triples

phronmophobic21:03:34

Even if the references only pertained to triple stores, that would be helpful

hiredman21:03:39

And in that model a transaction would be a set of assertions and retractions of triples

hiredman21:03:33

But not all sets like that can be mapped to a sql table (for example)

hiredman21:03:00

You can't just retract a single attribute, you have to retract the whole row or nothing

phronmophobic21:03:21

Right. Assertions and retractions seems kind of low level (especially compared to the query side). In theory, it seems like it should be possible to ā€¢ given a specter path, a query, and a schema ā€¢ automatically translate that into a transaction that targets any of the available datastores

phronmophobic21:03:13

like you can run datalog against any datastore, or just an in memory datastructure

phronmophobic21:03:53

it should be possible to write an update description that is as expressive as specter's navigators that would update any datastore (it already works on in memory data structures)

hiredman22:03:45

You will need to make lots of decisions on things like the above mentioned "what if I retract a triple on a sql store?" Is it an error if you don't retract everything else in the same row too? Or does it automatically imply they all get dropped? Or is a nop?

phronmophobic22:03:06

Are there any options that work just on triple stores?

hiredman22:03:08

And as you make decisions about that it makes your approach less universal

hiredman22:03:17

Datomic is a triple store

phronmophobic22:03:56

Right, but afaict, the transaction interface is much less expressive than the query interface

hiredman22:03:00

Transaction functions(or whatever they call it) let you run arbitrary functions on the value of the database

hiredman22:03:29

They just have to bottom out/expand into a set of assertions and retractions

phronmophobic22:03:01

right. Are there any tools/libraries that help with that?

hiredman22:03:50

I would really not go with spectres navigators as an example

hiredman22:03:51

Like, basically a navigator is just a database query

phronmophobic22:03:55

To summarize, query feels very well documented (guides, APIs, examples) and expressive. The options for Updates/transactions seem less well documented and expressive.

lilactown22:03:42

what else would you like to express?

hiredman22:03:03

But tightly coupled to a concrete data structure, where a relational/tuple store query language is not

phronmophobic22:03:02

This is just off the top of my head, so it's not a great example: Given several todo lists (work, home, hobby), toggle all work todo list items created more than a year ago. (Toggle would be logical negation, even though that use case doesn't make sense. I just wanted an example that updated an attribute using a function rather than setting the value) This is just an example, but I'm looking for something that is composable similar to the way queries are composable.

hiredman22:03:19

Data has logical relationships with other data

hiredman22:03:25

But when we use data in programs it has physical relationships (as things are connected in memory etc) as well

hiredman22:03:44

And spectre as a tool is way to into physical relationships

hiredman22:03:49

the logical basis for query languages is first order logic, the academic treatments of datalog use first order logic directly, and sql does its best

hiredman22:03:47

the problem is logic is atemporal while update/assignment/etc requires time

hiredman22:03:52

so you might look at something like dedalus https://www2.eecs.berkeley.edu/Pubs/TechRpts/2009/EECS-2009-173.html which explicitly models time

phronmophobic22:03:36

I'm not saying specter is the right answer, but what I like about it is that 1. paths can be reused for either query or transformation 2. complex paths can be composed of simpler paths. Nothing about specter is tied to concrete data structures (it could work with dbs as well). It's more similar to lens in that you just need to implement select and transform. It seems like you should be able to reuse datalog, EQL, etc. for transformation and I was hoping to find an example.

hiredman22:03:57

but that is already the case

phronmophobic22:03:11

can you provide an example?

hiredman22:03:18

a datomic query is the "path"

hiredman22:03:04

so you have a transaction function that runs a query for the todo items, then asserts and retracts them

hiredman22:03:28

for a lot of the abstractions you listed, they really don't have transactions, which is why their updates suck, you can't do a read and expect things to be consistent with that read when you write

phronmophobic22:03:51

maybe I'm just missing something, but I can't find any examples that reify the transaction description. I found this todomvc (which is kind of old), https://github.com/madvas/todomvc-omnext-datomic-datascript/blob/master/src/cljc/todomvc/queries.cljc#L48

phronmophobic22:03:52

1. It's doesn't even do toggle, it reads and then writes in two separate transactions. 2. It's not composed of simpler pieces

lilactown22:03:01

one of the things that makes query languages like datalog, EQL, SQL etc. "worth it" are the fact that you can give them to another process or machine to run. you're essentially moving the compute of the result to where the data lives. datomic and datascript IME run under an assumption that whatever you're modifying can fit in memory, and to just tell them the facts that have changed - and then transaction functions fill in the gap when that's not the case.

hiredman22:03:39

yeah, I mean, random code on github is terrible šŸ™‚

lilactown22:03:16

maybe some other corners of tech that deal with writing/updating large (i.e. won't fit in memory) data would yield some more ideas?

phronmophobic22:03:29

Do you have an example that you would recommend?

hiredman22:03:46

how do I keep hitting that checkbox

3
lilactown22:03:20

I mean, first thing that comes to mind is SQL šŸ˜›

hiredman22:03:33

it shows the transaction function takes a db as an arg, and that the result of the function is a list of assertions and retractions

phronmophobic22:03:06

yea, SQL actually does have a way to combine the query with an update. I was assuming EQL, datalog, etc would also

hiredman22:03:12

the db that the tx function takes can be queried etc

phronmophobic22:03:38

I know that it's theoretically possible. I was hoping to find some references/guides/anything where someone is already doing that

hiredman22:03:37

Dunno, I just know that exists from watching datomic talks, I've never used datomic

hiredman22:03:11

We do have some code at work that treats a database row as an atom, it implements IAtom and does a compare and set in the database of the row contents

šŸ‘€ 3
hiredman22:03:30

Which is a similar kind of thing in the small

hiredman22:03:33

You want to express something like given some formula that is true for this version of the database this other formula must true in the next version

hiredman22:03:21

The first formula is used to query, and find all the triples that make it true, the second formula needs to generate a set of assertions and retractions that would make it true

hiredman22:03:52

And then you actually need some way to talk about different versions of a database (transactions)

phronmophobic22:03:26

There's a few different types of updates with increasing complexity 1. updating a specific attribute 2. updating multiple attributes on a single entity 3. updating an entity's attribute and the attribute of another entity related through a foreign key attribute 4. Updating several entities related that may be linked through attributes. The first 2 are fairly straightforward. Starting with 3), it gets more interesting

phronmophobic22:03:54

> And then you actually need some way to talk about different versions of a database (transactions) Most databases let you just put everything in a single transaction, right?

lilactown22:03:12

I think the insight is that the language needed to specify the kinds of updates you'd want to do is close to the level of clojure.core

hiredman22:03:40

All of those updates boil down to the same thing

lilactown22:03:55

and at that point, why not ship your database with clojure.core? hence, transaction functions

hiredman22:03:07

Assertions and retractions of triples

hiredman22:03:49

It depends, but I believe most of those query dsls things can run against anything

phronmophobic22:03:50

I agree. Are there examples of using EQL, GraphQL, datomic, datalog where the transactions updates are constructed of simpler pieces?

lilactown22:03:33

EQL and graphQL are not concerned with updating a store

lilactown22:03:49

their "mutations" are essentially function calls

phronmophobic22:03:29

ok, for applications that use those query languages, do they compose their transactions of simple pieces?

lilactown22:03:49

not from the client perspective, anyway

hiredman22:03:59

I dunno, maybe ask in #datomic

lilactown23:03:39

IME most graphQL and EQL services run some procedure on the backend that executes some SQL, or calls another service that updates some store

hiredman23:03:17

Yeah graphql very much does not have transactions

hiredman23:03:29

Mutations are the wild west

lilactown23:03:38

which makes sense on the whole. a lot of systems are built in a CQRS style and so updating a store might look nothing like querying it beyond the facade of a service call

phronmophobic23:03:41

If queries can be generic, flexible, and composable, is it the wrong question to ask if updates/requests/mutations can be structured to be generic, flexible, and composable?

lilactown23:03:59

I think that we generally accept more restrictions on how we express our reads than how we want to update something

hiredman23:03:23

I dunno, the whole thing is fraught because many of those tools are not just pass throughs, they can transform and merge data from multiple sources

hiredman23:03:46

It is hard enough doing transactional updates against a single store

phronmophobic23:03:55

We used to create REST apis for querying that were bespoke with the same justification. It seems at least plausible that specifying updates could have more structure.

lilactown23:03:25

like, EQL is strictly less expressive or powerful than datalog. but it meets a lot of use cases, and that lack of power is useful in many cases

lilactown23:03:39

if I were using a language that allowed me to declaratively express updates, does the costs in power outweigh the benefits?

hiredman23:03:51

I think it could be possible with a good transactional relational store

hiredman23:03:12

But almost all of your links are not one of those

hiredman23:03:43

They are middleware for presenting a particular style of query api

phronmophobic23:03:41

I'm sure there's a better approach than starting from scratch and building a custom solution every time. Are there any good examples for handling the update side that let you start with more than low level mutations?

hiredman23:03:50

Graphql isn't even relational, and has some many odd bits, I am not even sure how you could infer how to call mutations to update data returned by a query

phronmophobic23:03:25

GraphQL gives you a result. Assuming you have a schema, the original query, and can state what changes you would like, it should be possible to generate a transaction

hiredman23:03:07

With something like datalog, a query (a formula) can be used both to search a database for a set of facts that make it true, or (this you see mostly in prolog or core.logic) it can be used to generate the set of facts that make it true

hiredman23:03:24

Nah, graphql doesn't work like that

hiredman23:03:45

A mutation is basically an arbitrarily named rpc

phronmophobic23:03:20

Right. I'm not saying it helps you do that. I'm saying it would be possible to write a library that allows that

hiredman23:03:43

So an update can in theory be a pair of formula (queries),

hiredman23:03:20

One query to find facts that match another used to generate changes to be asserted

phronmophobic23:03:00

I think so, you would also need either an update function or a new value. I think you might also need a schema to make sure you include unique ids.

lilactown23:03:28

@U7RJTCH6J how would you specify an update like, "For each user, capitalize the first letter of every part of a name except <blacklist words>"

hiredman23:03:08

In theory no, because the bindings from first query should fill in any unknowns in the second

phronmophobic23:03:41

I'm imagining some syntax like: (update user-name-query capitalize-parts)

lilactown23:03:11

how would the remote server know what capitalize-parts is?

phronmophobic23:03:34

what remote server?

lilactown23:03:07

I guess that's not a req. then I'll ask slightly differently, how is capitalize-parts implemented?

phronmophobic23:03:25

regular clojure code

phronmophobic23:03:00

#(->> % split (map capitalize-if-not-blacklisted) join)

lilactown23:03:30

ah. i thought you were after something more declarative / composable

phronmophobic23:03:19

it's really being able to specify which entities/attributes that I was looking for composability

hiredman23:03:28

it wold be something like (update <query> (fn [<bindings from query that includes names>] <some-new-query-that includes the names as capitalize>))

phronmophobic23:03:18

I don't understand the new-query part. How does that work?

hiredman23:03:39

a query in datalog is a statement is first order logic

phronmophobic23:03:43

wouldn't the function just return the names capitalized

hiredman23:03:56

the result of the query is the set of data that makes that formula true

hiredman23:03:34

but you can reverse that and say, from this statement in first order logic, generate the data that makes it true

hiredman23:03:21

so the first query is run "forwards" as a query to find the data you want to make changes to, the second query is run "backwards" to generate the changes you need to make

phronmophobic23:03:31

I guess I was thinking more like:

(update user-name-query (map capitalize-parts))

hiredman23:03:02

maybe, it depends a lot on the structure of your results

lilactown23:03:09

I think you also need some way of validating that the result of the query matches whatever the update fn expects

hiredman23:03:25

a query is self validating

lilactown23:03:42

an update is destructive. invalid updates could trash your data

hiredman23:03:42

e.g. if the result doesn't match it is a bug in your query, so fix it

hiredman23:03:00

same with update-in

lilactown23:03:08

sure, I'm talking more operationally

lilactown23:03:15

I guess I'm stuck in this thinking that, most of these query langs (EQL, GraphQL, datomic datalog) exist in systems that have a server-client relationship. often, GraphQL and EQL are exposed to the internet. so there's some adversarial thinking that goes into the design as well.

lilactown23:03:45

maybe phronmophobic doesn't need that in his design

hiredman23:03:53

almost none of these things is strictly a first order logic langauge, so you may only be able to support subsets of them

phronmophobic23:03:54

I don't need that in my design

hiredman23:03:40

yeah, I think the question is flawed, in the way it conflates things that are not at all alike (databases and api middleware layers)

hiredman23:03:06

for api middleware layers, the fact that mutations are only exposed in an adhoc way is almost a feature

šŸ’Æ 3
lilactown23:03:22

the other consideration in EQL and GraphQL are that often they are often layers in front of many different stores, and it takes special care to ensure the relative correctness of what is basically a distributed transaction šŸ˜… (which is basically what hiredman is saying)

phronmophobic23:03:54

I agree that receiving updates from an untrusted client requires more adversarial thinking. Building mutations from scratch doesn't seem like a bonus

phronmophobic23:03:28

It would be great to have a solution that is as flexible and generic as EQL, datalog, etc. I would settle for a solution that works locally with a trusted client with datascript

lilactown23:03:38

like what if the way that your system reads data is from a SQL store, but the way you write is by putting a bunch of messages in kafka?

lilactown23:03:00

then I'm in agreement that the documentation for datascript is woefully lacking šŸ˜›

phronmophobic23:03:08

If there's an example for any of crux, autonormal, datomic,etc, that would be a good start as well.

lilactown23:03:35

you could probably do something like this in datascript:

(defn transact-q! [conn q f]
  (swap! conn (fn [db] (ds/db-with db (f (ds/q q db)))))

phronmophobic23:03:13

datascript also has transaction functions

lilactown23:03:23

where q is some datalog and f is some function that takes the result of the query and returns tx-data

lilactown23:03:47

ah I tried googling if datascript had arbitrary tx fns and couldn't find it

phronmophobic23:03:54

yea, that's more or less the plan. I was hoping there would be some prior art that I could cheat off of

lilactown23:03:32

autonormal you could do basically the same. autonormal doesn't have the notion of a conn, so you can handle the transactional guarantees however you like

lilactown23:03:48

but yeah, pathom and GraphQL aren't really meant to be fronts to a single store. autonormal is more of a utility library for reading, it doesn't deal with the whole time thing that hiredman was talking about

phronmophobic23:03:41

what I like about pathom is that it focuses on just specifying what data you need rather than how to get it. I still think it's possible to use something that's pathom like to say what changes you want without specifying how to run them.

phronmophobic23:03:08

in addition to queries being generic and composable

phronmophobic23:03:22

I don't need a generic way to handle multiple data stores, but I do plan on implementing a way to have some state in memory with an option to have some state stored more durably.

jjttjj23:03:44

Interesting convo. I don't have much to add but I did think of another example of a query/transform library that might be useful as another point of reference: Odin https://github.com/halgari/odin#transforming-data

šŸ™Œ 3
lilactown02:03:49

I think that as long as you can actually update all the data in one transaction, you can do that. even having two atoms, in a multithreaded context you could run into inconsistencies

phronmophobic02:03:47

the idea is that the atom would generally hold incidental, ephermeral, and generally less important state

hiredman22:03:00
replied to a thread:There are several query DSLs (GraphQL, EQL, pull syntax, Datalog, meander, SQL, specter, etc) that are fairly generic, flexible, expressive, and independent of datastore (ie. crux, datascript, datomic, in memory). It seems like mutations/updates/transactions should be equally flexible and generic, but I'm having trouble finding many good references. The closest thing I could find is https://github.com/redplanetlabs/specter navigators. Generally, most options for updating or inserting new facts seem less expressive, especially when updating info that refers to data that already exists in the datastore. I've looked at several different options, but compared to guides covering querying, the sections covering mutations/updates/transactions seem like the "then a miracle occurs" part of the documentation: https://docs.datomic.com/cloud/transactions/transaction-processing.html https://netflix.github.io/falcor/doc/DataSource.html#set https://relay.dev/docs/guided-tour/updating-data/graphql-mutations/ https://pathom3.wsscode.com/docs/mutations https://day8.github.io/re-frame/api-builtin-effects/#db https://github.com/mpdairy/posh#transact https://opencrux.com/reference/21.02-1.15.0/transactions.html https://github.com/cljfx/cljfx#event-handling-on-steroids https://book.fulcrologic.com/#Mutations https://edn-query-language.org/eql/1.0.0/specification.html#_mutations Are there any good resources or ideas I'm missing?

Transaction functions(or whatever they call it) let you run arbitrary functions on the value of the database

hiredman22:03:36
replied to a thread:There are several query DSLs (GraphQL, EQL, pull syntax, Datalog, meander, SQL, specter, etc) that are fairly generic, flexible, expressive, and independent of datastore (ie. crux, datascript, datomic, in memory). It seems like mutations/updates/transactions should be equally flexible and generic, but I'm having trouble finding many good references. The closest thing I could find is https://github.com/redplanetlabs/specter navigators. Generally, most options for updating or inserting new facts seem less expressive, especially when updating info that refers to data that already exists in the datastore. I've looked at several different options, but compared to guides covering querying, the sections covering mutations/updates/transactions seem like the "then a miracle occurs" part of the documentation: https://docs.datomic.com/cloud/transactions/transaction-processing.html https://netflix.github.io/falcor/doc/DataSource.html#set https://relay.dev/docs/guided-tour/updating-data/graphql-mutations/ https://pathom3.wsscode.com/docs/mutations https://day8.github.io/re-frame/api-builtin-effects/#db https://github.com/mpdairy/posh#transact https://opencrux.com/reference/21.02-1.15.0/transactions.html https://github.com/cljfx/cljfx#event-handling-on-steroids https://book.fulcrologic.com/#Mutations https://edn-query-language.org/eql/1.0.0/specification.html#_mutations Are there any good resources or ideas I'm missing?

https://docs.datomic.com/on-prem/reference/database-functions.html#transaction-functions has an example, but not of toggling