Fork me on GitHub
Drew Verlee15:03:47

Why use graphQL to service a single database over something closer to the database's native query language? E.g Why use Graphql to communicate with Datomic? I know this question is very open-ended and not well-defined. Feel free to take it in whatever direction you want!


Abstraction and limitation, which is especially helpful if you have many clients. In case you need to make changes to your DB in a way that affects queries, it's better to change a single GraphQL->DB_query layer than to change every client or add a non-trivial DB_query->DB_query layer. And of course you can replace "GraphQL" with any other querying mechanism.

👍 2

There is a lot of “mainstream” frontend tooling around GraphQL. Datalog can be a hard sell to frontend Javascript engineers.

👍 2

I don't think there is a reason if the API is just for internal consumption, and you are not in a massive org

👍 4

GraphQL can be limiting because everything has to fit into the inflexible type system.


I always thought the point of graphql is to have a unified API over heterogenous data sources that are spread over multiple backends.

👍 4

We used GraphQL for one app and, overall, I don't think we'd use it again for future apps. It's kind of a PITA with its structure and types and so on...


Also it has syntax.


it's unfortunate to put custom syntax on top of something that talks data.


Yea. For internal use, we just made something similar that just uses JSON.


If you do need a query abstraction then maybe have a look at pathom instead: It uses EQL (like datomic pull or spec2 select), so it's data driven. Might be worth checking out.


Yeah, I think if we really needed to surface a query abstraction, as opposed to just an API, we might look at

👀 4

GraphQL can allow the consumer of the data to have more control over the data they received This does put more of the responsibility on the consumer of that data to understand the value of that data. GraphQL is a very convenient way of exposing your data to others without knowing (or possibly limiting) what they want to do with it. Ultimately GraphQL is just another design choice. It can be put in front of one or more data stores what ever technology they are (so long as there are resolvers for the queries). GraphQL is often used to complement specific REST queries which tend to have a more specific design (hopefully) I've worked on two commercial projects with GraphQL and Clojure, like anything, if it's used well it's great, if not used appropriately then less so.


It feels like SOAP to me... 🙂

👀 2

SOAP and ebXML. Party like it's 1999 onwards 😂


IMO graphql was mindlessly adopted without much inspection or questioning, just like other trends clojure being so powerful and data-driven makes authoring bespoke endpoints (e.g. for a specific frontend use case) simple and easy. when considering that alternative, gql just gratuitously adds complexity and risks e.g. you wanted to give frontend devs "more agility", next month you're debugging performance problems or wondering about possible security/DoS holes.

💯 2

I’ve had the same question and eventually implemented a stack using pure clojure data structure to handle it. The stack “mounts” databases and other components into a single big lazy map which we call a “system”. When a request comes in, the newly gained information from the request will be “merged” into the “system” map. The resulting map then “matches” to a pattern contained in the request and returns only the information specified in that pattern.


The lazy map part of the stack has been released as the library.

👀 2

I mean, at a high level, it's like, why not just have the client pass a SQL query to the server? And the answer tends to be that you want a bit more safeguards, like some permission model on top, maybe there's more than one database and need to query over them all, maybe you don't want the client to handcraft SQL and possibly run some slow queries, etc. So an intermediate query language was added, aka, GraphQL

👀 2

In terms of permissions/auth: Correct me if I'm wrong, I'm not familiar with datomic. But I think you can derive a database (like a sql view) in some way, such that it only contains stuff that a given client should be able to read. Whenever you have different views on data with their own restricted access then you do something like this anyways. Or at least that's how I typically do it. Every query to a db like that is either wrapped in some authorization filter query or looks at a view that does the same. In terms of performance: A graphql layer that is sufficiently expressive will have a similar runtime complexity, plus some overhead right? How does it protect the client from slow queries if it can ask complex questions? The only way you can protect a client from certain slow queries and patterns is by restricting the expressivity of the query layer. But graphql isn't uniquely suited for this. You can again, provide a smaller, simpler view of things that can be looked at by the client without graphql.

Cora (she/her)14:03:33

graphql wins: 1. lots of client libraries across many languages 2. only pull the data your frontend needs 3. can wrap multiple backend services into a single, regular interface that frontends can consume 4. frontends can change what they pull without having to change the backend, in many cases lots of these wins amount to organizational wins that reduce the need for coordination and communication across teams, specifically you can have multiple frontend teams constantly changing the data they pull without having to consult with or need work from backend teams. if you don't have a need for that then it will seem like less of a win.

Cora (she/her)14:03:31

the technical differences between what's in graphql and what equivalent you could replace it with where you send a query to the server and have it execute it are pretty minimal, imo.

Cora (she/her)14:03:21

the big wins are in the organization, in how you organize work, and those are extremely important concerns


Conway's Law strikes again! ;)


Having to name every single intermediate object in GraphQL is definitely a pain.


Otoh, automatic validation of queries, mutations, their arguments and the returned data is very nice.


I think the same reasons architecturally you would pick pathom are the same you would pick graphQL. The other concerns (syntax & type system, adoption & tooling) are how you would compare the two IMO.


datalog & sql are similarly in a different problem space and you could compare the two similarly

🙏 2

@U02N27RK69K Only #1 and #3 are not deliverable by taking datalog queries from the frontend though. And #1 is kind of a self-fulfilling prophecy, when GraphQL was introduced, only #3 would have been unique to it as opposed to someone saying we should just take some datalog query as input from the frontend.

Cora (she/her)03:03:53

Im not evaluating these from the standpoint of "what if these were both introduced now", just comparing technology, I'm evaluating them in context. 1 and 3 are important, having existing technology is useful


But #3 is a lot more important than it looks, because like you said, different teams might already be using different databases, not all support the same datalog query language or share the same unified data-model. #3 is the real reason for GraphQL becoming popular.


Well, I kind of disagree though. If having existing technology was important, GraphQL would lose, because it was all brand new and different and had no libraries initially. Clearly it has unique properties that were so beneficial, that it didn't matter if it had no support anywhere.

Cora (she/her)03:03:13

the question is why use it now not why use it when it was first introduced, though

Cora (she/her)03:03:00

if the question was why did it become popular initially then sure, 3 is a heavy hitter


If you use it now "just cause it's popular", most likely it doesn't matter what you use, because you have none of the problems that justify using it. If you had the problems that justified using it, popularity wouldn't matter, because you'd have to use it to solve those problems.

Cora (she/her)03:03:19

I didn't say just because it's popular? it has popular support, which is important when building multiple frontends to the same data


Like, if you need #3, well you need #3, and datalog won't work, SQL won't work, and at this point, sure, reinventing a GraphQL-like might not be ideal given the tooling and existing know-how around it.

Cora (she/her)03:03:50

if you're building a clojure frontend to a clojure backend that calculus changes quite a bit

Cora (she/her)03:03:57

sure, 3 is important, that's why I listed it


Right, I just wanted to point out that #1 is kind of a beside the point. Like #1 is not a defeating argument, because if you need #3, solutions that deliver #1 but not #3 just aren't good enough. Now between two things that solve #3, sure, #1 can be the winning factor.

Cora (she/her)03:03:07

1 is important because writing quality libraries is non-trivial, writing them for multiple platforms is a huge pain

Cora (she/her)03:03:30

ahh I see what you mean

Cora (she/her)03:03:46

3 is pretty unique

💯 2

Exactly! I agree with all your points, but the essence of why GraphQL is good is #3, it's the differentiating factor, and it's why it became popular in the first place, and why people built support for it on multiple platforms and languages.

Cora (she/her)03:03:37

I mean it bridges multiple frontends and backends with a regular interface without teams needing to coordinate, it's not surprising this is a facebook technology given their systems

💯 2
Cora (she/her)03:03:26

I even enjoy using hasura with postgres for read-heavy apps because of how convenient it is to not have to iterate on the frontend and the backend at the same time. basic data querying over the web feels like it should be solved turnkey-like at this point and hasura feels pretty close to that

Cora (she/her)03:03:01

(for SPAs, I mean)


So if you've already got a bunch of different backend APIs that return data in some format. All use different databases. Some even return derived data that doesn't exist in the database but is computed in the backend. Or some pull from external services. Etc. Or if you don't want to force all team to use Datomic, and align on some single shared Datomic instance, or at least a single shared data model. Now you've got no choice but to use GraphQL, or to have a new backend API for every frontend query that under the hood fans-out to all these other backbends/DBs. Prior to GraphQL, there was nothing that could do this where the frontend can write arbitrary queries without needing a new backend API for each one. Now there is technically EQL in the Clojure world, but now that does become a question of like, okay but EQL is less known, not as well supported, might lack some features, etc. And that's because GraphQL was first/had Facebook fame

Cora (she/her)03:03:50

yeah whatever replaces it would need to be significantly better

💯 2

Now if I come back to OPs question, it's why use GraphQL for a frontend serving a single database. And that I would say, there's not any real reason. You could say... Why not? But you don't have a problem that necessitates the use of GraphQL, so it doesn't really matter if you use it or not in OPs context.

Cora (she/her)03:03:30

ah, right, I was reading that as sending queries over the web to that database. I don't want to accept sql from someone's browser

Cora (she/her)03:03:54

(db administration interfaces aside)


Ya, that's fair. Though now that I think about it, why don't we do that? Is it a security thing? Like why not have the browser client just send a SQL parameterized string + a map of parameters ?

Cora (she/her)03:03:08

because parsing and then validating sql? definitely a security thing

Cora (she/her)03:03:49

the crazy sql injection tricks people would come up with if you straight-up accepted sql? oof

Cora (she/her)04:03:02

that may be another benefit of graphql, a safe(r) query language that you can put controls on


Ya, that's what I'm thinking as well, but I wonder if what dgb23 mentioned than could mitigate that. I also don't know exactly why GraphQL is safer? We never found any GraphQL injection attacks? I guess also SQL is annoying because everything is relational, and I feel frontend tend to want hierarchical data, hance the "graph" in GraphQL. But Datomic would provide that as well.


Seems there's a ton of them actually :rolling_on_the_floor_laughing:


But SQL injection risks are extremely well-known and it's "easy" to protect yourself from them. We seeing thousands of attempted SQL injection attacks every day -- but they're all easily blocked. That's a completely different matter from accepting SQL queries from the frontend (OMG -- that terrifies me!).


Having re-read this thread a few times now -- the joys of long-lived discussions -- yes, totally agree that if you already have a giant mess of disparate systems you've allowed to grow up independently (cough, Facebook) then GraphQL looks really attractive. But if you've been paying attention and actually had standards and a modicum of control, then GQL is a corporate monster in search of a problem.


Do keep in mind also how GraphQL allows you to adopt it by reading a spec, and all the decisions are made for you already. So very low cognitive overhead to get started.


Also, for me a +1 of GraphQL is that it's not RESTful -> you don't have to think in terms of resources, verbs etc and get into arguments about them -> you just create an API and you only think about names.


an interesting pattern that has been adopted by some GraphQL libraries is, at compile time, hash the query and only send the hash to the backend. This way you can iterate on the query in your front end code, but your backend is not actually executing arbitrary GraphQL queries. This improves caching, security and allows backend teams to optimize those query executions over time. the interesting thing is you could do the same with SQL and datalog...


also just going to say +100 to what @U7PBP4UVA is saying 😄 less decisions is better.


We are definitely optimising queries with a specific use case in mind. We haven't done yet the hashing thing, but we should; in a mostly-private schema, you shouldn't be accepting any old random query at runtime.

Drew Verlee16:03:50

@U7PBP4UVA elaborate on why it's a pain (see quote below) to name intermediate objects in a graphQL (query)? Can't parts of the tree traversal be encoded and shared so that the devs can focus on a more local context? > Having to name every single intermediate object in GraphQL is definitely a pain.


An example (if I guessed what he meant correctly):


That isn't even a complete list for what is needed to create an 'issues' connection field.


What @U08JKUHA9 said. You have to come with names like FooConnection, FooResult, FooField, FooFieldWithValue etc etc. This only comes into play when defining the schema, when writing the queries you only use the field names.


The “connection” concept is arguably the best thing about GraphQL (I know it’s not really part of GraphQL per se). Baking pagination, sorting, and filtering into every cardinality-many reference at every level in your API is so important, and well worth the ceremony IMO. Yes, you could build a REST API that handled pagination as well and as consistently--but, in practice, people don’t. And then you end up with really intractable performance problems as your data scale.


Yea the concept is good, just a little verbose. You could imagine having a malli spec for a similar API that would be more concise and have fewer names without losing anything, for example. Or even a TypeScript type, since it allows more inlining.


Late for the discussion, but interested in the topic, (I am new to Clojure), I read with pleasure Roy T. Fielding's dissertation. GQL takes you in a direction very different from REST. The discoverability of GQL is achieved in REST via hypermedia, so, you are not supposed to have any kind of info out of band, that's why browsers work beforehand without any previous knowledge about your web app. Data queries are already included by design with the introduction of URI and specifically query-strings, so you should be able to translate a query-string into any underlying data query without difficulty. As for the aggregate API "feature" is also included by design in REST, you are supposed to introduce a new resource which not necessarily map to a single resource under the hood, back is supposed to be able to proxy in order to compose making use of proxy caches via ETag header and finally, resource identity is where specific back implementation fits in when out of the box CRUDs aren't enough. Regarding the "feature" of reducing payload by instructing the server to send specific data, that doesn't seems to be too difficult to achieve in Clojure where everything is a map.


Here a lib I have been playing with on translating uri into a db query using honey.sql

Drew Verlee22:06:53

@U03S5ALURGQ My contention is simply that if your client is primarily backed by one data source, then having the client communicate directly in the language of that data source is preferable to introducing a layer of indirection. As an example, Datomic has a tree pull syntax, and It's unclear why i would need an additional layer. (it's confusing to me that graphQL uses the word graph, but as far as i can tell doesn't allow for graqph queries, only tree queries, or i'm i missing something?) That library is interesting, whats the advantage of putting the query in the URL? I'm fairly sure the answer is "because the URL is a key to the state of the application which includes the query" but i might be missing something.

Drew Verlee22:06:18

But, yea, what that library is doing is more or less what I advocate for, but it's not a popular idea :thinking_face:


In REST, clients are not supposed to know any implementation details about the data source, I mean clients are supposed to support mime-types and behave according to the content type the server sends. I know SPA hasn't helped with the understanding of this idea and I hope in the near future clients move to mime-type based approach instead of fat-clients like SPA. The approach of sending the query to the server looks to me like an approach where the client knows too much about the backend and the implementation details, more like a RPC.

Drew Verlee23:06:24

@U03S5ALURGQ Datomic datalog and can be backed by multiple data sources (postgres, dynamodb), it's already an abstraction layer. The same is true for most variants of SQ (dynamoDB has a sql implementation i'm sure...). They both reveal the same implementation details that a standard rest API would.

Drew Verlee23:06:20

I think having the FE use the data source language is rare because the FE community isn't often tasked with pulling the data all the way to the client and there are security issues that might be hard to overcome if your not dealing with a query language that isn't captured inside a data structure that can be easily modified.

Drew Verlee23:06:51

Michael Drogalis did a good job presenting some of these ideas in a keynote about designing with data

Drew Verlee23:06:54

i'm not sure i understand what it would mean to move towards a "mime-type" based approach. What's the overlap with mime types, and the query language? The mime-type could tell the server the request was (for example) sql, but i think that's a small gain, as the endpoint the client was using will already have to be configured to handle the request in far more detail. So if the mime-type was just "text" it would work, though maybe it would be better if the protocol was more specific?


@U0DJ4T5U1 the mime-type almost all of us use today is general purpose, for instance. application/json, text/html, text/yaml, etc. None of those ones is domain-specific, this is the reason a tailor-made app should be created for each specific web API. Using hl7-fhir as an example of a well-defined standard, a new mime-type, let's say: application/hl7-fhir may be registered in IANA. So anyone can create a fhir generic client/browser capable to interact and understand Content-Type: application/hl7-fhir. As part of the mime-type, this client also knows about the supported query parameters because this is part of the mime-type specification, just like it happens with HTML where the browser has support for tags and attributes. That way both, the client and server can implement support for search via query parameters and serve data as specified in the mime-type. That's what makes me believe mime-type is a way of answering questions about queries over REST. Hopefully, mime-types should be created for each specific domain, and generic domain-specific clients/browsers can exist based on mime-type and hypermedia. By having such a client app, it doesn't seem to be a good idea to have any server-implementation-specific query but use plain query-string instead.


Thank you for sharing, I am already watching it