Clojurians Log v2

I usually use the DB schema as the source of truth. Everything else flows from it. And even if something needs to be changed in the DB without affecting the domain model (or the inverse - the data model should be changed but the DB should not be affected), it can be done with views relatively pain-free. And on the Clojure side it's just maps everywhere. As you said, specs/schemas are only for APIs and they don't get into the nitty-gritty of every single model (although they could, you can even generate them at run time with DB introspection). I don't hide the fact that an ID is an integer or a UUID, that model A refers to model B via b_id, and so on - no magic, everything is a direct translation to/from what the DB needs/provides.

Agreed: just hash maps for domain models. I prefer qualified keys, and will map between the domain model keys and the database keys if necessary but there's usually nothing wrong with just using the raw results from next.jdbc and treating the db rows as your domain model in a lot of situations.

That's an interesting take. So you both place the DB with it's schema at the center of your architecture? Like a 1:1 mapping? And if so: what's left programattically for Clojure to do in your projects? It seems most data wrangling than could be done very efficiently with modern SQL by the DB itself...

Yes, 1:1 mapping. An occasional view in between when required. Some thin layer around the DB to select/update just the "public" keys and not touch/return any "service" keys. A practical example - a column that's a denormalized variation of a few other columns that cannot be moved to a view due to the complexity of the logic behind it, so I use Clojure code to populate that column, and it gets updated automatically via notifications and a background service; it's used only for a specific version of a full-text search - never for display or changes. Unlike Sean, I don't like using fully qualified keywords for my data. I always maintain the structure of the data, never merge maps of different nature together, and I find FQN keywords mostly annoying to work with. Again, a practical example - 10 different entities that are of different nature but all have a name field. In all cases, it's the same exact thing from every aspect. I tried using FQN keywords, and that led to a proliferation of :model1/name, :model2/name, etc. keywords. With a simple :name field I can have all the name-related code have a simple universal API. With FQN fields, I must either create 10 interfaces or pass around the actual names of the name field, which cannot be done in a neat way. > what's left programattically for Clojure to do in your projects? Do you have all day? :) Not all projects are the simplest CRUDs. And even the simplest CRUD, in my experience, eventually might have a day in its life where someone says "it needs this little feature", and that feature ends up being impossible to implement at the DB layer. So, anything that's not related to the data in the DB. Or anything that would be too much for any language available to the DB (like plpgsql) to handle - either due to the limits of the language itself or due to poor developer experience. Just some specific things that I can think of without even looking at any of my code: • Anything HTTP-related - this is a huge swath of functionality on its own • Anything that could in theory be handled by the DB in general but cannot be handled in practice due to some limitations of some third-party DB services • Files in general • l10n/i18n • XML validation via XSD schemas, XML generation and in-place updates • Audio data analysis and generation • Scheduled background tasks, similar to cron • Ironically, SQL generation • Spellchecking • Third-party authentication/authorization • Shelling out to existing programs

For me, the DB is really mostly just a persistence layer for my domain model, so there's often only fairly minimal mapping needed between them. At work, there are a few places where the persistent format is notably different, and we have mapping functions there from SQL JOIN results to simpler domain model hash maps.

Thanks @p-himik for your elaborate answer! > Do you have all day? 🙂 Right now: I could talk about this the whole day. 😅 > Not all projects are the simplest CRUDs. I sure know that. Most of the time I don't even think about CRUD, as I believe it's a reduction of real world problems to technical jargon and mapping real world problems to the hardware and technical implementations (like relational databases) we use as programmers. But yes, as I asked specifically about domain modeling there's a lot CRUD like things involved. Like creating an entity, updating an entity, relate one entity to others and so on. And of course all the nitty-gritty of branching out all the possibilities of doing this or that in this or that case can't be modeled well in the database. But all the other things you mentioned - maybe we're now leaving the domain of Clojure and this thread should be better kept in another channel - I believe are most likely (besides very technical, hardware based problems) outside of a "pure" domain model. At least in regard of a "clean architecture": > The way you keep software soft is to leave as many options open as possible, for as long as possible. What are the options that we need to leave open? They are the details that don’t matter. All software systems can be decomposed into two major elements: policy and details. The policy element embodies all the business rules and procedures. The policy is where the true value of the system lives. The details are those things that are necessary to enable humans, other systems, and programmers to communicate with the policy, but that do not impact the behavior of the policy at all. They include IO devices, databases, web systems, servers, frameworks, communication protocols, and so forth. (Robert C. Martin - Clean Architecture) But of course, as it is with alle things: you can heavily argue, that this approach is often not very practical and can lead to heavy over-engineering. And still: why I asked the question in the first place: Clojure with it's emphasis on maps and seqs seems to be a perfect match for modeling the "policies" whom Martin speaks about in his kind of "clean" architecture. 😬

Ugh, sorry but I cannot take "Uncle Bob" seriously. In every bit of his writing that I've seen, every even paragraph contradicts every odd paragraph. I'm exaggerating of course, but only slightly. And this part - "The details [...] do not impact the behavior of the policy at all" - was at the epicenter or CORBA, which I like to mention so much every once in a while as a thing that conceptually could've never worked properly, precisely because of the assumption that "details don't impact the behavior". So at this point I'm not really sure what your question is, or even if there is one.

@p-himik The question still is: how do experienced Clojurians model their domain entities in their Clojure projects. How do they use - if at all - records or specs or plain maps for that. 😊 Just was thinking about your specific answer and it got me a little „off track“. 😅

Thanks @seancorfield so it‘s mostly just Clojure maps you use, without any necessity to define records?

Records are useful when you need to use protocols or when you for some reason need a concrete Java type. The first part became a bit less relevant now that protocols can be marked as respecting metadata on some objects, like Clojure maps. But not all protocols are marked as such, and metadata isn't always the most ergonomic solution.

My take on all of this is it's better to keep things as simple as possible for as long as possible. Why? Because you can't predict the future and changing something simple is easier than changing something complex. The logical extension of this thought is to have less abstractions (as each abstraction is a layer of complexity that bleeds implementation details to layers calling it). Therefore starting with a simple 1 to 1 mapping to the data source is the starting point, add mappings and translations on demand as required. Once you have a definite pattern that is bound tightly to the businesses whole purpose then you can abstract the commonality and complexity. So, for example, I avoid DRY for as long as possible. In fact with current IDE's and AI why not repeat yourself often as it's easier to find and refactor repeated code than to try and adapt an abstraction to do something different (especially as the callers may diverge over time hence wait until the abstraction is stable and proven). This approach is the whole reason I love Clojure, all domain data can be simply basic data structures which are proven stable abstractions.

Plain Clojure maps for domain entities. Specs if you want (we like Spec at work). No records. Note that Alex Miller has said that he would de-emphasize records for domain entities if there is a new edition of Clojure Applied (the 1st ed pushes records for domain entities quite hard). And I agree with Eugene about "Uncle Bob" -- so much of his stuff is "ivory tower" pronouncements about how we should do everything "ideally" (and I really think his advice is tied to an older way of building software).

Fully agree with agile_geek above, with a caveat that DRY can be construed differently. You don't want to inline your whole utils.clj. :D Although I sometimes still do use DRY with business-level abstractions before the "a posteriori moment". Most of the time it works our just fine, probably because my intuition works alright. Sometimes it end up needing a second take, and in my experience most such abstractions end up just as splittable as the alternative repetitions are combinable. So for me personally it's a moot point.

And I agree with Eugene about "Uncle Bob" -- so much of his stuff is "ivory tower" pronouncements about how we should do everything "ideally" (and I really think his advice is tied to an older way of building software).Altho I disagree with much Uncle Bob says I think I've seen him recently change his mind on a lot of Clean Code stuff (cos he uses Clojure himself now!) and he admits the patterns were developed for when we had separate teams for UI, middleware, domain logic and databases.

In case anyone is in need for explicitly worded out alternatives, this is a nice read/watch: https://clojurians.slack.com/archives/C03RZGPG3/p1761830923934309?thread_ts=1761753196.212429&cid=C03RZGPG3

Nice 👍 this went in a quite favorable direction, which seems to answers most of my questions! 😅 It‘s always good to take out the old stumbling blocks like Uncle Bob. I am really looking forward to watch/read your recommendations.

Sorry for my imprecise questions, I am still very much at the exploration stage 😅 I guess what I actually want is to see a repo where I can just see how it's done so I can ask more precisely along the lines of "why do it that way"

Well, if you can believe it, Sean has just what you need. :D https://github.com/seancorfield/usermanager-example Of course, this is just one of many such repos, but still probably the most pertinent given that the author is right here.

Sweet, thanks, see y'all in a month when I am done digging through this:grin:

Godspeed!

I wouldn't argue that usermanager is "best practice" but it is a pragmatic way to get started. It certainly has the persistence intermixed with the model's "logic" (which is very minimal).

Cool. So what's your conclusion? 😁

I'm asking because I am still very unsure about the whole persistence thing, it's likely the biggest thing keeping me from using clojure in production

Really? That's one of the biggest turn offs for me in most of other languages. :) The lack of persistence, I mean.

@aaronrebmann We've had Clojure in production for nearly 15 years, and have a massive MySQL database. What are you still unsure about with persistence that we could talk about? (assuming you mean persistence = database and not persistent data structures)

Oh, right, my bad - forgot what this particular thread was about.

Yes, I mean persistence as in database. I think the problem is that I haven't found any best practice guide or similar and am afraid of shooting myself in the foot - because of the impression that as soon as I write into a database I feel like I'm leaving pure-land, so whenever I think about how to approach this I gravitate towards event sourcing which in turn might be over engineering in many cases. So in my mind there's currently only two choices: no persistence at all or massive over engineering 😄

I consider writing Clojure data to rows in MySQL to be a pragmatic choice: next.jdbc makes that about as "simple" as possible -- get-by-id returns a single hash map, find-by-keys returns a vector of hash maps, insert! writes a single hash map, update! updates rows based k/vs from a hash map, delete! deletes rows based on k/vs from a hash map. There are various strategies to keep I/O and other side effects out of pure code, but I find a lot of those to be overengineering 🙂

Much depends on how much business logic you have, separate from any CRUD operations you need to perform. I don't have too many concerns about intermixing some CRUD code into otherwise pure code -- it's so easy to spin up a throwaway database instance with Docker these days.

Can you elaborate on what you mean by that intermixing?

I'm not sure how to reword that... so I'm not sure what you're asking...?

> as soon as I write into a database I feel like I'm leaving pure-land What Sean meant, I believe, is that leaving pure-land is not that huge of a deal when some workflow requires it. I.e. it's alright to do impure stuff when you need persistence. More than alright, actually - you cannot do persistence without it at all. Even if it's event sourcing. But lots of people do sweat over it and invent various approaches to mitigate impurities. The most frequently mentioned motivation that I've seen is "ease of testing", but with ephemeral DBs created with Docker et al. it's trivial to test something that's not pure. Blasphemy to many, I query my DB right inside some HTTP handlers. Well, maybe not directly under route definitions, but I don't invent 5 different components and indirection layers just to be able to save and query a user preferences object.

You may be interested in Datomic and other Clojure-centric database systems ( e.g. Datahike, Crux, XTDB) that are built around Entity-Attribute-Value tuples, and are queried with Datalog graph patterns rather than SQL table joins. Some people find domain modelling easier with graph-shaped data.

FYI: XTDB is a bitemporal document database, not EAV. With the SQL API it looks a lot like a regular database with tables etc, but schema-less. It still has a Datalog-adjacent query language (XTQL) but that isn't the focus of current work. Disclaimer: I love XTDB but haven't been able to use it in production.

(XT1 was more similar to Datomic, but XT2 is intended to be familiar to "regular" database users)

My apologies, I've never used it. On my to-do list. I had the strong impression you queried it with graph patterns in Datalog.

XT1, yes. XT2, no. XT2 is amazing.

no performance issues, no such thing as a read only repl

I think a popular option is to expose the repl over a local port and do ssh port forwarding. If you, for example, do nrepl over http you can encrypt it via ssl and add your own auth middleware, but it gets complicated quick and depends on clients knowing how to auth

Just remember to give nrepl an explicit port!

iirc nrepl picks a random port if none is assigned

Which is great in dev, but unfortunate in prod where a random port may be exposed to the internet

Using a firewall is a good practice to have a safeguard against accidentally exposing sensitive ports

the repl does hold handles to *1 *2 *3 which could have memory pressure consequences depending on what you are looking at if you hold a session open, otherwise agreed that it's not a perf issue

at work for a while we had a separate JVM for nrepl that was only connected to a read only replica of the DB. not quite the same because it saw no production traffic, but better than nothing.

thank you all!