If you are wanting to do static site generation, cryogen and bootleg are good choices

if self-hosting isn’t a requirement and you’d like to try runnable code instead of just syntax highlighting, try http://nextjournal.com

I use codemirror and its syntax highlighter (via javascript) on my blog: http://thegeez.net/2017/04/28/data_transducers_machine_learning_clojure.html view source shows how it works

Maybe the webserver you use only lets you read the body once. You could check if this works for you:

If it's a string array, it's like this: "[Ljava.lang.String;" Just change it to whatever array type you need

awesome! That worked! Thanks 👍

Commanded is awesome, if I were working in Elixir I’d be building on top of it. I’ve been using its docs for inspiration in designing my own stuff

“I want event structs and handlers.” -> “I want maps and dispatch (i.e. multimethod).”

You can spec and do static checks to taste. (e.g. check the methods of the multimethod to make sure every command type has an entry)

that’s one of the conclusions I came to but I don’t know Clojure well enough to know if it’s the right path

but obviously that’s only a small part

Ah, this is interesting. I assumed it’d be just be something you could easily tack on

Yeah, no. Friends don’t let friends put records on the wire. Use maps.

Guess I was thinking of them as decorated maps, but they’re actually Java classes right?

exactly

Uuuuh, absent whatever context you’re thinking of, I’m gonna disagree pretty hard here.

The model is much saner than the industry standard.

That's my best practice advice absent any context. The cases where CQRS will be a win are the minority. So it's almost always best avoided. Obviously, my personal opinion.

If you’re thinking, “CQRS backed by a SQL data store is terrible and probably not worth the cost,” then yeah. That’s sane.

But datomic is basically CQRS embodied (depending on your definition).

And it’s super nice to work w/.

It's not CQRS at all

I know lots of people make these huge leap trying to compare them

it’s not a leap

but ok

But it's just a very different model

No. It’s different, but related in obvious ways.

Related the same way a for loop relates to reduce in my mind

The differences matters immensely here

They’ve taken the work out of creating “aggregates” by doing it by default.

You can debate whether you call txns “commands” or not. The end effect is the same.

They’ve also made it trivial to revisit aggregate states.

So yes. Big diffs. But the extremely important concept of “track everything that happens and don’t forget it,” is still there.

That's not really anything to do with CQRS

ok, not gonna debate semantics w/ you

You could make a CQRS implementation that has that property

But CQRS is an architectural pattern. It's like an implementation. It's not really a guiding principle

You're kind of talking about use cases almost. I just find it concerning to associate the two.

Yes. I’m sure it’s an architectural pattern devoid of any principles.

CQRS isn't what you have to reach for if you want to have a history of your transactional changes.

Just one option

Datomic being another 😜

There’s a reason people always tack on “ES.”

apparently

It’s so semantic police will not attack them.

But I'm not playing with the word semantics here

I'm talking about the semantics of the software design

If you want to call the pattern Datomic follows CQRS go for it

Did I say that?

But there still exists a different design which is different from how Datomic works, and some people call that CQRS as well, and they don't work the same way

You’re pretending I’m saying things just so you can argue.

And when I was more precise, you continue to make vague assertions about what “counts as CQRS.”

Ok, so maybe I don't know what you're saying. It seemed you were saying you think CQRS is great, and gave me Datomic as an example of why?

So, yes, you are playing word games.

No. I said CQRS is better than the industry standard because of certain properties it has.

> The model is much saner than the industry standard.

Oh, okay. So I guess we are in violent agreement then

😄

Wait, the CQRS model or Datomics?

Maybe I'm confused now

Both. They share certain properties. (But Datomic is a major improvement.)

I'm saying, CRUD is best for the average project, and choosing to implement CQRS instead is often a big mistake.

Using Datomic I consider is a third option

Which is a better choice then CQRS, and depending on your requirements, CRUD as well

I mean, I’m not gonna hard disagree, except many projects need event history. It’s not an uncommon requirement.

Ya but you don't need CQRS

But saying, “prefer SQL to CQRS” is much softer than what you said initially 😛

You can just publish some separate events to some data warehouse store

Well, then you have another data store you gotta keep transactionally in-sync 😄

Or if you use say DynamoDB, it even supports publishing events automatically on a change. And I'm sure probably Postgress Sql has something similar

To my knowledge, postgres does not.

I wish it did.

Reading a log (a la datomic) is by far preferred.

But, Datomic (and I might be wrong), is not really a log. In that, the log is used by the writer, but then data is just records that are indexed, from which you can query slices no?

No, it has a reified log.

You can ask for it and troll it yourself.

You can replay and stuff?

I mean, okay fair enough if it keeps the writer log durably

In-memory sure. There’s no button for resetting the db back and re-playing.

But one of the data-removal strategies is to replay the log into a different db, removing data you don’t want anymore.

I was under the impression in was more simply a n-tuple store. With a log based single node writer to it

It is. But the log is reified as data as well.

In which case, the writer and reader use the same object model

Ya okay, that's nice

Super nice. And you can use it to, say, sync an external, eventually-consistent data store.

reliably

Ya agree

To be fair, I haven't used a DB that didn't have an event stream baked in in a while

So maybe the "industry standard" is way behind on my assumptions

I just feel I need to warn people venturing in CQRS

Being interested in it and trying to build it is very different from saying you're choosing to use Datomic or Crux or DynamoDB, etc as your database

Like the person asking for a library/framework for it

I've seen this path crash and burn multiple times before

Martin Fowler puts it well: > CQRS is a significant mental leap for all concerned, so shouldn't be tackled unless the benefit is worth the jump. While I have come across successful uses of CQRS, so far the majority of cases I've run into have not been so good, with CQRS seen as a significant force for getting a software system into serious difficulties

> Being interested in it and trying to build it is very different from saying you’re choosing to use Datomic or Crux or DynamoDB, etc as your database

this tho

Yeah, I’d agree with all of that. If you’re uncertain, you should pretty universally just start w/ Postgres and figure the rest out. But for many apps, you need a reified log (which isn’t provided by most RDBMS’s). And that mental model is super powerful once you have it.

Curious what your DB experience has been in. Not sure how many dbs have an event streams baked in.

Very interesting piece by Fowler. Thanks for highlighting that. I'd be interested to know if his thinking on this has changed any in the nearly ten years since he wrote that...

I've been working with AWS for the last 5 years. And mostly rely on DynamoDB. Though what we often do for audit/metrics/analytics is publish a separate log to SQS and batch process those into S3 files, which we later run queries using AWS Athena. We also maintain the last 2 years of data in a AWS RedShift cluster.

We need to do this anyways because we have data in multiple places across multiple services and teams. And the business needs a unified history view of all that. So having a separate publisher, everyone can just make sure to integrate with it where required no matter what they use for managing their data

It does mean data can drift. Since not all store will guarantee to be 100% in sync at all times. You can try and make the publishing of event atomic with your data storage, but when the storage layer isn't designed from the ground up to support this, it's almost impossible

The best we do sometimes is perform a kind of Saga of events. Like publish the event, make the change to the DB, if the change to the DB failed, publish a redaction event. And if the first publish failed, we'd fail the request itself.

But you can see there's still point of failures here, like failing to publish the redaction

We kind of just live with those

They'll often just disappear in the wash.

If I was modeling a transactional system moving money around, probably I'd want something which eleminates even those possible faults.

And then you might need a more integrated solution

I’m playing with CQRS/ES in toy project(s), but the event-based pattern is what I’m aiming for as the entire project is essentially a timeline and needs an audit trail

Are you explicitly saying to stay away from CQRS or ES?

Which you could build on DynamoDB, since it gives you a log stream with all these guarantees. Or Datomics would be a good option for that too I think

I'm kind of saying to stay away from both

I mean, in toy projects it's irrelevant, if the point is to learn

It's not that they don't have merits, but they are very complex systems

And you can easily turn a simple business use case into an accidental complexity mess

The effort to properly implement CQRS and ES, and then build all your requirements over them, is huuge, compared to some alternatives

Like you said, these are toy projects so I’m not sinking business-paid dev time on them, but it’s fair warning, thanks.

Ya, I mean they're entirely valid approaches, but I think they're often used because I guess they're the architecture "du jour", and using it for the wrong use cases is an anti-pattern

I can see that’d be the case, but it’s also why I wondered if there was any kind of framework to make it a little more smooth sailing. I can see that CQRS/ES as an architecture probably benefits a certain scale and attitude, but I want to give it a go mostly out of curiosity

The in-built log stream is interesting, although I don’t use AWS

I guess Datomic also just gives you an history trail for free

Datomics was one of the things that made me think Clojure would be a good fit

But I guess both of those are paid $$$ solutions

the project I’m playing with in Elixir uses Postgres, but Elixir brings its own benefits to this kind of architecture

Datomic has a free offering right?

I'm surprised PoatressSql doesn't support some form of it. I wonder if any free open source DB does

> 2 simultaneous peers and transactor-local storage only.

Ya, but I think for non-commercial use only

Oh that's nice. So a free tier then

Honestly, I’d not thought of a transaction log as a side effect at the DB level

I’m really just toying with different architectures so I can get a bit of experience with them

Ya, then it's worth looking into

Good learnings and experience for sure

But on the flip-side, I’ve spent a few days just figuring out the architecture and implementation, whereas I could have had half of the app running so far, haha.

Hoping it’s initial investment

😅 ya, most of my personal toy project end up like that

I'm more interested in messing around then delivering anything of value it seems hehe

Seems you kind of can do it with PG as well

https://wiki.postgresql.org/wiki/Audit_trigger_91plus

You'll have a separate table of logs which can record all insert, updates, deletes and truncates operation

Seems you can even just access the PostressSQL write ahead log directly

This describes how: https://aws.amazon.com/blogs/database/stream-changes-from-amazon-rds-for-postgresql-using-amazon-kinesis-data-streams-and-aws-lambda/

You don't need to use AWS for it. You can adapt the pattern.

So, Audit triggers assume an audit table.

If you’ve got everything recorded in a log, there’s no need for a trigger.

Yeah, you can access the WAL, BUT (and it’s a big but) you’re consuming resources on the server for the duration. Postgres does not keep the entire WAL (not by a long shot). So if you’re, for example, disk constrained, and your log reader gets behind, you risk hosing your entire system.

That's not how I'm reading the docs. The trigger is on your regular table, and what the trigger does is write a log to an audit table

> write a log to an audit table

ah, I see what you mean

but yeah, basically a form of: store the log in a sql table

Ya, it's not ideal for replicating to a different DB. I guess you could then export the audit table and import it elsewhere

But if you don't care about that. Just want an audit trail, it works

yeah, postgres has a pretty robust WAL replication story already

no need to side-car that

Ya, the wal2json approach seems nicer to me

You're right that you can get behind I guess

Wonder if there's a way to handle that, and throttle the writes to the wal

Or stop them in that case

Reading from the WAL is an interesting approach

I know Kafka can consume the pg wal

yeah that seems bad news bears to me

I'm guessing Datomic just starts failing on writes in that case

unless you’re massively over-provisioned, so resources are guaranteed to be not constrained.

Say It's writer node runs out of disk space?

uh, I dunno. Datomic doesn’t assume it can trim the log

I mean, it's a scenario I feel shouldn't be too hard to plan for

problem w/ postgres is: you’re interfering with its internal assumptions to a degree

The wal isnto disk correct?

you know, you’d think that

yeah

but I’m currently in a situation where getting behind on WAL reading would hose the whole app :derp:

g2g, dinner

thanks for your input

So you've got a big buffer. If you know your consumption rate, you can just plan and put some write limits to the DB.

Dunno if PG can be set to have a write limit. If not, you'd have to build an additional layer.

Eh, anyways

feels a bit circular though. you write to postgres then consume the WAL update and process that in your app.

I'd still contend these type of approaches I think are simpler (even if still complicated), then full on CQRS

Well, it depends on your app

Like, if you just want to model a timeline of some logical app domain entity. I wouldn't even do any of this. Just model it directly in your data model

Like if you want list of X, just have a table for it

There's kind of a difference between you need to build a audit like feature in your app. And having to audit the entire application data

I've been talking about the latter the whole time

Generally that's for things like government compliance, record keeping, reporting, business analytics, etc.

If all you care is like users being able to go in a view that shows their activity timeline, all of this is overkill

For that all you do is insert a new row for each new activity and then filter over the ones for your user sorted by date

In some activity table

And done, you now have user activity audit

Each user can go and see their history of activity

There's an internal we have, they modeled comments as ES, and it's the worse user experience ever. You type a comment submit, you see your comment, refresh the page, the comment is gone. All because of what you described. ES is naturally eventually consistent. It's a huge roundabout. Your comment was first sent as an event: Please create this comment. And then a reader later consumes it and creates the comment in the Queryable store. And only then can you refresh the app and see the comment

In the meantime, it's possible others commented before you, yet you were not yet seeing their comment

It's a mess

Nothing more frustrating to a user then doing something and seeing it disappear and later reappear

Oh, and one last thing 😋, CQRS and ES comes a bit from DDD (domain driven design). I love DDD. And just want to say. You can seperate your queries from your writes in your app, but still use the same DB for both. And that I'd say is a pretty nice pattern.

CQRS takes this principle from DDD, and applies it to the storage layer as well

But even DDD, is a pretty complicated arrangement, and you need a complex enough domain to justify using it too

But all it means is that, you should have entities structured in terms of your writes. Store those in the DB. They will protect your data invariants, and define the level of atomicity required.

Those entities you only retrieve for the purpose of modifying them

Then, you have seperate queries, only for the purpose of displaying data, not to be modified

Those queries can choose to return the data in whatever structure or shape, with whatever level of processing applied to them, since they are read only

But they could still be performed over the same darastore, same tables

They also accept collections

I don't think there was any reason then whatever whoever chose the name thought of calling it