your macro override clojure.core/future and then expands to the call to itself. that makes an infinite recursion

eventually recursion exhaust maximum callstack

oh, ok so I need to rewrite the 'future' from core internally? macro over macro is no go?

just rename your macro

ohhhh shieeet 🙂 yeah thanks

I still get some problems on Exception binding:

(defmacro future-log [& body]
  `(future
     (try ~@body
       (catch Exception e
         (println "Error occured:" e)
         (throw e)))))

yeah, during macro expansion e symbol gets resolved. you can use special notation e# to let macro know that this symbol needs special treatment. use the same symbol everywhere in the macro body to access it's value

more about this https://clojure.org/guides/weird_characters#gensym

Fun-with-macros aside, it might be more effective to use Thread.setDefaultUncaughtExceptionHandler

You could also do

(defmacro future [& body]
  `(clojure.core/future
     (try ...)))

although ofc you then need to not refer clojure.core/future into the ns that uses this and refer this in instead

which is doable with a :refer-clojure clause in the namespace.

Btw there’s logging-future in Timbre that does this

@U06BE1L6T thanks, that's great tip, its called logged-future

Ah sure, sorry

I have my own version called logging-future+ https://github.com/jumarko/clojure-experiments/blob/develop/src/clojure_experiments/concurrency.clj#L43-L47 That has an advantage of preserving the client's stacktrace which may be very useful when debugging problems (otherwise you may lack the context from where the future was called)

Like, i'm used to the ecosystem libraries being legos and bringing my own glue. I think thats actually better a lot of the time. But all the stuff like • Live View • Automatic Admin Panels • Easy AF background jobs without extra infrastructure • Standardized CI flows • Authorization schemes with auto generated tables for roles and such

Its mentally exhausting thinking of how to do half of that

and I feel like I've been around here enough that I know all the established options

agree, phoenix is fantastic, can recommend this video by José: https://twitter.com/josevalim/status/1613541885666000904

I’m not aware of a Clojure websockets library that’s useable across multiple nodes for example, so it’s not like we have all the bricks

also phoenix is using OTP so your webapp can be part of a larger OTP app and it’s not dictating an architecture for the whole thing

Even something comparatively basic like fastapi for Python is pretty far ahead of what we have for Clojure, last time I wrote a webapp

I think there is a valid observation in Clojure that framework-atizing building blocks is a problem - that being said yeah we are missing blocks

what do you think are the big ticket items?

I found yada a good batteries-included solution for quickly building REST APIs. It's sad that it's discontinued

> what do you think are the big ticket items? I like to think im on the right hand side of the dunning kreuger when I say I do not know

like I can start to describe the mechanics and structure of how Phoenix "does its thing", but I don't have enough true understanding to really break it down

like - every time the "let it crash" thing is explained to me i feel like I'm getting a marketing pitch more than an actual technical explanation

same vibes as my old roommate going "Scala stands for scalable language..."

but the fact that actors are a primitive of the runtime is used to create a non-trivial part of the magic

(at least for live view)

with respect to "let it crash", I think the classic erlang paper is really great, http://erlang.org/download/armstrong_thesis_2003.pdf

and its hard to break it down into "what just feels nice because its pre-glued" and "what is missing"

> I think the classic erlang paper is really great My immediate reaction to seeing "Philosophy" in the index and skimming that section in the singular minute after opening this is that people really haven't been exposed to the "cars have windows and can move. houses have windows and can't move, so its not the windows that make the cars go" style classical philosophy

feels strange, honestly

but, marketing is tangential to merit

I'm not sure what else to tell you other than maybe try to keep an open mind

oh yeah no i'm not commenting on the actual practice itself

as well as an appeal to > So there is this really important paper at the bottom here. And if you read this paper over and over again, which I recommend, you are going to see a couple of facts about systems. And it is another way in which systems are really different from programs.

I first stumbled on this paper when it was mentioned in the language of the system talk.

I guess one way to put it is that I am playing the game of "how much of what I am hearing from this person telling me to use technology X comes from their understanding of X and how much is them being religious about X" So in that space I am suspicious of "let it crash". The person who pitched Phoenix did not read this 300 page paper. I can't immediately turn around and go "we should make a system such that I can build clojure programs with the 'let it crash' philosophy and that will get us to where Phoenix is"

Same way rich hickey worship is a bit funky-fresh-culty-cringe

this is getting really tangential - i'm just sad they are doing better than us.

I agree that the short explanation isn't convincing. There is a longer explanation (eg. the paper), but there's not enough time to read and understand every longer explanation that's available. The only way I can see to make meaningful progress is to try and recommend the longer explanations that are actually worth the effort to read and understand.

we don't really have a Clojure-on-Rails, I guess

> we don't really have a Clojure-on-Rails, I guess True, but I think the defining property of rails is that the component pieces are not decomposable

In short, the idea behind letting it crash is instead of programming defensively, validating, etc, write the happy path and let someone else (supervisor) worry about it Think about it like every code piece you write being wrapped in a try catch block with configurable policy when you catch. The philosophical idea is that failures WILL happen and consistency isn't worth chasing, better focus on making things resilient and scalable. Does that sell you on the thesis?

i guess that's inevitable— there was the discussion of the websocket migration within a cluster earlier, briefly, it's hard to see how that wouldn't come with some infrastructural implications

We can glue any pieces together and make some "vertical experience" nice at any point. Thats what we do every time we write an app absent a rails. Thats what a yada/kit/luminous does. I'd loop back to we just don't have the bricks, and thats the bigger problem

I think let it crash is a useful idea generally, but the erlang runtime and ecosystem has lots of support for that style so it's not exactly as easy to pick up and run with when using other languages/runtimes.

> Does that sell you on the thesis? Yes, but in isolation it doesn't sell me on distributed actors being the ideal solution to make a "live" web page without javascript or elucidate an understanding of how the meta-properties of supervision trees and erlang processes differ from regular single machine threads with default exception handlers

I'm pretty sure we all write the web server/rest api parts of our things with a "let it crash" approach. That isn't the special part of Elixir

> regular single machine threads with default exception handlers this philosophy is baked into the erlang runtime. As an example, the BEAM vm (erlang's runtime) has a preemptive scheduler. That means if a process gets stuck in an infinite loop, it won't bring down the whole system.

right but being able to detect infinite loops and kill processes stuck in them isn't what clojure is missing - I don't think

there's other examples, but I think the preemptive scheduling is a really huge feature I wish I could have in other runtiems.

I hate nuance because im so bad at it

but this list exists > what do you think are the big ticket items? I just don't know how to arrive at it

I also don't think the "let it crash" is a huge part of what enables liveview. I think it's the general erlang ecosystem that supports making something like the phoenix framework.

one example here is that I’ve often experienced is that when our thread that’s accepting new websocket connections died because of some core.async error (more than 1024 pending puts or something like that) the whole app was broken. Failures aren’t this catastrophic in erlang/elixir with a proper supervision tree typically and a process is a great way to model a connection imo.

I do think that let it crash and only coding the happy path and a process per connection is a great combo that does enable the how easy liveview is. If the other part sends a message you don’t understand, the process dies and the client needs to reconnect.

> a proper supervision tree typically and a process is a great way to model a connection imo. Right - but we've had akka for awhile. What about akka was unsatisfactory/didn't lead to an equivalent jvm thing? It had supervision trees

virtual threads are one hypothesis, but that too is hard to "connect" to everything else for me

like - akka processes not being jvm processes could be "fixed" nowadays with vthreads

yeah, I do think virtual threads offer a chance to try this again

do virtual threads offer preemptive scheduling? I thought they didn't.

no, so not going all the way

it comes with a hefty performance implication though

I’d gladly take a 10x slower clojure for a lot of uses cases for the reliability of pre-emptive scheduling

but we’re recommending to turn off assertions and spec instrumentation in prod

I guess it’s also a harder sell to go from good to worse performance

are other people sensing the "hole" now too kinda? Like is it really just pre-emptive scheduling that makes it so we can't have

(defn stock-card [data]
  [:div 
    [:h1 (:id data)]
    [:div
       (:score data)]])

going back to the original idea of this thread. I think a lot of the frameworks are useful if you're trying to write applications that are not substantially different than what is already available. I like clojure because even though getting started may be slower, it's easier to build new and different software in the long run.

> I like clojure because even though getting started may be slower, it's easier to build new and different software in the long run. I generally thought this too, but I no longer see a path where I can build "the same thing but slower to start." It feels like I can only build "a worse thing"

like, my web apps are not reactive and do not have notifications. I make a worse admin dashboard, worse feature flag impl, etc

shameless plug: with clerk you can have that example above update live on the user’s screen. But it’s ignoring a bunch of other problems that phoenix is solving, auth and users being two of them.

I think one aspect is also about the number of users. Dividing our community between x libraries means fewer users per lib and results in fewer eyeballs and polish.

Also don't listen too hard to me - I'm also rapidly approaching my burnout point at work and with the clojure we've written.

I don't think there's any particular road block to building any of those features in clojure, but I don't know of any framework that provides them all out of the box.

coming from Elixir & Erlang I was certainly surprised how much harder it was for us to build something that was as reliable in Clojure (with core.async)

you migrated by choice?

yes, felt Clojure was a better fit for Nextjournal

but for the gameservers we were writing before I’d never pick something other than Erlang or Elixir

being able to ship hot-code upgrades without dropping a tcp connection is just 🤯

hard act to follow, yeah

BEAM is a great platform. It took Elixir for it to read web dev, though

I actually think the lack of frameworks in Clojure is a feature, albeit one that takes some getting used to.

this old talk seems particularly apropos to the discussion, https://www.youtube.com/watch?v=ShEez0JkOFw

Pheonix has 88k lines of code. It's not like there's magic involved.

I think the question is, why no one in Clojure cares to invest 88k lines of code for a full featured framework?

And I'm not even including the LOCs from Pheonix's dependencies. There's a bunch of required modules like pheonix template, pheonix view, pheonix pubsub, and ecto that are all a ton of LOCs

we better get cracking

Is there a business behind Pheonix? Could that be why?

I do recommend checking out Biff, it's nothing compared to Pheonix, as it clocks in at 3.7k LOC only. But it does automatic email based auth (passwordless), background jobs, light ORM, and auto deploys. Then imagine if another 100k loc was added to it...

Ya, Ecto is 70k loc, the PubSub module is 4k LOC, etc. So it's probably getting close to 150k to 200k loc for Pheonix web framework.

that's substantially more than Clojure itself

I am just using the github chrome plugin, so it might be counting guides and some non elixir code files as well to be fair, but still.

yeah it's a massive project

@U3JH98J4R What building blocks you feel is missing? Or like, what functionality are you struggling to figure out how to do in Clojure?

1. Say someone sends a payment. We want that paid invoice to update immediately on everyones screens.

that's mostly a cluster-coordination problem, right?

i'll think about it a little, there's no way in hell i'm actually going to finish this gui wavetable synth

> Say someone sends a payment. We want that paid invoice to update immediately on everyones screens. You brought out the big guns from the start eh 😛 Option 1: 1. Update the invoice in the DB 2. Have the clients poll the server 3. When polled by a client, the server just looks it up in the DB 4. Server can have a in-memory cache as well to speed up lookups to the DB This is the simplest option. Option 2: 1. Most Java (and thus Clojure) web servers support WebScokets 2. Have the client open a WebSocket connection 3. Send all clients an events to tell them to update This is more complicated, because you need to establish a websocket connection, some older browsers might not support it, so maybe you need a fallback mechanism, and on the server you have to track clientIds and what client you want to tell to update. This option is also much harder to scale to more than one server instance, because it's difficult to load balance. But as far as I know, this is also hard to scale in Pheonix, most examples assume you are running a single instance. And scaling options are pretty much the same for Pheonix or others.

Option 2 is not particularly bad to implement -- we just did that at work. We actually started with Jetty 9 and the default Ring adapter and glommed a bunch of Java interop on to enable WS (the hard part was figuring which parts of Jetty's two completely different WS class hierarchies were supposed to be used together -- the docs were awful!). We already use Redis Pub/Sub for a cheap and cheerful messaging hub, so wiring live updates through that and out to connected clients was reasonably straightforward. Then we switched to the sunng "Jetty9" adapter (actually Jetty 11) which has a really nice API for WS and dropped all the Java interop. The end result is pretty clean and easy to work with.

What's your Load Balancing setup? Any issues with stickiness or balancing the websockets?

Hah, now you're asking me hardware/networking questions! I believe we have a WebSocket profile enabled on the load balancer (F5 BigIP, I believe) and that is supposed to keep socket connections alive to specific servers. But even if a connection drops, and gets reconnected to a different server, the message hub aspect keeps every "working".

persistent connections don’t usually make scaling/load balancing tricky. it’s persistent state that makes scaling/load balancing tricky.

(and we're not Facebook/Twitter-scale so this stuff is "fine" without too much effort)

Ya, the LB part is where this gets quickly out-of-reach for say solo devs or small shops. But on the topic of Pheonix, it does not solve this problem either. Like I said, many tutorials simply advertise a single intance setup. Which could work for many low traffic websites. That setup is simple in Clojure as well, use Jetty websockets, or http-kit websocket APIs. You could also have a failover instance, for high availability, but you need to assume that there's always really just one server instance running, not both of them used at the same time to horizontally scale. What you can do with Elixir is something called Distributed Erlang, where paired with a service discovery mechanism, you can send messages to Actors running on other server nodes. They call it cluster mode. That can be used instead of Reddis for example, to sync server state between instances. Or it can be used to perform distributed computing. But even in Elixir, I've been told a lot of people choose Reddis, because using Distributed Erlang and having a cluster deployment isn't always that simple either. Or people simply use a DB as a way to sync.

> persistent connections don’t usually make scaling/load balancing tricky. it’s persistent state that makes scaling/load balancing tricky. From my understanding, because websockets are TCP based, many LBs don't work with it, since a lot of LBs load balance HTTP.

I'd say modern LBs seem to handle this just fine?

The other issue, I think is that often you don't want to open each instance to the public internet, so like you could have a LB take the HTTP request and have the WebSocket connection made to the server that returned the initial client request. But I think a better setup is for the LB to remember that the websocket was created to instance 2 and keep it open and proxying back/forth.

Ya, I know HAProxy does for example, and I'm sure most LBs can, but how easy it is to configure I don't know.

Additionally, you almost always have to handle disconnect/retries anyway. You can support load balancing like normal just by having good reconnect logic (usually).

When I would do chores like clean the floor, my dad would say that it should be clean enough to eat off of. So I would have to eat something off of it. Thats neither here nor there, the man's an alcoholic, but: Even though this conversation is interesting, I need to find a way to have an admin panel that doesn't require our CX team to refresh every few minutes along with a million other warts of tech debt that have nuked our velocity and caused really high turnover. I would not "eat off the floor" of these suggestions right now.

Sounds like you have a lot more problems there than Phoenix/Elixir/LiveView would solve 🙂

in isolation, yes

But it is at the point where despite being heavily, heavily invested in clojure and having the engineers for clojure and having the expertise for clojure we are considering nuclear options

With your kind of problems, I would simply decide to not have real-time refreshes? Or I would just do polling, like Option 1.

It's often the case that companies start to consider a nuclear rewrite when they're too deep in "debt" with their current system, regardless of technology -- and there's an element of "the grass is greener" over there in some other tech because it "instantly solves" pain points X, Y, and Z... but most cross-tech rewrites only work if you completely replace the team that wrote the original, because they are what caused the debt in many ways.

(I've been talking a lot about rewrites with someone here lately and this seemed a common theme in all the rewrites -- both cross-tech and same-tech, successful and failed: the problems that led to the rewrite were almost never actually technological problems, unfortunately)

Poor management, a lack of commitment to quality, poorly-elaborated requirements, no training budget, unrealistic deadlines... This stuff can get baked into every aspect of a company's "DNA" in terms of processes and thinking. Those situations will create tech debt with any language, any framework...

20/20 all the way 🙂

@U3JH98J4R Are you running a single instance server?

No, but we could

realistically, we could

So this shows a small chat backend using http-kit: https://github.com/http-kit/chat-polling/blob/master/src/main.clj that will work for single instance setup

yeah i could figure that out. I've also done a chat system with redis and keeping track of active connections with gql subscriptions

So it's like 60 loc 😛. Reddis would be needed to sync client map if you had a multi-instance server deployment That example uses long-poll, not websocket, but http-kit uses the same API for both, so you can have another route for websocket and use the same handler.

Apropos -- from Kent Beck's latest "Tidy First" newsletter: Big changes are risky. What if you pack everything in your covered wagon, you cross the plains without contracting cholera, you get to California, & you don’t like it?

I take it, it might be more a matter of, if the team got themselves into a mess, maybe they need a framework to give them structure and handle the harder parts for them? And maybe they would be more successful given less power and control, and having to use the framework standard mechanisms instead? I've actually heard this argument for why Clojure needs a framework as well, and to be honest, I think it is a good one. Some people don't know how to put thing together cleanly, or even get it working. I do think a framework can help certain teams, not just so they can do it prior to having the know-how, but they also get to learn from how the framework does it, and maybe after a lot of experience with the framework they might finally be able to do it on their own without.

one of the cool things about the erlang paper is that it not only talks about the ecosystem and runtime, but also the reality of building software with teams of people with different amounts of experience

> Apropos -- from Kent Beck's latest "Tidy First" newsletter: Big changes are risky. What if you pack everything in your covered wagon, you cross the plains without contracting cholera, you get to California, & you don’t like it? Well from my memory of the Apple IIE, the class period was barely long enough to maybe get across the Mississippi River. Best use of time was to practice shooting squirrels and elk. 😄

@U7RJTCH6J Yes, I've been feeling like this is the part we're missing in Clojure. OTP for example is just a set of tools for more advanced use-cases. I feel like it's the third layer, we have a cool language, we have good tooling now as well, the next level is providing utility libs and frameworks for advanced usage in real life scenarios. Here we've mostly been using Java's options through interop, but I think it's where it be cool to see Clojure grow.

I wish we could break this thread into like 5 subthreads, so many good topics. For additional reading on the Let-it-Crash, besides the Armstrong thesis: http://tomasp.net/blog/2015/failures/ I think it’s worth mentioning I think a lot of what is meant by “let is crash” is not “it’s okay that the whole system becomes unavailable,” but instead “provide an execution context where, built in by default without pervasive error-handling code, any unexpected situation from the smallest argument error to the largest component failure is automatically isolated (to its own lightweight thread(s)) and by default comes with a graceful recovery strategy (restart, and assume that your system can handle the vast majority of messages/conditions even if it fails on rare/exceptional ones. Note that a lot of these superpowers of Erlang/Elixir, and especially Phoenix are built on three layers of frameworks: the BEAM VM itself, OTP framework, and the Phoenix framework. So, like Rails, a lot of the superpowers only work when operating inside the framework. But for a lot of situations, these assumptions and frameworks work because there’s a genuine domain match — for Erlang/OTP, building network servers, and for Phoenix, building specifically web/http servers. Clojure as far as I can tell is actually much less opinionated/frameworked so far in terms of those system assumptions--a lot of the design work has gone into the language/FP/data structures itself rather than the system/process execution context, I think? Though a lot of Rich’s later talks allude to being aware of this stuff. It should be possible, given the herculean efforts mentioned above, to build a bunch of the frameworks in Clojure if we had the resources, and we’d be able to get everything except the BEAM/JVM differences. Although JVM itself is slowly evolving.) (I also think Jose has said he was linguistically inspired by Clojure, so to some degree they were able to benefit from the linguistic work on Clojure and then added it on top of the BEAM/Erlang systems superpowers)

Yeah, it's been an anfractuous but productive thread

I find the "let it crash" is a bit hyped up in Erlang, but like, most languages do that nowadays. Every enterprise Java app just catches all errors on a request and let the client retry for example. Most exception recovery is done by just wrapping a part of the code in a retry, etc.

yeah top-level catch at the top of a process handles most stuff. BEAM+OTP has some cool facilities for lightweight teardown + restart even across processes/in a distributed context — but these days it seems like people are getting that monitoring/restart facility from Kubernetes etc.

That would only be if you run it clustered. I think I'm missing some of the history a bit. Like what was the "other philosophy?". Was there another view for handling failures in distributed systems that was more popular at the time?

Armstrong mentions this Gray paper (1985) as precedent work for reliable systems: https://www.hpl.hp.com/techreports/tandem/TR-85.7.pdf It sounds like already in the 80s serious systems used dual redundant isolated processes. It sounds like before that people would just monitor the a single non-redundant process and wait for an operator to restart the system manually. (And programmers had to make a best-effort to not have errors in that process — which maybe leads to the Ada lineage of trying to prevent errors)

> It sounds like before that people would just monitor the a single non-redundant process and wait for an operator to restart the system manually. (And programmers had to make a best-effort to not have errors in that process — which maybe leads to the Ada lineage of trying to prevent errors) Hum, 1985, and we're talking manual operators waiting to manually restart the system. This corroborates my suspicion that the context is very outdated. (And so when used as a positive of Erlang nowadays it's kind of misleading and a bit of a hype-train)

Yeah that makes sense. Most situations these days can catch exceptions, and coarse-grain restart processes in a way that is Good Enough for the vast majority of circumstances (vs. the Erlang context of trying to do telecom-scale reliability, in the 80s).

And I do reckon that Erlang makes it very explicit and nice to do so with OTP. It also makes a bit more sense in an Actor model to need to emphasize these. And I agree that say in Java, a lot of that stuff is not built-in, if you have a database client that is instantiated on application start, and it gets in a corrupted state, your app might not be smart enough to try to restart the client and use the new one, so you might be stuck with an app using a broken DB client and no way to recover. OTP emphasizes always making sure things like that are able to restart when retries are not sufficient. What I've found more misleading is say on Elixir's official guides (which is one of my fav languages after Clojure by the way), you will find: > If you have prior programming experience, you may be wondering: “could we just guarantee the bucket does not crash in the first place?”. As we will see, Elixir developers tend to refer to those practices as “defensive programming” And I just can't fathom who has prior programming experience in other languages and is thinking, oh ya, the solution is simply to write a provably bug free program and run it over infallible hardware and infrastructure. It's also not what I personally refer as "defensive programming", which I think more as rejecting invalid input before attempting to execute code with it. It's also a little bit of bad advice in my opinion, where I would say you want to validate input to prevent crashes, and you want to retry on a crash first, because it might not be due to an issue with your state, and if retries fail, you might want restart (aka re-initialize your state to a known working place), which is worse than the other two because it could also cause in-transit things to fail that maybe were not affected.

chiming in, my Ripley library for Clojure was inspired by Phoenix LiveView… it’s not really similar in many cases, but the main gist is that you render a page, then can send callbacks back and when things change, a partial render is sent to browser via WS

clj-kondo will also give you linting for EDN files (in your editor or command line)

when you parse with edamame, you'll also get a "decent" error message about the location

ah thanks.. of course there’s a library, i just missed it.