This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
- # announcements (5)
- # architecture (79)
- # babashka (48)
- # beginners (148)
- # calva (57)
- # cider (31)
- # clara (2)
- # clj-kondo (4)
- # cljs-dev (6)
- # cljsrn (1)
- # clojure (10)
- # clojure-australia (4)
- # clojure-dev (5)
- # clojure-europe (13)
- # clojure-france (1)
- # clojure-nl (6)
- # clojure-provo (4)
- # clojure-uk (51)
- # clojurescript (38)
- # conjure (20)
- # core-logic (18)
- # cursive (28)
- # datomic (12)
- # duct (2)
- # emacs (16)
- # figwheel-main (3)
- # fulcro (43)
- # honeysql (17)
- # hoplon (3)
- # jobs (1)
- # meander (4)
- # mount (5)
- # off-topic (32)
- # pathom (6)
- # pedestal (5)
- # re-frame (1)
- # reagent (2)
- # reveal (8)
- # shadow-cljs (209)
- # spacemacs (5)
- # tools-deps (37)
- # xtdb (18)
I have an architecture question about the interaction of the component library and web frameworks like ring, pedestal etc. Usually web requests will need access to pretty much every component under the sun, on top of request-specific information (like who the user is etc). How is this to be modelled without having to pass in the entire system under a key in the request map?
With Integrant you can decompose a system into components and define dependencies for each component. A component could be a subset of routes.
We do the same with component — subsystems with dependencies, and it’s working well. I’m more asking about “don’t pass the system around” which is a common advice…
Same here. For example, our
web server component usually has dependencies on
monitoring components only.
Or maybe decompose a service layer into components and pass only needed dependencies per a service (component). It’s similar as you do in Spring etc. But I don’t know if this approach would be better/readable.
In our current code we’re creating a map with namespaced keys that correspond to various components that the requests need, and we pull those out in our handlers. But as gradually add more and more components, we have to expose those too, so it eventually degenerates to be a renamed version of our main component system.
> Usually web requests will need access to pretty much every component under the sun This strikes me as the root cause. It doesn't seem normal to me that every handler needs every component. That could mean any of these two: * there's a somewhat excessive amount of low-level components * maybe: create a higher-level component for them * or: don't create as many components - sometimes direct access can be OK * the app intends to be highly modularized, but it breaks its own module boundaries * this happens quite often which trying to use somehing like the Polylith but without any automatic means of ensuring that modules have a tractable dependency graph
I see some interesting points here, which leads me to ask: how many components do your apps have? It would fun to see the results of
(keys system) to see what kind of dimensions people use...
In our case we have roughly a dozen components (afk so I can’t run that code atm) - and our web server component needs: • the multi tenant support • The mongo connection factory • The Postgres connection factory • The solr instance • The notification component • The email component • Some config stuff
So some real low level stuff isn’t used from the web (eg we have some queue component that isn’t exposed)
I’m using integrant, but aside from that my components look like:
• postgres connection
• app config
• firebase system
• stripe system
However, I will need to add my own
I have to pass
app config to my HTTP Handler component (web component), yes.
And then as I add things, I would likely have to pass those as well, so I can see where your coming from, but haven’t reached it as of yet.
> I’m more asking about “don’t pass the system around” which is a common advice… That question was the interesting one to me as well. I would like to see an example of an alternative approach. I figured what I do is fine (for now) because it’s not a map with random things that I pass in, but a curated map….which maybe is the same thing, but with more discipline? 😆
As in, I can define which parts of the overall system my component will need in the system definition
of course, it's possible to have a component (say web handler), which require everything - but you never pass the whole system explicitly
having a bunch of keys in a map that you don’t care about isn’t an issue. it can, but doesn’t necessarily lead to some other problems like: • since every request may interact with any key, reasoning about the system becomes strained • producing information for the request that may not be needed may introduce noticeable overhead • others? It might be useful to further clarify the problem.
Another underlying problem might be that requests are strongly coupled to specific implementations of resources they need. For example, rather than just requiring some necessary user info, they are instead interacting with a mongo connection.
Yes, we use protocols for “interfaces” that are mature and we know every component is using them to avoid spreading internal details.
However, we still have some details for some things that are not very well mature yet. We prefer to feel the pain first and then refactor later. For that reason, we have integration tests which cover the interaction between components.
If I remember correctly, we have 31 components in our System. But the web server (and all remaining components) only depends on the DB and Monitoring for now. I think there’s nothing you can do if there’s a dependency between things. You can re-think if you really need a dependency. Our project is doing mostly data pipelines so they are more or less independent from each other. We are now breaking them into different “services”.
We have an "application" component which is reused across most of our web apps (which also have a "web server" component and a few others). That "application" component contains database (multiple connection pools), environment, caches, ElasticSearch, Redis, template engine (wrapper around Selmer that installs tags and filters at startup), and "SDKs" for communication with some of our subprojects, plus a few other things. Then there are a handful of other components that are used in some apps but not others (e.g., Microsoft Active Directory, email-based error logging, "presence" -- for tracking online members on our sites).
We do not try to create interfaces/protocols/APIs that hide all the implementation details except where we genuinely have multiple implementations.
we kind of lean on existing interfaces/protocols, some of our components implement clojure.lang.IFn, so you can look up a configuration value by calling the configuration component as a function
the nice thing about re-using interfaces and protocols is you can get mocks/stubs in tests "for free", so like if we need to mock the configuration component in a test for some reason, because it acts like a function, you can just use a function
IFn on a component is a nice affordance -- and work is where I got the idea to do that for
Interesting that we never run into 'too many dependencies passed in' problem in our applications - perhaps it's because we run a SOA?
So I guess it’s the application component that in the end acts as the “orchestrator” and gets everything else as a dependency I guess, right @seancorfield ?
I’m curious to see how component is used without records/protocols, is it just a map with dependencies then?
@orestis Several apps have that application component as a subcomponent, but that's where most of our system dependencies live because that's our "core" system across most of our apps.
When I said no "interfaces/protocols/APIs", I meant we don't write those to wrap subsystems -- which you see as examples sometimes of "how to use Component". I'm not talking about the two lifecycle methods of Component itself.
That said, we do have non-record implementations of Component's lifecycle. And
next.jdbc works that way too.
My original question came from an argument at work - why have a protocol and a record where you could just have a function?
So it's an empty hash map with a
start function and then a function with a
stop function attached.
Especially in the case of non-stateful components where you just need the dependencies
I asked Stuart Sierra about enhancing Component to run dependencies via metadata and he felt it was too narrow a need (because only a few things can carry metadata that aren't already associative). So, if you need dependencies, you need a hash map or a record (but you don't need the protocol implementation if the component has no lifecycle). And if you don't need dependencies, you can have anything that carries metadata.
See https://github.com/stuartsierra/component/issues/67 for his response.
To rephrase: most protocols have “this” as their first argument. Is there any point in making a protocol + implementation (reify or record) vs just defining a “public” function that takes “this” as a first argument where “this” is expected to be a hash map with dependencies?
The caller shouldn’t care because to the caller the dependencies are opaque - record or hashmap makes little difference
Right, so if you don’t care for that, you don’t really need protocols. Tests can always redef the public function if they need to stub out stuff.
a def is a global thing, so is a redef, so if you have two instances of a component, redefing the public function to behave differently for one of them becomes a chore
That’s true, but then you are back at square one if you want testability, right? A protocol or multimethod...
at my last job some people really hated records and protocols, so we had what was basically a fork of component where the lifecycle protocol was replaced with multimethods
it is ok, but you can't just
(reify ...) up something that those multimethods will do the right thing with in the middle of a test if you need to stub/mock
It’s just annoying that re-evaluating protocols breaks existing implementations and you have to reload everything.
our protocols are pretty all defined in a namespace like *.protocols which pretty much only contain protocols and almost never change (don't need re-evaluating)
which is a common pattern you can see https://github.com/clojure/core.async/blob/master/src/main/clojure/clojure/core/async/impl/protocols.clj and https://github.com/clojure/core.logic/blob/master/src/main/clojure/clojure/core/logic/protocols.clj
if you use multimethods, I recommend doing something similar, pull the defmultis (the interface definition) into a distinct file
(I wish I'd followed that pattern more rigidly with
next.jdbc -- I mistakenly put a few protocols in with other code!)
I do like the idea of a Component-variant that uses just functions (with metadata for dependencies as well as lifecycle hooks).
My takeaway from this is that if you want to be able to stub out things protocols are needed. Put them in another file. See if you can implement IFn then the protocol is just a function which can easily be stubbed.
Perhaps if a record cannot be avoided, defer all the logic in a plain function and implement the protocol by just forwarding the calls to the plain function, this way you can REPL away without having to tear down your system.
ClojureScript does it this way I think, a lot of protocols that just forward the call to a similarly named function...
clojurescript does the reverse of this, a lot of functions that forward the call on to a protocol function
I don’t hate anything 😅 just trying to figure out stuff, without cargo culting some approaches blindly.
About the IFn approach, is that where your protocol would have a single function so it’s replaced by a record that implements IFn?
It doesn't need to be "a record that implements IFn" -- it can just be a function.
It only needs to be a record if you have dependencies and you want it to implement IFn as well.
Right, I meant that to the caller the component is a function. If it’s a plain one or an IFn record or a closure is an implementation detail...