Fork me on GitHub
#integrant
<
2021-11-10
>
JAtkins00:11:50

So, this is absolutely a beginner question. I’m currently fiddling with Integrant instead of Mount for a simple project, and I’ve run into a weird code smell. I have a top level component (for example a mem-db) that is used widely in the system. I have a sub-component that is also widely used who’s functions will always need access to mem-db (let’s call this thing db-dependent-thing). I’m currently rewriting the functions in this namespace to have the signature [mem-db db-dependent-thing & args] , but this feels icky. The other two immediately obvious solutions are: 1. to exchange the args for [{:as sys :keys [mem-db db-dependent-thing]} & args] - where the first arg is the system, and 2. changing the ig/init-key for db-dependent-thing to store the mem-db inside the db-dependent-thing object. So, my question is: which (if any) of the above options seem the most idiomatic way forward based on your experiences? (sample ish code)

(defmethod ig/init-key :my.example/mem-db 
  [_ _]
  (atom {}))

(defn assoc-db [mem-db k v]
  (swap! mem-db assoc k v))

;; ...

(defmethod ig/init-key :my.example/db-dependent-thing
  [_ {:my.example/keys [mem-db]}]
  (assoc-db mem-db :initial :data)
  (atom {}))

(defn db-dependent-thing--action [mem-db db-dependent-thing & args]
  ;; do stuff
  )

rickmoynihan12:11:33

If I understand your question, you should do 2. Integrant is a dependency injection system, and components should close over their configuration and dependencies. It’s not the job of another component to give a component its dependencies; though it may supply it with runtime data — e.g. a http request. 1. seems to like you’re trying to pass the whole system around as an argument. You definitely shouldn’t do that. Within a component you may pass a context map that gives functions making up a given component their configuration and dependencies. You should only really access the system like that in a development / debugging context, not in the real code.

rickmoynihan12:11:16

Also your code snippet seems odd to me. Why does ::db-dependent-thing return a new (atom {})?

rickmoynihan12:11:44

it would be more helpful if we knew what these two components were

JAtkins18:11:34

This might be a bit more clear. I went with #2 here:

(defn -start-transactions-thread! [processing-system]
  (future
    (while (::running? @processing-system)
      (if-let [tx (-pop-tx! processing-system)]
        (-run-transaction! processing-system tx)
        (Thread/sleep (::threaded.sleep-ms-between-checks @processing-system))))))

(defmethod ig/init-key ::processing-system
  [_ {::keys [synchronous?
              ;; threaded.sleep-ms-between-checks

              ;; This controls the system that can be used by coeffects during processing. 
              ;; Should contain at least ::core-parser/parser-env and ::state/app-state
              coeffects-system
              running?]
      :as    opts}]
  (let [sys (-build-processing-system opts)]
    (when running? (-start-transactions-thread! sys))
    sys))
Basically, the tx processing system is an atom that contains a queue, and when integrant starts the system the tx processing thread should start as well. The fn -run-transaction! needs access to the system, since arbitrary things can be done against the system within the context of a transaction.

rickmoynihan10:11:18

Ok it sounds to me like you’re structuring things and thinking about integrant in the wrong way; but it’s hard to follow exactly what you’re doing and why. There are very few absolutes in software engineering, so I’d never say never — but I’m suspicious that you’re not using integrant how it’s intended. Integrant is good at representing and managing static systems, not so much dynamic ones. A fixed set of components are started and stopped together, in an order determined by their dependencies. It’s essentially a dependency injection system, so components in that system should never look at the whole system. If your tx-processor needs to see other components to do its job, then they should be provided to it as explicit dependencies i.e. ig/ref’s. I would typically expect something like a transaction manager to operate on runtime data. Runtime data — i.e. the data that is in flow; should not be in the integrant system at all.

rickmoynihan10:11:25

For example a classic web app is a static system. At a high level and simplified it will typically have the following static components: • a web server • a connection / connection-pool to a database • a handler (your apps function) Components do not come and go whilst the system is running, and no component see’s the whole system, or interacts with it directly.

rickmoynihan10:11:44

Further more the web-server won’t have direct access to the database, your app handler will be given that, and the handler given to the web server. The database will then be encapsulated and hidden from the web-server behind the handler.

JAtkins20:11:53

Hm. I think I understand what you are saying. It does clear up a bit of my confusion. I think I’m sorta riding the line of what integrant is designed to do. Basically, I’ve got maybe a dozen pieces that have a clear dependency order. They all maintain a bit of their own state, and the system can be started in 5 or 6 partial states. Partial states are possibly useful for testing. I could start the whole system I suppose, but I enjoy having an explicit configuration documenting the pieces of the system required for some bits of functionality. Maybe that isn’t the best tradeoff here though? Needs more thought. You are right though, the system currently stores its state in keys of the configuration. For example - the tx processing system contains a mutable queue. The parser contains a mutable pathom environment where additional functionality can be pinned on.

rickmoynihan08:11:04

I don’t know, so far I’ve not heard much that puts you outside of integrants sphere of relevance. FWIW in my experience, with a little bit of effort, integrant can work well with testing. I’m assuming by partial state; you just mean subsystem. If that’s the case integrant works quite well for this, as integrant lets you init any subsystem from the wider configuration, by simply passing (ig/init config [:subsystem-component/to-start ,,,]). Integrant will then ensure that any dependencies of those keys are also started. We use integrant a lot in many of our projects. The largest of which has almost 4000 lines of configuration; specifying 300 keys of configuration when running; though we also maintain several different profiles of configuration, essentially customer specific customisations, with two or three slightly different environment profiles overlaid per deployment. The system configuration is then essentially built by (apply meta-merge [base-profile customer-config env-profile]). It’s more complicated than this in practice; with a few more config files — but that’s the main idea. At this scale it can be a little awkward to manage the configuration; but 95% of the time you just add or edit a key in the right place and it just works.

Noah Bogart01:11:02

I’ve done the former and that’s how I’ve seen it done a couple times but I’m not sure if it’s idiomatic

Noah Bogart01:11:14

In my app I init the db connection (mongodb) to :system/db, and then in my compojure init, I add it to every request so all of my request handler functions have access to it too:

(defn wrap-db [handler mongo]
  (fn [req]
    (handler (assoc req :system/db (:db mongo)))))