Clojurians Log v2

I don't see how the code above would produce the error below. Given that the stacktrace mentions some-other-caller, the code was somehow modified. Can you post the actual, full code? Does what you're doing work if you unwrap (d/future ...)?

BTW I would rename db-conn to db-src. Since it's not a connection but a data source.

Thanks, I renamed it. It would be too much to post the real code here, but I have just figured out the problem - which leaves me with a new question... The problem occurred during an integration test in which I stub a few methods by overriding the integrant system config (as per your recommendation https://clojurians.slack.com/archives/C053AK3F9/p1753354632561999?thread_ts=1753348946.178649&cid=C053AK3F9). The stubbing itself works like a charm. However, in order to set up my fixtures I use this function:

(defn system-fixture
  ([system-overrides]
   (fn [f]
     (if (seq system-overrides)
       (println "Wrapping tests in system-fixture with stub overrides:" system-overrides)
       (println "Wrapping tests in system-fixture without stub overrides"))
     (letfn [(with-app-buildup-teardown [f]
               (core/start-app {:opts {:profile :test}
                                :overrides system-overrides})
               (f)
               (core/stop-app))]
       (cond
         (every? nil? [@core/system ig-repl-state/system])
         (with-app-buildup-teardown f)

         @core/system
         (do
           (core/stop-app)
           (with-app-buildup-teardown f))

         ig-repl-state/system
         (do
           (ig-repl/halt)
           (set-custom-prep! {:profile :dev} system-overrides)
           (ig-repl/go)
           (f)
           (ig-repl/halt)
           (set-custom-prep! {:profile :dev})
           (ig-repl/go))))))
  ([]
   (system-fixture {})))

(defn system-fixture
  ([system-overrides]
   (fn [f]
     (if (seq system-overrides)
       (println "Wrapping tests in system-fixture with stub overrides:" system-overrides)
       (println "Wrapping tests in system-fixture without stub overrides"))
     (letfn [(with-app-buildup-teardown [f]
               (core/start-app {:opts {:profile :test}
                                :overrides system-overrides})
               (f)
               (core/stop-app))]
       (cond
         (every? nil? [@core/system ig-repl-state/system])
         (with-app-buildup-teardown f)

         @core/system
         (do
           (core/stop-app)
           (with-app-buildup-teardown f))

         ig-repl-state/system
         (do
           (ig-repl/halt)
           (set-custom-prep! {:profile :dev} system-overrides)
           (ig-repl/go)
           (f)
           ;; (ig-repl/halt)
           ;; (set-custom-prep! {:profile :dev})
           ;; (ig-repl/go)
)))))
  ([]
   (system-fixture {})))

I am using the above function like so (use-fixtures :once (system-fixture system-overrides)), where system-overrides is a map used to override the integrant system

with-app-buildup-teardown should use try-finally so that stop-app is called even if (f) throws. The (cond ...) after that doesn't check when both @core/system and ig-repl-state/system are set. But maybe that's somehow impossible. In any case, don't use integrant.repl.state for tests. It's a global state, and you don't want that for tests. Use integrant.core itself, pass systems around as plain values or use your own dynamic vars in fixtures to store them instead. Likewise, there should be nothing like @core/system in tests. I don't know what it is, but it's a global deref'able thing which is unlikely to have a thread-local value. But dynamic vars are thread-local.

Sounds all good and logical. I'll get back if I can't get it to work, but you gave me really good pointers. Thank you very much!

Any time. :)

I have now changed my test utils to use integrant.core and dynamic system vars for my fixtures, like so:

(defn system-fixture [system-overrides]
  (fn [f]
    (let [test-system (create-test-system system-overrides)]
      (try
        (binding [*test-system* test-system]
          (f))
        (finally
          (halt-system! test-system))))))

(finally
  (halt-system! test-system))

;; Run test -> calls handler -> makes future with `other-handler`

;; within `handler`:
(d/future
  (other-handler db-src ...))

Halting the system causing issues is just a symptom of you not waiting for the work to finish. The root cause can lead to a plethora of other issues down the line, many of which could be near impossible to debug due to the inherent race'y nature. First of all, why do you need future inside of a test? Why not just use the required functionality directly?

The future is not inside my test, it is inside the function that I am testing

It's an integration test that shall test an entire request-response pipeline

Is the future returned the caller? If so, you can wait for it. If not, maybe you can make it be returned.

That makes sense. I guess I could return it and then wait (deref?) it in the test. Only issue here is that I wouldn't know how to properly serialize this... The function being tested is the entrypoint (basically "controller" if you will) of a web request. It returns something like this:

(response/ok
  {:data
    {:type "email-processing-job"
      :id incoming-email-processing-job-id
      :attributes {:status "started"}}})

> wait (deref?) Yes, just @ it and ignore the result. > The function being tested is the entrypoint (basically "controller" if you will) of a web request So nothing waits for that future, it's purely "fire and forget"?

Correct. The longer story is that I am actually calling an external service, which in turn calls a webhook on my server once it has finished, but only if my controller produced a non-empty vector of datapoints. In the case this vector is empty, it does not make sense to call the external service - but I still want my flow to continue. Since now the external service won't call my webhook, I am manually calling my webhook handler with a fire-and-forget future.

But why does calling your own webhook requires db-src? Surely when that third-party service does that, db-src is not required, at least not at the API level.

The simplest solution, if it's possible, would be to avoid passing such things around when they aren't passed around when the functionality is called in a "real" way. A long-term, more universal but more involved, solution would be something like the "Function Core, Imperative Shell" architecture. tl;dr: don't use side-effects willy-nilly, push them to the boundaries. In the case of a handler, you'd have the handler function itself and an impl function. The handler calls the impl, the impl returns a collection of things that must be done (effects) and the actual result, the handler acts on the effects and returns the result. Then, your test could use the impl and ignore any undesirable side-effects.

> But why does calling your own webhook requires db-src? Surely when that third-party service does that, db-src is not required, at least not at the API level. It's not that the API requires db-src to be passed. But my webhook handler (like all my controllers) gets passed a request map. Since I wrapped my server, routes and ring handler in my integrant system, request contains the db-src. Looks like this in my system config edn:

:server/http
 {:port 3000
  :host #or [#env HTTP_HOST "0.0.0.0"]
  :handler #ig/ref :handler/ring}

:router/routes
 {:routes #ig/refset :reitit/routes}

:router/core
 {:routes #ig/ref :router/routes}

:handler/ring
 {:router #ig/ref :router/core
  :api-path "/api"
  ...}

:reitit.routes
  {...
    :services
    {:db
      {:source #ig/ref :db/source}}
   ...}

(defn resume-flow [request]
  (let [db-src (get-in request [:reitit.core/match :data :services :db :source])]
    (other-handler db-src ...)

    (response/ok ...)))

I think what is confusing here is that I mixed up the terms "handler" up a bit with what I believe you'd call "impl"

> Integrant is all about passing around systems I wouldn't say so myself. I never pass around the whole system. If I have, say, 5 routes, I'll have at least 7 Integrant components - one for each route, one for the router, one for the actual server. Each route then can receive exactly what it needs and nothing more. Or, if you have 10s or 100s of routes, you can split them into some semantic groups and have those be separate components where each gets what it needs. That approach also means that you don't have to enrich the request data in any way - the request can stay separate from the Integrant-supplied values. But that doesn't really help you with your issue. > as opposed to, say, Mount which IIRC uses global vars Yeah, just don't use Mount, it's incredibly bad to work with in a sizeable system. Convenient in a few things, forbidding in quite a few. > I think based on what I described above, I am already doing this? I didn't necessarily mean passing db-src explicitly. I meant it being available somewhere in the arguments and being required. And if that functionality does need db-src then there's no easy way around it. (The "functional core, imperative shell" is not easy to just switch to - there's a bit of work involved.) > I think what is confusing here is that I mixed up the terms "handler" up a bit with what I believe you'd call "impl" I don't think it's a very clear term anyway, often context-specific. And here I meant "the function that gets called when a route is matched, accepts a request and returns a response". Anything that it calls is not a handler by itself, it's just an impl detail.

> If I have, say, 5 routes, I'll have at least 7 Integrant components - one for each route, one for the router, one for the actual server. > Each route then can receive exactly what it needs and nothing more. > Or, if you have 10s or 100s of routes, you can split them into some semantic groups and have those be separate components where each gets what it needs. This is actually what I was planning on doing, just so far with two endpoints did not feel the urgency. > That approach also means that you don't have to enrich the request data in any way - the request can stay separate from the Integrant-supplied values. This I do not understand. If I do not add it (e.g. db-src) to the request, then how else would I get access to it in my controller/handler? Where else does it come from?

;; config.edn
:some.db.ns/src
{...}

:some.ns/some-route-handler
{:db-src #ig/ref :some.db.ns/src}

(defmethod ig/init-key ::some-route-handler
  [_ {:keys [db-src]]
  {:get (fn [request]
          ... this here has both `request` and `db-src` ...)})

Ahh you’d make the handler a ig component as well. Okay that makes a lot of sense. Doesn’t my system config grow gigantic very quickly this way? Would one eventually split it up?

Sorry for all these questions btw, you already helped me a ton understanding this!!

No need to be sorry, that's exactly what #beginners is for. I wouldn't be here if I didn't want to answer. :) > Doesn’t my system config grow gigantic very quickly this way? If you keep using a component per handler - sure. But if there are pronounced groups, you can use refset and have a component per group. Or yes, you can split that EDN into multiple files. Hmm, might even go the Spring route and write some glue code to make components demand what they need so it would be automatically retrieved from the system if it's not already available... But I've never tried it and it sounds icky.

So many open questions about integrant just clicked into place. Am genuinely excited about this. Say I implemented this pattern (which I definitely will), then how does this solve my „stale system data in future" problem in the test run from earlier? I am not sure how I would adjust my calling structure in my main handler and the resume handler

Conceptually, you have this:

(defn impl [request db-src]
  (future (do-something request db-src))
  (make-response request db-src))

(defn handler [request db-src]
  (impl request db-src))

(defn impl [request]
  {:response (make-response request db-src)
   :effects #(do-something request db-src)})

(defn handler [request db-src]
  (let [{:keys [response effects]} (impl request db-src)]
    (run! future-call effects)
    response))

Ah, and what I mentioned about Integrant and separating db-src from request has nothing to do with the above. It was a separate comment.

Got it, thanks! Will give this a try and see how far I get.

> So when my external service calls the webhook (let's call it "resume-flow"), I have a fresh db-src from the request object. However, in the other case, when I call the other-handler (which is wrapped by resume-flow) manually, I need to pass in the db-src . Have you considered having your future actually make an HTTP request to yourself, pretending to be the external service, rather than try to bypass it and call the handler directly? It seems quite a bit simpler to me, as it requires no extra code path for the special case.

I kind of assumed that was an anti-pattern - but maybe it's not. Will think about it.

FWIW, @p-himik your last thread post reminded me of https://day8.github.io/re-frame/dominoes-30k/#domino-2-event-handling, when I dabbled in CLJS/re-frame. If I understood your example correctly, the analogy to re-frame would be roughly: coeffects --> (map that contains) db-scr event --> request . Makes me wonder if there is a similar library for CLJ that enforces clear separation of side effects like re-frame does.

Re-frame works in CLJ as well. But I don't think I'd use it there. And for something as simple as the approach I've described, you don't need any library at all.

> I kind of assumed that was an anti-pattern I'd say it depends on other of things. Suppose you have middleware that requires auth - now that future has to somehow do auth. Or middleware that gathers metrics - now you somehow have to differentiate between a third-party service hitting your endpoint and your own future hitting it. And so on. In other words, I'd say might be easy to go that route, but it's also likely to move you in the direction opposite of simplicity.

The nil is the actual return value of the function! The string you get printed out is the 'side-effect' that calling print outputs, but it actually returns nil in your program So if you evaluated (def myvar (print "hello")) "hello" would print out, and then if you eval'd myvar your repl would give you nil, because thats the actual value returned by print (because every function has to return a value)

Excellent, thank you! I sometimes get tripped up needing to know the why. Much appreciated.

Always good to ask questions! Side-effects (I/O) is an odd business :^)

This is a common issue that faces beginning programmers in languages that have a terminal-based shell (or REPL). The core issue here is that we have a single terminal in which two very different things are getting printed. On the one hand, the job of the shell is to print out the result of your functions; on the other hand, the job of print is to print out its argument. It just so happens that in a shell that runs in a terminal, those two print to the same place. As an aside, I've been pondering the value of starting students out with a shell for this very reason; the confusion goes away with time and experience, but there are alternative ways to introduce people to programming that do not create that confusion in the first place. The short version is that if the same code runs in a real program, this:

(defn f [] "a string")

(defn g [] (println "a string"))

https://clojure.org/guides/learn/syntax is a helpful page to read here

This particular area is covered there in the printing section but you might find this or subsequent pages to be generally useful!

Thank you, yes, I've been reading the docs, but keep forgetting about the guides. I"m currently working through "Clojure for the Brave and True". Slowly, but at a pace that works for me. I also have your book @alexmiller and look forward to digging into it. Thank you for the reminder about the Guides.

And seriously, thank you all. I've joined a few communities over the years like this but have never found one as helpful as Clojure.

Keep asking questions!

I have spent most most of my weekend reading the guides and REPLing with some of the examples. I plan to get back to "Clojure for the Brave and True" but the guides are definitely where I should have started. Too bad the tag is deprecated. A nice bright ing "Start here!!" would be appropriate.