Clojurians Log v2

Oh. Ohhh... It seems that whatever context is set up by around is only available inside it. Whereas my test looks something like this:

(defdescribe the-test
  {:context [setup-db]}
  (let [db-conn (get-db-conn)]
    ...))

hmmmm

i don't remember designing around to work like X-each, but it's possible i implemented it like that

The "Run Lifecycle Overview" section doesn't even mention around. But it is briefly mentioned before: > then the around hooks are called on the nested tests (if they exist)

if you wanna, open a github issue noting this so i can remember to look next time i'm at my computer

looking at the source on my phone, i don't know why around doesn't work as expected, but i'm missing tests for these cases so... dang

Done: https://github.com/NoahTheDuke/lazytest/issues/24

Huh. Maybe there is something else going on in addition to that issue. I thought I would be able to fix it with an extra (it "" nil) at the top - but no. Hmmm...

jk i think i do know the reason, had to think for a moment. the let binding happens immediately, when the test suite is constructed, and then the around is called when the test suite is run. these are different steps, and it's a little unclear from looking at the code

a solution is to move the let into the around, maybe use a dynamic variable instead of a let bind

I see. Does it not make sense to also defer the let bindings till the test is actually run? Otherwise it seems like it has potential of breaking other expectations. Like e.g. suppose that let does some heavy setup just for that test, and the test is disabled via metadata. The way you described it makes me think that the setup would still run, taking time and resources, and then would be discarded.

yes, that is a potential issue

there's a fundamental tension here, given clojure's emphasis on lexical scope and immutable variables

lazytest is built to emulate mocha or rspec, so one might write

describe("outer", function() {
  let dbConn;
  around(function(f) {
    dbConn = expensiveCall();
    f();
    dbConn.release();
  });
  it(...);
});

but that's not easily done in clojure with let unless you use dynamic variables or volatiles

But surely it's possible via macros? Or not... I just tried "fixing" it on my end by wrapping the body of the whole test in (it "sets up db-conn" ...), and the test passes now after 19 seconds, which is around what I'd expect. However, I put (expect false) in there. And the only thing that's reported is

fit2breed.breeder.test-matings-test
    test-matings-test
      √ sets up db-conn

Oh, and of course even if I use a dynvar, I cannot have any setup in the outer let at all, even though its results are needed by multiple it. So it seems that the only way for me to get that setup to work is to use another fixture that's specific for that very test..?

Yeah, this bit me a couple of times until I developed a better mental model of how the test stuff worked. The docs could give more examples to clarify the behavior, I think.

We "get away" with it since we tend to build a system Component in a ns-level fixture and that has a connection pool, so getting a connection repeatedly in a test is low overhead.

Right, for db-conn it's not that much of an issue, although it's still a tiny nuisance. But the real trouble comes from heavy setup needed by many its. Of course, I can wrap everything in delay, but I don't wanna. :) Or not?! WTH. With delays in the setup, the test runner doesn't even output anything. Did not expect a Saturday puzzle, heh.

The previous post here - two months ago - was also a confusion with let and it that I think more docs would address...?

The heavy setup issue is why I went for set-ns-connect! and split tests across multiple nses:grin:

Ooof.

I still don't have a good enough mental model to write up how around works across nested describe / it combos 🙃

lol maybe lazytest is too magical, whereas something like mocha is very obvious. each describe call results in a map that has :context and :children vectors, which can be passed around. it calls result in a TestCase object that wraps the body in a function.

in mocha, the functions must be written by the developer

I will note that before Lazytest we already used dyn vars with clojure.test fixtures quite a bit - the equivalent of around...

when you use a let binding in a describe, it's like saying (let [...] {:children [(fn [] (do stuff ...))]})

(and we didn't do setup in let - we did it in the fixtures which execute around the test)

all of this can be clarified, you're both right

lazy test can't code walk because people use macros that do all sorts of stuff, and we can't/won't macroexpand because that's a whole heap of trouble

but i can add documentation warning against expensive calls in lets or even using lets in describe blocks entirely

Is there no way to just make around work around the whole test body, regardless of it?

I think the design is solid, just not fully intuitive. Specific examples of heavy setup used across multiple tests in a single suite would help I think. Show how to do X (rather than just don't do Y).

lol i would say yes but your experiments are saying no, so i'm unsure

i have a lot of tests making sure that before and after and before-each and after-each all properly nest, but only a couple for around

i gotta add those in

My experiments are very superficial and don't involve changing lazytest itself, so they should be taken with a 1 m3 of salt. I'm just fiddling around to see how I can at least make it work on my end. And figure out that delay conundrum.

Alright, I think I'll crawl back to clojure.test for now, the behavior of lazytest is way too different for me to handle at the moment. delay works. I thought it didn't work because it surfaced another issue, one for which I have no answer at all. The particular test in question also uses ns-publics and retrieves project-specific :test-data from each var. And for some reason it doesn't work with lazytest. Namely, I have this fn:

(defn get-vars-with-test-data [ns]
  (filterv (fn [[sym var]]
             (when-let [td (:test-data (meta var))]
               [sym var td]))
           (ns-publics ns)))

Oh, hold on. How am I this dumb and how did it work in the first place? Jesus... Now I have to figure that out.

my apologies that lazytest has been such a difference lol, it's quite a different paradigm

Yeah... but the realization of my idiocy gave a newfound strength to pursue it for longer. A delay here and there is not the end of the world and so far to me it worth it that I get the output of each it right as it happens, as opposed to the output of testing being delayed till the very end of testing.

user=> (def ^:dynamic *one* nil)
#'user/*one*
user=> (def ^:dynamic *two* nil)
#'user/*two*
user=> (defdescribe a {:context [(around [t] (println "around 1 before") (binding [*one* 1] (t)) (println "around 1 after"))]}
  #_=>   (println "in a")
  #_=>   (it "test 1" (println "in test 1" *one* *two* "*one* *two*") (expect (zero? (- 1 1))))
  #_=>   (describe "nested 1" {:context [(around [t] (println "around 2 before") (binding [*two* 2] (t)) (println "around 2 after"))]}
  #_=>             (println "in nested")
  #_=>             (it "test 2" (println "in test 2" *one* "*one*") (expect (= 1 (* 1 1))))
  #_=>             (it "test 3" (println "in test 3" *two* "*two*") (expect (= 2 (+ 1 1)))))
  #_=>   (it "test 4" (println "in test 4" *one* *two* "*one* *two*") (expect (= 3 (+ 1 1 1)))))
#'user/a
user=> (run-tests *ns*)
in a
in nested
around 1 before
in test 1 1 nil *one* *two*
around 2 before
in test 2 1 *one*
in test 3 2 *two*
around 2 after
in test 4 1 nil *one* *two*
around 1 after
Ran 4 test cases in 0.00300 seconds.
0 failures.

{:total 4, :pass 4, :fail 0}

Yeah, it makes sense after realizing how it works. It's just that I myself would definitely prefer

around 1 before
in a
around 2 before
in nested
in test 2
in test 3
around 2 after
around 1 after

Right, the mindshift is that describe runs (as "normal" code) and builds a test suite with fixtures, and then run-tests runs the fixtures and tests in that suite. (does that sound right @nbtheduke?)

I updated the example above to show nesting of dynvar bindings.

yes, i believe that's right

Now the tests are prettier than ever, and I get the output right as it's ready, nice.

(The long full var name is needed for Cursive to turn the file_name:line_num part into a hyperlink).

that's so cool

is that a custom reporter?

Nah, built-in.

The colorful (f x y) form is made with Puget and fed into it.

(defn testing-scope
  "Reports the location of the var in a way that
  at least IntelliJ IDEA understands and highlights in its terminal."
  [form var]
  (let [{:keys [file line ns name]} (meta var)
        location (str (ns-name ns) "/" name
                      (when file
                        (str " (" (.getName (io/file file)) ":" line ")")))]
    (str (puget/cprint-str form {:width 1000}) " - " location)))

that is so cool!

Oh, and as I mentioned I use :test-data metadata on plain functions to specify test data for those functions. Very similar to :lazytest/test but does extra setup. Namely, sets up that db-conn and anything else that functions in a specific namespace might need. Maybe it would make sense to add something like this to lazytest as well - some kind of :lazytest/test-data with :lazytest/test-setup-fn or something like that, dunno.

do you have an example? i don't quite understand the connection between metadata and usage

The overall shape I currently have is like in the snippet.

Interesting. So your functions-under-test have their own are-style test data attached as metadata, and then you have your own test harness that explicitly runs the f.u.t. for each pair of input/output...

Yes, and you took note around a year ago or so. :D

that's clever

Oh, more than three years ago, actually. https://clojurians.slack.com/archives/C03RZGPG3/p1650317272864349

Haha... that's almost pre-history!