Fork me on GitHub
#beginners
<
2024-06-24
>
Jim Newton08:06:39

question about clojure.test … looking at the documentation of is it does not say (as far as I see) that it suppresses exceptions. But from my experimentation, it seems to be doing so. I’d like my test case to fail whenever is fails. maybe I should not be using is but rather something else. How should I rewrite the following so that the :no test does not executed because the :yes test failed?

(use '[clojure.test :only [is]])

(do (is (= :yes @(deref (future (throw (ex-info "test error" {:test 1}))))))
    (is (= :no @(deref (throw (ex-info "test error" {:test 2}))))))

Jim Newton08:06:32

In my actual code there’s only one literal is but it is in a loop. and it seems the loop continues, even though one of the is fails

Jim Newton08:06:43

ahh looking at the code for is and try-expr it seems that it purposefully continues even if the assertion failed.

(defmacro try-expr
  "Used by the 'is' macro to catch unexpected exceptions.
  You don't call this."
  {:added "1.1"}
  [msg form]
  `(try ~(assert-expr msg form)
        (catch Throwable t#
          (do-report {:type :error, :message ~msg,
                      :expected '~form, :actual t#}))))

Jim Newton08:06:20

What should I use rather than is or do I need to write my own is which has different semantics?

daveliepmann08:06:28

maybe thrown? — see clojure.test namespace docstring. it is admittedly an odd construction

Jim Newton08:06:05

@U05092LD5 not exactly sure what you mean. are you suggesting (is (thrown? …)) ? If so, no, that is not the semantic I want. I usually want to test for a particular value, but in case an exception is thrown when evaluating the expression, I want such reported, and the test to fail.

👍 2
Jim Newton08:06:18

sorry, if I misunderstood your suggestion

andy.fingerhut09:06:23

Unless I'm forgetting (which I might be), clojure.test/is by default is intended to run many test cases, reporting failures, but continuing to try other tests that might fail or succeed independently of that one.

Jim Newton09:06:51

yes, i’d like it to run the other tests defined by deftest, but if I have a loop that tries to call f on 100000 different input values, I’d like the loop to abort as soon an the first is finds a problem.

Jim Newton09:06:13

It sounds like is is the wrong candidate. I should be using something different.

👍 2
andy.fingerhut09:06:24

I mean, you don't need to use deftest, is, are, etc. You can write a file of Clojure code that executes whatever you want, printing or not printing whatever you want, doing a (System/exit 1) if anything goes wrong, according to what you think of as going wrong.

genmeblog09:06:46

what about just one is and a function which do 100k calls or a sequence of 100k results? something like: (is (my-100k-calls-wrapper) or (is (every? identity (repetedly 100k #(f (rand)))

Jim Newton09:06:08

I don’t want the system to exit if an assertion fails. I want the tester to report the failure and go on to the next deftest .

Jim Newton09:06:46

@U1EP3BZ3Q yes, one possibility is to transform the for loop around is, to an is around a every? loop. I may do that if my other attempts fail.

genmeblog09:06:17

(defn f [v] (/ 1 v))

(t/deftest some-facts-about-v
  (t/is (== 0.5 (f 2.0)))
  (t/is (every? identity (repeatedly 1000 #(f (rand-int 2)))))
  (t/is (== 2.0 (f 0.5))))

(t/run-test some-facts-about-v)
;; => {:test 1, :pass 2, :fail 0, :error 1, :type :summary}

Jim Newton09:06:43

It is a bit surprising to me that is has these semantics without that being documented in the docstring. The docstring claims that is is a generic assertion macro. I interpreted this as having assert semantics, but instead it suppresses the assert semantics. OK, now I know.

Jim Newton09:06:11

@U1EP3BZ3Q, yes I believe I mostly understand. However, that code transformation is not trivial in many cases. For example here is one particular case.

(deftest t-plus-associative
  (binding [*time-out* polynomial-time-out]
    (testing-with-timeout "plus associativity"
      ;; check associativity
      (doseq [p1 polynomials
              p2 polynomials
              p3 polynomials]
        (is (sut/poly-almost-equal 0.001
                                   (sut/poly-plus (sut/poly-plus p1 p2) p3)
                                   (sut/poly-plus p1 (sut/poly-plus p2 p3)))
            (format "Discovered non-associative input for poly-plus\np1=%s\np2=%sp3=%s"
                    p1 p2 p3))))))

Jim Newton09:06:51

What I’ve done for now is defined my own is macro which simply rethrows the exception rather than suppressing it as follows:

(defmacro util-try-expr
  "modified copy of closure.test/try-expr"
  [msg form]
  `(try ~(assert-expr msg form)
        (catch Throwable t#
          (do-report {:type :error, :message ~msg,
                      :expected '~form, :actual t#})
          ;; now RE-THROW the exception so the testing framework will advance to the next deftest
          (throw t#))))

(defmacro is
  "Generic assertion macro.  'form' is any predicate test.
  'msg' is an optional message to attach to the assertion.
  
  Example: (is (= 4 (+ 2 2)) \"Two plus two should be 4\")

  Special forms:

  (is (thrown? c body)) checks that an instance of c is thrown from
  body, fails if not; then returns the thing thrown.

  (is (thrown-with-msg? c re body)) checks that an instance of c is
  thrown AND that the message on the exception matches (with
  re-find) the regular expression re."
  ([form] `(is ~form nil))
  ([form msg] `(util-try-expr ~msg ~form)))

2
genmeblog09:06:17

Treat is as a recorder of the result. The goal of the suite is to record and report the whole picture of your code.

genmeblog09:06:51

Why do you want to stop testing after an exception?

genmeblog09:06:15

Test all the cases, correct buggy ones, and retest.

Jim Newton09:06:11

If I don’t stop after testing, then I get a log file with 7.5 million lines

2
genmeblog09:06:41

Was there recently in the similar situation (10k cases and most of the raised an exception hanging my emacs). Just limit cases at the beginning adopting testing scenario. Or go for every? and for

genmeblog09:06:02

Or temporarily do not use clojure.test in favour of some script.

genmeblog09:06:25

(funny coincidence: I was working on polynomials recently for fastmath library: https://github.com/generateme/fastmath/blob/3.x/src/fastmath/polynomials.clj)

Jim Newton09:06:23

@U1EP3BZ3Q what you are suggesting, although well intentioned, won’t really work. You are assuming I am trying to test and debug my own code. I’m actually trying to write tests to test all of my student’s code. The tests should be resilient so that any error that my students have in their code has a hope of being reported. Furthermore the tests are not run in batch, but rather in a docker image, using the test-report-junit-xml option. This interface, gathers all the test output into and xml which some other software (out of my control) displays in a user’s web browser.

Jim Newton09:06:49

if the output log is too large, the user never sees anything in the output web page, and never has a clue about what went wrong.

genmeblog09:06:29

Ok, got it! Now I understand your motivation.

2
Jim Newton09:06:44

basically my goal is not to debug my own code, but to write tests to test someone else’s code

Jim Newton10:06:27

I’m surprised if clojure.test doesn’t having any assert-like is. Of course I can copy my own is implementation into all of my projects.

Jim Newton11:06:17

I’ve updated the documentation with a note: https://clojuredocs.org/clojure.test/is

p-himik12:06:00

@U010VP3UY9X What is your test runner? Some test runners support fast failing, e.g. for Kaocha: https://cljdoc.org/d/lambdaisland/kaocha/1.91.1392/doc/cli-fail-fast-option

Jim Newton12:06:24

I’m using lein test, which test runner is that?

Jim Newton12:06:30

@U2FRKM4TW ouch, that sounds pretty scary to change the test framework. The risk is that all the surrounding code which launches the tests in my docker image might need to change. I’ve already had the issue that the xml file generated by lein-test-report-junit-xml "0.2.0" does not conform to that which the web framework (which is out of my control) is using. So after lein test finishes I already have to run a python program which reads in the broken xml file, to fix the inconsistencies. This was a horribly painful process, that I’d have nightmares if I had to revisit.

p-himik12:06:24

Maybe https://github.com/pjstadig/lein-fail-fast? It's old but might still work. Or maybe it can be used as a source of inspiration for writing something that works with the present day lein test.

Jim Newton12:06:29

what does “Top a testing run” mean? I don’t want the testing run to file, i just want the current test to fail, and the next test defined by the next deftest to start.

p-himik12:06:55

Ahh, that's indeed a bit different. But I'm 80% certain it can still be done. clojure.test is rather generic and extensible.

2
Jim Newton12:06:59

(defn wrap [form]
  `(do
     (add-hook
      #'clojure.test/do-report
      (fn [f# & args#]
        (let [result# (apply f# args#)]
          (when (contains? #{:fail :error}
                           (:type (first args#)))
            (System/exit 0))
          result#)))
     ~form))
No, it looks like the hook wraps the call to do-report and calls System/exit whcih would be catastrophic.

Jim Newton12:06:33

yes, it ought be be doable, because to me it is a really reasonable request.

p-himik12:06:33

Indeed. But it doesn't mean that it's the only approach to not running the next is.

2
Jim Newton12:06:27

Looking at the code https://clojurians.slack.com/archives/C053AK3F9/p1719218323960759?thread_ts=1719218139.516939&amp;cid=C053AK3F9 What I want (or think I want) is that after do-report finishes, the throwable t# should just get re-thrown. I’m not sure whether a hook around do-report could have access to the exception being thrown.

Jim Newton12:06:37

on the other hand, the value t# is being passed to the do-report call within the hashmap with key :actual

Jim Newton12:06:42

so maybe it is possible.

Jim Newton12:06:48

@U1EP3BZ3Q btw, how would you suggest to rewrite https://clojurians.slack.com/archives/C053AK3F9/p1719222371847929?thread_ts=1719218139.516939&cid=C053AK3F9to use every? or some How to generate an error message in a scope where p1, p2, and p3 are in scope, but is passed as the 3nd argument of is

Noah Bogart13:06:54

i think your solution of using a custom is macro is the correct one. i do something similar in the netrunner codebase, as it's not useful to continue a test case once a failure has been met (because the test cases are stateful).

2
Noah Bogart14:06:25

If you don't want the stack trace, you can also wrap deftest to catch the thrown exceptions and just not print them: ... ~(vary-meta name assoc :test (fn [] (try (do ~@body) (catch Throwable t#))))`

genmeblog21:06:55

@U010VP3UY9X what about this?

(deftest t-plus-associative
  (binding [*time-out* polynomial-time-out]
    (testing-with-timeout "plus associativity"
                          ;; check associativity
                          (is (every? identity
                                      (for [p1 polynomials
                                            p2 polynomials
                                            p3 polynomials]
                                        (sut/poly-almost-equal 0.001
                                                               (sut/poly-plus (sut/poly-plus p1 p2) p3)
                                                               (sut/poly-plus p1 (sut/poly-plus p2 p3)))))))))

Jim Newton10:06:16

@U1EP3BZ3Q, and if the every? fails, there needs to be a 2nd argument of is which indicates which values of p1, p2, and p3 identifies the counter-example. otherwise the fact that it fails is useless to the user.

Jim Newton10:06:41

@UEENNMX0T I quite like the stacktrace. ah but I guess your suggestion is to remember the millions of failures, but just suppress the many lines of stacktrace?

1
Jim Newton10:06:21

@U1EP3BZ3Q BTW this problem of the simple Boolean being returned from a existential or universal specifier is a larger problem which I address in a work under development: https://github.com/jimka2001/heavybool

👍 1
Noah Bogart14:06:37

yeah if you're trying to test lots of code, better to only get the reference that a test failed so you can mark the student's code as failing

genmeblog12:06:13

what about such construct?

(t/is (empty? (take 1 (for [a (range 10)
                            b (range 20)
                            c (range 30)
                            :let [res (* a b c)]
                            :when (> res 300)]
                        [a b c]))))
:when clause tests for wrong answer and this way we can filter out good cases to show the first wrong.

genmeblog12:06:38

The result of above is:

FAIL in () (form-init14545489860761380894.clj:428)
expected: (empty? (take 1 (for [a (range 10) b (range 20) c (range 30) :let [res (* a b c)] :when (> res 300)] [a b c])))
  actual: (not (empty? ([1 11 28])))

genmeblog12:06:52

or in your case something like:

(defn find-wrong-polys [polynomials]
  (for [p1 polynomials
        p2 polynomials
        p3 polynomials
        :when (not (sut/poly-almost-equal 0.001
                                          (sut/poly-plus (sut/poly-plus p1 p2) p3)
                                          (sut/poly-plus p1 (sut/poly-plus p2 p3))))]
    [p1 p2 p3]))

(deftest t-plus-associative
  (binding [*time-out* polynomial-time-out]
    (testing-with-timeout "plus associativity"
                          ;; check associativity
                          (is (empty? (take 1 (find-wrong-polys polynomials)))))))

growthesque09:06:13

noob question: so clojure maps aren't actually implemented as vectors, but only return kv pairs in vectors for faster analysis, kind of how map returns seqs? i.e. (first {:a 1}) doesn't return [:a 1] because that's how it's implemented but because it's acting as an interface (the map is the interface, i mean) ?

daveliepmann10:06:44

I believe those kv pairs are MapEntrys, not vectors, which are merely printed using vector syntax

daveliepmann10:06:11

(type (first {:a 1})) ; => clojure.lang.MapEntry

growthesque10:06:11

hmm, interesting.

daveliepmann10:06:07

map entries are an under-documented aspect of clojure IMHO

2
daveliepmann10:06:11

see for example https://clojureverse.org/t/why-only-vectors-into-map/4966 which is altogether logical despite featuring a scenario where vectors are treated as if they were map-entries

🙏 2
jpmonettas11:06:40

fwiw when seq is called on map like things they don't always return MapEntry but something that implements clojure.lang.IMapEntry :

(def m1 (array-map  :a 1))
(def m2 (hash-map   :a 1))
(def m3 (sorted-map :a 1))

user> (type (first m1))
clojure.lang.MapEntry

user> (type (first m2))
clojure.lang.MapEntry

user> (type (first m3))
clojure.lang.PersistentTreeMap$BlackVal

user> (instance? clojure.lang.IMapEntry (first m1))
true
user> (instance? clojure.lang.IMapEntry (first m2))
true
user> (instance? clojure.lang.IMapEntry (first m3))
true

👍 4
2
oddsor13:06:56

I’ve stumbled on this while writing a walk-algorithm that looked for vectors. Map-entries are also vectors implement IPersistentVector, so if you for instance try to take one element from all vectors in your nested structure during a walk, you get an exception you might not expect:

(vector? (first {:a 1}))
;=> true
(clojure.walk/postwalk (fn [x] (if (vector? x) (first x) x)) {:a 1 :b [1]})
;=> java.lang.IllegalArgumentException: Don't know how to create ISeq from: clojure.lang.Keyword 
Spent a lot of time trying to understand that error message 🙈
(clojure.walk/postwalk (fn [x] (if (and (not (map-entry? x)) (vector? x)) (first x) x)) {:a 1 :b [1]})
;=> {:a 1, :b 1}

🙏 2
growthesque14:06:45

ah, very useful, will make a note of this thanks

lepistane18:06:38

maybe bit basic overall but what are you favorite libraries and approaches for authentication and authorization? I've been using buddy jwt with access rules per route and so far so good. Though i've been wondering if there is a more elegant way to define access rules ? I haven't seen any new elegant solutions so i am kinda looking to explore what's out there (not saying new = better)

ghaskins23:06:15

This isn’t clojure specific, but we use open-policy-agent as a policy-decision-point for our clojure microservices (https://www.openpolicyagent.org/). It’s awesome

clojure-spin 1
ghaskins23:06:08

Our customers can define their own policy rules

Melody01:06:24

@U13AR6ME1 Seems very cool, I am gonna consider this when I come to the point that I need to implement things like this in my own programs.

dharrigan05:06:55

I've been looking at using and it's API

lepistane07:06:07

@U13AR6ME1 are you using https://github.com/anderseknert/clj-opa or have your own solution ? What's the context and setup you have?

lepistane07:06:53

@U11EL3P9U were you just exploring or you've used it in production?

dharrigan07:06:36

Just exploring atm, for future use. I think (given that I've been down this road many times), hand-rolling a solution (for me now) is tiresome 🙂

dharrigan07:06:40

Let someone else do it 🙂

ghaskins14:06:38

@U45SLGVHV we have our own solution using the OPA low-level API and JNA

ghaskins14:06:03

Most people use the sidecar approach, though, which would be compatible with clj-opa

ghaskins14:06:05

We have specific latency requirements and execution model, so the in-process model worked better for us

José Javier Blanco Rivero18:06:08

Have not tried it yet, but you can check out tempel https://github.com/taoensso/tempel