Fork me on GitHub
#testing
<
2019-01-07
>
Ben Grabow19:01:15

I have some test data (input and expected output) in a text file. I would like to parse the examples from this file and somehow get clojure.test to tell me which examples are failing. Should I try to do something like (doseq [...] (deftest ...)) or (deftest ... (are [x y] (= x y) ...)? I am also hoping to get the test runner to print the actual data from the files, rather than printing the code verbatim that I used to pull data from the files. E.g. I want (deftest file-parsing (is (= (parse "input-file.txt") (slurp "output-file.txt")))) to print the output file contents as the expected value rather than to print literally "(slurp \"output-file.txt\")".

seancorfield20:01:11

I'm not sure I follow @ben.grabow

user=> (deftest parsing (is (= (parse "foo") (slurp "user.clj"))))
#'user/parsing
user=> (parsing)

FAIL in (parsing) (NO_SOURCE_FILE:1)
expected: (= (parse "foo") (slurp "user.clj"))
  actual: (not (= "test" "(println \"top-level user.clj was loaded\")\n"))
nil
That prints the contents of the file (and the value returned from parse) -- is that not what you want?

seancorfield20:01:17

Ah, you don't like the expected value formatting -- clojure.test shows you the specific form from your code that you expected to the truthy and then the actual values that the assertion failed with (for = tests).

Ben Grabow20:01:09

Ah, makes sense. I had some really noisy stuff on the "expected:" line that was distracting me from the fact that the "actual:" line already has what I want.

Ben Grabow20:01:51

Any advice for the best way to generate multiple assertions or tests from a file that has multiple examples in it?

Ben Grabow20:01:00

I wanted to use are but it seems it works at macroexpansion time so I don't know how to do that without literal values.

seancorfield20:01:07

clojure.test isn't really designed for the sort of test you're trying to build -- it assumes a statically known set of assertions in code. I think your best bet is to write some code to read the two files and then write out a deftest with are and the input/output forms in the body -- write that to a file and then load-file it to read it back in and actually define the test.

seancorfield20:01:46

In other words, preprocess your input/output files into a Clojure file that you can then run like any normal test...

seancorfield20:01:03

Hmm... well... I guess you could write a deftest that sucked in the two files and then did a doseq over is but I'm not sure what your failures would look like...

seancorfield20:01:59

No, I think that wouldn't work here -- your inputs are code fragments/expressions, yes?

seancorfield20:01:33

And the outputs are (just) values. So you'd really need the (is …) to include the source from the input so you could see what was going on.

seancorfield20:01:44

I think you have to go the file generation route for that.

Ben Grabow20:01:57

I've parsed my files into a structure like [[input1 ouptut1][input2 output2] ...].

seancorfield20:01:30

But is the input a piece of code you need to evaluate, or just a plain value?

Ben Grabow20:01:35

This seems to give me what I'm looking for:

(deftest all-examples
  (doseq [[input output] examples]
    (is (= output (fn-under-test input)))))

Ben Grabow20:01:39

Just a plain value.

Ben Grabow20:01:03

(where examples is the data structure with the above format)

seancorfield20:01:14

Oh, so you're not really "parsing" the files -- you're just reading them in as values. They don't contain code?

seancorfield20:01:37

And it's all to test a specific function?

seancorfield20:01:51

OK, then the approach you have there is fine 🙂

Ben Grabow20:01:09

Great, thanks for the feedback!

seancorfield20:01:56

If you did a doseq over deftest, it would (attempt to) define a named test for each iteration (so you'd need different names) -- and then you'd still have to both execute that code and run tests afterward.

seancorfield20:01:27

But doseq over is gives you multiple assertions inside a single named test.

Ben Grabow20:01:21

I think the one thing this is lacking is the ability to see in the test runner's output what the value of input was. In the "expected:" line it shows the literal code "(fn-under-test input)" and in the "actual:" line it shows the resulting value from calling fn-under-test.

seancorfield20:01:49

Try (is (= [input output] [input (fn-under-test input)]))

seancorfield20:01:19

That may give you too much output to be useful but it should at least show both input and output values.

Ben Grabow20:01:34

That's a good idea. Another route seems to be to add a testing context label around the assertion: (testing input (is ...))

seancorfield20:01:21

Ah, I always think of that as taking a literal string but I guess it can taking computed values too?

Ben Grabow20:01:34

Sure looks like it! Here's my final test code that I'm satisfied with.

(deftest all-examples
  (doseq [[input output] examples]
    (testing (with-out-str (clojure.pprint/pprint input))
      (is (= output (fn-under-test input))))))

5
Ben Grabow20:01:16

Thanks for the help, Sean. You rock.