I have an interesting case where I call stest/check from the REPL (lein + cider nrepl) and it doesn’t finish. Anyone familiar with that one?

it does terminate when I run the tests from the command line

how do you know it won’t finish? did you solve the halting problem!?

I’m a human, I use heuristics.

seems to work with clj repl

$ clj -A:test:repl
user=> (stest/check `str)
({:spec #object[clojure.spec.alpha$fspec_impl$reify__2524 0x3a94964 "clojure.spec.alpha$fspec_impl$reify__2524@3a94964"], :clojure.spec.test.check/ret {:result true, :pass? true, :num-tests 1000, :time-elapsed-ms 19215, :seed 1540906061983}, :sym clojure.core/str})
user=> ^D
Borkdude@borkdude ~/Dropbox/dev/clojure/speculative (borkdude*) $ lein repl
nREPL server started on port 57033 on host 127.0.0.1 - 
(require '[REPL-y 0.3.7, nREPL 0.2.12
Clojure 1.10.0-RC1
Java HotSpot(TM) 64-Bit Server VM 1.8.0_45-b14
...

user=> (require '[speculative.core])
nil
user=> (require '[clojure.spec.test.alpha :as stest])
nil
user=> (stest/check `str)
;; still nothing after a looooong time

Have you tried at least a few times from the command line? Tests that probabilistically take a Long Time can be misleading like that with just a few data points

@gfredericks trying. hmm, now clj lingers after printing the result of stests:

$ clj -A:test -e "(require,'[speculative.core])" -e "(require,'[clojure.spec.test.alpha,:as,stest])" -e "(stest/check \`str)"
({:spec #object[clojure.spec.alpha$fspec_impl$reify__2524 0x3e14c16d "clojure.spec.alpha$fspec_impl$reify__2524@3e14c16d"], :clojure.spec.test.check/ret {:result true, :pass? true, :num-tests 1000, :time-elapsed-ms 15108, :seed 1540911818977}, :sym clojure.core/str})

it just takes a long time before it returns to the cmd line. maybe garbage collecting?

is it exactly 60s?

this is the output with time:

$ time clj -A:test -e "(require,'[speculative.core])" -e "(require,'[clojure.spec.test.alpha,:as,stest])" -e "(stest/check \`str)"
({:spec #object[clojure.spec.alpha$fspec_impl$reify__2524 0x3e14c16d "clojure.spec.alpha$fspec_impl$reify__2524@3e14c16d"], :clojure.spec.test.check/ret {:result true, :pass? true, :num-tests 1000, :time-elapsed-ms 14804, :seed 1540911992271}, :sym clojure.core/str})
clj -A:test -e "(require,'[speculative.core])" -e  -e "(stest/check \`str)"  25.76s user 1.40s system 35% cpu 1:16.86 total

I meant 60s following the end of the tests if so, then adding (shutdown-agents) after the stest/check call should help

Repro:

;; run with:

;; clj -Srepro -Sdeps '{:deps {org.clojure/clojure {:mvn/version "RELEASE"} org.clojure/test.check {:mvn/version "RELEASE"}}}' -i stest_time.clj


(ns stest-time
  (:require
   [clojure.test :as t]
   [clojure.spec.alpha :as s]
   [clojure.spec.test.alpha :as stest]))

(s/fdef clojure.core/str
  :args (s/* any?)
  :ret string?)

(stest/check `str)

(println "after stest")

(shutdown-agents)

(println "after shutdown agents")

maybe any? just generates loads of garbage?

that sounds very strange

you can try the script…

I mean it does, but I don't think a JVM needs to GC on shutdown

you just burn it all down and let the OS sort it out

maybe a System/exit helps

yeah it does… but for a test namespace that’s not an option maybe

maybe use jstack while it's "hung" to see what it's doing at the moment

kill -3 says there's an agent thread running generator stuff

so that code launches a future somehow

I think shutdown-agents might not be synchronous in its effect

so the fact that it waits longer when there's a future in the background that hasn't completed isn't surprising

but why there's a future in this case I don't know

(I haven't used clojure.spec.test.alpha much)

at clojure.spec.test.alpha$quick_check.invokeStatic(alpha.clj:310)
	at clojure.spec.test.alpha$quick_check.invoke(alpha.clj:303)
	at clojure.spec.test.alpha$check_1.invokeStatic(alpha.clj:336)
	at clojure.spec.test.alpha$check_1.invoke(alpha.clj:324)
	at clojure.spec.test.alpha$check$fn__3088.invoke(alpha.clj:412)
	at clojure.core$pmap$fn__8342$fn__8343.invoke(core.clj:6994)

check uses pmap

https://github.com/clojure/spec.alpha/blob/f23ea614b3cb658cff0044a027cacdd76831edcf/src/main/clojure/clojure/spec/test/alpha.clj#L411

ah well there you go then

gotta spend 15 seconds generating a few last giant piles of garbage in case somebody wants them

@borkdude the most recent alpha of test.check might be a bit better about this. I can't remember, and also don't know what version you get by specifying RELEASE

looks like org/clojure/test.check/0.10.0-alpha3

oh; nevermind then

for this kind of test it's probably best to use a smaller max-size for the gen/any

I’ll try that, thanks.

I wonder if gen/any should just have a smaller default size universally :thinking_face:

any snippet handy that I could re-use?

no, I'm not sure of the exact syntax for sizing control in spec

k, I’ll figure something out

if you don't mind affecting the whole generator, I think the stest/check call lets you pass max size

oh that would be nice

for finer grained control you'd probably set an override specific to the any? spec

@alexmiller if you want I can post that script in an issue if you think it’s something that should be improved over time

Hmm?

issue solved (see above thread).

didn’t realize that (pun intended)

I'm not sure if this is a bad idea or something but I'd love to always have stest/instrument switched on during development. Is there some way to keep functions instrumented after re-defining them in the REPL?

@martinklepsch I hook up stest/instrument with component. So when I restart the system, my new fns are instrumented.

somehow this also lingers after the test has executed, even with doall:

(ns spec-keys-test
  (:require
   [clojure.test :as t :refer [deftest testing is]]
   [clojure.spec.alpha :as s]
   [clojure.spec.test.alpha :as stest]))

(defn foo [n] n)

(s/fdef foo
  :args (s/cat :n number?)
  :ret number?)

(deftest repro-test
  (testing "output of check"
    (let [r (doall (stest/check `foo))
          c (count r)]
      (println "count" c)
      (is (= c (count (keep :spec r))))
      (is (= c (count (keep :sym r))))
      (is (= c (count (keep :clojure.spec.test.check/ret r)))))))

(t/run-tests)

I think shutdown-agents was mentioned before, but wasn't sure whether you had tried it. If your code calls future directly (or indirectly through one of several other functions), it can cause a 60-second pause at the end before the JVM exits: http://clojuredocs.org/clojure.core/future

e.g. pmap clojure.java.shell/sh