Fork me on GitHub
#test-check
<
2016-08-17
>
lvh02:08:02

woop woop 🙂

gfredericks02:08:12

it's more of a headache than it should be :(

lucasbradstreet14:08:13

@gfredericks: re our twitter conversation, what’s your solution for generators with high variance? I think maybe it’s ok, because the common case where you’re using parallel testing will be tests passing

lucasbradstreet14:08:27

if your tests are failing often, then generally parallelism isn’t really all that helpful anyway. Most of the time when I run test.check running overnight in #core JVMs, I do not expect to catch anything

gfredericks14:08:16

It can slow down even in the passing case

gfredericks14:08:29

Oh I forgot another goal

gfredericks14:08:41

Given a seed, the result should be deterministic

gfredericks14:08:26

So if you're preallocating work you have to give n/t trials to each thread

lucasbradstreet14:08:27

Yeah, I agree with that goal. That’s why I was saying that you have to partition the space and check whether all threads have passed X, where X is a failing iteration

lucasbradstreet14:08:45

I can see how it’ll slow down passing cases though

gfredericks14:08:52

If one thread has lots of slow trials the others will be idle while it finished

lucasbradstreet14:08:59

Yeah, that’s troublesome

lucasbradstreet14:08:08

Especially if you have a bunch of tests to run through

gfredericks14:08:14

So my alg has a queue of trials to run

lucasbradstreet14:08:22

Generally I’m just running one test overnight, on X cores

lucasbradstreet14:08:33

So this isn’t a problem, but I see where it’s a problem for a test suite

gfredericks14:08:05

And the threads just pull them off the queue

lucasbradstreet14:08:19

I was going to say, a priority queue might be the answer

lucasbradstreet14:08:36

Maybe you don’t even need to prioritise, now that I think about it

gfredericks14:08:16

The other problem I have is deciding how to manage the threads

gfredericks14:08:38

I suppose I could have it just use N futures...?

lucasbradstreet14:08:46

You also need to ID the test cases, in case there is some variance in when the cases finish?

gfredericks14:08:47

Maybe that's best actually

gfredericks14:08:58

Yes they're ordered

gfredericks14:08:40

So a lot of complexity comes from when you find the first failure but have to wait for all earlier jobs to finish first

lucasbradstreet14:08:51

Yep, I can see that

lucasbradstreet14:08:05

If you use futures, make sure to check for (Thread/interrupted) in the processing loop, so you can cancel the thread reliably after the test is done

gfredericks14:08:52

yeah cancelling things is a big deal too

gfredericks14:08:08

I don't know how bad it is to .stop a thread

lucasbradstreet14:08:31

Yeah, it doesn’t really work well

lucasbradstreet14:08:41

It’s really up to the thread to behave well

lucasbradstreet14:08:07

if you check whether you’ve been interrupted in between test iterations, you’ll be fine with just (future-cancel fut)

gfredericks14:08:18

C-c in nrepl calls .stop I think

lucasbradstreet14:08:49

future-cancel may do too, but it’s really up to the thread to behave properly

lucasbradstreet15:08:47

pretty sure thread.stop has been deprecated too

gfredericks15:08:44

yeah that's the entire reason for my hesitation

gfredericks15:08:09

it's tough in this case to ensure the thread behaves properly though since it's running user code, and so might hang

gfredericks15:08:47

so if I don't call thread.stop, the caveat to users is "if your test or your generator hangs, you're just going to have an extra thread on your jvm until you restart it, sorry"

lucasbradstreet15:08:03

if user code hangs, it’s not your problem in my opinion

lucasbradstreet15:08:10

best you can do is exit between iterations

gfredericks15:08:21

sure, but users have the expectation that they can C-c their tests and everything stops

gfredericks15:08:35

what is "exit between iterations"?

lucasbradstreet15:08:00

I agree, but there’s nothing you can do to guarantee that. User code can always cause situations where that is not true

lucasbradstreet15:08:26

I mean, a thread will, at minimum, stop between an test iteration

lucasbradstreet15:08:42

as far as I know, there may be other situations where their code can bail safely, but there can always be other situations where things are blocked and C-c is never going to help

gfredericks15:08:12

that's true when users are explicitly using threading constructs

lucasbradstreet15:08:26

when I say “may”, I mean, it depends on their code

gfredericks15:08:32

but for user code on the main thread I think it's always true that C-c in nrepl stops it

gfredericks15:08:11

I suppose this discussion overlaps with the "should tests run in parallel by default?" question

lucasbradstreet15:08:25

Ah yes, I can see how that is important

gfredericks15:08:41

I think using futures and using an atom to aggregate the results will simplify that code a lot

gfredericks15:08:23

so I'm glad about that;

gfredericks15:08:02

I'm a little grumpy about how this interacts with the reporter-fn, since I think that has to be nondeterministic now

gfredericks15:08:39

and the behavior when the reporter-fn throws an exception is potentially weirder

Alex Miller (Clojure team)18:08:27

calling .stop on a thread is bad - it frees all monitors which can badly impact others using them

Alex Miller (Clojure team)18:08:01

much preferred to interrupt

gfredericks20:08:49

@alexmiller: do you think it's a problem that nrepl does it on ctrl-c?

gfredericks20:08:28

should we be resigned to forcing repl users to restart their jvm if they execute an infinite loop?

gfredericks20:08:53

I see ctrl-c at the builtin clojure repl just kills the whole jvm

gfredericks20:08:26

I'm also curious if you have any sense for how realistic the monitor problem is to a user who isn't explicitly using monitors

Alex Miller (Clojure team)20:08:06

I don’t know that there is any better solution for nrepl, but that doesn’t make it good

Alex Miller (Clojure team)20:08:35

the thread stop thing is a real problem, which is why its deprecated. any code that uses Java locks will be affected and that includes code that is calling into the Clojure runtime in places where synchronization is used (every lazy seq uses synchronization).

Alex Miller (Clojure team)20:08:10

that said, typically at the repl, the person typing is the one likely to be affected and they will probably just start over with (mostly) fresh resources

Alex Miller (Clojure team)20:08:42

Java Concurrency in Practice is probably the best resource on these problems and their recommended alternatives

gfredericks21:08:16

Yeah that's about what I thought. Seems like there's not much you can do in a general context where you don't control all the code. I'm surprised it hasn't been worth fixing (or is too hard to fix) on modern jvms