Fork me on GitHub
#clojure-dev
<
2018-11-07
>
mfikes12:11:05

ClojureScript's unit tests pass with 1.10.0-beta5

gfredericks15:11:24

I spent thousands of milliseconds feeling guilty about test.check not having async support

ghadi15:11:08

> The fruition callback must be summoned when all affirmations have run. That sounds fun

gfredericks15:11:40

looking forward to figuring out what concepts rich chose these words to describe

andy.fingerhut16:11:04

@gfredericks Not sure if you are the right person to ask, but thought of you first. Apologies if you have heard the question N times already, and there is a nicely written summary I don't know about. Have you considered changing test.check's default data generators to never generate ##NaN "values", because they are not equal to themselves?

gfredericks16:11:51

@andy.fingerhut definitely thought about it -- the downside is that you could use that same justification to say it ought to generate NaNs, to expose bugs based on that very edge case

gfredericks16:11:45

I'd love it to be easier to control; one idea is reforming the generators API so that all generators (at least all the ones that are functions) take a generic opts parameter that lets options flow down to contained generators, so that you could somehow blanketly {:NaN? false}

gfredericks16:11:06

OTOH, thinking about this more, do you think it's just an issue with gen/any in particular?

gfredericks16:11:07

I think that's the only place that the choice is made for you in a way you can't easily control

andy.fingerhut17:11:03

The main issue that people have with automatic checking is that any collection that there is a ##NaN buried somewhere inside of is not equal to something that looks like it should be equal. Thus they have to write extra code to generate but then not do = checks on anything with ##NaN inside.

andy.fingerhut17:11:20

So it might be gen/any in particular that most people hit that issue with.

andy.fingerhut17:11:49

I can see people wanting sometimes to opt in to ##NaN, but maybe a default of opting out for gen/any might be more useful.

gfredericks17:11:07

it goes counter to the "test.check finds bugs you weren't even thinking of because it defaults to generating all these edge cases" approach

gfredericks17:11:23

moving toward "test.check finds the bugs you were thinking of and built the generators to find"

andy.fingerhut17:11:33

Asking on #clojure-spec channel for people's pain points is probably better than asking me, since I'm sure some others who frequent that channel have used property testing in anger much more than I have.

gfredericks17:11:39

there are softer versions of this in other generators

gfredericks17:11:03

e.g., should default string generators be full of unicode garbage?

andy.fingerhut17:11:35

IMO, yes 🙂 But those are = to themselves in Clojure.

gfredericks17:11:26

in an ideal world, users aren't going to assume that two values are equal to themselves unless they've spec'd things in a way that specifies no NaNs

andy.fingerhut17:11:40

##NaN might be in a unique position here. Even mutable Java objects are usually = / equals to themselves.

andy.fingerhut17:11:43

But property-based / generative testing using spec uses = for you, right?

andy.fingerhut17:11:59

I could be confused on that last statement.

gfredericks17:11:43

I can't think of how that's true

gfredericks17:11:51

the last statement I mean

gfredericks17:11:43

agreed that NaN is unique -- I don't think that contradicts my previous statement

andy.fingerhut17:11:46

Yeah, I am probably guessing something about the scenarios where others have come across this (that clojure.core/= is involved) when in fact it isn't. Now I'm confused 🙂

andy.fingerhut17:11:30

I may ask in that channel how it is that others are hitting this ##NaN issue.

gfredericks17:11:43

I'd guess users of any?

gfredericks17:11:49

which is probably a lot of people

andy.fingerhut17:11:29

The part I'd like clarity on (for my understanding) is which step they are doing that checks = on the collections.

gfredericks17:11:12

it might be more prone to happen when writing your own properties rather than just using spec's automatic properties

andy.fingerhut17:11:29

Probably hand-written checks of fn return values vs. their args.

gfredericks17:11:52

yeah I think putting them in map keys can also be weird:

$ clojure -e '(assoc {} [##NaN] 12 [##NaN] 13)'
{[##NaN] 12, [##NaN] 13}

alexmiller17:11:51

this is a rare case where I think having a dynamic var like *generate-NaNs* defaulted to false would be a reasonable choice

alexmiller17:11:46

they are particular weird objects and have caused me to waste far more time than they have saved me

gfredericks17:11:34

yeah, I don't doubt something along those lines is the right answer; I just feel like the argument from the maximal-mathematical-correctness-and-completeness-and-all-that-nonsense side is undertold and underappreciated

andy.fingerhut17:11:41

I can see a library dealing with floats/doubles and arrays of them would be the most likely case to want them in there explicitly. Even then they would require custom checking of fn ret values against args, if that spec feature was used.

andy.fingerhut17:11:17

I mean, I guess another approach would be to change clojure.core/= so that ##NaN was = to itself (quickly ducks for cover)

gfredericks17:11:34

yeah I was wondering who best to blame in this situation, and that was my best theory 🙂

andy.fingerhut17:11:00

I don't have a horse in that race -- just calling the race here.

gfredericks17:11:36

I don't think clojure equality is defined in terms of java equality in such a way that binds it to this behavior; but I generally assume that'd be too maybe-breaking and maybe-enslowening of a change to ever get considered

andy.fingerhut17:11:39

And anyone who coins words like 'enslowening' might enjoy not only this A Bit of Fry and Laurie sketch, but a bunch of others: https://www.youtube.com/watch?v=Z3515_x8R9c

👍 1
andy.fingerhut17:11:32

agreed, hence my ducking for cover in even daring to say such a thing out loud. Come to think of it, there has been a rash recently of bots impersonating long time Slack users and putting words in their mouth they would never ever say.

hiredman17:11:56

the pain of NaN with test.check hasn't come up with spec for me, but when using test.check directly. for complicated properties I really like the approach of finding a simplified model of the system being tested and generating operations that are run both against the system being tested and the model, and then the results are compared. for example the model for a distributed key value store might be a hash map. the NaN problem arises at the end when comparing results from the real system and the model

gfredericks17:11:05

so the system can tolerate NaNs because what it's doing doesn't make equality assumptions, but you can't easily verify the results of things?

hiredman17:11:52

yeah, for the key/value store case I think I am using the any-printable generator, and checking at the end that all keys exist in both the hash map and the kv store, and have equal values

gfredericks17:11:34

seems like an alternate = that works with NaNs could be a generally useful function

gfredericks17:11:49

clojure.core/===, of course

hiredman17:11:12

and what I ended up doing, after looking at what I would have to do to replace any-printable was define a custom comparison that looked at the pr-str of the values and returned true if it contained a NaN

😂 2
gfredericks17:11:13

(defn === [a b] (or (= a b) (and (re-find #"##NaN" (pr-str a)) (re-find #"##NaN" (pr-str b)))))

gfredericks17:11:04

@alexmiller do you think such a dynamic var would be read when creating the generator or when generating?

gfredericks17:11:43

(the fact that the answer's not obvious bothers me, and would be a usability issue regardless of the answer)

alexmiller18:11:43

What are the trade offs? Make a table

alexmiller18:11:04

(channeling Rich)

hiredman18:11:38

maybe a generic dynamic var like *generator-configuration* would allow for configuring deeply nested generators without altering the current api

gfredericks18:11:28

@hiredman do you have an opinion on which time the var would be read?

andy.fingerhut18:11:20

Strongly tempted to make a table of "channeling Rich" sayings ...

alexmiller19:11:19

There’s only like 3… - what problem are you trying to solve? - do you have a table of tradeoffs? - where’s the picture of your design?

arrdem20:11:01

can we get shirts?

andy.fingerhut19:11:50

If it would not be considered offensive or irksome to create such shirts ... That sounds like it might be an interesting little project to create a simple shirt design. 🙂 I would consider adding maybe some things he might not often say, but are great things to keep in mind: "state complects everything it touches"

arrdem04:11:33

you're a saint Andy

alexmiller07:11:42

not sure how Rich would feel about that

andy.fingerhut07:11:18

FYI, no actual physical shirts have been made yet.

andy.fingerhut07:11:50

If it turns out he likes the idea and wouldn't mind making a few $$ from it, happy to donate the design to the http://clojure.org Swag shop.

deliciousowl16:11:56

Clojure - The Least Worst Programming Language(TM)

andy.fingerhut18:11:57

Not really joking there -- one value of a good teacher, and/or books like Polya's "How To Solve It", is to get a person's mind in the habit of asking themselves good questions.

hiredman18:11:50

I think creation time makes the most sense, at least for me, a lot use time is inside things like defspec

hiredman18:11:16

and I am not even sure what you would put the binding around for use time checking for spec instrumentation

hiredman18:11:29

but there are still plenty of wrinkles with creation time, like spec's generators are actually thunks which return generators, so creation time may not be where you would expect

hiredman18:11:22

alternatively, an any-printable-but-not-nans generator would go a long way

seancorfield18:11:46

We don't have a predicate for testing a value is ##NaN?

seancorfield18:11:20

(and (double? x) (.isNaN x)) ... that's probably reflective ... ?

hiredman18:11:28

it isn't a matter of testing a scalar, it is a matter of generating some big tree where there is a NaN somewhere in a leaf

seancorfield18:11:05

Ah, and you need a version of = to work "automatically" for the whole data structure?

hiredman18:11:47

or just a generator that generates most clojure values but leaves out NaNs

gfredericks20:11:40

@seancorfield yes, which is also theoretically more useful than changing the generator

gfredericks20:11:03

is it really true that nobody ever needs to compare data structures with NaNs in them unless they're doing PBT?

andy.fingerhut20:11:06

If anyone today needs to compare data structures with NaNs in them, like maps, sets, and vectors, then they are today unhappy with clojure.core/= (or they are ok with = returning false when data structures contain one or more NaNs)

andy.fingerhut20:11:39

or they implemented your proposed === for themselves already

gfredericks20:11:14

my proposed === was a joke; it'd have to be rather more sophisticated

andy.fingerhut20:11:16

Not sure, but if anyone has done such a thing, I would think core.matrix would have hit this issue, or Dragan with his GPU code.

gfredericks20:11:53

does core.matrix use a java lib's matrices? if so then they probably already have custom equality baked in

andy.fingerhut21:11:26

perhaps so. That would make sense for it to do, for performance, but I haven't dug into its code.