Fork me on GitHub
#clojure-spec
<
2017-02-21
>
mike_ananev10:02:46

Hi! If I need (gen/sample ::somespec) about 1M or more, how to parallelize this operation? is there build-in tools? or just pmap or something?

gfredericks13:02:08

@mike1452 what do you need them for?

mike_ananev13:02:38

@gfredericks for data sample generation. I want to give samples to other team, which needs shape of our data for dev process.

mpenet13:02:39

@alexmiller what's the extra form parameter in s/valid? ?

mpenet13:02:45

doesn't seem to be documented

mpenet13:02:57

well, it's not used actually

alexmiller14:02:44

It's an alternate value to return if not valid

alexmiller14:02:46

Or maybe I'm reading that wrong

alexmiller14:02:57

Can't say I've ever used it

mpenet14:02:01

doesn't seem used at all but in the exception

alexmiller14:02:04

that’s not the current version of that code

gfredericks14:02:49

@mike1452 you can parallelize it however you like, that's kind of independent

mpenet14:02:01

@alexmiller it's the one linked from the api docs I think

alexmiller14:02:02

@mpenet looks like its used internally via pvalid?

mpenet14:02:21

I ll have a look

alexmiller14:02:22

yeah, the autodoc updating was broken by an issue in spec, so they are not the latest

alexmiller14:02:35

I have a patch pending to fix the code and get that working again

alexmiller14:02:43

that valid? form is used when you have an alternate form to supply for the error reporting

mpenet14:02:36

that's exactly what I wanted it to be 😄

mpenet14:02:43

I ll try it out

alexmiller14:02:42

I think maybe the only place it’s used is from within keys

alexmiller14:02:52

used with that extra form that is

mpenet14:02:19

so it's not supposed to be "public" ?

mpenet14:02:30

I mean subject to changes/removal

mpenet14:02:08

not sure it helps actually

mpenet14:02:47

I am wishing for a explain-info I think. Sometimes I need to add some context to explain data

mpenet14:02:10

I know I can wrap the whole spec and add this at a higher level, but it's a bit gross

alexmiller14:02:55

the valid? function should be considered public and I don’t think there are any expectation that api will change

alexmiller15:02:37

my impression is that Rich doesn’t want to provide extensions to modify explain (other than by writing your own spec and taking care of it there)

mpenet15:02:02

😞 care about more detail of my use case for a potential suggestion?

mpenet15:02:11

in short: that spec predicate is quite resource heavy, and upon failure the function wrapped by that predicate provides a lot of metadata about the failure (think a query parser output, with line/col num, explain traces etc). Right now not only I loose all this data but I need to re-trigger a parse to get back the metadata (or throw upstream, bypassing spec, which is ugly). In theory, one could just wrap and report the error manually, but this values are "deep" in a (spec validated) map.

mpenet15:02:10

open to suggestions on how to better handle this

alexmiller15:02:46

sounds like a good use case for writing a custom spec

mpenet15:02:27

you mean implementing the internal protocol?

mpenet15:02:41

is that "safe"?

mpenet15:02:49

I mean allowed

alexmiller15:02:00

it’s available to do but is subject to change while in alpha

mpenet15:02:00

ok, I guess I might just do this. That's how it used to work with Schema btw, seems quite an elegant way to do it

alexmiller15:02:10

it’s something I would avoid doing as much as possible

mpenet15:02:32

yes, same feeling.

mpenet15:02:57

works like a charm

carocad16:02:01

@mpenet I was under the impresion that explain-data was meant to provide that functionality: https://clojure.github.io/clojure/branch-master/clojure.spec-api.html#clojure.spec/explain-data

mpenet16:02:04

no explain-data just returns the data behind a failing spec, doesn't allow you to "customize" said data

mpenet16:02:18

same diff as ex-info vs ex-data (kinda)

carocad16:02:56

but isnt the data behind a failing spec the one that your parser already returned? I mean if your parser would be a speced fn then the data that you get already contains that custom information that you want. Or am I missing something?

mpenet16:02:33

no it's the spec form : something like #(try (parse q) (catch ExceptionInfo e ...))

alexmiller16:02:30

well, you’ll also get the failing value

alexmiller16:02:38

but not all the work that went into evaluating the failing value

mpenet16:02:55

hmm I don't think so

mpenet16:02:05

valid will return false, conform ::invalid

mpenet16:02:16

ex-data the form

alexmiller16:02:17

each problem in explain-data has the value that failed the spec

mpenet16:02:53

yes, the initial value true, but not the "detail" of the failed predicate (in my case an ex-info instance)

alexmiller16:02:55

maybe the exception case in particular is interesting

mpenet16:02:05

even then, that wouldn't make it as easy as it is not with the custom spec, it just feeds that extra data to the explain map

mpenet16:02:35

yes, imho it's something that might make sense in spec itself.

alexmiller16:02:37

well, what if when conforms fails with an exception, the ex-info was conveyed

alexmiller16:02:57

normally it just returns a truthy/falsey value (and nil/false can’t convey additional info)

alexmiller16:02:00

but exceptions can

mpenet16:02:14

that would be nice, that's what I meant with explain-info , it could return that

mpenet16:02:27

either accept a explain-info return, or the kw

mpenet16:02:44

but the reify Spec option is actually nicer in my case (for other reasons)

carocad16:02:48

how about (or (s/explain-data spec x) ::ok )

carocad16:02:49

returns ::ok when everything when well or the ex-info from your spec if it failed

mpenet16:02:16

@alexmiller using explain-info you would basically write your own conformer( try/catch potential exception in my case) and return the (explain-info {...}) on failure. I am not sure doing by default for any exception (or ex-info instance) is desirable

tjtolton19:02:48

Is core.spec able to use something else as keyword qualifiers besides the current ns? As I say that out loud, I realize how silly it would be if that were not the case, but I see a lot of the example code using the double colon keywords, which auto expand to current ns

alexmiller19:02:48

any ns is fine

tjtolton19:02:56

I don't suppose there's some kind of dynamic binding that would change the way the double colon autoexpands?

tjtolton19:02:21

::keyword => :com.mycompany/keyword

alexmiller19:02:45

you can use aliases

alexmiller19:02:04

(alias ‘a ‘com.mycompany)
::a/keyword

alexmiller19:02:35

the tricky part is that right now, com.mycompany must be an actual namespace

tjtolton19:02:45

that's great!

alexmiller19:02:11

you can work around that by doing (create-ns ‘com.mycompany)

alexmiller19:02:29

that is something we’ve talked about changing though

ag20:02:30

guys, I’m kinda stuck again. I’ve done it before (I think) now I can’t remember the right way of doing that. How do I properly generate things based on vector? So If I have generated vector of maps, and then for each item I need to generate something else

ag20:02:36

passing vector generator with fmap into a function with for kinda works, but I think this isn’t right approach, because I need to call gen/generate inside for

ag20:02:41

since it can’t return generator

zane20:02:09

> For example, say we wanted to generate a vector of keywords, and then choose a random element from it, and return both the vector and the random element.

ag20:02:58

@zane here in fn in just grabs random element from generated vector and then…

ag20:02:05

actually that may work for me

gfredericks21:02:38

Random element?

gfredericks21:02:19

Why did you generate a whole vector of them just to pick one out?

alexmiller22:02:55

well one reason you might do that is if you need a vector AND an element from that vector

ag23:02:29

actually @gfredericks comment in my case make sense.

ag23:02:26

ok, there’s still something I can’t figure out for the second case, where I have a predefined vector of maps and for each item I need to generate (something based of that map) and attach in a key to that map and finally return that modified vector

ag23:02:18

my solution requires to call gen/generate within for, which I don’t like.

ag23:02:47

I though test.chuck’s for would work for me, but I can’t get it right

gfredericks23:02:59

Calling gen/generate in general is not a proper way to build a generator

gfredericks23:02:23

And it shouldn't ever be necessary

gfredericks23:02:10

If you want to describe what you're doing I'm happy to help work it out

ag23:02:11

so I have list-of-accounts (predefined list) which is a vector of maps, e.g. [{:id 1 :type :customer} {:id 2 :type :investor},,,]. Now, for each item in that vector I need to generate a balance-update data and add it in :bal key and return that modified vector