Fork me on GitHub
#clojure
<
2019-08-20
>
Matheus Moreira05:08:00

hello, good morning! do you know if there is a simple way to use a function without arguments as a custom generator for a spec? something like:

(spec/def ::repository (spec/with-gen #(satisfies? AProtocol) (gen-from-function my-aprotocol-instantiator)))

Alex Miller (Clojure team)11:08:13

No, and there are good reasons not to do this. In test.check, the generators are the only things that create novelty (/ randomness). Importantly, they do so in a repeatable way (so you can retest with a known seed) and in a shrinkable way (so a failing test can be “shrunk” to a simpler failing test)

Matheus Moreira06:08:46

thank you for the answer. maybe i’ll have to rethink my strategy. 🙂

Matheus Moreira05:08:30

the only way that i found is to use a combination of gen/return and gen/fmap: return to “start” the generation and fmap to call the function ignoring the value generated by return; this is a very ugly hack. 🙂

borkdude07:08:46

@andy.fingerhut hmm yeah, I want a way to test this behavior on CI

kirill.salykin07:08:41

Would redefining in work in this case?

borkdude08:08:01

yeah that's what I've been trying

andy.fingerhut08:08:24

It seems perhaps good enough to test an interactive behavior like this interactively, and repeat such interactive tests if anyone messes with that function again?

andy.fingerhut08:08:39

and try as much as possible to "push I/O to the boundaries" of whatever library this is in, i.e. hopefully it is easy to separate this function and when it is called from most other functions, and most other functions can remain without I/O, or only file I/O, which can easily be tested with automation.

borkdude08:08:23

yeah, well, there's a PR that reads input from /dev/tty and the problem is, I don't know how that behaves on different platforms, so I would really like to test this on CI

andy.fingerhut09:08:27

Maybe there is a better way, but at some point you have to let folks try it interactvely on those different platforms and see if ti behaves correctly?

andy.fingerhut09:08:20

WIll CI actually test it on all the platforms you care about?

borkdude09:08:44

only Mac and Linux, but at least more than the one I'm using

Toby Clemson10:08:22

Hi all, I have the following function:

(defn ->attribute-value [attribute-spec value]
  (cond
    (as/view? attribute-spec :user)
    (->byte-buffer value)

    (as/name? attribute-spec
      #{:creation-time
        :last-access-time
        :last-modified-time})
    (->file-time value)

    (and
      (as/view? attribute-spec :posix)
      (as/name? attribute-spec :permissions))
    (->posix-file-permissions value)

    :default value))
In this case, attribute-spec looks like {:view :posix :name :permissions}, value is any object. I'd like to make this extensible by the consumer of the library as the number of views and attributes is not fixed. I can't use a multi method since I need to match on parts of the provided attribute spec. How would you go about making this extensible?

Toby Clemson10:08:49

I was thinking of using a protocol but I'd need some way of registering implementations if I took that approach

vlaaad10:08:10

looking at your code, I don't really understand what exactly is extensible

vlaaad10:08:24

what do you want to dispatch on?

Toby Clemson10:08:55

So my library calls this function internally and I want consumers of the library to be able to register new conversions for future views and names

vlaaad10:08:04

views? names?

vlaaad10:08:32

"view" means there is a key :view in input map?

Toby Clemson10:08:07

The attribute-spec looks like {:view <some-view> :name <some-name>}.

Toby Clemson10:08:40

So in some cases, I want to apply a conversion regardless of name, based only on view

Toby Clemson10:08:48

In others based only on name, regardless of view

vlaaad10:08:58

(defmulti ->attribute-value (juxt :view :name))?

Toby Clemson10:08:01

And others, more specifically based on a view and a name

Toby Clemson10:08:44

In that case how would I match any name for specific view?

vlaaad10:08:53

> I want to apply a conversion regardless of name, based only on view (defmethod ->attribute-value [:my-custom-view :default] ...)

vlaaad10:08:24

> In others based only on name, regardless of view (defmethod ->attribute-value [:default :my-custom-name] ...)

Toby Clemson10:08:25

In the view only case, the map coming in would look like {:view :user :name :some-name} so (juxt :view :name) would yield [:user :some-name].

Toby Clemson10:08:29

Which would look for (defmethod ->attribute-value [:user :some-name] ...) and wouldn't match a default.

Toby Clemson10:08:08

I think I'd need pattern matching or predicate dispatch to be able to achieve that. However in that case, it would be hard for users to register new matches or predicates.

vlaaad10:08:11

ohh, :default does not work in vector...

vlaaad10:08:18

that's strange

Toby Clemson10:08:32

:default is a bit of magic

vlaaad10:08:39

this works though:

(defmulti ->attribute-value (juxt :view :name))

(defmethod ->attribute-value [::my-custom-view ::default] [_] "my custom view")

(defmethod ->attribute-value [::default ::my-custom-name] [_] "my custom name")

(derive ::blep ::default)

(->attribute-value {:view ::blep :name ::my-custom-name}) 
;; => "my custom name"

vlaaad10:08:57

it's not really a magic, it's configurable and a part of a hierarchy

vlaaad10:08:27

my mistake, :default is a part of a multimethod, not part of a hierarchy, that's why it does not work in isa? check

Toby Clemson10:08:32

Yeah, what I mean by a bit of magic is it's hard coded into the definition of a multimethod

Toby Clemson10:08:09

If I use hierarchies, the user has to derive from :default for every possible name.

Toby Clemson10:08:00

Really all I need is a :* entry that matches anything, so that I can say (defmethod ->attribute-value [:user :*] ...) or (defmethod ->attribute-value [:* :creation-time] ...)

vlaaad10:08:20

I think it's possible to do with custom hierarchy

vlaaad10:08:43

you can supply custom hierarchy to multimethod

Toby Clemson10:08:34

It looks like for a (defmulti ...) you can override the hierarchy

Toby Clemson10:08:13

I'd need a way to say :* is a descendent of any other keyword

Toby Clemson10:08:17

Not sure how to achieve that

vlaaad10:08:20

if you look at isa? source to see how it uses hierarchy, you'll see this: (contains? ((:ancestors h) child) parent)

vlaaad10:08:05

your hierarchy can be a map with :ancestors key, that instead of whatever is there by default is actually a function that makes this code return true

vlaaad10:08:20

it will screw up derive for this particular multimethod, but you probably don't want that anyway?

Toby Clemson10:08:27

Yeah that would be fine

Toby Clemson10:08:12

Ok, I'll give it a go, thanks

vlaaad10:08:26

@toby924 behold the horrors of abusing implementation details!

(defmulti ->attribute-value (juxt :view :name)
  :hierarchy (atom {:ancestors (constantly #{:default})}))

(defmethod ->attribute-value [:my-custom-view :default] [_] "my custom view")

(defmethod ->attribute-value [:default :my-custom-name] [_] "my custom name")

(->attribute-value {:view :blep :name :my-custom-name})
;; => "my custom name"

Toby Clemson11:08:19

Does seem a little hacky though 🙂

Toby Clemson11:08:40

The docs are insistent that you use make-hierarchy for any custom hierarchy

vlaaad11:08:05

yep, it abuses implementation details

Alex Miller (Clojure team)11:08:13

No, and there are good reasons not to do this. In test.check, the generators are the only things that create novelty (/ randomness). Importantly, they do so in a repeatable way (so you can retest with a known seed) and in a shrinkable way (so a failing test can be “shrunk” to a simpler failing test)

Atif12:08:17

Hey Guys, I am trying to use http://clojurequartz.info/ for scheduling periodic jobs for one of my Clojure projects. I am wondering, what tools/libraries I can use to monitor (web UI) to monitor these jobs? Any sort of pointer would be very helpful. Thanks in advance. :spock-hand:

dominicm13:08:17

I think there are quartz dashboards, I remember one being mentioned in the documentation

dominicm13:08:13

https://github.com/royrusso/jwatch Google turned this up and also quartzdesk

👍 4
Atif14:08:26

Thanks. I had a look at it earlier and project doesn’t seem to be in active development and lacks documentation. Wondering, if there is something in Clojure?

dominicm15:08:09

I don't think so. The java quartz community is much larger than the clojure one.

Atif12:08:01

@U09LZR36F Cool. Thanks. Also, what’s best way to trigger scheduled jobs in Clojure?

dominicm13:08:32

Depends what you mean, quartz might fit the bill?

Atif11:08:06

@U09LZR36F Cool. Thanks. So, basically, I can use Qaurtzite to schedule jobs, then put a table that maps the logs/status of the jobs and then show it up on a web UI.

Suni Masuno13:08:16

I have a "what's the most beautiful way" kinda question. I have a function get-user that takes a user id and gets the rest from a database. I also have a looong ole list of users (currently in a vect, but I can change that if needs be). Whats the best/most beautiful way to parallelize the calls to get those?

Tuomas-Matti Soikkeli13:08:21

perhaps you’d like not hammer your database and write a single database query and join the user-id’s to it.

Suni Masuno13:08:11

That sounds far too easy on the DBAs. troll In all honesty, I'm hitting a service actually and the maintainer said they didn't have time for a batch request and told me to perform my requests in parallel. I stared in shock for a bit, shrugged, and told them they were free to be crazy. ⚰️ I'm kinda hoping they'll "ask me for an emergency fix" to batch after they see the performance logs?

alfredx13:08:12

if number of users/ids is below 10, might be fine using pmap, otherwise, using SQL like id=id1 or id2 or id in [id1,id2] should be the way to go.

Suni Masuno13:08:19

I know nothing of their service/db setup. Maybe there's some arcane path where batching doesn't help much?

alfredx13:08:46

is it Relational DB?

Suni Masuno13:08:15

I know nothing I'm afraid. ¯\(ツ)

Suni Masuno13:08:44

Even with couch, mongo, or dynamo I'd want batching if I were them. Maybe Cassandra doesn't care? But it's enterprise software, so I'd be a bit surprised. I really oughta ask more just for my own curiosity.

alfredx13:08:08

I think even Cassandra would tell difference between 1000 request/response and 10 request/response batches

Suni Masuno13:08:48

Right? But I'm just speculating on someone else's stack.

alfredx13:08:29

Sounds like a external partner integration scenario

vlaaad13:08:55

pmap if you want parallelize only this

Suni Masuno13:08:30

The only downside I know of there is that if one of them dies you end up chugging through the rest of the threads before you can move on, right?

enforser13:08:10

https://clojuredocs.org/clojure.core.async/pipeline-blocking might give you a little more control over the rate at which you hit the service? Depending on the length of time an average query takes, it may be worth it to "batch" your requests within pmap - to avoid spinning up a new thread for each user: i.e.

(apply concat (pmap (partial map get-user) (partition-all 100 user-ids)))
;; ^ will fetch 100 users in a single thread, instead of 1 user per thread. This greatly reduces the amount of time spent managing thread versus: 
(pmap get-user user-ids)

enforser15:08:25

ah, my bad - should be

(apply concat ...)

enforser15:08:30

it's to move the batches back into the same list

user=> (apply concat (pmap (partial map inc) (partition-all 3 (range 10))))
(1 2 3 4 5 6 7 8 9 10)
user=> (pmap (partial map inc) (partition-all 3 (range 10)))
((1 2 3) (4 5 6) (7 8 9) (10))

Suni Masuno13:08:58

Ohhh, that's pretty. 👏

emccue15:08:22

@borkdude In school the way we tested stuff like that was by first factoring out the input and output into Readable and Writable "things" (I forget the exact java interfaces we used)

emccue15:08:58

then for unit tests we would pump some input into the code and inspect its output

emccue15:08:17

in the simplest case by just passing a StringBuilder for the output

emccue15:08:18

so we had stuff like (pseudo-clojure)

emccue15:08:19

(we didn't actually write clojure, but i think you get the idea

borkdude15:08:09

@emccue that's a good strategy. one issue is that the PR reads from (io/reader "/dev/tty") and I have no clue how that behaves over different operating systems, that's why I want a test for that in CI

borkdude15:08:47

but maybe replacing /dev/tty by an explicit reader thing and testing that is sufficient

emccue15:08:58

Well there are really two components

emccue15:08:14

One, given a stream of characters does your thing do the right thing

emccue15:08:41

Two, is each specific source of input a valid source of input

emccue16:08:51

That second one is the harder one to validate

emccue16:08:14

But there is value in separating the first one even if you can't think of a way to do that validation

Lone Ranger21:08:04

Does anyone happen to have any experience using @seancorfield’s clj-new.generate feature and has a working CLI and associated generate function?

Lone Ranger22:08:38

ahh nvm found the source code 😊

g22:08:07

i’m finding during profiling that the thing seemingly taking up the most time is the recursion call. i’m not quite sure what to make of that, any insights?

hiredman22:08:43

what do you mean by "the recursion call" and how are you profiling?