Fork me on GitHub
#clojure
<
2020-06-08
>
somedude31403:06:55

Is there a way to destructure items from multiple levels? Say I have this map and I want to destructure form-params, lang, user: {:request {:form-params {:email "..."}} :lang :en :user "joe" :something "else"} I have (*fn* [{:keys [lang user]}] working but I am unsure how to include the nested form-params to the two items I have from one level up

hiredman03:06:03

:keys is actually sugar on the full destructing syntax

danielglauser13:06:47

Thanks for pointing back at the basics, I had never seen & remaining in sequential destructuring.

somedude31413:06:23

Thank you - I found the solution on the page you linked in the example before the very last

jumar10:06:25

I'm trying to understand clojure.core/pmap implementation and its "semi-laziness". I thought that it would always run at most the (+ 2 (.. Runtime getRuntime availableProcessors)) number of tasks concurrently; but it seems to share the chunk size of the underlying collection (32). Note: Runtime reports 12 processors in my case:

;; this should print only the first element
;; but it prints all the first 32 elements 
(first (pmap (fn [i] (println i) i) (range 40)))
Is that expected behavior?

jumar10:06:25

I'm trying to understand clojure.core/pmap implementation and its "semi-laziness". I thought that it would always run at most the (+ 2 (.. Runtime getRuntime availableProcessors)) number of tasks concurrently; but it seems to share the chunk size of the underlying collection (32). Note: Runtime reports 12 processors in my case:

;; this should print only the first element
;; but it prints all the first 32 elements 
(first (pmap (fn [i] (println i) i) (range 40)))
Is that expected behavior?

jumar10:06:35

I got into this because I was trying to do something similar but a bit simpler: basically realizing at most N values "ahead of time" but using only a single (caller's) thread for "throttling" of AWS log api requests (first launch up to N StartQuery API requests, then poll the results by calling GetQueryResults API until the query is done - one query after another)

p-himik10:06:49

I was assuming that it would use 2 * coreN threads. But it uses 32 threads just as well. Huh.

delaguardo10:06:07

pmap internally is using map. the only difference - it will wrap provided function into future. but chunking comes from map implementation to compare: (first (map (fn [i] (println i) i) (range 40))) will print 32 times as well

delaguardo10:06:32

my “rule of thumb” - do not use pmap for any kind IO operations. only for intensive but pure computations

jumar10:06:11

Yes, but that's not what I expected and I'm asking if this is intentional. Because I thought that it shouldn't run more than the <number of available processors + 2> tasks in parallel

jumar10:06:52

So even with "pure computations" I could have let's say 2 processors and I wouldn't want to run 32 tasks in parallel instead

delaguardo10:06:43

there is no “shouldn’t run more than the <number of available processors + 2> tasks in parallel” in pmap’s doc string unfortunately (

jumar10:06:43

Yep, but this limit #processors + 2 is often used/recommended as the max cap for concurrency of CPU intensive tasks. So it almost looks like a bug although I'm quite sure people are aware of this behavior and it has some rationale behind it...

delaguardo10:06:54

there is a workaround:

(defn re-chunk [n xs]
  (lazy-seq
   (when-let [s (seq (take n xs))]
     (let [cb (chunk-buffer n)]
       (doseq [x s] (chunk-append cb x))
       (chunk-cons (chunk cb) (re-chunk n (drop n xs)))))))
re-chunk your sequence with n - (+ 2 (.. Runtime getRuntime availableProcessors)) and then map will respect configured chunk-size

jumar10:06:06

Interesting, thanks for the example

delaguardo10:06:03

but I can not judge rationale for pmap’s behavior. never read anything about that

andy.fingerhut14:06:18

Another issue with pmap is that if the individual threads differ significantly in how long they take to finish, the parallelism can be significantly lower than the n+2, because in the absence of chunking weirdness it will not create threads further than that many 'ahead' of the last one that has finished.

andy.fingerhut14:06:49

In short, I think that if you want tight control of processor resources used at one time, pmap is rarely the right tool for the job, if ever.

andy.fingerhut14:06:10

Some of this is documented by Clojure users here: http://clojuredocs.org/clojure.core/pmap

andy.fingerhut14:06:44

I don't see a mention there anywhere of using Java ExecutorService directly, which is a semi-common recommendation for those that do want tight control over processor resources in use at one time.

jsa-aerial14:06:49

I would recommend using reducers for any sort of computation of this sort. Has worked exceptionally well for me.

jumar15:06:30

Thanks everyone, I was curious whether pmap’s weird behavior (running significantly more than number of processors +2 tasks - in common case up to 32- concurrently) has some justification or is a side effect of implementation). It’s hard to tell based on available information. Apart from that, my main use case is basically single threaded so I’ve been looking for an easiest way to launch (one by one) up to N jobs via api1 requests; then fetch results of the jobs via api2 one-by-one - as soon as the first api1 job finishes, N+1 api1 request can be executed, and so on. So I think I want a lazy seq with custom chunk size (which suggested re-chunk fn could solve); I admit this approach might not be appropriate given the usual advice of not mixing side effects with lazy collections but it looked as the most straightforward option.

ben.sless16:06:47

If you don't want to reinvent the wheel, use Claypoole, it provides a pmap equivalent implementation with an executor service https://github.com/TheClimateCorporation/claypoole

pmonks19:06:04

Given that recommendations to “not use pmap” seem to be a fairly frequent theme, are there any plans afoot to improve its implementation?

andy.fingerhut19:06:48

Given the desire to preserve backwards compatibility, and the existence of libraries like claypoole (and I believe others), and that the Clojure maintainers likely have many other things higher on their list of priorities they want to work on, I doubt there are any such plans.

pmonks19:06:38

Seems like backwards compatibility wouldn’t be hard to preserve, even while the implementation is redone. Code that relies on the chunking behaviour, for example, is relying on an undocumented implementation detail.

pmonks19:06:02

Like map, except f is applied in parallel. Semi-lazy in that the parallel computation stays ahead of the consumption, but doesn't realize the entire result unless required. Only useful for computationally intensive functions where the time of f dominates the coordination overhead.

pmonks19:06:19

↑ seems like a pretty broad contract for the implementation

pmonks19:06:04

And yes, it goes without saying that the realities of person-power and priorities always come into play.

andy.fingerhut20:06:43

Sure, probably better if you omit the mention of backwards compatibility from my answer, since it is likely a red herring. The other issues are most relevant here.

gerred15:06:50

before I write it - anyone know of a tool to convert CSS to garden? or at least a CSS parser that anyone recommends? I specifically want to convert https://tailwindcss.com/ into a library I can better build/manipulate/abstract for clj and cljs on with Garden, and I want to be able to easily re-release tailwind.cljc for each version of Tailwind. that said, there's a lot of utility beyond me just converting over tailwind.

gerred15:06:50

before I write it - anyone know of a tool to convert CSS to garden? or at least a CSS parser that anyone recommends? I specifically want to convert https://tailwindcss.com/ into a library I can better build/manipulate/abstract for clj and cljs on with Garden, and I want to be able to easily re-release tailwind.cljc for each version of Tailwind. that said, there's a lot of utility beyond me just converting over tailwind.

isak16:06:26

Since tailwind is based on data/configuration, it doesn't really make sense to parse the generated output for one configuration - you would lose the benefit of tailwind. You'd want to get the data/configuration and just write a renderer for it in clojure instead.

noprompt16:06:55

@ I have some stuff on the back burner but I’ve been so busy personally that I doubt any of it will see the light of day soon. As a community it would be awesome to have, at a minimum, a standard CSS parser.

gerred16:06:39

@ that's fair! I hadn't even considered that, I was thinking fully about the end generated CSS

gerred16:06:15

so maybe I do want a full port, because I'd want to use it as cljc

vincent.cantin06:06:37

I recommend using the library instaparse for your parser instead of writting something from scratch.

gerred22:06:48

Agreed - I've done too many handwritten parsers and, at least for my needs, I'm not looking at raw performance of a hand-tuned parser.

lilactown15:06:12

there’s a couple of projects that might be similar to what you’re thinking of

jake14215:06:56

What’s the idiomatic way to assert the absence of a value in Clojure? More details in thread.

jake14215:06:56

What’s the idiomatic way to assert the absence of a value in Clojure? More details in thread.

jake14216:06:00

;; At t0, jake had a partner:

(def me {:person/name "jake"
         :person/partner "sally"
         :time 0})

;; At t1, jake no longer has a partner.
;; Since this is an "enterprise" company where we "update in place," I want my SQL database
;; to UPDATE people SET people.partner = NULL WHERE people.name = 'jake'.

;; What is the idiomatic way to represent this in Clojure?

;; Performing an effect (e.g. setting NULL) based on the absence of a value (the lack of a
;; :person/height key in this map) seems to violate the principal of least surprise:

(def me {:person/name "jake"
         :time 1})

;; The absence of :person/partner feels like it represents the absence of knowledge,
;; not the lack of a value (e.g. "jake has no partner".)

;; On the other hand, Maybe Not fairly convincingly argued against doing this:

(def me {:person/name "jake"
         :person/partner nil
         :time 1})

;; I suppose one solution might be something like this:

(def me {:person/name "jake"
         :person/weight 172
         :person/partner ""
         :person/has-partner? false
         :time 1})

;; But good luck convincing your coworkers to add a "has-partner" column to your database.
;; And what if you don't know if "jake" has a partner? :person/knows-has-partner? That way
;; lies madness...

;; Another, more datomic-oriented way of phrasing this is "how do you indicate that you
;; want to perform a retraction?"

lilactown16:06:37

you could use dissoc and contains?

alexmiller16:06:10

you could look at datomic's transaction model for one answer - datoms are marked with add/retract (https://docs.datomic.com/cloud/transactions/transaction-data-reference.html#org06a7dc2)

alexmiller16:06:24

the map transaction form for datomic is sugar over that - you decide how to transform a map into assertions

alexmiller16:06:49

(and you might have different rules than datomic would for this given the difference in storage)

jake14216:06:32

that’s interesting, I’ll dig into this

alexmiller16:06:35

having a well-known map value to indicate removal at the storage layer is worth considering

alexmiller16:06:43

could be nil, could be retract etc

alexmiller16:06:58

list the alternatives, consider the ways it affects your code

alexmiller16:06:12

keep clear the difference between a domain entity and a transactional representation of entity modification

jake14216:06:50

> you decide how to transform a map into assertions and > keep clear the difference between a domain entity and a transactional representation of entity modification Nice way of framing this, that’s a boundary that I don’t think about frequently enough.

kyle19:06:39

Why doesn't clojure.edn/read-string use *data-readers*?

dpsutton19:06:51

i think the history document had a bit of info on this

dpsutton19:06:42

it can be passed the readers to use. but the history document explained very well that the reader should not need a compiler.

dpsutton19:06:17

> Note that the reader can produce those data structures given that text without invoking the compiler/interpreter. Interpreting those data structures as the Clojure language is an entirely orthogonal process.

seancorfield19:06:56

As a user of clojure.edn/read-string I would want very strict control over what tags it was able to read, since those involve calling functions.

dpsutton19:06:19

The critical property of this system is that any edn reader, without enhancement, can read any
edn file. Specifically, it can read tagged literals it doesn’t understand (i.e. for which there is no
handler), know that it doesn’t understand them, and allow for programmer-defined policy in that

seancorfield19:06:19

I would not want #launch-missiles 42 to be automatically processed in some random, external EDN file, just because I happened to have sneaky/missile-launcher as a dependency on my classpath (i.e., I don't want that project's data_readers.clj file to affect my ability to safely read EDN).

dpsutton19:06:57

as an aside, the history document is fantastic and gives a lot of background to the decision points and what you gain from those decisions

dpsutton19:06:16

the reader not requiring the compiler i had missed up until now so was great seeing that explicitly made

mat19:06:36

I'm wondering what the idomatic approach is here:

(defn edit-product
  "Update a property of a product. "
  [db id updates] ; updates is expected to contain at least one of {:name, :location, :unit} and nothing else.
    (-> (update :products)
        (sset updates)
        (where [:= :id id])
        (execute db))
I want to do two things: 1. Indicate to the caller what the valid updates are for edit-product (name, location, and unit) 2. Validate that the caller has provided at least one valid key, and no invalid keys. Two is easy enough to check on it's own, but is there a way that merges both concerns into one? I feel like spec might be the answer here?

mat19:06:57

Could you please give me an example of a spec that would validate that? And how would I use it? Would I just (s/validate ...) inside edit-product?

seancorfield20:06:42

(I'm not sure how familiar you are with Spec based on your question)

seancorfield20:06:01

I'd say s/fdef to declare what the :args of edit-product can be -- a function spec appears in the output of doc when someone calls that on your function. I think that answers #1 ?

mat20:06:06

I have, and I've used it a little but never in this context. I've used it when creating re-frame apps to validate the app-db state.

seancorfield20:06:44

And inside the function you could call s/valid? on the updates arg, reusing the same Spec that you used in :args to describe updates.

mat20:06:36

That looks like just the thing! Thanks Sean.

seancorfield20:06:59

So once you have #2 as (s/def ::updates ...) (or whatever), then you would reuse that in (s/fdef edit-product :args (s/cat :db any? :id any? :updates ::updates))

seancorfield20:06:56

And now you also have the bonus that you can call (clojure.spec.test.alpha/instrument) and calls to edit-products during dev/test will now be automatically checked to validate that only valid updates are passed in.

mat20:06:59

Awesome. I didn't even know s/fdef was a thing. You say during dev/test. Does that imply the check will be 'compiled' out ?

seancorfield20:06:31

The check is only added when you call (st/instrument) -- in your REPL or your tests.

seancorfield20:06:14

I mean, you could call it in your production code too -- some folks with critical apps do run instrumentation in production, I believe -- but that checking does add an overhead.

mat20:06:50

ha.. The output of the doc leaves something to be desired if you don't now what product-update validates 😄

pasquet.skullery.backend.db/edit-product
([db id updates])
  Update a property of a product. 
Spec
  args: (cat :db any? :id number? :updates :pasquet.skullery.backend.db/product-update)
  ret: any?

seancorfield20:06:44

You can then do (doc :pasquet.skullery.backend.db/product-update) tho'...

kyle19:06:49

@seancorfield @dpsutton okay, thank you for the help!

seancorfield20:06:30

@kyle If you want to safely read all tagged literals in EDN, you can use (edn/read-string {:default tagged-literal} "some string")

seancorfield20:06:05

tagged-literal and tagged-literal? were added in Clojure 1.7 and the resulting object can be inspected via :tag and :form.

seancorfield20:06:18

user=> (edn/read-string {:default tagged-literal} "#launch-missiles 42")
#launch-missiles 42
user=> ((juxt :tag :form) *1)
[launch-missiles 42]
user=> (mapv type *1)
[clojure.lang.Symbol java.lang.Long]
user=> 

bfay21:06:44

How would one call a variable-arity java method from Clojure? I'm trying to call a method that has this signature allFlagsState(LDUser user, FlagsStateOption... options) Not sure how to pass 0, 1, or more params for options.

lukaszkorecki21:06:57

You can pass them as an array

lukaszkorecki21:06:45

(into-array FlagStateOption [op1 op2 op3])

ghadi21:06:56

@bfay mmm LaunchDarkly

bfay21:06:00

perfect, thank you

bfay21:06:05

haha you know it

ghadi21:06:11

Good service, really miserable Java API

bfay21:06:22

the new 5.0.0 API-breaking changes are a bit frustrating