Thanks for pointing back at the basics, I had never seen & remaining in sequential destructuring.

Thank you - I found the solution on the page you linked in the example before the very last

I got into this because I was trying to do something similar but a bit simpler: basically realizing at most N values "ahead of time" but using only a single (caller's) thread for "throttling" of AWS log api requests (first launch up to N StartQuery API requests, then poll the results by calling GetQueryResults API until the query is done - one query after another)

I was assuming that it would use 2 * coreN threads. But it uses 32 threads just as well. Huh.

pmap internally is using map. the only difference - it will wrap provided function into future. but chunking comes from map implementation to compare: (first (map (fn [i] (println i) i) (range 40))) will print 32 times as well

my “rule of thumb” - do not use pmap for any kind IO operations. only for intensive but pure computations

Yes, but that's not what I expected and I'm asking if this is intentional. Because I thought that it shouldn't run more than the <number of available processors + 2> tasks in parallel

So even with "pure computations" I could have let's say 2 processors and I wouldn't want to run 32 tasks in parallel instead

there is no “shouldn’t run more than the <number of available processors + 2> tasks in parallel” in pmap’s doc string unfortunately (

Yep, but this limit #processors + 2 is often used/recommended as the max cap for concurrency of CPU intensive tasks. So it almost looks like a bug although I'm quite sure people are aware of this behavior and it has some rationale behind it...

there is a workaround:

(defn re-chunk [n xs]
  (lazy-seq
   (when-let [s (seq (take n xs))]
     (let [cb (chunk-buffer n)]
       (doseq [x s] (chunk-append cb x))
       (chunk-cons (chunk cb) (re-chunk n (drop n xs)))))))

https://clojuredocs.org/clojure.core/chunk#example-5c9cebc3e4b0ca44402ef6ec

Interesting, thanks for the example

but I can not judge rationale for pmap’s behavior. never read anything about that

Another issue with pmap is that if the individual threads differ significantly in how long they take to finish, the parallelism can be significantly lower than the n+2, because in the absence of chunking weirdness it will not create threads further than that many 'ahead' of the last one that has finished.

In short, I think that if you want tight control of processor resources used at one time, pmap is rarely the right tool for the job, if ever.

Some of this is documented by Clojure users here: http://clojuredocs.org/clojure.core/pmap

I don't see a mention there anywhere of using Java ExecutorService directly, which is a semi-common recommendation for those that do want tight control over processor resources in use at one time.

I would recommend using reducers for any sort of computation of this sort. Has worked exceptionally well for me.

Thanks everyone, I was curious whether pmap’s weird behavior (running significantly more than number of processors +2 tasks - in common case up to 32- concurrently) has some justification or is a side effect of implementation). It’s hard to tell based on available information. Apart from that, my main use case is basically single threaded so I’ve been looking for an easiest way to launch (one by one) up to N jobs via api1 requests; then fetch results of the jobs via api2 one-by-one - as soon as the first api1 job finishes, N+1 api1 request can be executed, and so on. So I think I want a lazy seq with custom chunk size (which suggested re-chunk fn could solve); I admit this approach might not be appropriate given the usual advice of not mixing side effects with lazy collections but it looked as the most straightforward option.

If you don't want to reinvent the wheel, use Claypoole, it provides a pmap equivalent implementation with an executor service https://github.com/TheClimateCorporation/claypoole

Given that recommendations to “not use pmap” seem to be a fairly frequent theme, are there any plans afoot to improve its implementation?

Given the desire to preserve backwards compatibility, and the existence of libraries like claypoole (and I believe others), and that the Clojure maintainers likely have many other things higher on their list of priorities they want to work on, I doubt there are any such plans.

Seems like backwards compatibility wouldn’t be hard to preserve, even while the implementation is redone. Code that relies on the chunking behaviour, for example, is relying on an undocumented implementation detail.

Like map, except f is applied in parallel. Semi-lazy in that the parallel computation stays ahead of the consumption, but doesn't realize the entire result unless required. Only useful for computationally intensive functions where the time of f dominates the coordination overhead.

↑ seems like a pretty broad contract for the implementation

And yes, it goes without saying that the realities of person-power and priorities always come into play.

Sure, probably better if you omit the mention of backwards compatibility from my answer, since it is likely a red herring. The other issues are most relevant here.

cc @U06MDAPTP

Since tailwind is based on data/configuration, it doesn't really make sense to parse the generated output for one configuration - you would lose the benefit of tailwind. You'd want to get the data/configuration and just write a renderer for it in clojure instead.

@U05509S91 I have some stuff on the back burner but I’ve been so busy personally that I doubt any of it will see the light of day soon. As a community it would be awesome to have, at a minimum, a standard CSS parser.

@U08JKUHA9 that's fair! I hadn't even considered that, I was thinking fully about the end generated CSS

so maybe I do want a full port, because I'd want to use it as cljc

I recommend using the library instaparse for your parser instead of writting something from scratch.

Agreed - I've done too many handwritten parsers and, at least for my needs, I'm not looking at raw performance of a hand-tuned parser.

;; At t0, jake had a partner:

(def me {:person/name "jake"
         :person/partner "sally"
         :time 0})

;; At t1, jake no longer has a partner.
;; Since this is an "enterprise" company where we "update in place," I want my SQL database
;; to UPDATE people SET people.partner = NULL WHERE people.name = 'jake'.

;; What is the idiomatic way to represent this in Clojure?

;; Performing an effect (e.g. setting NULL) based on the absence of a value (the lack of a
;; :person/height key in this map) seems to violate the principal of least surprise:

(def me {:person/name "jake"
         :time 1})

;; The absence of :person/partner feels like it represents the absence of knowledge,
;; not the lack of a value (e.g. "jake has no partner".)

;; On the other hand, Maybe Not fairly convincingly argued against doing this:

(def me {:person/name "jake"
         :person/partner nil
         :time 1})

;; I suppose one solution might be something like this:

(def me {:person/name "jake"
         :person/weight 172
         :person/partner ""
         :person/has-partner? false
         :time 1})

;; But good luck convincing your coworkers to add a "has-partner" column to your database.
;; And what if you don't know if "jake" has a partner? :person/knows-has-partner? That way
;; lies madness...

;; Another, more datomic-oriented way of phrasing this is "how do you indicate that you
;; want to perform a retraction?"

you could use dissoc and contains?

you could look at datomic's transaction model for one answer - datoms are marked with add/retract (https://docs.datomic.com/cloud/transactions/transaction-data-reference.html#org06a7dc2)

the map transaction form for datomic is sugar over that - you decide how to transform a map into assertions

(and you might have different rules than datomic would for this given the difference in storage)

that’s interesting, I’ll dig into this

having a well-known map value to indicate removal at the storage layer is worth considering

could be nil, could be ::retract etc

list the alternatives, consider the ways it affects your code

keep clear the difference between a domain entity and a transactional representation of entity modification

> you decide how to transform a map into assertions and > keep clear the difference between a domain entity and a transactional representation of entity modification Nice way of framing this, that’s a boundary that I don’t think about frequently enough.

Could you please give me an example of a spec that would validate that? And how would I use it? Would I just (s/validate ...) inside edit-product?

Have you seen https://clojure.org/about/spec ?

(I'm not sure how familiar you are with Spec based on your question)

I'd say s/fdef to declare what the :args of edit-product can be -- a function spec appears in the output of doc when someone calls that on your function. I think that answers #1 ?

I have, and I've used it a little but never in this context. I've used it when creating re-frame apps to validate the app-db state.

And inside the function you could call s/valid? on the updates arg, reusing the same Spec that you used in :args to describe updates.

That looks like just the thing! Thanks Sean.

So once you have #2 as (s/def ::updates ...) (or whatever), then you would reuse that in (s/fdef edit-product :args (s/cat :db any? :id any? :updates ::updates))

And now you also have the bonus that you can call (clojure.spec.test.alpha/instrument) and calls to edit-products during dev/test will now be automatically checked to validate that only valid updates are passed in.

Awesome. I didn't even know s/fdef was a thing. You say during dev/test. Does that imply the check will be 'compiled' out ?

The check is only added when you call (st/instrument) -- in your REPL or your tests.

I mean, you could call it in your production code too -- some folks with critical apps do run instrumentation in production, I believe -- but that checking does add an overhead.

ha.. The output of the doc leaves something to be desired if you don't now what product-update validates 😄

pasquet.skullery.backend.db/edit-product
([db id updates])
  Update a property of a product. 
Spec
  args: (cat :db any? :id number? :updates :pasquet.skullery.backend.db/product-update)
  ret: any?

You can then do (doc :pasquet.skullery.backend.db/product-update) tho'...