Fork me on GitHub
#clojure
<
2019-11-16
>
alexbaranosky00:11:41

something like:

(defn select-ns-keys [m namespace']
  (into {}
        (keep (fn [[k v]]
                (when (= namespace'
                         (namespace k))
                  [k v])))
        m))

(select-ns-keys {:a/key 1
                 :a/val 2
                 :b/key 3
                 :b/val 4}
                "b")
;; => #:b{:key 3, :val 4}

alexbaranosky00:11:04

can be done more efficiently with reduce-kv probably if you care

kulminaator12:11:47

what kind of simple solutions would you propose to distribute work amongst 64 worker threads ? right now i'm using pmap for comfort but i need more parallelism ( i'm calling a blocking api and other side is slow on latency, but can do a bunch of ops in parallel)

alexbaranosky14:11:50

I'd start really simple. An arrayblockingqueue and a bunch of Threads. Then if I wanted "more powa" use an Executor. Pmap is not likely ideal for throughput.

👍 1
jsyrjala16:11:20

claypoole has e.g pmap where you can specify your own thread pool.

kulminaator12:11:30

just rolling my own copy pmap with a variable number instead of the fixed cpu core related one ?

kulminaator14:11:08

will give it a shot if my pmap fork should fail me 🙂

tianshu14:11:36

why there's no spec for defprotocol and defrecord?

alexmiller14:11:49

Executors are designed for this - there’s no reason not to just start there

kulminaator16:11:26

indeed, this turned out easier than expected with the help of reify , callable and mapping deref on the results

kulminaator15:11:50

my remote workers that i invoke over the api do all the heavy lifting, i just need to tell them what to do and wait for their rather compact output (i literally tell them 3 strings to operate on and they haul away for 4-5 seconds before they reply), so performance at this point is not important, i just need more parallelism than 10 🙂. the interface of pmap is very comfortable and fits my usecase well.

kulminaator16:11:18

I will probably roll with the Executors, they will satisfy my needs

kulminaator16:11:39

i expect the spread to be better when the work is something other than "just return that value as a string" 🙂

kulminaator16:11:57

yep, even throwing a little sleep into the callable spreads the work out nicely

borkdude15:11:36

Is it by design that binding doesn't act like other let-like macros?

Clojure 1.10.1
user=> (def ^:dynamic x)
#'user/x
user=> (def ^:dynamic y)
#'user/y
user=> (binding [x "hello" y x] y)
#object[clojure.lang.Var$Unbound 0x58294867 "Unbound: #'user/x"]

mfikes16:11:05

binding is not like let… bindings are done in parallel

borkdude16:11:37

yes, but why?

mfikes16:11:52

(As an aside, there was a bug surrounding this in ClojureScript until 1.10.439)

mfikes16:11:34

It would be interesting to dig up the rationale for parallel binding. (I don’t know it offhand.)

andy.fingerhut16:11:28

I do not know the full rationale, but part of it might be that the things being bound via binding are all expected to already be def'd globally earlier, before the binding was eval'd, but let bound names can be purely local there, not existing anywhere else in the code.

andy.fingerhut16:11:47

That's more of a maybe-hint-at-the-reason than an answer to your question

sogaiu16:11:24

it looks like binding uses var on each thing before calling push-thread-bindings -- is that perhaps why things being bound need to be def'd before?

mfikes16:11:19

Looks like the existing semantics were being established / documented in 2009 https://clojure.atlassian.net/browse/CLJ-152

borkdude19:11:33

why is the stringwriter empty in this example?

$ clj -e "(def w (java.io.StringWriter.)) (push-thread-bindings {#'*out* w}) (try (println \"hello\") (finally (pop-thread-bindings))) (prn \">\" (str w))"
#'user/w
hello
">" ""
I'm trying to debug a problem with a custom binding macro

borkdude19:11:28

oh wait, the empty let in binding is actually doing something?

$ clj -e "(def w (java.io.StringWriter.)) (let [] (push-thread-bindings {(var *out*) w}) (try (println \"hello\") (finally (pop-thread-bindings))) (prn \">\" (str w)))"
#'user/w
">" "hello\n"

andy.fingerhut20:11:01

I don't know exactly how that is behaving, but probably has something to do with how Java method pushThreadBindings uses the variables dval in Var.java

theeternalpulse20:11:17

when defining a record that implements a protocol, is it general practice to access the keys via the defrecord argument signature, and only use the this as part of the protocol definition for doing assoc/update?

andy.fingerhut20:11:47

The (let [] ...) in the defmacro of bindings appears to a casual reader as if it could be replaced with (do ...) and everything would behave the same, but your code examples above, and the care with which most of Clojure's implementation is written, leads me to believe the let is significant there somehow.

hiredman20:11:22

It is, because each form in a top level do becomes a separate compilation unit

hiredman20:11:27

Let causes it to all be compiled and run as a unit

andy.fingerhut20:11:01

There is no explicit top level do in borkdude's examples. Is there an implicit one?

seancorfield20:11:37

@theeternalpulse I've been told (by Alex, and maybe others) that it is both idiomatic and faster to reference the record's fields directly by their declared name, rather than use (:field this)

hiredman20:11:52

The behavior without the let is equivalent to the behavior of a top level do

seancorfield20:11:18

So now I only use this in the protocol definition syntax and when manipulating the record as a whole.

seancorfield20:11:48

(and I often use _ if the body doesn't need to reference this)

dpsutton20:11:09

Is “compilation unit” a technical term in the compiler or just a way to say they might not share thread bindings or something?

borkdude20:11:01

Interestingly when implementing a Clojure interpreter I also chose to handle forms in do as separate compilation units. I'd like to hear more about this as well. Is the body in a let not handled as a do block?

hiredman20:11:04

I am not sure why the binding is behaving that way, and not at a real computer to investigate

hiredman20:11:19

I just know why let gets you different behavior

hiredman20:11:20

The compiler (and hence the eval in the repl) handle a top level form at a time

borkdude20:11:39

user=> (let [] (def x 1) (def y z))
Syntax error compiling at (REPL:1:19).
Unable to resolve symbol: z in this context
user=> x
#object[clojure.lang.Var$Unbound 0x29ea78b1 "Unbound: #'user/x"]
user=> (do (def x 1) (def y z))
Syntax error compiling at (REPL:1:15).
Unable to resolve symbol: z in this context
user=> x
1

borkdude20:11:13

I thought let had an implicit do and would be handled the same as if you had a real do

hiredman20:11:37

But it is a do still wrapped in a let

hiredman20:11:53

The do isn't top level

borkdude20:11:23

so first the entire let expression is analyzed, if there are any errors like unresolved symbols, none of the side effects will have happened

borkdude20:11:51

maybe I should revisit that behavior in my clojure interpreter..

hiredman20:11:16

It is complicated, because there are compile time and evaluation time side effecst

hiredman20:11:19

So for example, at compile time in your example, the var x was created, but because of compilation errors, the code was never run, which is why x is unbound

hiredman20:11:43

Runtime is a better word, not evaluation time

borkdude20:11:02

yeah. right now I handle every do the same, even inside a let expression, so maybe I'd have to revisit

andy.fingerhut20:11:35

Perhaps an aside-type detail: There are Clojure/JVM macros that cause mutations during macro-expansion. Fun!

borkdude20:11:02

maybe this isn't that well defined, just an implementation detail?

borkdude20:11:09

if there's docs about this, I'd like to see them

hiredman20:11:25

Google the gilardi scenario

andy.fingerhut20:11:57

Not "official" docs, but worth reading for some understanding of existing behavior.

borkdude20:11:33

ok, so breaking up do in separate compilation units was already a change

andy.fingerhut20:11:44

I woudn't be surprised if some/all of that is different in detail for ClojureScript, and/or not applicable somehow.

andy.fingerhut20:11:33

but have no detailed knowledge of ClojureScript in that area to say for sure. Just something to be mindful of if you care about cross-platform code.

borkdude20:11:14

yeah. I had some difficulties when writing tools around spec related to the order how things are analyzed in CLJS (e.g. try / catch), because analysis and macroexpansion also have side effects (e.g. in the spec implementation)

borkdude20:11:43

there was something like: the finally block was analyzed before the body

andy.fingerhut21:11:12

I know they have a lot more in common than they have different, but I suspect someone getting into these level of details of Clojure/JVM and ClojureScript would have a deep appreciation of a statement like "The United States and Great Britain are two countries separated by a common language." ( https://en.wikiquote.org/wiki/English_language )

andy.fingerhut21:11:45

Also, thinking about it a bit more, perhaps a rationale for binding being 'parallel binding' rather than sequential like let is, is because making it sequential would probably require pushing N binding frames onto the stack, rather than only one, and then popping N off when the scope of the binding was exited. let doesn't create any kind of run-time stack in its implementation, and the sequential behavior is so very very useful. Common Lisp has parallel let and sequential let*, and I know Rich mentions explicitly in an early talk on Clojure to Common Lisp audience that Clojure's let is CL let*, by explicit choice.

andy.fingerhut21:11:46

So let is sequential because it doesn't cost any more implementation-wise to make it sequential, and it is so useful for it to be sequential in many contexts, and binding being parallel is perhaps due to efficiency concerns.

borkdude21:11:31

makes sense!

borkdude21:11:25

I noticed when implementing binding myself, I did the parallel implementation first, just because it was easier. Maybe that's how let got into Common Lisp, because of laziness on the implementor's behalf 😉

andy.fingerhut22:11:08

Unless someone wrote it up in a history of Lisp paper somewhere, that decision is probably over 50 years old now

andy.fingerhut04:11:42

It seems like to get to the origin of Common Lisp's let vs. let* and why they both exist, one would have to go back to the several popular flavors of Lisp that existed before Common Lisp was created, and see what they had. I suspect several of those prior Lisp flavors had both, and some may have had only one. I would guess that there were some very old Lisp implementations where let* was more expensive at run time than let, by at least enough for efficiency-minded programmers to care.

borkdude12:11:23

Funnily enough I made the same mistake as CLJS: https://github.com/borkdude/sci/issues/164