Fork me on GitHub
#clojure
<
2020-04-25
>
potetm00:04:12

@noisesmith How is it different than swapping into the atom in your code directly, (instead of calling tap>)?

potetm00:04:43

I’ve never actually understood the use case vs, e.g., calling def inline in the code

seancorfield00:04:57

@potetm The tap> can remain as-is, even in production. You can add (and remove) as many "taps" (watchers) as you want and they can each do different things.

seancorfield00:04:45

It allows you to have tap> in a running process, attach to it via a REPL, add some tap instrumentation, do some debugging, remove the tap instrumentation and then move on...

noisesmith00:04:31

right, no weird defs / printlns sneaking into the codebase

noisesmith00:04:23

I do wish tap> had a predicate / filter mechanism, but I'm also glad it's simple

potetm00:04:36

:thinking_face: I’m gonna have to consider that more. I usually refrain from connecting to a live process, so logging seems to be a better fit for my usage.

potetm01:04:03

If I want a REPL, I usually spin up a separate process to prevent accidents.

potetm01:04:36

Pretty rare that the problem is on a single box and not “the environment” (i.e. db+queues+network).

noisesmith01:04:09

I don't use tap> outside my dev repl, but it's still more ergonomic to control compared to printlns or defs

potetm01:04:52

Vs println, benefits are obvious. It’s hard for me to see how it’s better than def in a dev env.

potetm01:04:28

Seems like unnecessary indirection.

noisesmith01:04:48

with the swap! into atom, I get data from multiple sources in the order the tap> calls were hit

potetm01:04:00

@seancorfield Do you often use it like that? Connect to a prod repl?

potetm01:04:18

@noisesmith Sure. Swap into a def’d atom?

seancorfield01:04:42

I use tap> a lot of debugging -- because I use REBL and it can automatically accumulate results, and then let you browse them in tabular and graphical form.

potetm01:04:43

idk, no knocking your usage. More trying to convince myself.

noisesmith01:04:20

right, my tap function does a swap, maybe also attaching a time stamp and call stack

potetm01:04:32

idk there’s at least two other ways of doing that 🙂

potetm01:04:49

1. call a fn inside the swap 2. use a watch

noisesmith01:04:34

sure - but this way I only edit in one place to get the data (call tap> on each target), then separately refine the data gathering as I figure out what I need

👍 4
noisesmith01:04:00

and yes I'm doing exactly what you said - attaching data with the fn I provide to swap - with no risk of errors caused by stray debug code sneaking into the file

👍 4
potetm01:04:41

yeah okay, integrating with external tools: makes total sense

noisesmith01:04:52

then I can group-by, filter... use clojure generally

seancorfield01:04:23

And, yes, I do connect to REPLs in production and will add taps for debugging. In those cases, I'll usually redefine a few functions with tap> in them, then I'll call add-tap to get hold of the data to process/store it somewhere for analysis, then remove the taps (but leave the tap> calls in place).

potetm01:04:49

That sounds roughly like adding printlns and re-defing, no? (albeit it has all the benefits of data instead of bytes)

seancorfield01:04:58

One of the nice things about tap> is that you can put it into a very high traffic function and it simply drops data when there's too much being processed, so you won't block your production app 🙂

potetm01:04:56

Yeah I saw that in the impl. Still, not cost free to leave it there (though probably cost negligible).

potetm01:04:46

oh wait, I see what you’re saying

seancorfield01:04:58

Logging isn't cost free either, yet we use that for debugging a lot 🙂 and even putting a def into a function to capture values isn't free (and only captures one values at a time).

potetm01:04:41

Thing is: Logging can be configured ad nauseam, even at runtime. And a lot of effort has gone into optimizing them. I’ve seen it fail, but I certainly trust it a lot more than whatever fn I cook up to slide into a hot path.

potetm01:04:33

Well, to your point, my code isn’t in the hot path.

noisesmith01:04:00

and yes I'm doing exactly what you said - attaching data with the fn I provide to swap - with no risk of errors caused by stray debug code sneaking into the file

👍 4
potetm01:04:31

“add it to something that’s called 1000x a second” -> end up with a bunch of data, but not a stalled server

noisesmith01:04:57

tap> is internally rate limited and drops data if your debug fn is too slow. and you remove the tapping fn and now it's a noop

noisesmith01:04:23

acts like a dropping buffer on a chan

potetm01:04:51

yeah I see the impl

noisesmith01:04:27

oh I misread what you said here

potetm01:04:27

Yeah, I originally misunderstood Sean’s point. I thought he was talking about the impl, but I think it was more to do with the user experience.

potetm01:04:37

when debugging

seancorfield01:04:05

I would never have thought of tap> -- but it has been amazingly useful since Rich thought it up and added it. The debugging function I never knew I needed! 🤯

potetm01:04:03

Hmm… I’ll think on it more. I’m still skeptical that it’s worth the indirection except in the case of tooling.

potetm01:04:26

I appreciate ya’lls thoughts and sharing ya’lls experiences @seancorfield and @noisesmith

potetm01:04:15

It’s nice knowing I can get really good, informed opinions around here 😄

potetm01:04:37

Opinions worth mulling over, worth maybe changing my mind over

murtaza5208:04:30

(defn median [ns]
(let [ns (sort ns)
cnt (count ns)
mid (bit-shift-right cnt 1)]
(if (odd? cnt)
(nth ns mid)
(/ (+ (nth ns mid) (nth ns (dec mid))) 2))))
In the above function (from rosetta stone) what is bit-shift-right achieving?

dpsutton09:04:30

dividing by 2

dpsutton09:04:50

(map #(bit-shift-right % 1) [1 2 4 6 8 10]) -> (0 1 2 3 4 5)

dpsutton09:04:15

in base 10 it "pushes" the number and drops off the last one. so 143 -> 14 and is effectively dividing by 10. in binary the base is 2 not ten so when pushing over its not dividing by 10 but by 2

Vincent Cantin16:04:51

I would like to propose the macro implies to be added to clojure.core

(defmacro implies [condition implication]
(if ~condition
~implication
true))
is this the right place to talk about it?

Vincent Cantin16:04:35

This macro is useful when used in a conditional expression. It’s an alternative to the less readable (or (not condition) implication)

Vincent Cantin16:04:14

exemple of usage

potetm16:04:34

1. If you want to discuss language features, #clojure-dev or https://clojure.atlassian.net/projects/CLJ/issues is the place 2. I do not expect such a proposal to get very far

Alex Miller (Clojure team)16:04:50

Actually, the preferred place to make such a request is https://ask.clojure.org with a request tag

✔️ 8
potetm16:04:30

good to know

Vincent Cantin06:04:35

Thanks @U064X3EF3 I won’t request it as an enhancement because now I think that adding one more function in the core library would only amplify its learning curve unnecessarily.

potetm16:04:21

The use case is already covered. What you’re asking for is purely syntactic and, as you’ve shown, easy to achieve with a macro.

potetm16:04:10

Aside: As frustrating as that might sound, it’s an important design goal that Clojure give users the power of the language maintainers.

Vincent Cantin16:04:19

that’s not frustrating, don’t worry

Vincent Cantin16:04:10

maybe you are right, this should not be in clojure.core

dominicm16:04:22

Java interop question, given a ring path like "/foo/bar", how do I convert that into a (presumably) windows path, but preferably something where I don't care and could use nio.Path or similar.

dominicm16:04:33

I suppose something like File/separator is what I want 🙂 Replace / with File/separator

potetm16:04:55

@dominicm depending on what you’re doing, Java might take care of that for you

potetm16:04:29

e.g. you can always do (io/file "foo/bar/baz") and get a proper OS-dependent path

dominicm16:04:45

Oh really? Huh. I wasn't aware of that.

dominicm16:04:54

(I haven't run windows in a very, very long time)

potetm16:04:59

Yeah, Java did a half-decent job at actually making file-handling cross-compatible.

potetm16:04:17

(last time I had to do something like this, a REPL running in a VM was really, really helpful)

seancorfield17:04:41

Yes, forward slashes work via Java on both platforms (good) but you have to be careful about any code that either accepts paths as input from any config or user data, or constructs filesystem paths from filenames (or portable /-separated paths). Writing correct cross-platform code can be tricky, even with Java's help 🙂

💯 4
seancorfield17:04:57

The other thing now with Windows 10 and WSL, you can also map c:\path\to\file.ext on the Windows side to /mnt/s/path/to/file.ext in WSL (Chlorine for Atom does this so it can support connecting from Atom on Windows to a REPL running on WSL).

mauricio.szabo17:04:52

Yes, I remember when I caught a bug because Java returned /C:/Some Folder/Some File as a valid windows path...

seancorfield17:04:54

I'm still surprised by that one, since it indicated different behavior on different JDKs, right?

mauricio.szabo17:04:57

I don't think so... I remember that I tested on Hotspot OpenJDK, OpenJ9, and Graal and they all gave the same results

mauricio.szabo17:04:55

I think it's more of a case to be aware that Java can return a path that will not work with some libraries' code. (that, or Microsoft decides to accept / as path separators, but I would not bet on that one 😄)

potetm18:04:39

I was just reading this Reddit post: https://www.reddit.com/r/Clojure/comments/g7qyoy/why_does_orm_have_omg_complexity/ Anyone know what Rich meant when he said, “What’s the dual of value? Is it covalue? What’s a covalue? It’s an inconsistent thing.”

hiredman19:04:34

A value is a fixed unchanging thing, in category theory the prefix "co" means the reverse or inverse, so a covalue is like something always changing

potetm19:04:21

yeah, that I know. How does it relate to ORMs and “duals?”

hiredman19:04:22

But in that particular talk I believe covalue was a dig at another keynote

potetm19:04:55

(I don’t know what “dual” meant in that context.)

hiredman19:04:05

Erik meijers

hiredman19:04:49

He was a big fp/category theory guy. I don't really recall his talk, but the covalue stuff was I am sure added to Rich's talk in response

potetm19:04:29

@hiredman any idea the particular talk? Or even what conference Rich’s talk was at? Any idea how/whether it relates to ORMs?

hiredman19:04:05

Both were at strangeloop 2011

potetm19:04:26

ah, I’ll watch that

hiredman19:04:24

My first time at strangeloop, it was fantastic

potetm19:04:30

hmm… I can’t seem to find a recording 😕

jaihindhreddy05:04:53

I remember Alex Miller saying that Eric Meijers requested the recording not be published.

sveri19:04:31

Hi, I have a function like this:

(defn wrap-api [rout dev?]
(-> (if dev? (#'rout) (rout))
do-something))
Now if I try to compile that, the compiler complains it cannot resolve the var rout I assume #' does not work for function parameters? Is there a different way to do that?

hiredman19:04:12

Var qoute is only for vars

potetm19:04:58

#'rout translates to (var (quote rout))

potetm19:04:23

@U0677JTQX this does not appear correct

sveri19:04:06

Thank you both

hiredman19:04:38

No, #'rout translates to (var rout) no quote

potetm20:04:31

(but thanks for posting in the main channel for bystanders)

joshkh20:04:15

is it possible to read a def'ed expression as a string?

(defn some-function [] (+ 1 2 3))

(magic-reader some-function)
=> "(defn some-function [] (+ 1 2 3))"

dpsutton20:04:04

there's clojure.repl/source

joshkh20:04:29

i wasn't able to get clojure.repl/source to work with non-core functions:

(defn some-function [] (+ 1 2 3))
=> #'myns/some-function
(clojure.repl/source some-function)
Source not found
=> nil

dpsutton20:04:56

they need to be in source files and not just loaded in the repl but required

dpsutton20:04:45

"Prints the source code for the given symbol, if it can find it.
This requires that the symbol resolve to a Var defined in a
namespace for which the .clj is in the classpath.

Example: (source filter)"

dpsutton20:04:07

because it uses line number metadata to read it from the source

joshkh20:04:15

hmm. so if i defn my function in a namespace, load the namespace into a repl and call source on the symbol, it should work? silly example:

(ns mynamespace)

(defn some-example []
(+ 1 2 3))

(defn provide-source []
(clojure.repl/source some-example))

joshkh20:04:15

> but required nevermind, i got it. thanks @dpsutton 🙂

noisesmith15:04:26

also clojure.repl is injected into clojure by clojure.main when running a repl, but isn't guaranteed to be present in your app - require it if you use it

👍 4
stephenmhopper22:04:43

Is partition-all supposed to work with async/pipeline? I have a transducer and it looks like everything is just getting split into batches of size 1 and passed on from partition-all even though there’s far more items than that on the channel.

phronmophobic22:04:10

do you have some sample code?

stephenmhopper22:04:57

Yeah, my transducer looks something like this:

(comp (map transform-rows)
(partition-all 100)
(map some-batched-fn))
If I run that with transduce , my data comes through in batches of 100. If I ask async/pipeline to process the data, then everything comes through in batches of 1.

stephenmhopper22:04:31

I’m guessing it has something to do with this from the pipeline docs:

Because
it is parallel, the transducer will be applied independently to each
element, not across elements
What should I use if I want to process a sequence of data in parallel with my aforementioned transducer?

phronmophobic22:04:22

seems like it should be possible as long as you’re partitioning before you pipeline. let me give it a shot. one sec.

stephenmhopper22:04:32

Yeah, I’m probably going to move away from pipeline and just replace my map calls with pmap for simplicity. Thanks for the help!

phronmophobic23:04:47

sounds good, but for future reference, it seems like this is possible as long as you partition before hand. you can do this easily with clojure.core.async/pipe :

(defn some-batched-fn [xs]
(map inc xs))

(def to-chan (async/chan 30))
(def from-chan (async/to-chan (range 20)))

(async/pipeline 3 to-chan (map some-batched-fn)
(async/pipe from-chan (async/chan 1 (partition-all 3))))
(loop [x (<!! to-chan)]
(when x
(prn x)
(recur (<!! to-chan))))

hiredman22:04:26

A lot of stateful transducers like partition-all do not work in pipelines

stephenmhopper22:04:16

@hiredman I don’t know if you can see my comments from the thread above. What’s my best alternative then if I want to parallelize my transducer?

hiredman22:04:07

Hard to say, to some degree it depends on how you expect partitioning to work in parallel

hiredman22:04:40

pipeline preserves order when executing in parallel, but if you wanted partitioning to be have exactly like partitioning does on a sequence that eliminates parallelism

hiredman22:04:59

So maybe you can split your transducer around partition, a pipeline before, partition on the output channel, then feed the output into another pipeline for the bit after

stephenmhopper22:04:16

Yeah, that could work. I think my transducer is basic enough that I’ll probably ditch async in this particular case and just use pmap` for the parts before and after the partitioning. Thanks @hiredman

hiredman22:04:56

pmap makes me sad

potetm22:04:55

alternatives?

hiredman22:04:40

I use executors a lot

potetm23:04:45

loads of ceremony for a parallel pass over a sequence tho

potetm23:04:13

I keep meaning to look into reducers for this, but never had a strong need.

hiredman00:04:00

Reducers can only process vectors or maps in parallel because they use the tree shape structure to divide and conquer. The linear structure of a sequence isn't amenable

potetm00:04:47

ah, well, that’s a detail I didn’t know 😄

potetm00:04:53

Learning something already!