Fork me on GitHub
#clojure
<
2018-11-28
>
johnj01:11:22

if you have something like (into [] a-transducer (range n)) - does the range sequence gets created and cached here?

hiredman01:11:03

there is no seq

hiredman01:11:13

range returns a thing that both implements Iterable and IReduce, both of which have custom fast paths when processed via reduce, which into does

johnj01:11:18

gotcha thanks, was just learning about reducible collections

johnj01:11:47

looking at the source, in these case is using transduce instead of reduce but yeah, same concept

Alex Miller (Clojure team)02:11:32

and range doesn’t actually cache anything

johnj02:11:08

@alexmiller if you do (map foo (range n)) - the values of range don't get stored in memory?

Alex Miller (Clojure team)02:11:37

that’s not the code you have above

Alex Miller (Clojure team)02:11:50

and what happens is much different

johnj02:11:10

oh ok, though you were saying it in general

Alex Miller (Clojure team)02:11:13

when using a range as a reducible (as with into), the whole thing just boils down to a single loop over the data and no values are cached in memory

Alex Miller (Clojure team)02:11:08

even when used as a seq, it builds chunks that store only the start/step/count values, not the actual data values

johnj02:11:01

neat, so something not to worry about to much

johnj02:11:40

what makes a collection reducible? what are properties must it satisfy?

Alex Miller (Clojure team)02:11:03

and has a reduce method

Alex Miller (Clojure team)02:11:07

the into example above is primarily running this loop

Alex Miller (Clojure team)02:11:33

where f is the transducer

Alex Miller (Clojure team)03:11:38

this is not the only path to being reducible, but it’s the fastest

Alex Miller (Clojure team)03:11:48

you can also participate in the CollReduce protocol

jaawerth03:11:41

for building non-caching reducible things on the fly, there's also eduction

johnj03:11:56

so basically it needs to know how to perform a reduce on itself

Alex Miller (Clojure team)03:11:26

vectors and maps also know how to do this

johnj03:11:31

in the case of long range

Alex Miller (Clojure team)03:11:50

and the results of cycle, iterate, repeat

johnj03:11:03

is there a predicate for reducibles?

johnj03:11:23

(doc reducible?) returned nil 😉

jaawerth03:11:32

you can use instance? though

Alex Miller (Clojure team)03:11:59

#(or (instance? clojure.lang.IReduceInit %) (satisfies? clojure.core.protocols/CollReduce %))

jaawerth03:11:24

@alexmiller that reminds me, has there ever been any consideration of a for variant for producing a similar reducible/iterable? I remember poking around a while back thinking there'd be one. I figure it just doesn't come up enough as an optimization to be necessary, particularly since you can usually achieve the same thing with eduction, iterate, etc, but while we're talking about it....

Alex Miller (Clojure team)03:11:09

I’m not sure that makes sense

Alex Miller (Clojure team)03:11:58

reducible gets it power by eagerly consuming the source, so doesn’t make sense to make one reducible from another

jaawerth03:11:29

just curious!

johnj03:11:06

does the sequence fn just does the opposite of a transduce?

Alex Miller (Clojure team)03:11:26

not sure what that would mean, but no :)

johnj03:11:22

heh, let me reread this

Alex Miller (Clojure team)03:11:34

the sequence function, when used with a transducer makes an incrementally computed sequence

jaawerth03:11:46

mostly it's just there so you can generate a lazy-sequence by composing transducer functions (map, filter, take, etc) without having to use them as lazy sequence functions - saves the overhead of the intermediate sequences

Alex Miller (Clojure team)03:11:07

there are some differences from lazy sequences as each intermediate step is fully computed

Alex Miller (Clojure team)03:11:19

transducers are a push model, not a pull model

Alex Miller (Clojure team)03:11:47

so for example, lazy sequences can compute infinite intermediate sequences and you can use as much of that as you like

Alex Miller (Clojure team)03:11:10

but sequence with a transducer would work forever producing the intermediate step

Alex Miller (Clojure team)03:11:20

most commonly this comes up with mapcat

jaawerth03:11:52

oh, yeah, ike if you had a filter that's never satisfied it would run forever trying to get to a value that satisfies the filter even if you only try and get a single item from it

Alex Miller (Clojure team)03:11:28

well, that may be true of both

jaawerth03:11:41

hahaha yeah I realized that after I said it, whoops

Alex Miller (Clojure team)03:11:08

like (take 2 (mapcat #(repeat 1000 %) (range 5))) only needs to produce 2 values

jaawerth03:11:11

actually, that explains why the core transducer functions don't just do, like, ([pred col] (sequence (filter pred) col)) for the lazy-seq-producing form, instead using lazy-seq directly

Alex Miller (Clojure team)03:11:32

but (into [] (comp (mapcat #(repeat 1000 %)) (take 2)) (range 5)) will produce 1000 values and give you 2

jaawerth12:11:39

@alexmiller actually, thinking about this further, are you sure this is the case? take short-circuits the whole pipeline (even to the point of preventing chunked loading when using sequence) - (into [] (comp (mapcat #(repeat 1000 %)) (map #(do (println %) %)) (take 2)) (range 5)) only prints 2 lines, for example, as does (sequence (comp (mapcat #(repeat 1000 %)) (map #(do (println %) %)) (take 2)) (range 5))

jaawerth12:11:18

(into [] (comp (mapcat (fn [v] (iterate #(do (println %) %) v))) (take 2)) (range 5)) also successfully terminates despite the infinite iteration in the mapcat, printing 1 line

jaawerth13:11:39

My read on it is that while transducers are push-based, the fact that it's all driven by a reductio pipeline lets you nevertheless control it via reduced, avoiding realizing values that will never be used except in unavoidable situations that also apply to lazy sequences. At least, I can't think of one that would apply to a transducer pipeline + sequence but not a lazy-seq

jaawerth03:11:33

which I always do when I write my own because I am a lazy fool

Alex Miller (Clojure team)03:11:46

both seqs and transducers have their place

jaawerth03:11:27

oh sure. I just mean, when I write my own custom transforms, I like to follow teh same pattern as core and make them work either as transducers or sequence functions based on arity. but I usually just take the shortcut of using sequence. this conversation has me reconsidering that, though

Alex Miller (Clojure team)03:11:39

yeah, I wouldn’t do that

Alex Miller (Clojure team)03:11:59

they really are different

jaawerth03:11:20

yeah, I see why it's a bad idea now that you point out the effects of the push behavior. thanks

jaawerth03:11:32

there's likely also more overhead for the same result

johnj03:11:00

@alexmiller your into and concat example was to demonstrate what the sequence fn does with a transducer? that it fully computes intermediate values

johnj03:11:16

or as you better said, each intermediate step is fully computed

johnj03:11:50

finding a use case for sequence, is like when you want a lazy result of intermediate steps

Alex Miller (Clojure team)03:11:30

that use case is imo small

johnj03:11:29

yeah, I guess is not used much in practice, or at least can think of any right now.

pablore03:11:23

Has anyone here successfully deployed a “create-react-app” app with ring? I’m having trouble to work with registerServiceWorker.js, it says on the console log:

No internet connection found. App is running in offline mode.

pablore03:11:03

I’m using ring.middleware.resource/wrap-resource and it works

todo05:11:08

Is https://www.reddit.com/r/Clojure/comments/z0o6u/robbit_reddit_apibots_in_clojure/ state of the art for getting started with "write a reddit bot in clojure" ?

theeternalpulse07:11:02

the github link doesn't work, and the author doesn't seem to appear either

cheatex14:11:37

Hi. I'm trying to write a test with cljs.test. I want to have all the names from my main module right available in the test module. Without prefixes or explicit enumeration. Can I do that?

orestis14:11:30

You mean without requiring them?

cheatex14:11:38

Basically (:use [app.main :only [multiply]]) works while (:use [app.main]). While the docs say that both are correct.

cheatex14:11:58

@orend Any way that works.

orestis14:11:26

(:require [app.main :refer :all])

orestis14:11:48

I believe :use is discouraged these days.

cheatex14:11:11

Still doesn't compile Only :as, :refer and :rename options supported in :require / :require-macros;

cheatex14:11:14

Wrong message, real one > Don't know how to create ISeq from: clojure.lang.Keyword

orestis14:11:24

Can you paste your entire ns form?

dpsutton14:11:11

did clojurescript drop support for :all? am i misremembering that?

orestis14:11:32

Oh, is this ClojureScript?

dpsutton14:11:33

drop or possibly never support?

dpsutton14:11:40

cljs.test was mentioned

cheatex14:11:35

yep. Does it handle ns differently?

orestis14:11:57

Ah, yes. :all is not available in ClojureScript.

orestis14:11:06

No idea why though.

cheatex14:11:16

Wow. Didn't expect incompatibility on this level. Thanks!

orestis14:11:52

It most likely is an issue with bundle size — but you could ask on #clojurescript for details.

orestis14:11:02

As a “workaround”, you could do (:require [app.main :as a]) and use a/foo, a/bar. It’s an extra two characters on every occurrence, so just a little bit more onerous than the naked case.

orestis14:11:21

I agree though that sometimes for tests it’s nice to be able to just bring everything in.

orestis14:11:11

@masta please consider unpinning your message.

orestis14:11:11

Yeah, it’s discouraged in this Slack, only for admins to use. Thanks 🙂

Audrius14:11:21

How do you guys do Profile Clojure projects? I have tried JVisualVM bur it gives me some cryptic info. Please advice.

orestis14:11:35

Can you share some more info? What’s the cryptic info look like, what are you trying to measure?

Audrius14:11:32

https://imgur.com/a/sPlHqKl This is what I mean. Nothing resembles my actual code.

orestis14:11:19

Are you using code from the namespaces importer.integrity and clj_xml_validation?

orestis14:11:37

Meaning, is your code in those two namespaces?

orestis15:11:53

I’m not an expert on this, nor do I know if there’s a better way to do profiling on Clojure, but with a little effort this output is readable: clj_xml_validation.core$validator_from_schemas.invokeStatic is equivalent to a call to function (clj-xml-validation.core/validator-from-schemas ...)

orestis15:11:44

importer.integrity$differing_contexts_for_global_ids$iter_4304__4308$fn__4309$fn__4314.invoke is a little bit trickier, as it probably involves some anonumous function inside importer.integrity/differing-contexts-for-global-ids.

orestis15:11:13

The top one, clojure.lang.Util.equiv is the = function, IIUC.

Audrius15:11:57

thanks I will have a closer look!

Audrius15:11:05

Do you know any other tools?

dazld20:11:06

that might help you out a bit

dazld20:11:05

and - don’t forget you can (and should) name functions too eg, (fn foo [a b] (+ a b))

dazld20:11:18

which will help a ton for those anons

jumpnbrownweasel15:11:33

The hot spots can be confusing because you're not seeing a tree of method calls. Look in the Threads tab or Thread CPU time (can't remember if both of them have the method call tree).

enforser15:11:44

Another issue which I have run into with profiling in Clojure is that laziness can give you misleading results. i.e. If I construct a large lazy sequence in an inefficient manner, then I will not see the CPU spike at construction time (where the problem is), but whenever the list is actually evaluated and used

Eyal16:11:07

Hi guys, ever since I've upgraded from AWS Java SDK V1 to V2, I've been getting about 80 lines of warnings about version ranges. From reading [1] I understand that this issue is not going to be resolved soon, but is there anything I can do to suppress these messages for just these dependencies? [1] https://github.com/technomancy/leiningen/issues/2251