Fork me on GitHub
#beginners
<
2017-12-27
>
drewverlee00:12:50

I assume you can reduce over a lazy-seq without any trouble. This code seems to running a bit longer then i would think though (about 3 mins so far)... The file is 10gigs though.

(with-open [in-file (io/reader (io/resource kf))]
  (->> (cd-csv/read-csv in-file :separator \|)
       (sc/mappify {:header schema})
       (reduce
        (fn [coll {:keys [customer]}]
           (conj coll customer))
        #{})
       count
       println))

drewverlee00:12:05

thats semantic-csv

noisesmith00:12:09

you could replace the reduce with (into #{} (map :customer)) (leaving the rest the same) and that should be faster

noisesmith00:12:44

since your reduce is equivalent to mapping :customer across the input

noisesmith00:12:49

(then putting in a set)

drewverlee00:12:17

oh, right. Why would that speed things up?

noisesmith00:12:05

because into uses a transient which is marginally faster for building collections, and it's likely (map :customer) inlines better (edit - for larger collections the speedup can be better than marginal)

drewverlee00:12:03

is it possible to leverage multiple threads over chunks of the lazy sequence? Kind of like a map reduce?

sabbatical201701:12:33

What's the idiomatic way to do a map that has some state? I have a function f that basically takes x and returns y, but also takes in a "state of the world" s and returns a "changed state of the world" new-s, so that (f x s) is [y new-s]. Given an initial state s0, and some vector of xs like [x0 x1 x2 ...], I would like to compute y0 which comes from (f x0 s0) being [y0 s1], then y1 which comes from (f x1 s1) being [y1 s2], and so on for y2, y3, etc. (Also I would like to get the final value of s.)

sabbatical201701:12:21

If it wasn't for the s stuff, this would just be (map f xs).

sabbatical201701:12:32

There is probably some kind of Haskell higher-order function that does this, but of course I am interested in doing this in idiomatic Clojure.

noisesmith01:12:37

that's a reduce

noisesmith01:12:46

haskell calls it fold

sabbatical201701:12:06

I thought that maybe I can do something with reduce or reductions, but I could not see a nice way to arrange it.

noisesmith01:12:38

you can use a hash map as your accumulator, and hold your state-of-the-world plus your ys under two keys

sabbatical201701:12:57

@noisesmith Ah yes, that works very nicely, thank you!

naylyn.gaffney04:12:33

What's the most computationally efficient way of mutating elements of a Java array? So far I've tried doseq + aset and it seems terribly slow.

donaldball04:12:23

I would guess use loop/recur to walk the index and make sure you type hint and deal with boxes correctly.

donaldball04:12:59

dotimes is probably a better higher level form than loop/recur on second thought

noisesmith05:12:45

@naylyn.gaffney you could check out amap and areduce - you can run into pitfalls with numeric boxing with loops if you are not careful but those functions abstract that stuff a bit

noisesmith05:12:02

also be sure to check for runtime reflection and numeric boxing in general http://insideclojure.org/2014/12/15/warn-on-boxed/

naylyn.gaffney05:12:21

Thanks @donaldball and @noisesmith

naylyn.gaffney05:12:38

Having no experience with Java prior to Clojure it feels like there are a lot of landmines I need to avoid.

noisesmith05:12:32

for performance of numerics there are a bunch of gotchas - honestly it's often easier to write java code and call it from clojure instead

noisesmith05:12:09

but for correctness of programs that use a lot of threads (and readability of that code) - clojure can really excell

noisesmith05:12:40

the reflection and numeric boxing are a tradeoff for using a language that can't be effectively statically analyzed - there are benefits too (repl based development, being able to define things piece by piece without restarting the whole thing) but it makes optimal numeric performance tricky sometimes

naylyn.gaffney05:12:45

yeah I'm seeing that, it's starting to feel like I'm writing something closer to java than clojure at this point

sabbatical201716:12:23

What causes Math.abs to be available in my Clojure code? In one of my source files, the REPL says CompilerException java.lang.ClassNotFoundException: Math.abs. The same thing works in another file though. I have no idea why. I have not done anything to refer (or anything like that) Math in either file.

alexmiller16:12:11

All classes in java.lang are automatically imported for you in every namespace

scriptor17:12:01

@sabbatical2017 it should be (Math/abs -123)

sabbatical201717:12:35

@scriptor @alexmiller Thank you! I've been blind to that Math.abs instead of Math/abs ... too much Python exposure 😃

drewverlee18:12:55

Does anyone have any good resources about modeling the abstractions of your system as a FSM or maybe something like state Charts? I’m curious about the idea if creating a more contractual model for designing systems and modeling them as a FSM seems like one way todo this. For example a simple game might have something like player-one -> player-two -> game-over It wouldn’t capture all the possible things that could happen to the program but would be a means to express the business domain.

noisesmith18:12:55

@drewverlee this might not be what you are looking for, but petri nets are a formal language for designing asynchronous stateful systems that has a well defined semantics where you can make proofs about eg. whether it halts or deadlocks - similar to fsm but with no shared clock

pcbalodi18:12:43

I think reduce-fsm does somehting simiar - https://github.com/cdorrat/reduce-fsm

drewverlee18:12:52

Wow lots of responses. I’ll take a look at these in a bit. Thanks!

jgh22:12:23

how would I take this object and turn it into something I can access the data of #object[clojure.core$future_call$reify__8097 0x3f7533a9 {:status :ready, :val 0}]?

admay22:12:45

(def x (future 10)) --> @x

admay22:12:57

That will return the thing that x is defined to be

admay22:12:33

You can also use the deref function explicitly, (deref x)

noisesmith22:12:29

you can also use realized? to see if it is done yet (otherwise deref will block until it is)

noisesmith22:12:53

you can also provide an optional extra arg to deref to tell it how long to pause and wait for it to complete before giving up

jgh22:12:08

it’s definitely been realized since this is part of a result on a blocking function call, im not creating the future

jgh22:12:21

and in the repl calling (realized?) returns true

noisesmith22:12:23

right, just mentioning that in general that is a concern

jgh22:12:41

im just not sure how to get at the data heh

noisesmith22:12:51

like admay said - @ aka deref

admay22:12:59

A common pattern with realized? is (if (realized? x) @x ( ... ))

jgh22:12:13

weird im just getting 0 when i do deref

jgh22:12:22

oh is the status part of the future itself

noisesmith22:12:25

there’s also future-cancel which will stop it (if it is sleeping or waiting on IO at least)

noisesmith22:12:48

but that’s a no-op if it’s realized iirc

jgh22:12:29

ah yeah so @ is doing the trick, i assumed :status was part of the return data for the future but i guess it’s an internal thing

jgh22:12:50

thanks guys 🙂

lee.justin.m22:12:07

I’m trying to figure out how go and <! behave in the context of the JS environment. The go documentation say that any visible calls to <!, >! and alt!/alts! channel operations within the body will block. First: what does “visible” mean? Second, does a function which uses a go block return immediately or does it only return once the go block finishes?

tbaldridge23:12:46

It returns a channel immediately.

tbaldridge23:12:00

No values will appear on the channel until the go completes.

tbaldridge23:12:50

@lee.justin.m The macro rewrites the body of the block into a state machine that with attach callbacks to the channels wherever theres a >! or <!

lee.justin.m23:12:29

okay so visible literally means textually visible because go is a macro

tbaldridge23:12:43

although the "visible" bit is a bit strange, since <! and >! can be inside a macro and go will still see them

tbaldridge23:12:49

but it doesn't dig into function calls.

lee.justin.m23:12:10

i see. so basically one needs to pass the channel around until ready to consume, then use it with a go block and the <! macro.

tbaldridge23:12:38

Well if you just want to consume, then use put! and take!

lee.justin.m23:12:58

oh. i was just following the readme from cljs-http

tbaldridge23:12:59

the go blocks run in a task queue in JS. When the browser isn't busy the pending go blocks will run

lee.justin.m23:12:30

i’m trying to map all this stuff to promises and/or callbacks and having a hard time of it

tbaldridge23:12:22

Well, it's a lot of the same sort of thing. Let's say you have (http-get url callback) in JS.

tbaldridge23:12:31

You could write this as (http-get url (fn [result] (reset! a result)))

tbaldridge23:12:56

Or we could put that value on a channel: (http-get url #(put! ch %))

tbaldridge23:12:45

but we may want to notify something when that put is done: (http-get url #(put! ch % done-cb))

tbaldridge23:12:05

So what core.async is doing is transforming calls to >! and <! into calls to (put! ch val done-cb) and take!

tbaldridge23:12:29

And a simplified level, when we have a callback we have to run later, core.async is calling (js/setTimeout .. cb) to queue it up

lee.justin.m23:12:58

I see. So ergonomically go blocks are more like promises or async/await and put!/take! are more like callbacks. The upshot seems to be that you need to use the right functions from core.async to interact with the channel (which of course makes total sense)

lee.justin.m23:12:21

I get that they are not implemented the same, but that’s very helpful.

tbaldridge23:12:01

actually you're right, async/await in JS are very much the same sort of transform