Fork me on GitHub
#clojure
<
2016-07-21
>
pataprogramming03:07:56

I broke down and worked out the integer arithmetic I was too lazy to write first time around. But I'm still interested in what the actual timeouts are for 4Clojure.

artemyarulin07:07:42

Hi, maybe somebody can help, I’ve noticed that clojure.test/is doesn’t fail the test if checks happens on another thread

(clojure.test/deftest b
     (let [o (promise)]
       (.start (Thread. (fn []
                          (clojure.test/is false)
                          (deliver o 1))))
       @o))

joost-diepenmaat08:07:18

@artemyarulin: AFAICT you can call clojure.test/is in another thread, but you must make sure that it gets called before the main thread’s test returns

artemyarulin08:07:55

hm, this test passes all the time for me

joost-diepenmaat08:07:12

looking at your example code I may be wrong. let me check

artemyarulin08:07:25

I’ve added @promise just to be sure that thread got finished. Yeah, thanks

artemyarulin08:07:59

Basically I was able to dig down till (clojure.test/inc-report-counter :fail) which gathers all the errors before creating a report. But I’m not that familiar with all those dynamic things,

joost-diepenmaat08:07:14

@artemyarulin: not sure why that works, but I have some test code running multiple threads (and go blocks) using clojure.core.async and then it’s all ok

joost-diepenmaat08:07:23

(clojure.test/deftest b (async/<!! (async/thread (Thread/sleep 3000) (clojure.test/is (= false true)))))

artemyarulin08:07:44

yeah, I’ve noticed that, I’ve never had a problem with core.async

joost-diepenmaat08:07:06

I think async/thread does some capturing/rebinding of the current dynamic environment

artemyarulin08:07:32

oh, may be. I’ll check the source, thanks for the hint

joost-diepenmaat08:07:02

that’s from core.async

artemyarulin08:07:29

Originally I’ve found an issue in https://github.com/funcool/promesa library where (p/then (p/promise 1) (fn [_] (is false)) never fails, but I guess it’s related to the way how it handles new threads creation.

artemyarulin08:07:46

Thank you @joost-diepenmaat , I’ll try to apply same things to promesa lib then

joost-diepenmaat08:07:05

There was a bunch of stuff added related to threads and dynamic bindings in clojure 1.3 - see clojure.core/future and clojure.core/binding-conveyor-fn

joost-diepenmaat08:07:08

before 1.3 all thread creation fns lost their dynamic bindings. AFAIK that doesn’t happen anymore unless you use the java Thread API directly.

artemyarulin08:07:32

yep, it could be the case

joost-diepenmaat08:07:03

I think that’s probably the cause of your problems. good luck 🙂

lmergen08:07:40

ok, what is the best way to process a file in chunks in Clojure, without resorting to http://java.io functionality ?

lmergen08:07:03

i'm thinking that, if i could someway expose it as a core.async channel, this will give me the best results

joost-diepenmaat09:07:40

@lmergen if it’s text, just use line-seq

lmergen09:07:54

it's binary (video) and doesn't fit in memory

joost-diepenmaat09:07:38

I would go with something similar to line-seq (so lazy seq of chunks) unless you’re dealing with pipes / other interfaces where the data is being added while processing

joost-diepenmaat09:07:36

reading that link, something like (convert .. (seq-of … ))

joost-diepenmaat09:07:56

to convert the input stream to a sequence of bytebuffers

joost-diepenmaat09:07:51

but that’s also dependent on if you need to extract information out of the file or just copy it around

rickmoynihan09:07:24

lmergen: you could probably transduce it

lmergen09:07:57

rickbeerendonk: wouldn't that still require me to convert the file in a seq of chunks ?

rickmoynihan09:07:22

lazy-seqs will work... but if you're streaming bytes and video you might not want to pay the costs of laziness

lmergen09:07:36

well i'm using manifold streams right now

rickmoynihan09:07:36

it's been a very long time since I looked at manifold

rickmoynihan09:07:57

do you need it to be async? You can probably just open a memory mapped file on it and reduce/transduce an xform over the bytebuffers or something like that

rickmoynihan09:07:10

You could look at iota which I believe does this kinda thing but for text files: https://github.com/thebusby/iota

cyppan09:07:34

Hi guys, wondering what's the cleaner approach to log things in a library supposed to be used in your projects (basically some ring handlers and middlewares) which already has its own logging approach. A Logger protocol...?

cyppan09:07:47

answering myself ^^ SLF4J would probably be the best option

serce10:07:36

Please vote for creating a docs here. I'm sure it will be popular http://stackoverflow.com/documentation/clojure/commit

cyppan11:07:08

Yes I went with clojure tools logging and slf4j api thanks

luxbock12:07:28

I have frequency map of ordered data, i.e. {0.25 1, 0.5 3, 0.65 2, 1.0 4}, and I'm trying to come up with an algorithm for choosing n keys such that they cover as much of the values as possible. For example in the above map, I'd get the scores for each key as follows:

(let [m {0.25 1, 0.5 3, 0.65 2, 1.0 4}
      cost (fn [x [k v]] (* (Math/abs (- x k)) v))]
  (into {}
    (for [[k] m]
      [k (apply + (map (partial cost k) m))])))
But I'm not sure how I would go about choosing the best n values. Has anyone done something similar before?

luxbock12:07:22

the frequency maps have from few to upto 100 items, which I can filter down to eliminate very unlikely candidates beforehand

mccraigmccraig12:07:41

@luxbock: you could sort the map entries by frequency descending and take the first n values ?

luxbock12:07:19

@mccraigmccraig: yeah I think that would work for my given example, but now that I look at the results it would give for the data I have, I think I need to tweak my cost function

luxbock12:07:39

I need to come up with some better examples of the problem

mccraigmccraig12:07:54

@luxbock: in general though, if there is a value function and you want to select the maximum value in n records, can you not sort the records by value descending and take the first n ?

luxbock12:07:20

Here's how an example set of data looks: http://i.imgur.com/uPIt07f.png Doing a simple sort on the costs of each key looks like this (shortened slightly for brevity):

([0.5 0.31638572425017447]
 [0.55 0.3359974424552431]
 [0.45 0.3967740060451058]
 [0.65 0.5147116949546616]
 [0.35000000000000003 0.6094803534061847]
 [0.7000000000000001 0.6496919321088118]
 [0.3 0.7581667054173449]
 [0.7500000000000001 0.7940304580330158]
 [1.0 1.550482445942804])
 
so choosing the next value in order would not help me in increasing the "coverage" as the next value would be very close to the first one.

luxbock12:07:05

intuitively for this data set I would choose 0.5, 0.3 and 1.0 for n of 3 values

mccraigmccraig12:07:36

ah, so are you wanting to divide your data into quantiles then, and pick the boundary values ?

luxbock12:07:46

yes, exactly

mccraigmccraig12:07:58

so sort by score, calculate the cumulative score for each record in the sorted order, then partition using boundary values of total-score / n ?

luxbock13:07:28

what are the boundary values in this case?

simonewebdesign13:07:10

Newbie here, can someone help me figure out what's wrong with my recursive function?

simonewebdesign13:07:32

I'm not getting any errors, just a failed test

mccraigmccraig13:07:31

@luxbock: the multiples of total-score / n... so if n is 4 then [(/ total-score 4) (* 2 (/ total-score 4) ) (* 3 (/ total-score 4))]

mccraigmccraig13:07:46

i.e. quartiles in that case

mccraigmccraig13:07:49

@simonewebdesign: what are you trying to achieve with that fn ?

simonewebdesign13:07:56

I have a nested data structure of items, I just want to collect the ones marked as "good"

mccraigmccraig13:07:24

@simonewebdesign: ah, i think you meant recur rather than reduce

mccraigmccraig13:07:38

but that's not a very idiomatic way to do it in clojure

mccraigmccraig13:07:50

(filter :good collection) will do it, or if you wanted just the ids then something like (->> collection (filter :good) (map :id))

mccraigmccraig13:07:45

oh, hold on... my bad 🙂 i didn't spot the children

mccraigmccraig13:07:06

recursion is reasonable

simonewebdesign13:07:00

I got a similar function working in JS, but in this one for some reason acc is always empty

Chris O’Donnell13:07:39

@simonewebdesign: Is there a reason you didn't define the-good-one outside of your good-ones definition?

mccraigmccraig13:07:51

ah, you aren't giving the reduce an initial accumulator

mccraigmccraig13:07:11

no, you are that

mccraigmccraig13:07:43

i'll shut up now... too many threads in my head

cyppan13:07:31

the problem is the acc value being always returned at the end of your the-good-one function

cyppan13:07:54

should be the else branch of the (if (seq (item :children))

cyppan13:07:37

(defn good-ones [collection]
  (defn the-good-one [acc item]
    (if (item :good)
      (cons (item :id) acc)
      (if (seq (item :children))
        (reduce the-good-one acc (item :children))
        acc)))
  (reduce the-good-one [] collection))

cyppan13:07:19

but I agree, the-good-one defined outside would be more idiomatic, or defined as a letfn or let

simonewebdesign13:07:51

works! I just got the ids in reversed order but that should be easy to fix 🙂

cyppan13:07:06

try conj instead of cons 😉

simonewebdesign13:07:33

thank you very much!

luxbock13:07:46

@mccraigmccraig: sorry I still have trouble following

luxbock13:07:56

I think I may need to brush on up basic statistics 🙂

mccraigmccraig13:07:16

@luxbock e.g. sort your scores by key order, e.g. [[0.1 3] [0.2 4] [0.3 5] [1.0 1]], then calculate cumulative scores [[0.1 3 3] [0.2 4 7] [0.3 5 12] [1.0 1 13]] then partition by fractions of the total score... e.g. for quartiles 13/4, 13/2, 39/4 giving you quartiles of [[[0.1 3 3]] [] [[0.2 4 7]] [[0.3 5 12][1.0 1 13]]]

mccraigmccraig13:07:33

that example highlights a problem though - quantiles may turn up empty

luxbock13:07:49

@mccraigmccraig: thanks, had to go eat but I understand your suggestion now, I'll need to try it on a few examples and see if it matches up with my intuition for what I'm attempting to do

mccraigmccraig14:07:03

ha, those aren't quantiles anyway, i'm not thinking straight... for quantiles you just want to divide the list into equal sized groups... so [[[0.1 3 3]] [[0.2 4 7]] [[0.3 5 12]] [[1.0 1 13]]]

tap14:07:02

Curry On Conf talk records are out How To Win Big With Old Ideas - David Nolen Detail: http://www.curry-on.org/2016/sessions/how-to-win-big-with-old-ideas.html Video: https://www.youtube.com/watch?v=JlgVSt-WWkA Doing data science with Clojure: the ugly, the sad, the joyful - Simon Belak Detail: http://www.curry-on.org/2016/sessions/doing-data-science-with-clojure.html Video: https://www.youtube.com/watch?v=6tZQqwil1c0 Distributed Configuration with Clojure - Renzo Borgatti Detail: http://www.curry-on.org/2016/sessions/distributed-configuration-with-clojure.html Video: https://www.youtube.com/watch?v=FKa24oHsHbA

luxbock14:07:27

@mccraigmccraig: I cheated a little bit and got the quantiles from a library function, but unfortunately those don't look quite like what I'm looking for either

Chris O’Donnell15:07:20

@simonewebdesign: Wrote my own nonrecursive solution using zippers for practice. It's below if you're curious.

(require '[clojure.zip :as z])
(defn good-ones [coll]
  (let [zipper (z/zipper :children :children (fn [node children] (assoc node :children children)) coll)]
    (loop [loc zipper goods []]
      (if (z/end? loc)
        goods
        (recur (z/next loc)
               (if-let [good (-> loc z/node :good)]
                 (conj goods good)
                 goods))))))

simonewebdesign15:07:29

looks like a pretty interesting solution, especially since I never used zippers

Chris O’Donnell15:07:43

They're great for traversing tree-like structures

timgilbert15:07:48

Hi all... Does anyone know of a good recursive map-merge function that I can just (require) from a library rather than typing in one of the seemingly hundreds of stack-overflow answers?

cyppan15:07:14

yes no more clojure.contrib.. I wish it to be implemented in https://github.com/plumatic/plumbing for example, I think it's one of the most used utility lib

cyppan15:07:17

looks like Rich has rejected the contrib http://dev.clojure.org/jira/browse/CLJ-1468

ccann16:07:58

This is a super biased place to ask this question but here goes: I have the opportunity to pitch clojure as our primary server-side language for data intensive tasks (recommendation engine, data transformations, ETLs) — I have a couple years of experience writing Clojure full time but my colleague only knows Ruby and Javascript and has never seen FP or lisps. That said, I’m willing to be a teacher and he seems willing to learn. I could meet him in the middle with something like Python, with which I also have considerable experience, but I’m intrigued by Go as a dark horse candidate (maybe unwisely) with familiar imperative syntax and a nice growth curve.

ccann16:07:29

Any seasoned engineers out there who have gone through a similar process with advice to give?

cyppan16:07:51

we don't have super big data needs at my company but I've found Clojure to be pretty nice for our analytics pipeline. Shines especially at ETL stuff, but we keep avoiding the need for distributed computation, for which something like Scala+Spark would be probably a good match. I've heard good things about Onyx too.

ccann16:07:20

I’ve used clojure for an an analytics pipeline before and I found it to be a good fit too

cyppan16:07:43

Go could be indeed a good choice given its fastness and its consistent primitives, easy to learn and write, more and more companies bet on it. But this syntax... 😛 personally I just can't.

jjfine17:07:10

I find myself doing this a bunch in clojure.test and I'm wondering if there's a better way: (with-redefs [mocked-fn #(when (= :expected-arg %) :expected-return-value) ...

candiedcode19:07:46

has anyone done docker-compose and clojure?

candiedcode19:07:31

so in my docker-compose file

candiedcode19:07:28

i’m expecting when i run docker-compose up, that the image would run and i would have a repl connected

candiedcode19:07:56

and what i see is

candiedcode19:07:43

so the command runs successfully, but then exits,

candiedcode19:07:03

if i run that image that was created, i can interact with the repl with no problem

abnercarleto19:07:13

docker-compose -f your-file.yml -d

abnercarleto19:07:16

Do you want to run lein repl in docker?

candiedcode19:07:43

i mean i can if i connect to the docker image that gets created via docker-compose

abnercarleto19:07:42

docker runs a command and exit

abnercarleto19:07:51

do you need execute in interactive mode

abnercarleto19:07:09

try docker-compose run lein repl

candiedcode19:07:39

docker-compose run lein repl ERROR: No such service: lein docker-compose run "lein repl" ERROR: No such service: lein repl docker-compose run [lein repl] ERROR: No such service: [lein

candiedcode19:07:32

thanks for the help, i can make it run, i just thought my understanding in docker-compose that the image would run and not exit immediately

ddellacosta19:07:55

we’re pretty off-topic at this point but you’ll still need docker to connect to a container you’ve started via docker-compose, regardless

candiedcode19:07:16

okay back to all things clojure

ddellacosta19:07:18

and if you’re simply exposing a port, then you don’t need to do that via docker

cezar23:07:03

wonder if I could get some help here. I'm trying to do something really simple. I need to find matching ranges in an interval tree. I found this library

(require '[org.dthume.data.interval-treeset :as it])
which looks like it may be able to do what I need but the documentation for it is... let's just say not the greatest. So here is the issue. Given this interval tree:
(def cts (into (it/interval-treeset :as-interval :span)
               [{:span [6 9] :id 4} {:span [6 8] :id 3} {:span [3 5] :id 2} {:span [0 2] :id 1}]))
all I want is to be able to get something like:
(it/gimme-thevalues cts 7)
=> {:span [6 9], :id 4} {:span [6 8], :id 3}
but for the life of me I can't figure out how to get that from this scant documentation: http://dthume.github.io/data.interval-tree/codox/org.dthume.data.interval-treeset.html

cezar23:07:11

the closest call I found that worked for just one the first interval was this:

(it/first-overlapping cts [7 7])
=> {:span [6 9], :id 4}

candiedcode23:07:29

@cezar: what about select-overlapping

candiedcode23:07:57

(second (it/select-overlapping cts [7 7]))

cezar23:07:57

that seems to return something I don't understand

candiedcode23:07:12

[prefix selected suffix]

cezar23:07:35

oh, I see

cezar23:07:53

this ain't going to be very fast though as it returns the entire tree back to me 😞

dadair23:07:55

Hello! Quick question: I have a seq of strings that I want to randomly separate into 4 nearly-equal sized groups. I’ve looked at split, group, partition, and their variants, and not quite sure how to achieve that, as I don’t really have a clear-cut predicate. Can anyone point me in the right direction? Thanks!

Alex Miller (Clojure team)23:07:05

Partition-all by count divided by 4?

Alex Miller (Clojure team)23:07:30

Shuffle first if you want random

dadair23:07:56

Will look into that, thanks!