I broke down and worked out the integer arithmetic I was too lazy to write first time around. But I'm still interested in what the actual timeouts are for 4Clojure.

Hi, maybe somebody can help, I’ve noticed that clojure.test/is doesn’t fail the test if checks happens on another thread

(clojure.test/deftest b
     (let [o (promise)]
       (.start (Thread. (fn []
                          (clojure.test/is false)
                          (deliver o 1))))
       @o))

@artemyarulin: AFAICT you can call clojure.test/is in another thread, but you must make sure that it gets called before the main thread’s test returns

hmmm

hm, this test passes all the time for me

looking at your example code I may be wrong. let me check

I’ve added @promise just to be sure that thread got finished. Yeah, thanks

Basically I was able to dig down till (clojure.test/inc-report-counter :fail) which gathers all the errors before creating a report. But I’m not that familiar with all those dynamic things,

@artemyarulin: not sure why that works, but I have some test code running multiple threads (and go blocks) using clojure.core.async and then it’s all ok

(clojure.test/deftest b (async/<!! (async/thread (Thread/sleep 3000) (clojure.test/is (= false true)))))

yeah, I’ve noticed that, I’ve never had a problem with core.async

I think async/thread does some capturing/rebinding of the current dynamic environment

oh, may be. I’ll check the source, thanks for the hint

that’s from core.async

Originally I’ve found an issue in https://github.com/funcool/promesa library where (p/then (p/promise 1) (fn [_] (is false)) never fails, but I guess it’s related to the way how it handles new threads creation.

Thank you @joost-diepenmaat , I’ll try to apply same things to promesa lib then

There was a bunch of stuff added related to threads and dynamic bindings in clojure 1.3 - see clojure.core/future and clojure.core/binding-conveyor-fn

before 1.3 all thread creation fns lost their dynamic bindings. AFAIK that doesn’t happen anymore unless you use the java Thread API directly.

yep, it could be the case

I think that’s probably the cause of your problems. good luck 🙂

thank you!

ok, what is the best way to process a file in chunks in Clojure, without resorting to http://java.io functionality ?

i'm thinking that, if i could someway expose it as a core.async channel, this will give me the best results

https://github.com/ztellman/byte-streams also looks interesting, though

@lmergen if it’s text, just use line-seq

it's binary (video) and doesn't fit in memory

I would go with something similar to line-seq (so lazy seq of chunks) unless you’re dealing with pipes / other interfaces where the data is being added while processing

reading that link, something like (convert .. (seq-of … ))

to convert the input stream to a sequence of bytebuffers

but that’s also dependent on if you need to extract information out of the file or just copy it around

lmergen: you could probably transduce it

rickbeerendonk: wouldn't that still require me to convert the file in a seq of chunks ?

lazy-seqs will work... but if you're streaming bytes and video you might not want to pay the costs of laziness

well i'm using manifold streams right now

it's been a very long time since I looked at manifold

do you need it to be async? You can probably just open a memory mapped file on it and reduce/transduce an xform over the bytebuffers or something like that

You could look at iota which I believe does this kinda thing but for text files: https://github.com/thebusby/iota

Hi guys, wondering what's the cleaner approach to log things in a library supposed to be used in your projects (basically some ring handlers and middlewares) which already has its own logging approach. A Logger protocol...?

answering myself ^^ SLF4J would probably be the best option

Please vote for creating a docs here. I'm sure it will be popular http://stackoverflow.com/documentation/clojure/commit

cyppan: timbre and tools.logging have slf4j backends: https://github.com/clojure/tools.logging https://github.com/ptaoussanis/timbre

Yes I went with clojure tools logging and slf4j api thanks

I have frequency map of ordered data, i.e. {0.25 1, 0.5 3, 0.65 2, 1.0 4}, and I'm trying to come up with an algorithm for choosing n keys such that they cover as much of the values as possible. For example in the above map, I'd get the scores for each key as follows:

(let [m {0.25 1, 0.5 3, 0.65 2, 1.0 4}
      cost (fn [x [k v]] (* (Math/abs (- x k)) v))]
  (into {}
    (for [[k] m]
      [k (apply + (map (partial cost k) m))])))

the frequency maps have from few to upto 100 items, which I can filter down to eliminate very unlikely candidates beforehand

@luxbock: you could sort the map entries by frequency descending and take the first n values ?

@mccraigmccraig: yeah I think that would work for my given example, but now that I look at the results it would give for the data I have, I think I need to tweak my cost function

I need to come up with some better examples of the problem

@luxbock: in general though, if there is a value function and you want to select the maximum value in n records, can you not sort the records by value descending and take the first n ?

Here's how an example set of data looks: http://i.imgur.com/uPIt07f.png Doing a simple sort on the costs of each key looks like this (shortened slightly for brevity):

([0.5 0.31638572425017447]
 [0.55 0.3359974424552431]
 [0.45 0.3967740060451058]
 [0.65 0.5147116949546616]
 [0.35000000000000003 0.6094803534061847]
 [0.7000000000000001 0.6496919321088118]
 [0.3 0.7581667054173449]
 [0.7500000000000001 0.7940304580330158]
 [1.0 1.550482445942804])
 

intuitively for this data set I would choose 0.5, 0.3 and 1.0 for n of 3 values

ah, so are you wanting to divide your data into quantiles then, and pick the boundary values ?

yes, exactly

so sort by score, calculate the cumulative score for each record in the sorted order, then partition using boundary values of total-score / n ?

what are the boundary values in this case?

Newbie here, can someone help me figure out what's wrong with my recursive function?

I'm not getting any errors, just a failed test

@luxbock: the multiples of total-score / n... so if n is 4 then [(/ total-score 4) (* 2 (/ total-score 4) ) (* 3 (/ total-score 4))]

i.e. quartiles in that case

@simonewebdesign: what are you trying to achieve with that fn ?

I have a nested data structure of items, I just want to collect the ones marked as "good"

@simonewebdesign: ah, i think you meant recur rather than reduce

but that's not a very idiomatic way to do it in clojure

(filter :good collection) will do it, or if you wanted just the ids then something like (->> collection (filter :good) (map :id))

oh, hold on... my bad 🙂 i didn't spot the children

recursion is reasonable

doh!

I got a similar function working in JS, but in this one for some reason acc is always empty

@simonewebdesign: Is there a reason you didn't define the-good-one outside of your good-ones definition?

ah, you aren't giving the reduce an initial accumulator

no, you are that

@codonnell: not really

i'll shut up now... too many threads in my head

the problem is the acc value being always returned at the end of your the-good-one function

should be the else branch of the (if (seq (item :children))

(defn good-ones [collection]
  (defn the-good-one [acc item]
    (if (item :good)
      (cons (item :id) acc)
      (if (seq (item :children))
        (reduce the-good-one acc (item :children))
        acc)))
  (reduce the-good-one [] collection))

but I agree, the-good-one defined outside would be more idiomatic, or defined as a letfn or let

works! I just got the ids in reversed order but that should be easy to fix 🙂

try conj instead of cons 😉

thank you very much!

@mccraigmccraig: sorry I still have trouble following

I think I may need to brush on up basic statistics 🙂

@luxbock e.g. sort your scores by key order, e.g. [[0.1 3] [0.2 4] [0.3 5] [1.0 1]], then calculate cumulative scores [[0.1 3 3] [0.2 4 7] [0.3 5 12] [1.0 1 13]] then partition by fractions of the total score... e.g. for quartiles 13/4, 13/2, 39/4 giving you quartiles of [[[0.1 3 3]] [] [[0.2 4 7]] [[0.3 5 12][1.0 1 13]]]

that example highlights a problem though - quantiles may turn up empty

@mccraigmccraig: thanks, had to go eat but I understand your suggestion now, I'll need to try it on a few examples and see if it matches up with my intuition for what I'm attempting to do

ha, those aren't quantiles anyway, i'm not thinking straight... for quantiles you just want to divide the list into equal sized groups... so [[[0.1 3 3]] [[0.2 4 7]] [[0.3 5 12]] [[1.0 1 13]]]

Curry On Conf talk records are out How To Win Big With Old Ideas - David Nolen Detail: http://www.curry-on.org/2016/sessions/how-to-win-big-with-old-ideas.html Video: https://www.youtube.com/watch?v=JlgVSt-WWkA Doing data science with Clojure: the ugly, the sad, the joyful - Simon Belak Detail: http://www.curry-on.org/2016/sessions/doing-data-science-with-clojure.html Video: https://www.youtube.com/watch?v=6tZQqwil1c0 Distributed Configuration with Clojure - Renzo Borgatti Detail: http://www.curry-on.org/2016/sessions/distributed-configuration-with-clojure.html Video: https://www.youtube.com/watch?v=FKa24oHsHbA

@mccraigmccraig: I cheated a little bit and got the quantiles from a library function, but unfortunately those don't look quite like what I'm looking for either

@simonewebdesign: Wrote my own nonrecursive solution using zippers for practice. It's below if you're curious.

(require '[clojure.zip :as z])
(defn good-ones [coll]
  (let [zipper (z/zipper :children :children (fn [node children] (assoc node :children children)) coll)]
    (loop [loc zipper goods []]
      (if (z/end? loc)
        goods
        (recur (z/next loc)
               (if-let [good (-> loc z/node :good)]
                 (conj goods good)
                 goods))))))

looks like a pretty interesting solution, especially since I never used zippers

They're great for traversing tree-like structures

Hi all... Does anyone know of a good recursive map-merge function that I can just (require) from a library rather than typing in one of the seemingly hundreds of stack-overflow answers?

yes no more clojure.contrib.. I wish it to be implemented in https://github.com/plumatic/plumbing for example, I think it's one of the most used utility lib

@timgilbert: https://github.com/puppetlabs/clj-kitchensink/blob/cfea4a16e4d2e15a2d391131a163b4eeb60d872e/src/puppetlabs/kitchensink/core.clj#L311-L332 or https://github.com/flatland/useful/blob/48e517a1514ecfed769d17ea9f27041c6db6bdb3/src/flatland/useful/map.clj#L148-L155, though neither seem to be actively developed at the moment

looks like Rich has rejected the contrib http://dev.clojure.org/jira/browse/CLJ-1468

This is a super biased place to ask this question but here goes: I have the opportunity to pitch clojure as our primary server-side language for data intensive tasks (recommendation engine, data transformations, ETLs) — I have a couple years of experience writing Clojure full time but my colleague only knows Ruby and Javascript and has never seen FP or lisps. That said, I’m willing to be a teacher and he seems willing to learn. I could meet him in the middle with something like Python, with which I also have considerable experience, but I’m intrigued by Go as a dark horse candidate (maybe unwisely) with familiar imperative syntax and a nice growth curve.

Any seasoned engineers out there who have gone through a similar process with advice to give?

we don't have super big data needs at my company but I've found Clojure to be pretty nice for our analytics pipeline. Shines especially at ETL stuff, but we keep avoiding the need for distributed computation, for which something like Scala+Spark would be probably a good match. I've heard good things about Onyx too.

I’ve used clojure for an an analytics pipeline before and I found it to be a good fit too

Go could be indeed a good choice given its fastness and its consistent primitives, easy to learn and write, more and more companies bet on it. But this syntax... 😛 personally I just can't.

I find myself doing this a bunch in clojure.test and I'm wondering if there's a better way: (with-redefs [mocked-fn #(when (= :expected-arg %) :expected-return-value) ...

@jjfine: take a look at clj-fakes: https://github.com/metametadata/clj-fakes#patch-function-with-stub

has anyone done docker-compose and clojure?

I have

so in my docker-compose file

i’m expecting when i run docker-compose up, that the image would run and i would have a repl connected

and what i see is

so the command runs successfully, but then exits,

if i run that image that was created, i can interact with the repl with no problem

try this

docker-compose -f your-file.yml -d

Do you want to run lein repl in docker?

i mean i can if i connect to the docker image that gets created via docker-compose

docker runs a command and exit

do you need execute in interactive mode

try docker-compose run lein repl

docker-compose run lein repl ERROR: No such service: lein docker-compose run "lein repl" ERROR: No such service: lein repl docker-compose run [lein repl] ERROR: No such service: [lein

thanks for the help, i can make it run, i just thought my understanding in docker-compose that the image would run and not exit immediately

we’re pretty off-topic at this point but you’ll still need docker to connect to a container you’ve started via docker-compose, regardless

okay back to all things clojure

and if you’re simply exposing a port, then you don’t need to do that via docker

wonder if I could get some help here. I'm trying to do something really simple. I need to find matching ranges in an interval tree. I found this library

(require '[org.dthume.data.interval-treeset :as it])

(def cts (into (it/interval-treeset :as-interval :span)
               [{:span [6 9] :id 4} {:span [6 8] :id 3} {:span [3 5] :id 2} {:span [0 2] :id 1}]))

(it/gimme-thevalues cts 7)
=> {:span [6 9], :id 4} {:span [6 8], :id 3}

the closest call I found that worked for just one the first interval was this:

(it/first-overlapping cts [7 7])
=> {:span [6 9], :id 4}

@cezar: what about select-overlapping

(second (it/select-overlapping cts [7 7]))

that seems to return something I don't understand

[prefix selected suffix]

oh, I see

this ain't going to be very fast though as it returns the entire tree back to me 😞

Hello! Quick question: I have a seq of strings that I want to randomly separate into 4 nearly-equal sized groups. I’ve looked at split, group, partition, and their variants, and not quite sure how to achieve that, as I don’t really have a clear-cut predicate. Can anyone point me in the right direction? Thanks!

Partition-all by count divided by 4?

Shuffle first if you want random

Will look into that, thanks!