Fork me on GitHub
#clojure
<
2017-02-28
>
bbloom00:02:27

i stumbled across some code that uses arity overloading on fn without defn… i forgot that was even possible - does anybody actually make consistent use of that? i’d be curious to hear about it

trptcolin02:02:44

bbloom: i feel like i used to use that to golf on 4clojure

bbloom02:02:02

heh, super useful 😛

bronsa00:02:34

@bbloom transducer impls

bbloom00:02:54

ah yeah, i guess so

qqq00:02:04

I found that made transducers unnecessairly difficult to understand

qqq00:02:19

thigns like (f) = init value, (f x) = finalize it, (f r x) = reduce it

qqq00:02:30

argh, give me 3 functions with intuitive names pls 🙂

bbloom00:02:30

i’m still not quite sure why transducers aren’t a protocol that is reified

bronsa00:02:46

@bbloom i think because of comp

bbloom00:02:05

i also don’t understand why it was important comp worked 🙂

bronsa00:02:11

me neither

bbloom00:02:27

the haskell lens lib does something similar

bbloom00:02:41

they so desperately wanted the . operator to compose lenses, that it’s really difficult to tell wtf is happening

bbloom00:02:05

i realize that it’s too late to make clojure.core/comp defined on a protocol, but coulda just called it pipe or something

bbloom00:02:35

but thanks for the thoughts 🙂 appreciated b/c i was totally failing to recall those things

hiredman00:02:57

operator overloading is overrated

bbloom01:02:26

well it’s not so much overloading as it is bending the abstraction to match the existing signature so that overloaded behavior can be accomplished without overloaded dispatch

tbaldridge01:02:28

It also has to do in part with the way conj (and + ) works

joshjones01:02:53

@kgofhedgehogs in an effort to use core libs at all cost 😉 , presenting another solution for your "fill with zeroes":

(defn zero-vec
  [v]
  (vec (for [e v]
         (if (vector? e)
           (zero-vec e)
           0))))

hiredman01:02:35

conj was, if I recall, updated to work that way after transducers

hiredman01:02:58

or "with transducers"

tbaldridge01:02:41

right, but + has always worked that way: (+) => 0, (+ 1) => 1, (+ 1 2) => 3

tbaldridge01:02:26

because of that, reduce was "updated" (renamed transduce) to call (x) and (x acc), and the rest kindof fell out of that.

tbaldridge01:02:37

I agree, I'd prefer protocols, but from what I can tell, that was the development path

bbloom01:02:03

i’m not sure i follow - can you pass a transducer “xform” as the “f” parameter to transduce?

bbloom01:02:23

the arity overloading of the xform arg is unrelated to the arity overloading to the f arg, or am i mistaken about that?

hiredman01:02:15

the xform arg doesn't overload arity

hiredman01:02:21

if I recall

bbloom01:02:27

well internally

tbaldridge01:02:33

@bbloom the first thing transduce does is (xform rf)

tbaldridge01:02:39

and then uses that as the rf in a reduce

hiredman01:02:47

the function returned by xform and the 'f' need to match, because xform is X -> X and its argument is 'f'

bbloom01:02:34

i guess that makes sense

bbloom01:02:52

still, i find the whole set of machinations to be deeply confusing

bbloom01:02:56

and extremely error prone

bbloom01:02:04

i’ve tried to write some transducers and have failed badly

bbloom01:02:09

and the failure modes are super surprising

bbloom01:02:22

using existing transducers is considerably easier, but not without similar failure modes

hiredman01:02:27

well, the lack of reification means the parts don't come with ready made labels, so it is hard to talk about them

bbloom01:02:38

yeah - names: useful 🙂

bbloom01:02:54

somebody should make a language with really useful associative data structures 😉

tbaldridge01:02:57

it's a bit of a learning curve....I didn't understand them until I got a 3hr brain-dump from Rich on them.

bbloom01:02:12

lol, that seems unfair

hiredman01:02:30

"hey, I heard about transducers, they seem cool; but I can't find them, just a bunch of function composition"

bbloom01:02:03

@tbaldridge did you reify a protocol in pixie? i seem to recall you did “transducers at the bottom"

pandeiro02:02:32

how can I silence calls to clojure.tools.logging/info made by a library I'm using?

seancorfield03:02:08

@pandeiro If you’re using log4j you can change the reporting threshold for the namespaces in that library via log4j.properties.

pandeiro03:02:07

thanks @seancorfield - hadn't included a logging lib dependency but i will try that route

seancorfield03:02:32

I can’t remember what tools.logging uses by default but I suspect you can control that too.

olfal03:02:56

if anyone here is a #component master, any help would be much appreciated 🙂

qqq04:02:59

in datomic, how does :in $ ?title differ from :in $ [?title ...] ?

tbaldridge06:02:28

@qqq in the second example ?title is a collection of values that form a logical "or"

qqq06:02:16

@tbaldridge: thanks; realized that in a later part of the datalog query langauge

qqq06:02:24

in datascript/datomic, in a

(d/q '[:find .... :in .... :where .... ] db)
is there a way to run a "project" over the results? or do I have to call map after getting the queries

chrisetheridge06:02:43

in terms of performance, how “negative” is using partial over #( … ) ?

chrisetheridge06:02:50

or is the difference neglibable ?

mpenet07:02:31

depends, with partial as soon as you go over 3 args it calls apply, othewise it's just as fast if not more

mpenet07:02:19

also #() expands to fn* without doing any destructuring & more etc

mpenet07:02:40

but in practice they are almost identical, it shouldn't really matter

chrisetheridge07:02:56

so its down to personal preference, really?

genmeblog09:02:50

Having this code

(def n (rand))
(defn calc [v] (* v v))

(def do-iter (atom true))

(defn my-loop []
  (when @do-iter
    (Thread/sleep 1000)
    (println (str "calc: " (calc n)))
    (recur)))

(def f (future (my-loop)))

genmeblog09:02:24

I can redefine n and calc to see changed result immediately

genmeblog09:02:46

but I can't redefine my-loop

genmeblog09:02:54

Is it because it's run in different thread? If not, why?

genmeblog09:02:07

(I know I can put this function in atom and reset atom)

mpenet09:02:07

you could have all the vars n/calc/ do-iter as local bindings of the loop

mpenet09:02:24

and recur (or not) with whatever new value you need

henriklundahl09:02:26

@tsulej, my-loop is "redefined", but the new function is not used by the thread currently running (calling the function previously bound to my-loop). You can extract the meat out of my-loop to a new var e.g. my-fn and call that from my-loop.

rauh09:02:03

@tsulej Use (my-loop) instead of (recur), though be aware that this will grow your stack

genmeblog10:02:02

@mpenet the idea is to enable live coding, I want to rebind/change values externally

genmeblog10:02:48

@henriklundahl right, good advice, thank you

genmeblog10:02:51

@rauh I believe it's opposite. (recur) in tail position is optimized by Clojure ("Note that recur is the only non-stack-consuming looping construct in Clojure.")

rauh10:02:54

?? That's what I meant with the above statement.

rauh10:02:35

If you use my-loop instead of recur, you can change the behaviour during "runtime" but you'll be growing the stack every second.

genmeblog10:02:55

yeah... sorry, this is when you read in rush

rauh10:02:49

@tsulej Happens 🙂 . If you want to avoid that (ie growing your stack) you can pack the future inside of my-loop function.

genmeblog10:02:48

I considered this solution too, but my function is called 25 times per second and runing every call new thread is too much overhead (I suppose)

genmeblog10:02:59

so I ended with an atom

rauh10:02:49

I expected this code to be dev-only, no?

rauh10:02:58

Also, future doesn't create a new thread

genmeblog10:02:23

but solution gave by @henriklundahl is enough for my case

rauh10:02:26

You can create thousands of futures, for cheap. They're backed by the agent threadpool

genmeblog10:02:42

doesn't? ah... right

genmeblog10:02:56

I'll check it too

genmeblog10:02:07

and yes, solution is kind of dev-only in this case

rnandan27310:02:49

anybody from china? wanted a small test to be done

rnandan27310:02:19

if the following is accessible http://www.materiall.com. FYI this is a full stack clojure app

rnandan27310:02:00

pls ignore, i got reply for my query

compro11:02:59

I think I found an outdated info on a clojure page

compro11:02:18

Where and how should I report it?

jcf12:02:51

@compro https://github.com/clojuredocs is where the repos live I think. I don't think it's an official Clojure project out of Cognitect, but one of the Cognitect team is in the GitHub org.

compro12:02:31

@jcf thanks for your help. Was trying to learn clojure.

jcf12:02:15

You're welcome! Happy to help out with any questions you may have. 😉

compro12:02:11

Was following the page and now I am stuck at lein test which doesn't seem to run the tests successfully and prints a long output.

jcf12:02:20

java.lang.Exception: No namespace: command-line-args.core likely means you've got a file missing, named incorrectly, or haven't required it.

moxaj12:02:30

check your directory names

moxaj12:02:34

should use underscores

jcf12:02:50

What does the namespace in command_line_args/core_test.clj look like?

not-raspberry12:02:17

If that fails, paste the results of find . -name '*.clj' in the project.

compro12:02:58

./project.clj
./src/command_line_args/core.clj
./test/command_line_args/core.clj
./test/command_line_args/core_test.clj

jcf12:02:34

You need to remove ./test/command_line_args/core.clj.

jcf12:02:59

Files in both src and test will be considered by Clojure/Leiningen when you require something. The src and test bits are left off resulting in two files that could be loaded when you require command-line-args.core.

compro12:02:17

Got it. But now having new issues.

lein test command-line-args.core-test

lein test :only command-line-args.core-test/pairs-of-values

FAIL in (pairs-of-values) (core_test.clj:9)
expected: (not (= {:server "localhost", :port "8080", :environment "production"} (parse-args args)))
  actual: (not (not true))

Ran 1 tests containing 1 assertions.
1 failures, 0 errors.
Tests failed.
./src/command_line_args/core.clj:
(ns command-line-args.core)

(defn parse-args [args]
  (into {} (map (fn [[k v]] [(keyword (.replace k "--" "")) v])
                (partition 2 args))))
./test/command_line_args/core_test.clj:
(ns command-line-args.core-test
  (:require [clojure.test :refer :all]
            [command-line-args.core :refer :all]))

(deftest pairs-of-values
  (let [args ["--server" "localhost"
              "--port" "8080"
              "--environment" "production"]]
    (is (not (= {:server "localhost"
            :port "8080"
            :environment "production"}
           (parse-args args))))))

jcf12:02:39

@compro fixing the indentation and using clojure.core/not= makes it a bit easier to see what the test is trying to do:

(deftest pairs-of-values
  (let [args ["--server" "localhost"
              "--port" "8080"
              "--environment" "production"]]
    (is (not= {:server "localhost"
               :port "8080"
               :environment "production"}
              (parse-args args)))))
Why are you testing that the value is not equal? Don't you want it to be equal?

compro12:02:13

@jcf it worked. Need to read a bit more.

Matt Butler13:02:11

Hi, I'm looking for a way to parallelise I/O while consuming a lazy-seq. My current implementation looks like this

(doseq [element lazy-seq]
  (some-io element))
Is this a simple way to parellelise this operation without realising the entire seq, producing a seq (throw any return values away) and is safe to perform side effects in? Simply put a parallelised doseq Is core.async a good option for this?

joshjones13:02:22

@mbutler no, core.async has nothing to do with parallelism, rather, it is concerned with making asynchronous events appear synchronous. When you say “without realizing the entire seq” — what determines, in your case, how much of the sequence you want to realize?

Matt Butler13:02:36

I guess i would want to realise each element as it is consumed

Matt Butler13:02:32

and produce nothing as a result, similar to the behaviour of doseq. It can be slightly eager, each element it not large but the total sequence may be millions in length

joshjones13:02:43

you seem to be wanting to make a sequential decision (as it’s consumed), while executing things in parallel ...

joshjones13:02:28

it sounds like pmap may be a good candidate. it is semi-lazy

joshjones13:02:57

(pmap #(do-some-work %) lazee-seq)

Matt Butler13:02:38

Yes i had considered pmap but unfortunately it builds a seq out of the return values of each iteration and returns it

Matt Butler13:02:53

leaving me with a giant seq in memory, which i was hoping to avoid

Matt Butler13:02:31

I dont know how much memory a million element seq of nils uses

Matt Butler13:02:33

or if its a problem

joshjones13:02:20

ok, then consider a java.util.concurrent executor service:

(def lazee-seq (range 8))

(def tp (Executors/newFixedThreadPool 8))

(defn do-some-work [n]
  (Thread/sleep 1000)
  (println "Done with task " n))

(run! (fn [element]
        (.submit tp #(do-some-work element)))
      lazee-seq)

Matt Butler13:02:52

That looks pretty good 🙂

dm313:02:35

there’s no backpressure here though

dm313:02:00

not sure if the default executor queues are unbounded

dm313:02:39

they are

dm313:02:22

so you might accumulate all those closures in the executor’s queue if the lazee-seq evaluates much faster than the processing

Matt Butler14:02:34

Is this a problem because there is a limit to the size of the queue? @dm3

Matt Butler14:02:18

The I/O work (http request) is what takes the majority of the time currently which is why i was hoping to parellelise it 🙂

dm314:02:40

you will parallelise it, you just need to think about memory usage

dm314:02:21

using core.async or Manifold is not a bad idea

dm314:02:28

as these tools make backpressure explicit

dm314:02:40

unlike lazy seqs

dm314:02:26

(I’m not a big lazy seq producer, maybe there’s a way to do that properly with them too :))

mpenet14:02:40

I do that quite often with core.async, an input (chan) + a dotimes over N go blocks for parallelism that take on the main chan from these blocks, and an ouput chan for results it's kind of a "light" version of core.async/pipeline(-async). You get backpressure, bounded buffer, low resource usage etc etc

mpenet14:02:51

the source of pipeline & co in core.async is a good example of this pattern I think

Matt Butler14:02:08

@mpenet that sounds like roughly what I imagined

mpenet14:02:36

the bookeeping when disposing of the whole pipeline is important tho, be aware of that

mpenet14:02:48

like closing chans with pending takes and so on (or not continuing to send jobs on closed chans)

Matt Butler14:02:33

I will give that a read with eager eyes 🙂

Matt Butler14:02:36

Thanks a bunch

tap14:02:18

I think pipeline-blocking is a perfect match for this. https://clojuredocs.org/clojure.core.async/pipeline-blocking. Put element from doseq to from channel and use slidding-buffer or dropping-buffer as to channel Something like

(pipeline-blocking 10 (chan (dropping-buffer 1)) (map some-io) input-chan)

Matt Butler14:02:24

@tap awesome, ill give this a go too, thanks :thumbsup:

joshjones14:02:30

@mbutler was playing around a little more with this — while core.async gives you the backpressure support you need, it also adds a lot of overhead for what you’re trying to do, which is just limit the producer. Just dug this up I used a month ago or so for the same purpose, shown as an example here:

(let [sema (Semaphore. 50)]
  (doseq [e (range 100)]
    (.acquire sema)
    (println "permits remaining: " (.availablePermits sema))
    (.submit tp #(do
                   (do-some-work e)
                   (.release sema)))))

dominicm14:02:10

@tap wouldn't a sliding buffer cause a slow down upstream to make entries start dropping?

tap14:02:50

dominicm: yeah, you’re right. The operation in sliding buffer is unnecessary. Dropping buffer is better

joshjones14:02:33

this lets you still use an unbounded queue in the executor if you want, but limits the number of items in the queue to 50, as it will wait to acquire the semaphore before submitting the task. then, in the task itself, release the semaphore. works great I think for what you need, lets you still doseq over the elements, and is far simpler

joshjones14:02:17

you can use a cookie cutter (def tp (Executors/newFixedThreadPool 8))

Matt Butler14:02:56

That does seem nice and simple, really appreciate you playing around, ill give that a go 🙂 :thumbsup:

tolitius14:02:59

@mbut: when working with core.async + real back pressure don't forget buffered channels have limit after which it'll start throw on puts: https://github.com/clojure/core.async/blob/master/src/main/clojure/clojure/core/async/impl/channels.clj#L152-L156

joshjones14:02:16

no worries — dealing with j.u.c primitives will turn off some who want as little java interop as possible, but j.u.c has an amazing set of easy to use tools that are faster and more flexible than what clojure offers in its core library

tolitius14:02:47

by "real" back pressure I mean data that is consumed from other systems (i.e. message queue, topic..)

joshjones14:02:32

@tolitius yes but he would likely be >!! and >!, parking and blocking, instead of async put!ting, I would imagine … should not run into that, if so, an error in construction

tolitius14:02:32

core.async won't help you with the real network back pressure which is built in in TCP/IP..

dominicm14:02:20

@joshjones I think people fear the j.u.c primitives as they're not blessed as "safe" necessarily. I know I fear using anything except the occasional atom.

joshjones14:02:40

@dominicm how are they not “safe” ?

dominicm15:02:25

@joshjones Most concurrency primitives (in life, not in j.u.c necessarily here) I've come to consider as tools with many trade-offs. Locks are one such thing, fine if always used in X way. If those trade-offs aren't made extremely clear, then they tend to be used incorrectly.

joshjones15:02:47

well, clojure implements most/all of its safe concurrency features right on top of j.u.c … for example, clojure.lang.Atom has a single field, state, which is a java.util.concurrent.atomic.AtomicReference

tolitius15:02:31

@joshjones: perhaps. I just find it quite often to be a misconception when talking about core.async and back pressure. network is a really important part in things we build, but since "back pressure" term is thrown around, it is always good to keep in mind the "real" back pressure (i.e. blocking, congestion control, etc.)

joshjones15:02:43

but I understand what you mean — a layer of abstraction removes some uncertainty and adds convenience @dominicm

dominicm15:02:22

@joshjones also the blessing from Rich makes me feel (perhaps incorrectly) like they're "safe" to use. They've been given the simplicity stamp of approval.

Matt Butler15:02:37

I didn't know that about atom @joshjones, thats great 😆

Matt Butler15:02:20

Thanks for the advice @tolitius, ill bear that in mind if i have reason to use core.async 🙂

joshjones15:02:35

yes, I get it — but concurrent programming of any kind requires a special attention to its logic, or it quickly can become a nightmare, so best to be careful always 🙂

dm315:02:14

can also highly recommend http://jcip.net/ which should alleviate some of the fear you might have about the j.u.c 🙂

joshjones15:02:25

would be great to see an updated version of that ^^

mpenet15:02:03

that's a great talk

tbaldridge15:02:45

@tolitius I think people often mis-understand that limit of 1024 pending puts....

tbaldridge15:02:36

it has nothing to do with the buffer size, it's the number of threads that can be in a parked/blocked state on the channel, and it's there because in most (all?) other queues, blocking puts/takes are a unbounded queue.

tbaldridge15:02:16

There's nothing stopping you from doing (chan (* 1000 1000 1000)) and then having one billion go blocks put one billion items into the channel.

tbaldridge15:02:54

it's just that you can't have more than 1024 blocks parked at one time. And the argument could be made that if you are trying to do something like that you have a design flaw in your program

Alex Miller (Clojure team)15:02:06

the only time I’ve seen someone hit that is when their program was broken to begin with

danielsz16:02:56

There's a great talk by @ghadi about the trade-offs and pitfalls of core.async. It has very little views which is a real shame because it's one of the best stuff you'll find on the Internetz. (aside the excellent material coming from @tbaldridge ) https://www.youtube.com/watch?v=DPpEZ3_XowU

eslachance16:02:44

Would anyone perhaps know of a good knowledgebase/article/howto system build on Clojure?

eslachance16:02:38

Hmm. Well I'm more looking with something with a front-end, if possible.

danielsz16:02:08

Just another suggestion.

eslachance18:02:05

Thanks for the suggestions guys, I'll look into it!

eslachance16:02:58

Open-source if possible obviously so I can muck around in the code 😉

tolitius17:02:16

@tbaldridge

=> (def c (async/chan))
=> (doseq [i (range 1024)] (async/put! c i))
=> (async/put! c 42)

java.lang.AssertionError: Assert failed: No more than 1024 pending puts are allowed on a single channel. Consider using a windowed buffer.
                          (< (.size puts) impl/MAX-QUEUE-SIZE)
a windowed buffer would "work" by dropping messages once the 1024 pending "parks" is reached. I see three issues with it: 1. Naming: "put" is overloaded with "put into the channel's buffer" and "put into the park queue" 1.1 Error message suggests that the buffer that is currently used does not work (it throws with this buffer, consider using another buffer) 2. 1024 is an arbitrary number. Why can't I have more than 1024 pending jobs parked? Saying that if I do have more, I am doing it wrong is.. wrong. Although I can certainly may do it wrong. 3. In order to avoid #2 limitation I usually build my own "throttle" mechanism, otherwise the high frequency data, that has a luxury to use the true TCP/IP congestion control, falls apart (`Assert failed: No more than 1024 pending puts are allowed`) once it comes in to the system.

joshjones17:02:57

If 1024 is not the number, what should it be? It seems that if one is doing lots of async puts onto a channel whose buffer is full, then there is an issue that needs to be dealt with

joshjones17:02:58

I suppose one answer could be, “you can’t put! on a channel whose buffer is full, at all"

Alex Miller (Clojure team)17:02:06

1024 was considered to be high enough, that if you see it, you are probably doing something wrong

tolitius17:02:22

@joshjones yes, the way tcp/ip deals with this issue it lets the producers know to slow down and the congestion window becomes much smaller, and then grows slowly

tolitius17:02:16

@alexmiller that could be true, but this is hardly an argument for 1024. what is this number based on?

Alex Miller (Clojure team)17:02:29

it is not based on anything

Alex Miller (Clojure team)17:02:52

it was considered “sufficiently big” to proceed with working on the rest of the implementation and deferring further work on it

Alex Miller (Clojure team)17:02:26

the thought being that with time we would have more experience with whether that number is good/bad/or should be configurable

Alex Miller (Clojure team)17:02:07

certainly there is a strong belief that there should be a limit as unbounded queues are bad

tolitius17:02:59

I agree, it does make sense to deal with unboundness of the queue. making it configurable with a good default (seems like 1024 works? for most) would probably be better. another approach, which would be really useful, but I think would not fly in the async nature of core.async and its fixed thread pools size, is to (have a config to) block, throttle on q limit reached.

tolitius17:02:58

which (in case of the external systems) would delegate the back pressure to the network

tolitius17:02:15

i.e. let the network handle it, it's sufficiently good at it

Alex Miller (Clojure team)17:02:49

but what would you block? put! is async

tbaldridge17:02:21

Right async tcp plus coreasync seems like a good fit.

tolitius17:02:02

> put! is async it is until it throws. so it could block instead when the MAX-QUEUE-SIZE is reached. not by default of course, but something like:

(async/throttle number (async/put! is async))
just thinking outloud. I build my own threshold and I stop putting things to a channel when N jobs are pending there. (usually based on file handles and other things). But working with the data that is coming in from the network it proved to be really useful (to me).

Alex Miller (Clojure team)17:02:40

I think making put! blocking breaks the rest of the design

tolitius17:02:09

right, that is why "I think would not fly in the async nature of core.async and its fixed thread pools size" 🙂

Alex Miller (Clojure team)17:02:17

if you want blocking/parking, use >!!/>!

Alex Miller (Clojure team)17:02:42

if you want feedback without blocking, you can use offer!

hiredman17:02:53

I am always surprised that people actually use put!

hiredman17:02:04

only if you complete ignore the part of the document above that

tbaldridge17:02:09

So if you want 10000 tcp connections why not use a fixed size buffer?

tbaldridge17:02:03

Put is pretty much the only good way to interface with async libs

tbaldridge17:02:37

Sure if you want to hang the async libs dispatch thread

hiredman17:02:51

which you need to do if you don't want to break feedback

tbaldridge17:02:26

No you can limit input as well. Think of a callback based async http lib.

tolitius17:02:16

@hiredman >! has the same MAX-QUEUE-SIZE limitation that put! does @alexmiller yes, I do usualy use >!! to consume from external systems. however the problem is not simply consuming messages from the pipe, but also do the additional work within go blocks down the road. those go blocks (loops) read as fast as they can from these channels.

hiredman17:02:47

I said >!! not >!

hiredman17:02:04

entirely different animal

tolitius17:02:40

@hiredman: just in case if you meant put! (which you said) vs. >!

hiredman17:02:56

@danboykis it suggests put! as a replacement for (go (>! ch x)), but once you have a go block like that you have already lost

danboykis17:02:08

hiredman if I am understanding you correctly you can create a new chan do a put! to it in the callback and then read from said channel using !<

danboykis17:02:31

hiredman, it recommends changing a go block to a put!

danboykis17:02:46

i'm not sure which part you are disagreeing with

hiredman17:02:18

put! is a low level thing, and when you use it you are saying "hey, I am going to ignore the feedback mechanism that comes with channels because I have thought this through" and that is basically never a good default state

tolitius17:02:41

@alexmiller the problem comes down to marrying a "non async" consumption (read from a message broker, a tcp socket, etc..) with the system that internally uses go blocks. throttling is one way to deal with it. there could be better ways of course.

tbaldridge17:02:05

Wat no it's not. Put is just a channel op plus a callback

hiredman17:02:30

a channel op that will never block, so no feedback

danboykis17:02:09

what I am saying is that you can get feedback from it

tbaldridge17:02:11

Sure it will

hiredman17:02:55

when will it block?

tbaldridge17:02:07

And the callback will be executed when it completes

tbaldridge17:02:25

It's all the same code just a different ui

tbaldridge17:02:59

So pick the cleanest ui

hiredman17:02:49

sure, you can build something that communicates back pressure on top of put!, but it is easier to actually use the existing constructs that communicate back pressure, and I have yet to see anyone actually communicate back pressure when using put!, they just put! in their callback and move on

hiredman17:02:37

for the naive case, when if you haven't thought about how back pressure is handled, you will get a better behaving system using the blocking/parking ops. Assuming you don't do something silly like (go (>! ...)) which, again, abandons backpressure

danboykis17:02:08

here's some pseudo code

(let [c (a/chan)] (a/go (http-call-with-callback #(a/put! c %)) (a/<! c)))

hiredman17:02:05

#(a/>!! c %) should replace #(a/put! c %) to be clearer

hiredman17:02:26

so for the message handler for a message queue example

hiredman17:02:08

you have a core.async system, you want to pump messages in to from a message queue. you set up a thread that pulls messages from the queue and pushes them in to the core.async system

hiredman17:02:09

using put! in that thread you will hit the 1024 limit, using !!> the thread pulling messages will be blocked until the message is "in" the core.async system

tbaldridge18:02:11

Or leverage the message queue's ack API to not have more than X message in flight. Or only ack it (or request the next message) inside the put!s callback

hiredman18:02:40

I am not saying >!! is the only correct way

tbaldridge18:02:46

No reason to tie up a OS thread for that.

tbaldridge18:02:20

But it does depend on the MQs interface

hiredman18:02:40

I am saying if you do something without completely thinking it through (which as programmers we never do, of course) >!! is more likely to result in a better behaved system

tbaldridge18:02:54

Clojure: begging people to think about their software designs since 2007

tolitius18:02:34

@tbaldridge > Or leverage the message queue's ack API to not have more than X message in flight yep, that is one way to throttle, but it relies on the ack / X message in flight feature of the producer. sometimes this is just a tcp socket. for which a throttle can also be built once the packets are defragmented.. but requires the whole other layer of throttle. It would be really helpful if core.async had an optional throttle built in, since it knows "what the message is" + "what the limit is"

ghadi18:02:38

well i didn't look through it 😃, seems nice

tbaldridge18:02:48

To the original question though every time I've hit the 1024 issue, it's been because of a lack of back pressure or a queue size that's undefined

hiredman18:02:04

sure, which is sort of similar to how >!! builds a back pressure mechanism on top of put!

tbaldridge18:02:15

So I don't get it, aren't tcp sockets buffered with fixed size buffers. If you stop reading when a channel is full, won't that push pressure back onto the other end of the tcp socket?

hiredman18:02:31

yes, assuming the pile of abstractions between you and tcp doesn't break that somehow

tbaldridge18:02:12

@tolitius I must be missing some thing here, how are you hitting 1024 puts if you are properly handling backpressure with put!

hiredman18:02:42

https://lists.w3.org/Archives/Public/public-whatwg-archive/2013Oct/0217.html apparently at some point some browsers didn't properly communicate back pressure for websockets

tbaldridge18:02:10

Seems like !!> or put + nio select shoul work

tbaldridge18:02:35

Yeah httpkit is ugly that way too.

tolitius18:02:49

@tbaldridge I am not hitting 1024 since I don't put more than X number of messages to the channel (I throttle it). But without throttling there are ways: socket => >!! => (go-loop [msg (<! ..)] ... (ask-another-system-something-async callback)) in case a producer (stream from a socket) is faster than the "another-system", jobs queue size grows.

tbaldridge18:02:49

So stop reading from the socket?

tolitius18:02:08

right, that's the manual throttle

timgilbert18:02:08

Hi, got another thing where I'm sure I'm just missing an obvious function from the standard lib. I have a seq of pairs [[:a 1] [:a 2] [:b 3] [:c 4]] and I want to construct a map from the first value to a set of the second values: {:a #{1 2} :b #{3} :c #{4}}. I thought (zipmap) might do it but it doesn't seem quite right. I can get it working with loop/recur, but I'm wondering if someone can help me find a more simple/elegant solution

timgilbert18:02:47

Ah, I'm thinking merge-with and conj might be the way

timgilbert19:02:58

Well, it's not remotely pretty, but I did come up with a (group-by) solution...

(defn value-sets
  "Given a seq of pairs, return a map from the first element of each pair to a set of
  all the last elements.

  (value-sets [[:a 1] [:a 2] [:b 3] [:c 4]])
  ; => {:a #{1 2} :b #{3} :c #{4}}"
  [pairs]
  (let [pair-values (fn [pairs]
                      (into #{} (map last pairs)))]
    (as-> pairs $
          (group-by first $) ; {:a [[:a 1] [:a 2]], :b [[:b 3]], :c [[:c 4]]}
          (zipmap (keys $)
                  (map pair-values (vals $))))))

cgrand19:02:35

@timgilbert not standard lib but for all group-by related stuff there's my xforms lib: (into {} (x/by-key (x/into #{})) your-coll)

timgilbert19:02:16

Oh, that looks awesome @cgrand, will look into it. And it's cross-platform too! Thanks!

jr19:02:41

(reduce (fn [m [k v]] (update m k (fnil conj #{}) v)) {}
[[:a 1] [:a 2] [:b 3] [:c 4]])

tanzoniteblack19:02:24

@jr anytime you have the pattern reduce (fn [m [k v]] ... you probably want to use (reduce-kv (fn [m k v] ...)) instead

alexbaranosky19:02:26

clojure
(->> [[:a 1] [:a 2] [:b 3] [:c 4] [:c 4]]
     (group-by first)
     (map (fn [[x y]]
            [x (set (map second y))]))
     (into {}))

tanzoniteblack19:02:54

(just a random performance tidbit)

alexbaranosky19:02:16

;; => {:a #{1 2}, :b #{3}, :c #{4}}

timgilbert19:02:37

Thanks for the suggestions, these are great! I think I favor @jr's concise reducer, I always forget how useful fnil is

jr19:02:57

fnil is super handy

timgilbert19:02:19

FWIW, transforming that into reduce-kv is slightly cumbersome because my input is a literal seq of pairs, not a map per se

jr19:02:01

yeah I wouldn't suggest doing that since the data structure we are reducing isn't associative in the way reduce-kv expects

alexbaranosky19:02:40

yeah reduce-kv isn't a good optimization for this case imho

cgrand19:02:18

Using a transient would help.

timgilbert19:02:18

For my use-case I'm I/O bound and expecting small inputs anyways, but would you make the accumulator map a transient?

Alex Miller (Clojure team)19:02:59

I have found that benchmarking stuff like this does not always match my expectations (so it’s worth doing if you care)

Alex Miller (Clojure team)20:02:38

based on what you’ve said, I would pick the one that reads the best and ignore the perf difference though

schmee20:02:13

s3 is down

hiredman20:02:28

took slightly more than half the internet with it

qqq20:02:40

read it on http://news.ycombiantor.com this morning, was not expecting it to affect me

Alex Miller (Clojure team)20:02:31

cloudfront seems to be masking the problem on http://clojure.org and http://clojurescript.org atm

schmee20:02:56

how awesome is that?

Alex Miller (Clojure team)20:02:30

https://twitter.com/ian_surewould/status/836645989972918272 - presenter doing a talk about S3 realizes S3 is down during the talk

Alex Miller (Clojure team)20:02:52

ever have one of those days?

qqq20:02:06

This must be a supernatural sigh for "don't use datomic"

qqq20:02:20

Can't even access the docs.

jstew20:02:06

The thing that sucks about the outage is trying to explain it to a client, and explaining that you can't just swap out storage to "something else".

Alex Miller (Clojure team)20:02:15

looks like some stuff is coming back

alexbaranosky20:02:30

transients would help the code get more confusing to read 🙂 Only optimize after profiling, and save yourself some time and code complexity 😛

alexbaranosky20:02:26

gotta backup the data to a separate availability zone?

alexbaranosky20:02:51

yeah, that's a royal PITA...

qqq20:02:39

installing boot on another machine; it worked, via s3; now wondering if it's safe or if it's mitm-ed

hlolli20:02:25

what's a good way to traverse over a queue, call a boolean function and remove some items from the queue, without taking the items from one queue and put them into another. Queue can be of course any list operation here. I think using reduce is maybe slowing down this process, because the data is moving from a to b, while I'd like to just keep them in the same queue that Im traversing trough.

hlolli20:02:38

yes, that came to my mind too. But wanted to ask if someone know some tricks. The only thing I don't like about remove is that Im not always removeing, I could call a function on some conditions. But I think remove will do it for sure...

hlolli20:02:51

well, let's not make it more complex, I go with remove.

eraserhd21:02:34

Today's puzzler:

dev=> (clojure.core.unify/unify [#:a{:b {:c/d 42, :db/id "foo"}} {:e/f 43, :x/y "foo
"}] '[#:a{:b {:db/id ?x :c/d 42}} {:e/f 43 :x/y ?x}])
nil
dev=> (clojure.core.unify/unify [#:a{:b {:db/id "foo", :c/d 42}} {:e/f 43, :x/y "foo
"}] '[#:a{:b {:db/id ?x :c/d 42}} {:e/f 43 :x/y ?x}])
{?x "foo"}

eraserhd21:02:07

While

dev=> (clojure.core.unify/unify {:a :b, :c :d} {:c :d, :a :b})
{}

eraserhd21:02:34

oh,

dev=> (clojure.core.unify/unify {:a '?x, :c :d} {:c :d, :a :b})
nil
dev=> (clojure.core.unify/unify {:a '?x, :c :d} {:a :b, :c :d})
{?x :b}

hiredman21:02:53

I get {?x :b} for both

hiredman21:02:27

hah, on clojure 1.5.1 which is what you get if you clone the project and run lein repl

hiredman21:02:25

so on 1.8 I get the nil there

hiredman21:02:30

the breakage happens between clojure 1.6 and 1.7, which is where I think clojure made some changes to hashing, which changed things like the iteration order of maps

eraserhd21:02:20

I figured it relied on that.

eraserhd21:02:09

Ah, it seems that it is supposed to work, but it's untested: http://dev.clojure.org/jira/browse/UNIFY-6

eraserhd21:02:34

I'll create an issue

eraserhd21:02:50

UNIFY-9 created. I'll look into fixing this and attaching a patch, 'cause this is kind of up my alley.

eraserhd21:02:48

@alexmiller Are fixes to core.unify welcome and/or likely to be merged?

PB22:02:03

Hey all. I'm getting the following error when trying to access a service that has a let's encrypt cert: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target. I'm using the latest verison of clj-http with the :insecure? true option set. Am I missing something?

dominicm22:02:49

petr: the simplest answer is to update java

PB22:02:25

java -version openjdk version "1.8.0_121" OpenJDK Runtime Environment (build 1.8.0_121-8u121-b13-0ubuntu1.16.04.2-b13) OpenJDK 64-Bit Server VM (build 25.121-b13, mixed mode)

PB22:02:28

It's pretty recent

dominicm22:02:39

Hmm 8_101+ is intended to have let's encrypt support I believe

PB22:02:48

That's awesome, thank you so much

PB22:02:01

It seem to be a JVM thing but this can't mean that this is impossible?

dominicm22:02:15

You can do it

dominicm22:02:34

There's a clojure article about on it

Alex Miller (Clojure team)22:02:08

@petr I believe this is typical when the root cert is not in the default store for your jdk

Lambda/Sierra22:02:23

I've hit that cert error with LetsEncrypt too.

Alex Miller (Clojure team)22:02:39

you can certainly add it yourself manually

Lambda/Sierra22:02:48

I assume you have to update the Java certificate store, which is a hassle but doable.