Fork me on GitHub
#clojure
<
2017-02-09
>
chbm00:02:34

using pipeline-blocking, do I have reasonable expectation the functions will be started in message order ? more specifically if my xf does calling a network service in normal operation the network calls will start in message order ?

hiredman00:02:44

results will come out the order data when in, but pipeline runs your xf in parallel according to the number you give it

hiredman00:02:50

I suspect the degree out of out of orderness that can happen is bounded, by n (the parallelism number you pass to pipeline-blocking), but I am not sure

hiredman00:02:28

(require '[clojure.core.async :as async])

(def counter (atom 0))

(def to (async/chan))

(def from (async/chan))

(async/pipeline 8 to (map (fn [value] [value  (swap! counter inc)])) from)

(async/go
  (dotimes [i 1000]
    (async/>! from i))
  (async/close! from))

(count (for [[x y] (async/<!! (async/into [] to))
             :when (not= x y)]
         true))
;;=> 980

hiredman00:02:38

no proof, but it would appear that the max out of order it gets is n+2, which I think matches other n+2 behaviors pipeline has

chbm01:02:11

that’s pipeline, not pipeline-blocking

hiredman01:02:52

I don't mean that issue is directly applicable

hiredman01:02:06

it is another instance of n+2 popping up from one of the pipeline functions

chbm01:02:19

pipeline go-loops over a channel with n buffer, -blocking spawns threads

chbm01:02:08

oh async/thread

hiredman01:02:56

the only difference in implementation between pipeline and pipeline-blocking is the use of async/go or async/thread

hiredman01:02:28

pipeline-async diverges more from those two

chbm01:02:11

i’m not sure i understand how pipeline-blocking provides parallelization

hiredman01:02:21

that issue up above is a result of pipeline-async's "parallelism" being related to the buffering behavior of the pipeline as a whole(n+2), and less directly to the number of "workers" (n)

chbm01:02:25

oh (dotimes n)

hiredman01:02:29

which relates to what you are asking about, because I suspect the max out of order you will see using pipeline and pipeline-blocking is dependent on the buffering behavior of the pipeline, which is some how 2+n

chbm01:02:02

yeah there is no guarantee those threads won’t preempt each other

hiredman01:02:41

correct, the threads introduce out of orderness, but the size of the output buffer (which is n) limits how far ahead a thread can get

hiredman01:02:02

the +2 is some how added in the copying machinery

chbm01:02:38

in my use i need to do parallelism 1 then manage the side effects

hiredman01:02:57

I am not sure what the point of using 1 with pipeline would be

chbm01:02:25

yeah, at that point it’s just go-loop

chbm01:02:41

i was going with the pipeline thing

flyboarder02:02:12

Last day for ClojureRemote tickets!

eslachance03:02:15

Is anyone else having issues deploying to clojars? It just keeps telling me 401 unauthorized but I know I have the proper credentials.

yedi05:02:14

is there a built-in fn / more idomatic way of doing (cons :new-1 (rest '(:1 :2 :3)))

tolitius05:02:07

@yedi: can it be a vector?

tolitius05:02:38

=> (assoc [:1 :2 :3] 0 :new-1)
[:new-1 :2 :3]

yedi06:02:32

unfortuantely its a seq

yedi06:02:39

but no biggie

yedi06:02:08

another q: is there a standard variable name for a boolean returning function you might send to another function

qqq08:02:00

Are there any 3d modeling tools written in clojure (or cljs)? Something where I can model in OpenGL, then render it with a ray tracer.

danielcompton08:02:57

@eslachance if you're still having trouble, pop in to #clojars and we can help

achesnais08:02:03

Based on a very quick test I see that using partial is somewhat faster than using the #() notation. Anyone can enlighten me on why that would be?

ilevd08:02:39

Maybe because with partial there is only 1 function call, and with #() - two

rauh08:02:36

@achesnais Normally #() is faster, can you post code?

achesnais08:02:32

You are right – my initial test was with the time macro, but using criterium I see #() is faster. What’s the explanation though?

cgrand15:02:51

achesnais: the JIT. Using partial you only create new instances of existing (generic classes). Since they exist they are already optimized. when you use #(…) a new class is defined for the closure. So the first few time you use it you pay for classloading, interpretation, optimization etc. However once all of these are paid for you end up with an optimized specific class vs an optimized generic class.

tbaldridge15:02:49

Also, (credit to Rich for bringing this to my attention the other day). There is a perf difference between #( 3 %) vs (partial 3), due to Clojure compiler inlining

tbaldridge15:02:30

Partial inserts * as a var, that you have to call through when invoking the function, while #(* ...) gets compiled as #(clojure.lang.Numbers/mul ...) which removes the var overhead.

schmee16:02:13

huh, so #() is both shorter and faster than partial?

tbaldridge18:02:12

no, it's more like the compiler can optimize #() better sometimes

achesnais05:02:32

Thanks for the detailed answer @U3E46Q1DG and @U07TDTQNL – fascinating stuff!

danielstockton08:02:26

Maybe because partial returns a variadic function?

cgrand15:02:02

danielstockton: no because partial is use varargs only for 4+ args

danielstockton15:02:03

Right, but the function it returns is still multiple arity. I thought there might be some overhead to this, it has to dispatch the right one.

cgrand15:02:49

one arity is one method, method selection is done at compile-time

danielstockton15:02:00

Ah I see, thanks.

m3tti09:02:28

hi there is clojure.data.xml/parse lazy ??? because i'm trying to load an 400mb large file and in the repl it takes damn long to write it into a variable.

quartz11:02:21

anyone here ever had any luck getting pedestal and sente to work together?

bcbradley12:02:00

is it possible to create an anonymous multimethod?

gfredericks12:02:20

not using the normal api for multimethods

pbostrom14:02:05

I wrote up a blog post on debugging a memory leak in production (it turns out it was a badly memoized fn): https://techblog.roomkey.com/posts/memory-leak.html

schmee14:02:43

nice read!

igrishaev15:02:58

when I log something with clojure/tools.logging, I see a couple of lines in the console:

Feb 09, 2017 6:09:22 PM my.ns invoke
INFO: my log message
The question is, how to remove the first line?

sineer16:02:10

What does the #' macro do? as in: (jetty/run-jetty #'app {:port port :join? false})

val_waeselynck16:02:11

@sineer it's the var-quote reader macro, equivalent to (var app) https://clojure.org/reference/reader

sineer16:02:01

So if I understand the use of it in (jetty/-jetty #'app {:port port :join? false}), it means if app is undefined then the fn will be called with a single arg arity correct?

tbaldridge16:02:44

@sineer So let's back up and look at (jetty/-jetty app ...) This will work, but the problem is that if you redefine app it won't show up without a server restart. This is because every use of a var implies a deref of the var ((.deref jetty/-jetty) (.deref app) ...)

tbaldridge16:02:59

However, if you use the var itself, via #'app that implied deref is disabled, and so you're passing in the direct var itself. Which means that jetty will invoke the var #'app on every incoming request. So now your changes to #'app show up without needing a server restart.

tbaldridge16:02:19

It's a way of moving the point of .deref from setup-time to run-time.

sineer16:02:27

Awesome 🙂 Thank you very much! I'm surprized I didn't know about this one yet... very clever!

negaduck18:02:10

How to make sure only given keys are in spec? I have (s/def ::good-state (s/keys :req-un [::name ::active-panel])) and it doesn’t complain when the state also has :whatever true.

joshjones18:02:06

spec does not allow this through using s/keys — what you can do, if you must, is add an additional predicate to ensure that only the keys you want are in the map. however, the question to ask is, what does it hurt, if :whatever is present in the map?

joshjones18:02:57

spec says, “validate what is there”, not “validate what is there, and ensure that nothing else is there"

negaduck18:02:20

@joshjones, it doens’t hurt, I’m just exploring what can be done

joshjones18:02:44

a simple example to ensure that the map has only 1 key:

(s/def ::a string?)
(s/def ::mymap (s/and (s/keys :req [::a])
                      #(= 1 (count (keys %)))))
(s/explain ::mymap {::a "abc" ::b 2})
;; val: #:sandbox.spec{:a "abc", :b 2} fails spec: :sandbox.spec/mymap predicate: (= 1 (count (keys %)))

joshjones18:02:30

so, you can use the additional custom predicate to perform whatever test you want on the given map

sveri18:02:54

@negaduck The idea about maps in general is that they are open. Leaving the possibility to add what you need and every function picks what it needs

negaduck18:02:35

on the second thought, it can hurt when there is a typo in code that updates a key, e.g., in re-frame event handler

joshjones18:02:24

You mean a typo that updates the wrong key?

negaduck18:02:49

simple :req doesn’t catch :active-panel :hmoe-pane

joshjones18:02:15

:req-un would ensure that you have both :active-panel and :hmoe-pane — what is the specific case that is problematic?

negaduck19:02:21

yes, wrong example.

negaduck19:02:54

here is a solution

joshjones19:02:59

that is one solution, as is the one above, but I don’t think you’ve given a problem that this solves yet?

joshjones19:02:52

specifically, when does the presence of a key in a map present a problem that ignoring it won’t solve?

negaduck19:02:19

joshjones,

(re-frame/reg-event-db
 :set-active-panel
 [check-spec-interceptor]
 (fn [db [_ active-panel]]
   (assoc db :actiev-pneal active-panel)))

negaduck19:02:02

now db has both :active-panel and this one with typo

joshjones19:02:40

so, it feels dirty because it shouldn’t be there, but no one will ever know .. right?

negaduck19:02:12

a user clicks a button and nothing happens

joshjones19:02:40

is there something more that spec should be checking about the :active-panel key itself that would inform you that it should have been set differently? (sorry, not a re-frame guy)

sveri19:02:25

What if a man, says something deep in the woods and no one can hear him. Is he still wrong?

asolovyov19:02:41

Weird stuff: I'm catching TimeoutException, but I get uncatched CheckedOperationTimeoutException in logs. This one subclasses TimeoutException. I have no idea what is going on here, but maybe someone can enlighten me? Should catching parent exception type catch all children?

hiredman19:02:12

are you using the go macro from core.async?

hiredman19:02:49

yeah, catching a superclass should catch all subclasses, so you should check to see if the exception is actually being thrown where you think it is

negaduck19:02:14

joshjones, yes, here spec checks if the :active-panel is exactly one of two options: (s/def ::active-panel #{:home-panel :about-panel}), this is straightforward.

sveri20:02:16

I think the question here is not, if the map contains a key you did not expect. The question is more or less, if there is a bug in your code that adds this key and the value. So it could be useful as a measure to find a very rare species of bugs.

dominicm20:02:49

Is there any reference to the function that clojure uses to look up reader literals? I have a need to do it dynamically, I've implemented something that follows the docstring, but wanted to know if I could kill the work altogether.

asolovyov20:02:29

@hiredman no, I'm not using go macro, it's a pretty regular code

hiredman20:02:35

so the usual possibilities, the exception isn't being thrown where you think it is, or it is being caught before it gets to your handler and logged

negaduck21:02:08

why after expansion it tries to use current-namespace/any? instead of cljs.core/any? here: (macro whatever [] (any? 1))?

spieden21:02:21

@negaduck i’m guessing quote vs syntax quote?

negaduck21:02:33

probably. Thank you.