Fork me on GitHub
#clojure
<
2016-07-30
>
wei00:07:42

yes, I’d like it to be in ascending order from left to right (that’s depth first I believe)

wei00:07:31

@lvh: are you referring to clojure.walk/walk and keeping a counter in the form arg

lvh00:07:07

clojure.walk/prewalk, presumably; you keep the counter external to that call

lvh00:07:46

(atom 0), I’d presume; getting a new id by doing (swap! count inc)

wei00:07:52

works well, i think:

(defn index-map [m]
  (let [counter (atom 0)] (prewalk #(if (map? %) (assoc % :id (swap! counter inc)) %) m)))

lvh00:07:45

that looks about right 🙂

wei02:07:29

one problem with that function above is that nested maps get transformed too- I wanted to break out of a branch after a node is found. specter’s walker does that so here’s my updated solution, feedback appreciated

(defn transform-with-index [m f]
  (let [counter (atom -1)]
    (com.rpl.specter.macros/transform [(com.rpl.specter/walker map?)]
                                      #(f (swap! counter inc) %) m)))

wei03:07:24

@codonnell from #C0FVDQLQ5 with a cleaner solution:

(transform (subselect (walker map?)) (partial map #(assoc %2 :id %1) (range)) [[{} {}] {} {}])
[[{:id 0} {:id 1}] {:id 2} {:id 3}]

lvh14:07:01

It seems like lein replaces :jvm-opts by default now; how do I just add something (the old behavior)?

lvh14:07:27

I’m trying to add -Djava.awt.headless=true for my tests so it doesn’t spawn a window when running unit tests.

akiva14:07:23

I have :jvm-opts in ~/.lein/profiles.clj and it seems to still be working for me.

lvh15:07:18

akiva: Sure, but IIUC that disables tiered compilation, right?

lvh15:07:37

I’m trying to use this for tests; I guess I probably still want tiered compilation 🙂

akiva15:07:38

Hmm. I honestly don’t know.

akiva15:07:42

I suppose you could add the tiered compilation flag to the project.clj by hand like you used to have to do with older eversions of lein.

lvh15:07:28

yeah; that’s what I’ve done now

lvh15:07:51

I’m just not super happy with specifying things about tiered compilation when all I want to specify is headless awt

akiva15:07:25

Understood. It doesn’t seem like it should be this fussy.

borkdude15:07:42

I've never done much with alter-var-root but what is the difference between just re-defing a var, except that alter-var-root can alter atomically?

hans15:07:52

@borkdude: idiomatic clojure has def only on the top level of a file and uses alter-var-root in the rare case that the root binding needs to be modified.

kaosko17:07:10

Hi guys. Using migratus, rollback doesn't seem to do anything at all. no errors, nothing. how do I debug this thing?

asolovyov19:07:48

Hey all! I have a problem with performance, which is looking pretty weird for me - but I'm a newbie to JVM. So the thing is that when I try to test my app's performance with wrk, it replies ok for some time and then starts replying very slowly - can't reproduce that on my laptop, but it's working that way on production server. I connected to the app with YourKit profiler and I see that for some time there is a lot of writes to socket (~110 MB/s) and 100% CPU load and at some point socket write speed drops to 50 KB/s and CPU load drops to 0. After about 30-40 seconds it resurrects and starts replying fast again. Thread count during that time is consistently high. I did a thread dump during "dead" phase and it shows a lot of threads waiting for SynchronousQueue$TransferStack, which seems to me like 'waiting for a connection'. Right now it's with Netty (Aleph), before that it was with Jetty9 (jet), and it was the same, and it's the same with http-kit. So... Any ideas where to look next to find the source of the problem?

Alex Miller (Clojure team)19:07:35

probably worth checking whether you have excessive GC going on - YourKit can tell you that

asolovyov19:07:44

checked that already

asolovyov19:07:58

(that's me from reddit thread btw :-))

asolovyov19:07:24

I even tried to switch to G1GC just in case - and obviously (because I don't have any problems) - it didn't help

Alex Miller (Clojure team)19:07:37

in these periods you’re not doing work, so that implies you are waiting

Alex Miller (Clojure team)19:07:43

what are you waiting on?

Alex Miller (Clojure team)19:07:08

if it’s I/O / sockets then maybe something is misconfigured in your network and/or sockets

asolovyov19:07:20

how do I determine what I'm waiting for?

Alex Miller (Clojure team)19:07:27

look at your thread dump :)

asolovyov19:07:38

353 threads waiting for notification on lock: SynchronousQueue$TransferStack

Alex Miller (Clojure team)19:07:16

synchronous queues require both producers and consumer threads to be there at the same time - they are 0-length queues

Alex Miller (Clojure team)19:07:27

are you waiting for a producer or a consumer?

asolovyov19:07:45

not sure :)) maybe I need another tool for thread dump

Alex Miller (Clojure team)19:07:57

they look the same regardless :)

Alex Miller (Clojure team)19:07:37

353 threads is a lot - is this a big box?

asolovyov19:07:38

http://spotify.github.io/threaddump-analyzer/ - I use this one and I'm not sure what I'm waiting for 🙂

serce19:07:47

Try to look at safepoint pauses -XX:+PrintSafepointStatistics -XX:PrintSafepointStatisticsCount=1

asolovyov19:07:55

fairly big, 8 cores and 32gb of ram

Alex Miller (Clojure team)19:07:12

I wouldn’t worry about safepoint stuff until much further down the list

asolovyov19:07:25

that's what aleph started for itself, I had jetty at 200 threads usually

asolovyov19:07:09

"manifold-pool-2-1341" #1449 daemon prio=5 os_prio=0 tid=0x00007f1b4c191800 nid=0xc09 waiting on condition [0x00007f1b01f90000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00000003d98e2220> (a java.util.concurrent.SynchronousQueue$TransferStack)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
    at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
    at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
    at io.aleph.dirigiste.Executor$Worker$1.run(Executor.java:52)
    at manifold.executor$thread_factory$reify__29095$f__29096.invoke(executor.clj:36)
    at clojure.lang.AFn.run(AFn.java:22)
    at java.lang.Thread.run(Thread.java:745)

asolovyov19:07:35

this looks like it waits for incoming connection rather for my app, right?

serce19:07:16

You have a lot of threads doing work. If one thread couldn't reach a safepoint other threads will be waiting for it. And of course, you'll get a thread dump only when it reach the safepoint. AFAIK, yourkit can't report you safepoint pause problems

Alex Miller (Clojure team)19:07:17

probably, I don’t know aleph that well

Alex Miller (Clojure team)19:07:45

@serce that’s all true, but unlikely to be important in my experience

asolovyov19:07:51

@serce: ok, so I just do this and it prints if there are any problems to stdout?

asolovyov19:07:12

@alexmiller: I mean I have no ideas where to look at and that's quite easy to do, so I can try it 🙂

Alex Miller (Clojure team)19:07:28

I don’t think it’s going to tell you anything useful, but go for it :)

asolovyov19:07:40

wow it prints a lot :))

madvas19:07:04

In REPL I can use (Math/abs 1), but when I want to load file into repl where it is used I get

CompilerException java.io.FileNotFoundException: Could not locate clojure/math__init.class or clojure/math.clj on classpath.
Does anybody know what might be an issue?

Alex Miller (Clojure team)19:07:07

you need to understand more about the big picture of your system, how your resources are being used

Alex Miller (Clojure team)19:07:56

@madvas: Math is a Java class in the JDK, not a Clojure ns

asolovyov19:07:59

yeah... but right now what I know is that my system almost stops reading from sockets and writing to sockets, and just keeps threads in the pool waiting for... IDK what

asolovyov19:07:27

also, I can't reproduce that on my laptop as well, which means something is configured differently?..

asolovyov19:07:56

my laptop is os x though

asolovyov19:07:28

@alexmiller: when you say 'how your resources are being used', that's more about the "server" as the system rather than just jvm, right?

Alex Miller (Clojure team)19:07:46

yes, primarily that’s what I meant

Alex Miller (Clojure team)19:07:13

doesn’t sound like you are having system resources problems with memory or JVM problems with heap/GC

Alex Miller (Clojure team)19:07:50

if it is a network issue, that’s a different set of tools, and I haven’t done that kind of work in a long time

serce19:07:00

@asolovyov: Also, do not rely on YourKit, try to get details staticstic from JVM using keys, the samples would be -XX:+PrintGCApplicationStoppedTime, it won't print timestamps, but show you all the pauses your JVM has

asolovyov19:07:17

heh, ok! thanks!

asolovyov19:07:50

is there an easy way to block exit from a clojure program?

lmergen19:07:19

You can derefence a promise that will never be filled to block forever

asolovyov19:07:37

ah, nice 🙂

lmergen19:07:43

@(promise) will do the trick

asolovyov19:07:55

@alexmiller: it seems you were right, I did the simplest possible app with aleph (which does 30k rps on my laptop), and on the server it sharply drops in performance in about 8 seconds %)

josh_tackett20:07:18

Looking for clojure developer with experience in the following libraries to finish building out webdriver system:

 -  
 - 
 - 

Terms: Ongoing hourly contract
Location: Remote OK 
Start: Aug 8th