Hey guys. We’re running a ring server (on jetty) and we’re running into a “too many open files” exception. Doing `lsof |grep our-uberjar|wc -l\ shows *16207

162075 open handles on its own jar.

I can’t find if this is a known problem somewhere in our stack or that we messed something up.

This is on 3.13.0-43-generic #72-Ubuntu SMP Mon Dec 8 19:35:06 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

with ring-jetty-adapter 1.3.0 / org.eclipse.jetty/jetty-server "7.6.13.v20130916"

Any ideas what may be going on? My first guess is that response streams for bundled resources aren’t being closed properly.

@joost-diepenmaat: have you tried lsof -p {pid} to see which fds it's owning?

hmm

watch -n 1 "lsof -p 8662|grep $our-jar.jar|wc -l"

shows an increasing number

going quickly from ~ 100 to 811 right now

and then at some point it drops again.

1321

doesn’t look like the ridiculously high numbers of beffore

now it’s back to 303

and climbing

lsof |grep $pid|grep $jar shows a lot higher numbers than lsof -p $pid|grep $jar

maybe it’s opening and closing many FDs quickly and you’d get overlapping sets in lsof without the -p

@ragge < 3000 open FDs seems a lot more reasonable than > 160000

but what are the actual fds it's holding onto? lots of regular files? which ones? sockets?

bunch of sockets and lots of open handles on the running uberjar

~ 580 TCP connections, 930 jars, 90 other

and of those, which ones are static, and which ones are changing?

which FDs are changing? let me see..

oh, just saw your first message, ~160k open file handles on it's *own* jar sounds suspicious

wild guess is that it's opening resources on each request using URLConnection

yeah but I don’t acutally get 160K open handles using lsof -p

that’s when I do lsof|grep $pid

the lsof -p option lists a varying number of FDs on the uberjar - between 100 and 2500

and the number seems to be increasing quickly and then drops back to 100 after reaching 3000

I’m not sure what that is about. Looks like it’s only closing the FDs when it’s running out of handles.

are you loading resources from the uberjar? like using io/resource?

yep

on each request?

my money is on JarURLConnection or whetever it's called then

most of the static assets in the site - stylesheets, js, css, are served using ring-resource middleware

we do serve all the “large” assets from a separate server, but the basic styling assets and frontend code is served from the jar

Right now, I’m not sure if there’s a large problem; we’ve been running like this for over a year and this is the first time I’ve seen this exception.

Occurred about an hour ago, and it’s not been back since.

I’m assuming the 160K open FDs is an artifact of running lsof without the -p option with a process that opens and closes FDs continuusly

i think lsof -p will exclude child proceeses too, could also be the reason you're seeing different numbers

hmm

oh

it's easy to leak (not close in a timely manner) InputStreams using UrlConnection

they will eventually be closed anyway

but sometimes, during some circumstances (lot's of requests for instance) you could leak more than you have fds

so I can see how it work fine, then happen suddenly, then work again

i fixed a bug in boot recently that had a similar issue: https://github.com/boot-clj/boot/pull/228

strace was very helpful there

you could try that too

I think I’ll give strace a shot.

I used something like sudo strace -e open,close -p {pid} -o trace.log

thanks @ragge! I’ll take a closer look

is there something more compact than this to get the values for a set of keys from a map? either in core or in another library?

user=> (vals (select-keys {:a :b :c :d :e :f :g :h} [:a :c :g]))
(:h :d :b)

using map and get, not that much more compact, but avoids building intermediate map just to get vals:

boot.user=> (def m {:a :b :c :d :e :f :g :h})
#'boot.user/m
boot.user=> m
{:a :b, :c :d, :e :f, :g :h}
boot.user=> (map (partial get m) [:a :c :g])
(:b :d :h)

@timvisher: ^^

@timvisher: juxt is your friend user> ((juxt :a :c :e) m) [:b :d :f]`

anybody with experience with riemann here? I'm trying to receive events through graphite transport and get errors like that - WARN [2015-09-16 14:56:35,174] nioEventLoopGroup-3-1 - riemann.transport.graphite - Graphite server received malformed message (java.lang.ClassCastException: io.netty.channel.socket.DatagramPacket cannot be cast to java.lang.CharSequence): #<DatagramPacket DatagramPacket(/127.0.0.1:65318 => /0:0:0:0:0:0:0:0:2003, SimpleLeakAwareByteBuf(UnpooledUnsafeDirectByteBuf(ridx: 0, widx: 105, cap: 2048)))>

I'm not even sure where exactly this error happens, let alone how do I fix that...

can I maybe extract full traceback from riemann?

@timvisher @ordnungswidrig nice with juxt, can simplify my example to also just (map m [:a :c :g]), works where keys are not functions too

@joost-diepenmaat: not much to add, but we are seeing the same problem occassionally, I'd be curious to know if you find anything

@ordnungswidrig: man! that’s awesome

Is there any transducer sort?

@profil: There is not. In general if a transducer would have to wait until the sequence ends to actually output anything, then there is no transducer for it since it makes limited sense.

Yeah thats what I thought..

I am transforming a big list with transduce using drop and map second, then using conj as the reducing function. After this I want it sorted then calculate the mean and some quantiles, it feels kinda ugly not doing it in a sequence

@crankyadmin: that is clj-refactor telling you you have a broken namespace

(setq cljr--debug-mode t) should give you more info

Is there a Clojure library that supports XML element namespaces? This is a requirement for interfacing with an external service and data.xml does not support this feature (although there’s a seemingly dead proposal to fix this: http://dev.clojure.org/display/DXML/Namespaced+XML).

@profil: you should benchmark using eduction instead of transduce in this scenario - (sort (eduction (drop 1) (map second) coll))

it might be faster

there are several competing effects going on here so it's hard for me to predict which is better

eduction is going to effectively build an iterator and sort will build a chunked seq over that iterator

if you're doing transduce with conj you should really just do into (which automatically uses transients) - (sort (into [] (comp (drop 1) (map second)) coll))

that will conj into a vector with transients, then use a seq that traverses the vector directly (which is not chunked, but vector seqs are using the underlying data structure, so generally very fast)

if you do benchmark those, I'd be curious to know the input size and timing results :)

kind of an interesting reducing function would be one that inserted results into a sorted collection as they arrived

for example (into (sorted-set) (comp (drop 1) (map second)) coll)) - if data has no dupes

then you traverse the input exactly once, extract the right data, and drop it directly into a sorted output collection

Ahh, cool. The data can contain duplicates, but thats not very likely. I need to get this assignment done, I will try to bench it in the weekend 😄 The data sets is a simple neural network consisting of 1000 neurons, 500 patterns, and this will be sampled for 50000 time steps, so its not huge, but it still takes a few hours

@alexmiller: How would I bench it? using time or something more sophisticated?

well if it takes hours, then time should be sufficient :)

If it takes hours, you could probably use a physical stopwatch

Anyone know of any libraries for interacting with nameservers?

@mocker: https://github.com/brweber2/clj-dns

@beppu: That's the only one I've found (hasn't been updated in several years)

Does it not work?

Yeah, getting NullPointerExceptions for some basic lookups.

Think it may be the only option though.

Anyone got a nice alternative to this without flatten? (reduce + (flatten [[1 2] [3 4]]))

one level of flatten is apply concat

or reduce into

(apply + (reduce into [] [[1 2] [3 4]]))

(apply + (apply concat [[1 2] [3 4]]))

reduce into has the nice property of almost looking like english words that mean what it's doing

(transduce cat + [[1 2] [3 4]])

I think that's my fave

@gtrak: Thanks 😄 kittens!!

something something floss digestive tract something streams

@mocker: if your needs are light, a Java library may be a better option

@danielcompton: Agreed, it might also be a good opportunity for me to learn how to write a clojure library that wraps around the java library.

(basically, an updated clj-dns)