Fork me on GitHub
#clojure
<
2018-11-27
>
hiredman00:11:49

most operating systems have various limits they impose on processes at different levels, the number of open files is one of them

hiredman00:11:20

my guess would be you are doing a lot of stuff with :as-stream without consuming the stream or closing it

kenny00:11:53

@hiredman Hmm. This application is run in a docker container. Here's the output of ulimit -a .

time(seconds)        unlimited
file(blocks)         unlimited
data(kbytes)         unlimited
stack(kbytes)        8192
coredump(blocks)     unlimited
memory(kbytes)       unlimited
locked memory(kbytes) 16384
process              unlimited
nofiles              1048576
vmemory(kbytes)      unlimited
locks                unlimited
I believe the relevant number here is nofiles 1048576. There's no way that many requests were sent before this exception was hit. It's more in the range of 5-10k. This is the whole map passed to request. Nothing with :as-*.
{:method         :get
 :async?         true
 :url            (str base-url "/query")
 :query-params   {}
 :socket-timeout timeout-ms
 :conn-timeout   timeout-ms}

kenny00:11:33

My callback fn looks like this:

(fn [resp]
  (async/go
    (async/>! ch (merge (normalize-response resp)
                        {:timeout-ms timeout-ms}))
    (async/close! ch)))
Do I need to close something in resp?

hiredman00:11:41

I would run it outside of docker (hard to tell how that effects limits) and I would check the kernel limits(which are maybe in /sys or /proc somewhere) which would override anything set by ulimit

hiredman00:11:13

and I would figure out somewhere to actually count the number of requests made

hiredman00:11:35

I wouldn't be surprised if you do need to close something in resp

hiredman00:11:51

I've never used :async? true

kenny00:11:25

Will look for that. Some places say the default max for a Docker container is 1024, others say differently. Needs a bit more reading. I'll add a request count log to see if that sheds any light. The docs don't say anything about needing to close something https://github.com/dakrone/clj-http#async-http-request but I also wouldn't be surprised if something was missed.

hiredman00:11:54

I would look at what is in the resp map

hiredman00:11:18

if :body is some kind of stream (which maybe :async? implies :as-stream) then you will need to close it

kenny00:11:10

:body is a string.

hiredman00:11:40

do you have something upstream of the calls to clj-http that is limiting the number of calls?

kenny00:11:41

It has this :streaming? true. Not sure what that's talking about though.

hiredman00:11:30

because you don't have any limiting downstream with the callback like that

kenny00:11:22

Nothing explicit. Should there be?

hiredman00:11:23

I could easily see that combined with pipeline-async generating http requests as fast as the cpu can

kenny00:11:22

I don't think that is happening here. Although there is no explicit bound, it is called by mapping over some DB vals which can be verified to be a low number (though bounding it still sounds like a good idea).

kenny00:11:58

Adding the number of requests metric would shed some light on that one though. I can try doing what you describe locally to see if the same error is produced.

kenny00:11:03

Doing this worked without the above exception. Interestingly, it did block for a few seconds before returning.

(dotimes [_ 2000]
  (http/request
    {:method :get
     :url    ""
     :async? true}
    (fn [resp]
      (dosync
        (alter req-count inc)))
    (fn [resp]
      (dosync
        (alter req-count inc)))))
According ulimit -a, "Maximum number of open file descriptors (-n) 1024". Though I may need to check somewhere on the system.

kenny00:11:07

I could try doing the same via a clj REPL in a docker.

kenny00:11:28

Cannot repro that exception in a Docker container in a clj REPL with the above code.

kenny00:11:54

I was able to reproduce by starting the docker container like this docker run -it --ulimit nofile=1024:1024 java:8 bash and running the above code!

theeternalpulse05:11:11

is there an easy way when using time to only display the time output but not the result of the expression evaluation

theeternalpulse05:11:52

mainly in the repl

theeternalpulse05:11:19

was looking for a dynamic varbinding

mfikes05:11:42

If you are worried about huge amounts of output, you could evaluate (do x nil) where x is the thing you prefer not to see.

mfikes05:11:50

You'd have to be careful if x is lazy.

theeternalpulse05:11:26

haha, I was experimenting with the performance of realizing a lazy sequence in different ways so it works out

theeternalpulse05:11:20

man I forget so many little fundamental things with clojure

dpsutton05:11:38

i often use (count x) so i know the order of magnitude of the thing and if it conforms to my expectation

dpsutton05:11:59

or (keys x) or some kind of aggregate

jumar11:11:39

For those who use timbre: How do you include common request context (like user-id, request-id, etc.) into all your log messages? I've found with-context but that seems only to add :context key to the data but actually don't log anything:

(log/with-context {:user "juraj" :account "current"} (log/info "Hello"))
2018-11-27 11:50:38 Jurajs-MacBook-Pro.local INFO [xyz.core:50] - Hello
Perhaps I need to define custom output-fn as suggested here: https://github.com/ptaoussanis/timbre/issues/243

mbertheau16:11:05

How can I write (conj (if (empty? xs) [] xs) x) more concisely?

benoit17:11:38

@mbertheau Did you mean to check for nil with empty?. If so you could do ((fnil conj []) xs x).

mgrbyte17:11:56

is that not just (vec (seq xs))? sorry, misread

mgrbyte17:11:13

(-> xs seq vec (conj x))) works tho

mbertheau17:11:21

@me1740 that looks good, thanks!

jaihindhreddy-duplicate17:11:15

That may work for your use case but note that it (conj (vec xs) x) doesn't do the same thing as (conj (if (empty? xs) [] xs) x). The output of the former will always be a vector regardless of the type of xs.

laseray17:11:38

Question: Does anyone know where there is info on how to compile modular code with leiningen/boot (that needs module-info.java)?

enforser19:11:33

Does anybody know if there is a way to set a hard limit on the number of times criterium will execute a bench mark? I'm using it to measure quite a large operation, and it's taking a very long time to execute - to the point that I'm willing to sacrifice accuracy in favour of speed.

taylor19:11:03

is the quick-benchmark function still too slow?

enforser19:11:35

Thanks! I will look at tweaking the vars. Yeah, it is still too slow. The function itself takes about 1.5 minutes to run once, so I'm hoping to get down to about 20 executions

taylor20:11:34

if it takes that long, I'm not sure you're getting much benefit out of using Criterium, but I could be wrong

taylor20:11:37

I think one of the goals of Criterium is statistically sampling many invocations then doing some math stuff to pick up the most representative samples, deemphasizing the effects of JVM-stuff/GC/whatever

taylor20:11:21

I might consider breaking that huge operation down and benchmarking smaller pieces of functionality, if possible

taylor20:11:32

something like this might be useful for you too https://github.com/ptaoussanis/tufte

enforser02:11:19

@U3DAE8HMG just saw this now. You're totally right. I ended up abandoning Criterium and creating something myself for what I was doing. tufte looks great, I think I will be able to use that! thanks a lot for the suggestion

jrychter19:11:22

I'm looking at migrating from leiningen+cljsbuild+figwheel to clj+figwheel (using deps), as @dnolen suggested to me on Twitter. But I'm confused: how do people build uberjars using deps?

jeroenvandijk08:11:02

You can use boot-clj for both. Here is an example to build an uberjar with deps and boot-clj https://github.com/dundalek/closh/blob/b16bdd1ce5111c726a21ca642d7a465ed02adf88/build.boot#L11-L20

dnolen19:11:02

but the other thing is that this is not an all or nothing situation

dnolen19:11:21

you can use lein for some parts, and perhaps just deps for dev

dpsutton20:11:07

there is a nice plugin for lein that reads from deps.edn. makes it easy to use lein deploy to private repos etc

jrychter20:11:15

Thanks. But having spent some time looking into this, I don't think I'm ready for migration yet. It's too costly for me right now, especially if I end up with yet another build system to manage in addition to the existing ones.

mkvlr20:11:16

and https://github.com/hagmonk/depify might help with the migration

dnolen20:11:28

@jrychter don’t be lured by a false dichotomy - you can still benefit from what I’ve been suggesting with just Leiningen

dnolen20:11:15

what is deps giving you … nothing … that’s the value add

dnolen20:11:52

ClojureScript works directly from the command line - you don’t need something else, ditto for Figwheel main

dnolen20:11:20

but you can use Leiningen in exactly the same way - just drop the plugins

dnolen20:11:04

I did that for years lein run ...

jrychter20:11:31

by "plugins" you mean cljsbuild?

dnolen20:11:39

lein :aliases even let’s you give CLI boilerplate a name

dnolen20:11:32

whatever plugins you’re using that aren’t paying their way or complicating your build unnecessarily

dnolen20:11:36

I will say I’m slightly confused that you use both cljsbuild and figwheel

dnolen20:11:40

since figwheel subsumes cljsbuild

jrychter20:11:56

What is complicating my life is the interaction of lein profiles, cljsbuild builds, and figwheel configs. Of these three, cljsbuild provides the least value. But these things sound easy for simple examples and quickly get hairy for larger projects. I have things like getting the SHA from git into a version file, which then gets read by a macro at compile time, to get immutable versioned resources from clj/cljc/cljs.

jrychter20:11:31

I use figwheel for development, and cljsbuild is left for building the production system. I guess I could remove that indeed.

jrychter20:11:57

Removing leiningen would be much more difficult at a first glance.

dnolen20:11:07

well figwheel main has feature parity with figwheel

dnolen20:11:23

one way to clean up your project.clj is to get build config out of there

dnolen20:11:29

and into some EDN file on the classpath

dnolen20:11:45

:aliases will get you shortcut

dnolen20:11:52

lein cljs-dev, lein cjls-prod

dnolen20:11:19

(that might even work w/ existing figwheel, don’t remember)

dnolen20:11:53

recently I’ve stopped putting build stuff in project.clj - it feel gratuitous now

jrychter20:11:48

Yes, that sounds like a good first step. I will need to get started on this eventually, as I got to the point where I need to produce multiple ClojureScript apps, some preferably split into modules.

dnolen20:11:01

in anycase - dropping lein is orthogonal to simplifying your cljs setup

jrychter20:11:48

That is true. In general, I am under the impression that the various build tools have considerable overlap in functionality, as each tries to be an all-encompassing solution.

jrychter20:11:37

I mean, even the ClojureScript compiler isn't just a compiler in the traditional sense: it's a build system in itself.

dnolen20:11:27

I would take it step further

dnolen20:11:46

all the integrations are just sugar over what ClojureScript already handles just fine on it’s own

dnolen20:11:07

and figwheel and shadow-cljs just do extra stuff

dnolen20:11:58

another thing to consider is that ClojureScript is generally dev time only

dnolen20:11:08

it’s not clear to me that this stuff should be expressed in project.clj

jrychter20:11:14

Well, for building, true. But figwheel also provides reloading for dev. Which kind-of overlaps REPL functionality, as with clj we've been recompiling single forms or entire files for ages using nrepl.

dnolen20:11:17

all stuff just becomes JS on some CDN

dnolen20:11:33

i.e. ClojureScript is never a real dep, only a pre-deploy dep

jrychter20:11:09

Not really, in my case. It's very convenient to package the produced app into the uberjar as well, into resources/

macrobartfast20:11:25

This is a long shot, but does anyone know of anyone who can't make the Conj who had a ticket? I'm in Durham and would love to go, but the conference is out of reach for me right now. I wasn't originally going to be in town and so wasn't going to be able to make it, but things changed.

dnolen20:11:26

that doesn’t require ClojureScript to be in there

dnolen20:11:32

doesn’t matter where the assets live

jrychter20:11:20

Oh, so you mean that the ClojureScript build configuration should not live in leiningen? That I definitely agree with. I can't see a need for it there in particular. Just saying that it is a dependency of a complete uberjar.

dnolen20:11:03

@jrychter it’s a dependency to produce the uberjar

dnolen20:11:05

but it should not be inside

dnolen20:11:24

since it can serve no purpose

dnolen20:11:38

unless your server is compiling ClojureScript on the fly for client

enforser20:11:57

@macrobartfast I would try the #events channel as well

jrychter20:11:00

I'm not sure I understand you. The resulting .js app is something I do want in my uberjar. The ClojureScript compiler isn't. And the configuration does not need to be in project.clj.

jrychter20:11:41

BTW, just to amuse you, to top it all off I have a toplevel Makefile, too, which actually runs leiningen 🙂 (mostly for building external JS/CSS dependencies and for post-build tasks like brotli compression).

dnolen20:11:58

@jrychter got it - then we’re on the same page

dnolen20:11:31

it’s sound like just breaking out all the cljsbuild stuff would simplify your project config quite a bit then

jrychter20:11:42

I think so. Thanks for getting my mind out of a rut!

jaawerth21:11:16

speaking of app architecture and build configuration, I've been slowly going through https://github.com/juxt/edge - a lot to unpack in there but some neat stuff particularly with how they're building everything around the clojure cli tools in conjunction with small utils (many of which are also juxt projects)

☝️ 4
dominicm21:11:51

👋 maintainer here, let me know if you have questions. #juxt is a good place to ask. If you have feedback, I would love to hear it!

parrot 4
jaawerth22:11:25

Thanks! No questions thus far - it's helped a lot with the inevitable decision fatigue that comes with project layouts and organizing scripts around a minimalist build tool. Thanks for maintaining it, and being so responsive!

jaawerth21:11:53

kind of a neat window into what I imagine to be their internal clojure SDK

jrychter21:11:37

This is nice, a good complete complex example — these are very helpful.

jaawerth22:11:35

I thought so too - gave me lots of ideas to play with for my own setup

Azzurite23:11:08

umm... should it be possible for this exception to be generated? 😄 ArityException Wrong number of args (-1) passed to: controls/grid-pane clojure.lang.Compiler.macroexpand1 (Compiler.java:6917)

Azzurite23:11:38

how was I able to pass -1 arguments?

noisesmith23:11:53

tl;dr macros have two invisible args, and the arity check output message was doing the real arg to conceptual arg count translation wrong

dpsutton23:11:13

apologies that was a similar cljs one. here: https://dev.clojure.org/jira/browse/CLJ-1279

Azzurite23:11:27

interesting 🙂 thanks

dpsutton23:11:55

and it appears this will be resolved in 1.10