This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2018-11-27
Channels
- # adventofcode (1)
- # announcements (4)
- # beginners (120)
- # calva (5)
- # cider (12)
- # clara (3)
- # cljdoc (48)
- # cljs-dev (33)
- # cljsrn (4)
- # clojure (124)
- # clojure-dev (43)
- # clojure-europe (2)
- # clojure-italy (168)
- # clojure-nl (2)
- # clojure-spec (7)
- # clojure-uk (79)
- # clojurescript (50)
- # core-logic (6)
- # cursive (12)
- # datascript (1)
- # datomic (8)
- # devcards (2)
- # emacs (5)
- # events (2)
- # figwheel-main (6)
- # fulcro (18)
- # graphql (42)
- # hyperfiddle (3)
- # jobs (1)
- # luminus (2)
- # nrepl (5)
- # off-topic (59)
- # onyx (5)
- # parinfer (2)
- # pathom (10)
- # pedestal (2)
- # portkey (3)
- # re-frame (24)
- # reagent (6)
- # reitit (54)
- # remote-jobs (1)
- # ring (5)
- # shadow-cljs (75)
- # spacemacs (35)
- # sql (22)
- # tools-deps (16)
- # unrepl (10)
most operating systems have various limits they impose on processes at different levels, the number of open files is one of them
my guess would be you are doing a lot of stuff with :as-stream without consuming the stream or closing it
@hiredman Hmm. This application is run in a docker container. Here's the output of ulimit -a
.
time(seconds) unlimited
file(blocks) unlimited
data(kbytes) unlimited
stack(kbytes) 8192
coredump(blocks) unlimited
memory(kbytes) unlimited
locked memory(kbytes) 16384
process unlimited
nofiles 1048576
vmemory(kbytes) unlimited
locks unlimited
I believe the relevant number here is nofiles 1048576
. There's no way that many requests were sent before this exception was hit. It's more in the range of 5-10k.
This is the whole map passed to request
. Nothing with :as-*
.
{:method :get
:async? true
:url (str base-url "/query")
:query-params {}
:socket-timeout timeout-ms
:conn-timeout timeout-ms}
My callback fn looks like this:
(fn [resp]
(async/go
(async/>! ch (merge (normalize-response resp)
{:timeout-ms timeout-ms}))
(async/close! ch)))
Do I need to close something in resp
?I would run it outside of docker (hard to tell how that effects limits) and I would check the kernel limits(which are maybe in /sys or /proc somewhere) which would override anything set by ulimit
Will look for that. Some places say the default max for a Docker container is 1024, others say differently. Needs a bit more reading. I'll add a request count log to see if that sheds any light. The docs don't say anything about needing to close something https://github.com/dakrone/clj-http#async-http-request but I also wouldn't be surprised if something was missed.
if :body is some kind of stream (which maybe :async? implies :as-stream) then you will need to close it
do you have something upstream of the calls to clj-http that is limiting the number of calls?
I could easily see that combined with pipeline-async generating http requests as fast as the cpu can
I don't think that is happening here. Although there is no explicit bound, it is called by mapping over some DB vals which can be verified to be a low number (though bounding it still sounds like a good idea).
Adding the number of requests metric would shed some light on that one though. I can try doing what you describe locally to see if the same error is produced.
Doing this worked without the above exception. Interestingly, it did block for a few seconds before returning.
(dotimes [_ 2000]
(http/request
{:method :get
:url ""
:async? true}
(fn [resp]
(dosync
(alter req-count inc)))
(fn [resp]
(dosync
(alter req-count inc)))))
According ulimit -a
, "Maximum number of open file descriptors (-n) 1024". Though I may need to check somewhere on the system.I was able to reproduce by starting the docker container like this docker run -it --ulimit nofile=1024:1024 java:8 bash
and running the above code!
is there an easy way when using time to only display the time output but not the result of the expression evaluation
mainly in the repl
was looking for a dynamic varbinding
If you are worried about huge amounts of output, you could evaluate (do x nil)
where x
is the thing you prefer not to see.
haha, I was experimenting with the performance of realizing a lazy sequence in different ways so it works out
ah that does it
man I forget so many little fundamental things with clojure
i often use (count x)
so i know the order of magnitude of the thing and if it conforms to my expectation
For those who use timbre
: How do you include common request context (like user-id
, request-id
, etc.) into all your log messages?
I've found with-context
but that seems only to add :context
key to the data but actually don't log anything:
(log/with-context {:user "juraj" :account "current"} (log/info "Hello"))
2018-11-27 11:50:38 Jurajs-MacBook-Pro.local INFO [xyz.core:50] - Hello
Perhaps I need to define custom output-fn
as suggested here: https://github.com/ptaoussanis/timbre/issues/243@mbertheau Did you mean to check for nil with empty?
. If so you could do ((fnil conj []) xs x)
.
That may work for your use case but note that it (conj (vec xs) x)
doesn't do the same thing as (conj (if (empty? xs) [] xs) x)
. The output of the former will always be a vector regardless of the type of xs
.
Question: Does anyone know where there is info on how to compile modular code with leiningen/boot (that needs module-info.java)?
Does anybody know if there is a way to set a hard limit on the number of times criterium will execute a bench mark? I'm using it to measure quite a large operation, and it's taking a very long time to execute - to the point that I'm willing to sacrifice accuracy in favour of speed.
there's this dynamic var you could tweak https://github.com/hugoduncan/criterium/blob/develop/src/criterium/core.clj#L72
maybe this one too https://github.com/hugoduncan/criterium/blob/develop/src/criterium/core.clj#L75
Thanks! I will look at tweaking the vars. Yeah, it is still too slow. The function itself takes about 1.5 minutes to run once, so I'm hoping to get down to about 20 executions
if it takes that long, I'm not sure you're getting much benefit out of using Criterium, but I could be wrong
I think one of the goals of Criterium is statistically sampling many invocations then doing some math stuff to pick up the most representative samples, deemphasizing the effects of JVM-stuff/GC/whatever
I might consider breaking that huge operation down and benchmarking smaller pieces of functionality, if possible
something like this might be useful for you too https://github.com/ptaoussanis/tufte
@U3DAE8HMG just saw this now. You're totally right. I ended up abandoning Criterium and creating something myself for what I was doing. tufte looks great, I think I will be able to use that! thanks a lot for the suggestion
I'm looking at migrating from leiningen+cljsbuild+figwheel to clj+figwheel (using deps), as @dnolen suggested to me on Twitter. But I'm confused: how do people build uberjars using deps?
You can use boot-clj for both. Here is an example to build an uberjar with deps and boot-clj https://github.com/dundalek/closh/blob/b16bdd1ce5111c726a21ca642d7a465ed02adf88/build.boot#L11-L20
@jrychter juxt has a think called pack https://github.com/juxt/pack.alpha
there is a nice plugin for lein that reads from deps.edn. makes it easy to use lein deploy
to private repos etc
Thanks. But having spent some time looking into this, I don't think I'm ready for migration yet. It's too costly for me right now, especially if I end up with yet another build system to manage in addition to the existing ones.
@jrychter here's even more choices https://github.com/clojure/tools.deps.alpha/wiki/Tools#packaging
and https://github.com/hagmonk/depify might help with the migration
@jrychter don’t be lured by a false dichotomy - you can still benefit from what I’ve been suggesting with just Leiningen
ClojureScript works directly from the command line - you don’t need something else, ditto for Figwheel main
whatever plugins you’re using that aren’t paying their way or complicating your build unnecessarily
What is complicating my life is the interaction of lein profiles, cljsbuild builds, and figwheel configs. Of these three, cljsbuild provides the least value. But these things sound easy for simple examples and quickly get hairy for larger projects. I have things like getting the SHA from git into a version file, which then gets read by a macro at compile time, to get immutable versioned resources from clj/cljc/cljs.
I use figwheel for development, and cljsbuild is left for building the production system. I guess I could remove that indeed.
Yes, that sounds like a good first step. I will need to get started on this eventually, as I got to the point where I need to produce multiple ClojureScript apps, some preferably split into modules.
That is true. In general, I am under the impression that the various build tools have considerable overlap in functionality, as each tries to be an all-encompassing solution.
I mean, even the ClojureScript compiler isn't just a compiler in the traditional sense: it's a build system in itself.
all the integrations are just sugar over what ClojureScript already handles just fine on it’s own
Well, for building, true. But figwheel also provides reloading for dev. Which kind-of overlaps REPL functionality, as with clj we've been recompiling single forms or entire files for ages using nrepl.
Not really, in my case. It's very convenient to package the produced app into the uberjar as well, into resources/
This is a long shot, but does anyone know of anyone who can't make the Conj who had a ticket? I'm in Durham and would love to go, but the conference is out of reach for me right now. I wasn't originally going to be in town and so wasn't going to be able to make it, but things changed.
Oh, so you mean that the ClojureScript build configuration should not live in leiningen? That I definitely agree with. I can't see a need for it there in particular. Just saying that it is a dependency of a complete uberjar.
@macrobartfast I would try the #events channel as well
I'm not sure I understand you. The resulting .js app is something I do want in my uberjar. The ClojureScript compiler isn't. And the configuration does not need to be in project.clj.
BTW, just to amuse you, to top it all off I have a toplevel Makefile, too, which actually runs leiningen 🙂 (mostly for building external JS/CSS dependencies and for post-build tasks like brotli compression).
it’s sound like just breaking out all the cljsbuild stuff would simplify your project config quite a bit then
speaking of app architecture and build configuration, I've been slowly going through https://github.com/juxt/edge - a lot to unpack in there but some neat stuff particularly with how they're building everything around the clojure cli tools in conjunction with small utils (many of which are also juxt projects)
👋 maintainer here, let me know if you have questions. #juxt is a good place to ask. If you have feedback, I would love to hear it!

Thanks! No questions thus far - it's helped a lot with the inevitable decision fatigue that comes with project layouts and organizing scripts around a minimalist build tool. Thanks for maintaining it, and being so responsive!
umm... should it be possible for this exception to be generated? 😄 ArityException Wrong number of args (-1) passed to: controls/grid-pane clojure.lang.Compiler.macroexpand1 (Compiler.java:6917)
it's a bug with macros. https://dev.clojure.org/jira/browse/CLJS-1516
tl;dr macros have two invisible args, and the arity check output message was doing the real arg to conceptual arg count translation wrong
apologies that was a similar cljs one. here: https://dev.clojure.org/jira/browse/CLJ-1279