This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2019-06-14
Channels
- # announcements (2)
- # aws (12)
- # aws-lambda (5)
- # beginners (42)
- # calva (56)
- # cider (16)
- # clj-kondo (1)
- # cljs-dev (45)
- # cljsjs (1)
- # cljsrn (25)
- # clojure (171)
- # clojure-europe (3)
- # clojure-italy (16)
- # clojure-losangeles (2)
- # clojure-nl (49)
- # clojure-spec (2)
- # clojure-sweden (3)
- # clojure-uk (11)
- # clojurescript (84)
- # component (11)
- # core-async (12)
- # core-logic (2)
- # cursive (8)
- # datomic (41)
- # events (2)
- # fulcro (48)
- # graalvm (1)
- # graphql (1)
- # hoplon (12)
- # jackdaw (1)
- # jobs (1)
- # jobs-discuss (45)
- # joker (5)
- # keechma (10)
- # nyc (3)
- # off-topic (14)
- # pathom (16)
- # qa (1)
- # re-frame (22)
- # reagent (12)
- # reitit (4)
- # remote-jobs (1)
- # shadow-cljs (40)
- # spacemacs (3)
- # timbre (3)
- # tools-deps (29)
it does a lot of the things we like
(ins)user=> (def h (java.util.LinkedHashMap.))
#'user/h
(ins)user=> (.put h :a 0)
nil
(ins)user=> h
{:a 0}
(ins)user=> (:a h)
0
(cmd)user=> (.put h :b [:c :d])
[:c :d]
(cmd)user=> (get-in h [:b 0])
:c
but not others
(ins)user=> (h :a)
Execution error (ClassCastException) at user/eval323 (REPL:1).
java.util.LinkedHashMap cannot be cast to clojure.lang.IFn
(def validate-attr-re #"^[A-Za-z][A-Za-z\-.0-9]")
(re-matches validate-attr-re "asdf") ;; => nil
what am I doing wrong here?or a *
if you intend to match java style identifiers
There is a Clojure lib containing an ordered-map
that preserves insertion order, too, and is more Clojurey: https://github.com/clj-commons/ordered
If it's "given that I have list of the keys (in the right order" then read from the list, pulling the values in the order of the list. Or am I missing something here? Do you want all the values in one call?
That terminology also applies to rounding (e.g. https://en.wikipedia.org/wiki/Rounding#Round_half_down) > round half down (or round half towards negative infinity)
in simpler words it means that if it truncates to the smaller integer rather than to the biggest integer
can you say which exact function you've read the "Truncates toward negative infinity" so we can craft examples to demonstrate it?
cljs.user> (rem -2 1000)
-2
cljs.user> (mod -2 1000)
998
are the classic examples i think“Truncates toward negative infinity” means if you write your numbers with negative infinity on the left and positive infinity on the right, truncation will move to the nearest integer to the left of your floating point number.
https://biblio.ugent.be/publication/314490/file/452146.pdf big pdf but check section 2.3 Div-Dominant Definitions
hey everyone, how can i supply a custom implementation of keys
for a deftype based on a map?
there are multiple interfaces needed to implement a clojure persistent hash map, one solution would be to implement them all, with most of them delegating directly to an underlying hash-map
there's no simple built in way to do this, but there's a lib that tries to make it easier https://github.com/ztellman/potemkin "def-map-type"
the summary for that lib is a good caveat :D
user=> (pprint (supers (class {})))
#{java.lang.Object clojure.lang.MapEquivalence java.io.Serializable
clojure.lang.IEditableCollection clojure.lang.ILookup
clojure.lang.Associative java.lang.Iterable clojure.lang.IObj
clojure.lang.APersistentMap java.util.Map clojure.lang.Counted
clojure.lang.Seqable clojure.lang.IFn clojure.lang.IMapIterable
java.util.concurrent.Callable clojure.lang.IHashEq
clojure.lang.IKVReduce clojure.lang.IMeta clojure.lang.AFn
java.lang.Runnable clojure.lang.IPersistentCollection
clojure.lang.IPersistentMap}
nil
that's a lot to implement by hand
How do you set limits (CPU / MEM) for you Clojure applications in kubernetes / docker? What is your method to determine min,max CPU / MEM usage?
I've never set a CPU limit, but mem limits are just an arg to java
and if you care about mem enough to limit it, definitely start your app with java rather than some build tool
I believer newer jvms detect that they are running in docker and size themselves (memory wise) according to the container limits
OK - the docker part doesn't really change anything here (other than the ability to set limits to docker itself I guess), and I'd advocate for using clojure to build a classpath then running java itself using that classpath
there's extensive documentation for java itself regarding how it allocates and manages memory
between the man pages, and oracle docs
only one way which come to my mind: start with something and if it is too low increase 😉
oh so you are asking how to pick a number?
OK, I misread, and thought you were asking how to impose that limit
use a profiler and look at resource usage under something near typical load
visualvm, yourkit, hprof are all good options depending on what kind of UI you like
surely you can create incoming load?
my idea is to deploy with high limitation and see how many resources it takes. After some times (a few days?) set limits based on metrics
you can run hprof or some other metric collecting agent in prod, sure
just not really sure how people do it and I would like to hear how others do it in everyday practice 🙂
profiling is how I do it
first there, then in a uat/staging env
usually by then I have a good idea of what the upper limit should be
with hprof you just add an agent to the vm on startup, no gui or extra program needed, then you collect and analyze the output files
it's pretty uninvasive
I've never successfully gone lower than 50 megs
for anything near real
I've never tried to limit clojure CPU usage
We set memory limits via jvm flags (`-Xmx2G`) after observing our services running in Docker after a week
because we're still on java8 , after moving to java11 you can use docker limits to control heap size
so using docker's own logging / metrics to pick the limit?
yeah, we have a tiny service (in python) which scrapes container CPU and memory usage and reports to statsd->collectd->stackdriver
I don't have any experience with prometheus (just read things about it) so can't really compare. But coming from statsd+graphite world, Stackdriver is quite nice - it supports tagged metrics, collectd has native support for it and it just works. UI isn't great though, I'd prefer to use grafana
If you're into running your own stuff, then yeah - it will be cheaper. If you have stuff to do other than ops then having a hosted service is cheaper 😉
one thing to note: setting the heap size is not going to control the total jvm memory usage
You'd have to run on Java 11 (or 10 I think) - you'll have to dig around the internet to find out the version + jvm flags combination
btw @kwladyka you might want to deploy your clojure apps as uberjars as you can control more aspects of the running process via JVM flags (and your container will only need the JVM, no clojure tooling required)
@kwladyka Several of our production processes run with 1G heap but we have a couple that require more to run smoothly. Our public REST API Server has a 4G heap (although, last I looked, I think it could run happily in 2G maybe?). Most of those running with 1G could go lower but, hey, memory is cheap 🙂
@seancorfield it is cheap as long as you don’t pay for it from your own money 😉
We build uberjars for deployment but do not AOT anything. So we run with java -cp /path/to/the.jar clojure.main -m our.entry.point
which starts Clojure and runs our.entry.point/-main
. Plus a bunch of JVM options of course 🙂
clojure does a bunch of things you don't need to do in prod - you already know your classpath by the time you've deployed (and if not it's as simple as running a clj command line and caching the output)
because clojure is just a shell script for running the jvm, and it has no useful features for prod
(beyond knowing the classpath, which is easy to cache when deploying and reuse)
@seancorfield I think we were talking in the past about clojure
> java
, because of this AOT issues and best practice. Am I right?
We run everything with clj
/`clojure` locally in dev/test, but we build uberjars (with clojure
!) to run on QA/production. They are just easier to deal with because a) it’s just one artifact rather than a whole source tree b) they already have all their dependencies baked in so startup doesn’t need to check the server’s local Maven cache and maybe fetch dependencies from Maven/Clojars.
@kwladyka How much traffic are you going to have? I bet you could start with a 256M heap and see how it goes.
@kwladyka there were at least two recent discussions related to this recently, although not specific to docker. And I don't think your issue is really docker-specific. Here I'm dumping several resources that may help: ----------------------------------------------------------- 5 tips for proper Java Heap size: https://www.javacodegeeks.com/2012/07/5-tips-for-proper-java-heap-size.html * this may give you some hints on estimating proper heap size for your java app * Check also How to estimate memory consumption? https://plumbr.io/blog/memory-leaks/how-to-estimate-memory-consumption * Other resources * https://dzone.com/articles/hardware-sizing-for-javajee-products * https://stackoverflow.com/questions/35555084/finding-memory-requirements-of-java-application * https://softwareengineering.stackexchange.com/questions/206878/how-can-i-determine-the-minimum-requirements-of-a-piece-of-software * How well does Clojure perform when it comes to memory footprint? https://stackoverflow.com/questions/4058430/how-well-does-clojure-perform-when-it-comes-to-memory-footprint * Is Clojure too heavy-weight for a hobby project? https://www.reddit.com/r/Clojure/comments/7cgywh/is_clojure_too_heavyweight_for_a_hobby_project/ * some real numbers from testing Clojure microservice
What others mentioned about docker is that with Java 10 there's a better support for configuring heap for apps running inside containers via XX:+UseContainerSupport
-> see https://medium.com/adorsys/jvm-memory-settings-in-a-container-environment-64b0840e1d9e
Thank you all. I will try it. Just thinking if I will achieve something valuable spending time for all this analysis vs set it high and change after a week on production. 😉
and with JVM args you can fine tune it - or let JVM do it for you based on some "goals" you specify
but sure, you may well to start without using any JVM args and let it to use whatever it likes (usually 1/4 of available memory for the heap) You may end up with under-utilized machine though
Isn't it the case that if you had a container/VM/whatever with a 1G limit for the memory used by all processes running on it, and you ran a JVM with a max heap of 2G, then as soon as that JVM process tried to allocate over 1G, the OS would kill that JVM process?
Well I have never done it before in that way. Analysis all segments and set limitations for it sounds like I can easy make a mistake and do it worst, than not touch it 🙂
@U0CMVHBL2 Depends on the "overcommit" settings I think
But if you start the JVM with a heap limit maybe a bit less than 1G, then the OS should never kill it for that reason, and the JVM's GC should start collecting if it ever reaches that JVM heap limit (and likely GC at other times, too).
hmm not sure, I didn’t have such issue. OS never killed my app because of it. At least I think so 🙂
It is pretty easy to make that situation happen, with the right settings in OS and JVM max heap size 🙂
The JVM always has a max heap size, whether you set one explicitly, or not. If not, the JVM has ways of calculating one for you.
So…. how can you be confident to set all this limitations about heap etc. for Java process?
You could waste GBs of memory on a big machine with long-running application that generates lot of objects even for fiarly trivial app. But people don't usually deploy a trivial service on 244 GB machine. I think for your use case it may not be a well invested effort/time. Start simple, give it 512-1024M memory (limited by docker) and see how it goes
is there a way to define a constant field in a record? For example having (:name (MyRecord. _ _))
always return a constant name?
e.g. massively incomplete PoC
user=> (deftype T [m] clojure.lang.ILookup (valAt [_ k] (case k :name "foo" (get m k))))
user.T
user=> (:name (T. {:a 1}))
"foo"
the spec seems to suggest it must be a vector, but the docstring doesn’t say so, also clojuredocs gives no example of this
when I tried something I got this:
(memfn toString "")
Syntax error macroexpanding clojure.core/fn at (REPL:1:1).
("") - failed: Extra input at: [:fn-tail :arity-1 :params] spec: :clojure.core.specs.alpha/param-list
target144 - failed: vector? at: [:fn-tail :arity-n :params] spec: :clojure.core.specs.alpha/param-list
that’s why I thought the args should be a vectormakes a lot of sense. but the arg names cannot be used anywhere else than side the memfn
call right? they are just placeholders, without any re-use
> name may be type-hinted with the method receiver’s type in order to avoid reflective calls. I guess the docstring could be clearer on that
Any additional notes like those written down somewhere like here https://clojuredocs.org/clojure.core/memfn would be nice, if anyone felt so inclined.