Fork me on GitHub
#clojure
<
2019-06-14
>
noisesmith00:06:38

it does a lot of the things we like

(ins)user=> (def h (java.util.LinkedHashMap.))
#'user/h
(ins)user=> (.put h :a 0)
nil
(ins)user=> h
{:a 0}
(ins)user=> (:a h)
0
(cmd)user=> (.put h :b [:c :d])
[:c :d]
(cmd)user=> (get-in h [:b 0])
:c

noisesmith00:06:15

but not others

(ins)user=> (h :a)
Execution error (ClassCastException) at user/eval323 (REPL:1).
java.util.LinkedHashMap cannot be cast to clojure.lang.IFn

lilactown00:06:46

(def validate-attr-re #"^[A-Za-z][A-Za-z\-.0-9]")

(re-matches validate-attr-re "asdf") ;; => nil
what am I doing wrong here?

lilactown00:06:04

I need to know if a string is valid based on a regex

lilactown00:06:47

ah. needs a +

Lennart Buit05:06:58

or a * if you intend to match java style identifiers

andy.fingerhut01:06:48

There is a Clojure lib containing an ordered-map that preserves insertion order, too, and is more Clojurey: https://github.com/clj-commons/ordered

the2bears01:06:05

If it's "given that I have list of the keys (in the right order" then read from the list, pulling the values in the order of the list. Or am I missing something here? Do you want all the values in one call?

dharrigan10:06:50

#announcements would be a good place.

👍 4
pinkfrog13:06:58

Modulus of num and div. Truncates toward negative infinity.

pinkfrog13:06:05

what does this mean “Truncates toward negative infinity.”

pinkfrog13:06:31

(mod -2 3) is 1 not a negative number

bronsa14:06:00

truncates towards, meaning -inf,..,-2,-1,0,1,2,..,+inf it truncates in <- direction

aisamu14:06:34

That terminology also applies to rounding (e.g. https://en.wikipedia.org/wiki/Rounding#Round_half_down) > round half down (or round half towards negative infinity)

bronsa14:06:08

in simpler words it means that if it truncates to the smaller integer rather than to the biggest integer

bronsa14:06:24

1.8 -> 1 instead of 2

pinkfrog14:06:24

it supports floats, i see.

pinkfrog14:06:11

hmmm. still didn’t got it. any example?

pinkfrog14:06:27

i am aware of the ceiling, flooring meaning.

dpsutton14:06:06

can you say which exact function you've read the "Truncates toward negative infinity" so we can craft examples to demonstrate it?

dpsutton14:06:54

cljs.user> (rem -2 1000)
-2
cljs.user> (mod -2 1000)
998
are the classic examples i think

manutter5114:06:20

“Truncates toward negative infinity” means if you write your numbers with negative infinity on the left and positive infinity on the right, truncation will move to the nearest integer to the left of your floating point number.

pinkfrog14:06:54

(mod -2 1000)

pinkfrog14:06:25

i didn’t see the truncation. (mod 5.1 1000)

pinkfrog14:06:31

gives 5.1, not 5

dpsutton14:06:43

https://biblio.ugent.be/publication/314490/file/452146.pdf big pdf but check section 2.3 Div-Dominant Definitions

g17:06:35

hey everyone, how can i supply a custom implementation of keys for a deftype based on a map?

noisesmith17:06:58

there are multiple interfaces needed to implement a clojure persistent hash map, one solution would be to implement them all, with most of them delegating directly to an underlying hash-map

noisesmith17:06:50

there's no simple built in way to do this, but there's a lib that tries to make it easier https://github.com/ztellman/potemkin "def-map-type"

noisesmith17:06:08

the summary for that lib is a good caveat :D

noisesmith17:06:40

user=> (pprint (supers (class {})))
#{java.lang.Object clojure.lang.MapEquivalence java.io.Serializable
  clojure.lang.IEditableCollection clojure.lang.ILookup
  clojure.lang.Associative java.lang.Iterable clojure.lang.IObj
  clojure.lang.APersistentMap java.util.Map clojure.lang.Counted
  clojure.lang.Seqable clojure.lang.IFn clojure.lang.IMapIterable
  java.util.concurrent.Callable clojure.lang.IHashEq
  clojure.lang.IKVReduce clojure.lang.IMeta clojure.lang.AFn
  java.lang.Runnable clojure.lang.IPersistentCollection
  clojure.lang.IPersistentMap}
nil

noisesmith17:06:50

that's a lot to implement by hand

kwladyka17:06:17

How do you set limits (CPU / MEM) for you Clojure applications in kubernetes / docker? What is your method to determine min,max CPU / MEM usage?

noisesmith17:06:42

I've never set a CPU limit, but mem limits are just an arg to java

noisesmith17:06:10

and if you care about mem enough to limit it, definitely start your app with java rather than some build tool

kwladyka17:06:34

I am running everything in kubernetes so in docker containers

kwladyka17:06:52

by clojure command

hiredman17:06:54

I believer newer jvms detect that they are running in docker and size themselves (memory wise) according to the container limits

☝️ 4
noisesmith17:06:56

OK - the docker part doesn't really change anything here (other than the ability to set limits to docker itself I guess), and I'd advocate for using clojure to build a classpath then running java itself using that classpath

dpsutton17:06:22

9+ right? 8 and below looked at host rather than container?

hiredman17:06:43

something like that

kwladyka17:06:47

so backing to my question: still I don’t know how to set resources limits 🙂

noisesmith18:06:12

there's extensive documentation for java itself regarding how it allocates and manages memory

noisesmith18:06:22

between the man pages, and oracle docs

kwladyka18:06:43

I was thinking more about container limitation

kwladyka18:06:04

So I have to “guess” what is min and max for app

kwladyka18:06:12

but not really sure how to do it right

kwladyka18:06:36

only one way which come to my mind: start with something and if it is too low increase 😉

noisesmith18:06:05

oh so you are asking how to pick a number?

kwladyka18:06:18

sorry if it wasn’t clear

noisesmith18:06:25

OK, I misread, and thought you were asking how to impose that limit

noisesmith18:06:45

use a profiler and look at resource usage under something near typical load

noisesmith18:06:05

visualvm, yourkit, hprof are all good options depending on what kind of UI you like

kwladyka18:06:23

In practice I will never have the same load as in production

noisesmith18:06:26

surely you can create incoming load?

kwladyka18:06:28

my idea is to deploy with high limitation and see how many resources it takes. After some times (a few days?) set limits based on metrics

noisesmith18:06:49

you can run hprof or some other metric collecting agent in prod, sure

kwladyka18:06:26

just not really sure how people do it and I would like to hear how others do it in everyday practice 🙂

noisesmith18:06:48

profiling is how I do it

kwladyka18:06:03

so profiling on your private computer first?

kwladyka18:06:22

or in real environment?

noisesmith18:06:25

first there, then in a uat/staging env

kwladyka18:06:38

ok thx 🍻

noisesmith18:06:40

usually by then I have a good idea of what the upper limit should be

noisesmith18:06:04

with hprof you just add an agent to the vm on startup, no gui or extra program needed, then you collect and analyze the output files

noisesmith18:06:11

it's pretty uninvasive

kwladyka18:06:15

What is min in your experience? like for hello world with ring?

noisesmith18:06:30

I've never successfully gone lower than 50 megs

noisesmith18:06:36

for anything near real

kwladyka18:06:47

wow 50 MB is super low for Clojure 😉

kwladyka18:06:51

what about CPU?

noisesmith18:06:08

I've never tried to limit clojure CPU usage

kwladyka18:06:04

thanks for sharing how you do it

kwladyka18:06:58

Anybody doing it differently and want to share how?

lukasz18:06:06

We set memory limits via jvm flags (`-Xmx2G`) after observing our services running in Docker after a week

lukasz18:06:43

because we're still on java8 , after moving to java11 you can use docker limits to control heap size

noisesmith18:06:43

so using docker's own logging / metrics to pick the limit?

lukasz18:06:07

yeah, we have a tiny service (in python) which scrapes container CPU and memory usage and reports to statsd->collectd->stackdriver

kwladyka18:06:30

How would you compare stackdriver vs prometheus-operator?

lukasz18:06:14

I don't have any experience with prometheus (just read things about it) so can't really compare. But coming from statsd+graphite world, Stackdriver is quite nice - it supports tagged metrics, collectd has native support for it and it just works. UI isn't great though, I'd prefer to use grafana

kwladyka18:06:51

I heard it is 10-100x more expensive then prometheus-operator 😉

kwladyka18:06:03

But I don’t have my own experience about it

lukasz18:06:44

If you're into running your own stuff, then yeah - it will be cheaper. If you have stuff to do other than ops then having a hosted service is cheaper 😉

lukasz18:06:12

one thing to note: setting the heap size is not going to control the total jvm memory usage

lukasz18:06:30

there's off-heap data, memory pre-allocated by the jvm for scheduling etc

lukasz18:06:43

so you can set the heap to 50M and the jvm process will still consume 250M

kwladyka18:06:09

hmm I was thinking to set limitations only for container

lukasz18:06:57

You'd have to run on Java 11 (or 10 I think) - you'll have to dig around the internet to find out the version + jvm flags combination

lukasz18:06:03

btw @kwladyka you might want to deploy your clojure apps as uberjars as you can control more aspects of the running process via JVM flags (and your container will only need the JVM, no clojure tooling required)

seancorfield18:06:27

@kwladyka Several of our production processes run with 1G heap but we have a couple that require more to run smoothly. Our public REST API Server has a 4G heap (although, last I looked, I think it could run happily in 2G maybe?). Most of those running with 1G could go lower but, hey, memory is cheap 🙂

kwladyka18:06:12

@seancorfield it is cheap as long as you don’t pay for it from your own money 😉

seancorfield18:06:40

We build uberjars for deployment but do not AOT anything. So we run with java -cp /path/to/the.jar clojure.main -m our.entry.point which starts Clojure and runs our.entry.point/-main. Plus a bunch of JVM options of course 🙂

8
🍻 4
noisesmith18:06:10

best way to run clojure

4
kwladyka18:06:57

hmm so not clear clojure command? Is it real difference?

noisesmith18:06:22

clojure does a bunch of things you don't need to do in prod - you already know your classpath by the time you've deployed (and if not it's as simple as running a clj command line and caching the output)

kwladyka18:06:26

I mean are you able to compare it why choose JVM over clojure

noisesmith18:06:51

because clojure is just a shell script for running the jvm, and it has no useful features for prod

noisesmith18:06:15

(beyond knowing the classpath, which is easy to cache when deploying and reuse)

kwladyka18:06:08

@seancorfield I think we were talking in the past about clojure > java, because of this AOT issues and best practice. Am I right?

kwladyka18:06:13

But the context was different 🙂

seancorfield18:06:09

We run everything with clj/`clojure` locally in dev/test, but we build uberjars (with clojure!) to run on QA/production. They are just easier to deal with because a) it’s just one artifact rather than a whole source tree b) they already have all their dependencies baked in so startup doesn’t need to check the server’s local Maven cache and maybe fetch dependencies from Maven/Clojars.

👍 4
seancorfield18:06:14

@kwladyka How much traffic are you going to have? I bet you could start with a 256M heap and see how it goes.

kwladyka18:06:28

I am starting SaaS, I don’t have even finished app, so 0 😉

kwladyka18:06:51

But I want to pass through all things to do it right to learn things

jumar18:06:09

@kwladyka there were at least two recent discussions related to this recently, although not specific to docker. And I don't think your issue is really docker-specific. Here I'm dumping several resources that may help: ----------------------------------------------------------- 5 tips for proper Java Heap size: https://www.javacodegeeks.com/2012/07/5-tips-for-proper-java-heap-size.html * this may give you some hints on estimating proper heap size for your java app * Check also How to estimate memory consumption? https://plumbr.io/blog/memory-leaks/how-to-estimate-memory-consumption * Other resources * https://dzone.com/articles/hardware-sizing-for-javajee-products * https://stackoverflow.com/questions/35555084/finding-memory-requirements-of-java-application * https://softwareengineering.stackexchange.com/questions/206878/how-can-i-determine-the-minimum-requirements-of-a-piece-of-software * How well does Clojure perform when it comes to memory footprint? https://stackoverflow.com/questions/4058430/how-well-does-clojure-perform-when-it-comes-to-memory-footprint * Is Clojure too heavy-weight for a hobby project? https://www.reddit.com/r/Clojure/comments/7cgywh/is_clojure_too_heavyweight_for_a_hobby_project/ * some real numbers from testing Clojure microservice

🍻 4
jumar18:06:55

What others mentioned about docker is that with Java 10 there's a better support for configuring heap for apps running inside containers via XX:+UseContainerSupport -> see https://medium.com/adorsys/jvm-memory-settings-in-a-container-environment-64b0840e1d9e

kwladyka18:06:03

Thank you all. I will try it. Just thinking if I will achieve something valuable spending time for all this analysis vs set it high and change after a week on production. 😉

jumar18:06:24

sure, depends how much do you wanna "play" with it

kwladyka18:06:18

So while you can limit CPU / MEM for container why you do it extra using java?

jumar18:06:32

because former would limit the whole process

jumar18:06:49

But JVM process has multiple memory segments, notable the heap

jumar18:06:07

and with JVM args you can fine tune it - or let JVM do it for you based on some "goals" you specify

jumar18:06:10

but sure, you may well to start without using any JVM args and let it to use whatever it likes (usually 1/4 of available memory for the heap) You may end up with under-utilized machine though

andy.fingerhut18:06:11

Isn't it the case that if you had a container/VM/whatever with a 1G limit for the memory used by all processes running on it, and you ran a JVM with a max heap of 2G, then as soon as that JVM process tried to allocate over 1G, the OS would kill that JVM process?

kwladyka18:06:52

Well I have never done it before in that way. Analysis all segments and set limitations for it sounds like I can easy make a mistake and do it worst, than not touch it 🙂

jumar18:06:54

@U0CMVHBL2 Depends on the "overcommit" settings I think

andy.fingerhut18:06:29

But if you start the JVM with a heap limit maybe a bit less than 1G, then the OS should never kill it for that reason, and the JVM's GC should start collecting if it ever reaches that JVM heap limit (and likely GC at other times, too).

kwladyka18:06:08

hmm not sure, I didn’t have such issue. OS never killed my app because of it. At least I think so 🙂

andy.fingerhut18:06:55

It is pretty easy to make that situation happen, with the right settings in OS and JVM max heap size 🙂

kwladyka18:06:41

oh but I don’t use this extra parameters when run java to set limits

kwladyka18:06:31

but use the newest Java 🙂

andy.fingerhut18:06:34

The JVM always has a max heap size, whether you set one explicitly, or not. If not, the JVM has ways of calculating one for you.

kwladyka18:06:13

Do you limit CPU usage for your apps?

kwladyka18:06:14

So…. how can you be confident to set all this limitations about heap etc. for Java process?

kwladyka18:06:32

I mean how can you be sure you are doing it right

kwladyka18:06:48

it sounds a little unknown

kwladyka18:06:18

or demand a lot of effort

jumar18:06:25

You cant: you can try, measure and tune.

kwladyka18:06:47

hmm and is it worth it vs not using this parameters?

kwladyka18:06:25

How much memory you can save with what cost of performance?

kwladyka18:06:48

this is the point for me 🙂

jumar18:06:57

It's very application dependent

kwladyka18:06:42

from your retrospection and own experience for apps which you did it? 🙂

kwladyka18:06:32

just you know… do you think if it is really worth to do it?

kwladyka18:06:53

or doing because you were doing it 🙂

jumar18:06:36

You could waste GBs of memory on a big machine with long-running application that generates lot of objects even for fiarly trivial app. But people don't usually deploy a trivial service on 244 GB machine. I think for your use case it may not be a well invested effort/time. Start simple, give it 512-1024M memory (limited by docker) and see how it goes

kwladyka18:06:19

thanks for your thoughts

dbernal19:06:28

is there a way to define a constant field in a record? For example having (:name (MyRecord. _ _)) always return a constant name?

bronsa19:06:19

you can using deftype if you care for reimplementing the lookup semantics yourself

bronsa19:06:51

e.g. massively incomplete PoC

user=> (deftype T [m] clojure.lang.ILookup (valAt [_ k]  (case k :name "foo" (get m k))))
user.T
user=> (:name (T. {:a 1}))
"foo"

dbernal19:06:30

Ah I see, thank you

borkdude19:06:01

how is memfn supposed to be used with the args argument?

bronsa19:06:23

the first rule of memfn is there's no point in using memfn

borkdude19:06:24

the spec seems to suggest it must be a vector, but the docstring doesn’t say so, also clojuredocs gives no example of this

borkdude19:06:32

I don’t want to use it, but analyze it

borkdude19:06:40

for linting

johnj19:06:02

lint error 😉

bronsa20:06:21

user=> ((memfn cons el) () 1)
(1)

bronsa20:06:51

user=> ((memfn toString) 1)
"1"

borkdude20:06:36

ah: ((memfn substring start end) "foo" 0 1)

bronsa20:06:40

this is implicit, args are explicit

bronsa20:06:17

type hinting the method name is the way to type hint the target

borkdude20:06:33

when I tried something I got this:

(memfn toString "")
Syntax error macroexpanding clojure.core/fn at (REPL:1:1).
("") - failed: Extra input at: [:fn-tail :arity-1 :params] spec: :clojure.core.specs.alpha/param-list
target144 - failed: vector? at: [:fn-tail :arity-n :params] spec: :clojure.core.specs.alpha/param-list
that’s why I thought the args should be a vector

bronsa20:06:45

so in your example

bronsa20:06:47

user=> ((memfn ^String substring start end) "foo" 0 1)
"f"

borkdude20:06:27

makes a lot of sense. but the arg names cannot be used anywhere else than side the memfn call right? they are just placeholders, without any re-use

bronsa20:06:44

apart from type hint targets

borkdude20:06:53

oh you can also hint those args

borkdude20:06:20

> name may be type-hinted with the method receiver’s type in order to avoid reflective calls. I guess the docstring could be clearer on that

bronsa20:06:21

one major gotcha is you can't type hint with primitives

borkdude20:06:41

I can see why this isn’t used a lot

bronsa20:06:43

another reason why memfn is completely useless

andy.fingerhut20:06:44

Any additional notes like those written down somewhere like here https://clojuredocs.org/clojure.core/memfn would be nice, if anyone felt so inclined.