Fork me on GitHub
#clojure
<
2021-11-19
>
seancorfield15:11:49

GC question: we've been using the G1 collector in production for ages with JDK 11. We're in the process of moving to JDK 17. For now, we're still using the G1 collector. What's the state of the art these days with Clojure, GC, and JDK 17? Any words of wisdom from folks who are on JDK 17 and not using G1 in production?

Ben Sless17:11:16

What are you optimizing for? Throughput, response times, or a balance? you can refer to the https://github.com/bsless/stress-server/tree/master/results/out for comparisons, but generally, G1 is very good. If you care about response latency, you should try testing Shenandoah and ZGC

Ben Sless17:11:13

G1 generally offers a good balance

Ben Sless17:11:21

You should also try out GraalVMs JDK for its new JIT

seancorfield17:11:50

@UK0810AQ2 Thanks. Once we have a decent baseline of 11 vs 17 with the same GC (G1), and we roll 17 onto a few more servers, we may try G1 vs ZGC. I'll have a read of your data (I'll probably paste the link into our JDK 17 testing ticket in Jira for future reference too).

Ben Sless17:11:00

Ping me before you plan to get into it, I might focus on one specific scenario and run it for a long period to get more accurate results

seancorfield18:11:05

It will be Jetty, Middleware, Sync. I've just been looking at some of the .png files for that with G1 vs ZGC. Doesn't look like there's huge differences? ZGC seems to have better latency until it falls over, but G1 seems to have slightly more gradual failures?

Ben Sless18:11:29

Yes, but I'm worried in the high percentiles even one bad request can throw the result off so I'll give each GC algorithm an hour in 10 and 30 qps

seancorfield18:11:57

So far, our servers haven't come anywhere close to failure due to CPU overloading (in years) but occasionally we've seen memory leaks that have caused enough GC pressure to make servers unresponsive (solution: fix the memory leaks -- one of which was due to a JDK bug and we needed to upgrade the JDK!).

Ben Sless18:11:07

You don't have alerts on GC pauses and heap utilization?

seancorfield18:11:45

Yes, lots of stuff in New Relic 🙂

seancorfield18:11:25

Doesn't always help if a server keels over in the middle of my night tho'...

Ben Sless18:11:23

"Pager Duty Alert: you have one incident on ,,,," 😠

seancorfield18:11:24

Our data center folks have a different set of monitors and they restart processes for us if they trip those alarms. Very rare though.

Ben Sless18:11:57

one of the problems with restarting them is that you don't get to poke at them as they're catching fire

seancorfield18:11:06

If you have enough fires, you'll get one during office hours so you can poke at it 🙂 We just don't have very many fires these days (we still shudder with memories of the frequent fires we used to have -- before we switched to Clojure).

Ben Sless18:11:32

Some of those are just the result of weird things happening at large scales. We once had a machine constantly disconnect from Kafka, which triggered constant rebalancing, which caused huge lags. Weird things always happen

seancorfield18:11:21

True. We have one server in the cluster that periodically triggers one of our apps into heavy GC and it becomes unresponsive. The same JAR file running with the exact same configuration on all the other servers never does that -- only on this one server. No pattern to it, as far as we can tell. And none of the other JARs running on that server experience problems. As far as we can tell, everything about that server is identical to others that don't have this issue. Weird things indeed.

ghadi16:11:57

it's prod ready as of JDK 15

ghadi16:11:13

but got much much sweeter in JDK 16

dharrigan16:11:21

We tried the ZGC at work. It didn't work out very well for us

dharrigan16:11:28

We went back to the default, which is the G1

dharrigan16:11:54

We had massive memory consumption from ZGC - you really need a very large heap for it to play with. That was our discovery anyway.

ghadi16:11:19

out of curiosity which jdk version + heapsize?

dharrigan16:11:17

heapsize, about 2GB

seancorfield17:11:02

2GB feels like a fairly small heap to me. The only thing I've noticed so far after going from JDK 11->17 but keeping all the GC settings the same is marginally higher CPU usage by the G1 collection (from 0.1-0.2% to 0.2-0.3%) and that G1 survivor space seems to stay lower, so I guess G1 is a bit more aggressive in 17?

Ben Sless17:11:50

Different GC algorithms have different strategies. ZGC may need a bigger heap to be more effective

ghadi16:11:42

mem usage is one of the ZGC improvements for JDK 17

seancorfield16:11:03

Interesting. Thank you, both. We only have one app that normally runs with a large heap. Most of our apps have only a 1GB heap (a couple have 1.5-2GB).

John Conti17:11:38

I feel like this has come up for me before, but cannot remember if there was a worthy work-around. As it stands now a record with a protocol implemented satisfies? but an arbitrary object with method implementations added via with-meta does not. Is there a reasonable substitute implementation of a predicate that would return true for the object that implements all the methods of a protocol via meta-data? https://flexiana.com/2021/08/on-the-nature-of-clojure-protocols summarizes better than I: > Indeed, extending via metadata will not satisfy a protocol, it only makes the protocol’s functions work on an object with the correct metadata.

thumbnail16:11:18

At our company we use a library which ships a satisfies?, it's on http://github.com/nedap/speced.def

1
seancorfield17:11:45

@john493 https://clojure.atlassian.net/browse/CLJ-2426 -- part of the issue is that extension via metadata allows for just a subset of the protocol to be implemented so it's not clear whether a given object's metadata "satisfies" the protocol or whether it just provides enough implementation for a given use case. My feeling is that if you're calling satisfies? you are probably doing something wrong (in the same way that instance? should also make you question your choices). Sometimes, you do need to do explicit type checking but I'd say that it is mostly a "code smell".

John Conti17:11:15

I can understand this perspective. It was being used in a precondition just to make sure the component being passed in was actually the intended one. Easy to pass the wrong one when working with nested components. Definitely not in the logic of any app.

John Conti17:11:57

It might, for this application, be enough to just write a predicate on the metadata to see if the methods are there.

Lennart Buit17:11:18

(Somewhat related; satisfies? isn’t particularly fast, so you might want to be careful in using it often)

Ben Sless17:11:57

That's an understatement. It's astronomically slow in relation to any other core function

Kira McLean17:11:07

somewhat random question, not intending to nerd snipe anyone today, but just curious, what do you think is the most “normal”/idiomatic/clojure way of rounding numbers? I’ve seen (format ".%2f" val) or (-> val (* 100) Math/round (/ 100) double) , but curious what other people do

Kira McLean17:11:03

(not exactly equivalent.. the first one still needs to be coerced back to a number, which makes it not ideal)

hiredman17:11:51

that is what I would say, if it is for display format is fine, otherwise the latter (using 100.0 for the mult and divide and without the call to double)

Kira McLean17:11:28

oh that’s smart! nice one.

Ben Sless17:11:57

That's an understatement. It's astronomically slow in relation to any other core function

ghadi17:11:23

We should make it slower to discourage people using it

ghadi17:11:53

Give it a nice 50ms 💤

😄 2
Lennart Buit17:11:18

or; even more evil; give it an exponentially increasing 💤

Ben Sless17:11:20

At least emit a warning

Ben Sless18:11:40

Which can't be turned off

dwm82319:11:20

anyone have some time to have a discussion about a few issue im running into with clojure: • how do i set up a productive development environment? • java interop is difficult, how do I avoid using java libraries? • im not used to dynamic typing, is there a solution? expanding on this more, im having a lot of fun with clojure but the lack of tooling (or my knowledge of tooling) is causing a lot of difficulties for me. im also running into issue with interop causing a lot of hacky code, and dynamic typing is difficult for me, Im looking at some of my old code and I dont seem to fully understand it until runtime

dorab20:11:47

Happy to discuss, but #beginners might be a better channel for that. What sort of IDE are you already familiar with?

dwm82320:11:43

thanks, didn't notice that channel, might ask there as well. currently im using vscode with calva as the plugin for writing clojure and starting the repl

dwm82320:11:39

it seems to work pretty well, it has linting, syntax highlighting, and code completion. debugging doesn't seem well developed in calva though compared to these other features

dwm82320:11:55

I'm used to using a debugger to find and resolve some bugs, but ive heard some say a debugger is not needed and that repl + code structure + tests can solve the need for a debugger

dorab20:11:48

So, you seem to be reasonably well set on the IDE front. Correct. A debugger is less important in a REPL driven environment.

dorab20:11:50

As to Java interop, you should (re-)consider. There are many useful libraries in the java ecosystem. Some of them have Clojure wrappers. Even then, for many Java libs, direct interop is probably more common.

dorab20:11:14

Dynamic typing frees you up to explore much faster.

Colin P. Hill20:11:27

You might find spec or malli useful if dynamic typing is uncomfortable for you (I sympathize). They're not quite type systems, but they scratch many of the same itches.

dwm82320:11:04

thats a good point on dynamic typing, I am able to prototype much faster in the early stages. its later stages where I am getting a bit lost without the type system

dorab20:11:22

Definitely endorse the use of spec - once you understand the problem you are trying to solve. REPLs and dynamic typing are great for exploratory programming - to help you understand the problem.

dwm82320:11:56

I'll look into spec, ive heard some mention it solves a lot of similar problems the type system solves

dwm82320:11:13

do you know if spec or malli supports clojurescript as well?

Colin P. Hill20:11:14

Good naming, good docstrings, and judicious use of specs should mostly solve the "wait what was this again?" problem. In practice I usually find I only really crave specs for really complex map structures.

Colin P. Hill20:11:02

Yes, both are also available on cljs

dwm82320:11:46

good to know, I will look into both of these

roklenarcic20:11:45

do you have Java experience?

dwm82320:11:15

docstrings are another thing I struggle with here, I'm used to using tools that analyze doc strings and generate documentation. docstrings in clojure tend to be less verbose and more to the point, but im used to verbose documentation and lots of comments

dwm82320:11:54

Ive programmed in java and javascript, so my comparisions with clojure and clojurescript are coming from that perspective

dwm82320:11:52

its seems clj/cljs is a completely different way of doing things and thinking than in java and js

Colin P. Hill20:11:00

nothing stopping you from being verbose in your docstrings

dorab20:11:41

Look at cljdoc and codox for generating documentation from docstrings.

kwladyka20:11:43

this is exactly how I set up environment in cloud https://cto.wladyka.eu/posts/2021/best-devops-choices/

kwladyka20:11:34

for static frontend pages I use github pages

dwm82320:11:45

thanks, im looking into cljdoc and codox now, do you know if they support type annotations in docstrings? it seems in the clojure world types aren't typically documented in docstrings

dwm82320:11:58

although sometimes its inferred by variable names like s

dwm82320:11:31

thanks for the link, looking at this now

dwm82320:11:06

@U029J729MUP im looking at malli now, this looks interesting and maybe what I need to tighten up some code. is malli only runtime checked, will there be a performance penalty?

Colin P. Hill20:11:41

I haven't looked into malli personally, I just know by reputation that it's well-regarded as an alternative to spec

Colin P. Hill20:11:10

spec's strategy is to turn checking off by default and let you opt in (for unit tests, REPL sessions, dev environments, etc.), and I would take an educated guess at malli following a similar strategy

dwm82320:11:32

any chance there is editor support to use spec or malli schemas to check for errors while developing?

dorab20:11:58

Not sure what you mean by "type annotations", but there is some work to get specs into docs.

1
dorab20:11:01

Look at instrument in spec - to check for errors while developing.

Colin P. Hill20:11:36

Generally you will only get feedback from spec (and probably also malli) when the code runs, whether in the repl or in a test. There's https://github.com/arohner/spectrum out there for trying to do some of this statically, but it's extremely experimental.

Colin P. Hill20:11:08

> thanks, im looking into cljdoc and codox now, do you know if they support type annotations in docstrings? If you mean things like ^String, I would be extremely surprised if they did. Type hints are meant for the compiler, not for the developer. If you're trying to use them like static types, you're going to be disappointed and you're going to run into some frustrating cases.

cdeszaq20:11:25

clj-kondo (leveraged by Calva) is also a good tool for pointing out errors that are often found by IDEs via static type analysis and linting.

2
practicalli-johnny09:11:48

For myself, a productive development environment is having tooling that supports my repl driven development approach. https://practical.li/clojure/repl-driven-devlopment.html I also use clojure.spec when data crosses boundaries (database, http, JSON, etc) or to document non-trivial data structures in Clojure.

roklenarcic20:11:03

I added gen-class call to a ns. I expected that when my ns gets loaded this class is added to classloader, but that doesn’t seem to be the case. What am I missing?

hiredman20:11:03

gen-class only does anything when aot compiling

ribelo22:11:42

can I somehow implement notifyWatches for custom type?

ribelo22:11:03

in cljs it is possible