This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2021-11-19
Channels
- # announcements (3)
- # asami (3)
- # babashka (39)
- # beginners (65)
- # calva (13)
- # cider (4)
- # clj-kondo (69)
- # cljdoc (19)
- # cljs-dev (2)
- # clojure (90)
- # clojure-dev (10)
- # clojure-europe (61)
- # clojure-france (15)
- # clojure-nl (8)
- # clojure-uk (2)
- # clojurescript (28)
- # conjure (2)
- # core-logic (4)
- # cursive (8)
- # datalevin (5)
- # datascript (7)
- # datomic (14)
- # depstar (4)
- # events (1)
- # graphql (7)
- # holy-lambda (5)
- # jobs (5)
- # kaocha (1)
- # malli (14)
- # membrane-term (13)
- # missionary (13)
- # nextjournal (6)
- # off-topic (1)
- # polylith (15)
- # portal (10)
- # re-frame (35)
- # reitit (1)
- # remote-jobs (3)
- # schema (3)
- # sci (121)
- # spacemacs (6)
- # tools-build (8)
- # tools-deps (74)
- # xtdb (7)
GC question: we've been using the G1 collector in production for ages with JDK 11. We're in the process of moving to JDK 17. For now, we're still using the G1 collector. What's the state of the art these days with Clojure, GC, and JDK 17? Any words of wisdom from folks who are on JDK 17 and not using G1 in production?
What are you optimizing for? Throughput, response times, or a balance? you can refer to the https://github.com/bsless/stress-server/tree/master/results/out for comparisons, but generally, G1 is very good. If you care about response latency, you should try testing Shenandoah and ZGC
@UK0810AQ2 Thanks. Once we have a decent baseline of 11 vs 17 with the same GC (G1), and we roll 17 onto a few more servers, we may try G1 vs ZGC. I'll have a read of your data (I'll probably paste the link into our JDK 17 testing ticket in Jira for future reference too).
Ping me before you plan to get into it, I might focus on one specific scenario and run it for a long period to get more accurate results
It will be Jetty, Middleware, Sync. I've just been looking at some of the .png
files for that with G1 vs ZGC. Doesn't look like there's huge differences? ZGC seems to have better latency until it falls over, but G1 seems to have slightly more gradual failures?
Yes, but I'm worried in the high percentiles even one bad request can throw the result off so I'll give each GC algorithm an hour in 10 and 30 qps
So far, our servers haven't come anywhere close to failure due to CPU overloading (in years) but occasionally we've seen memory leaks that have caused enough GC pressure to make servers unresponsive (solution: fix the memory leaks -- one of which was due to a JDK bug and we needed to upgrade the JDK!).
Yes, lots of stuff in New Relic 🙂
Doesn't always help if a server keels over in the middle of my night tho'...
Our data center folks have a different set of monitors and they restart processes for us if they trip those alarms. Very rare though.
one of the problems with restarting them is that you don't get to poke at them as they're catching fire
If you have enough fires, you'll get one during office hours so you can poke at it 🙂 We just don't have very many fires these days (we still shudder with memories of the frequent fires we used to have -- before we switched to Clojure).
Some of those are just the result of weird things happening at large scales. We once had a machine constantly disconnect from Kafka, which triggered constant rebalancing, which caused huge lags. Weird things always happen
True. We have one server in the cluster that periodically triggers one of our apps into heavy GC and it becomes unresponsive. The same JAR file running with the exact same configuration on all the other servers never does that -- only on this one server. No pattern to it, as far as we can tell. And none of the other JARs running on that server experience problems. As far as we can tell, everything about that server is identical to others that don't have this issue. Weird things indeed.
We had massive memory consumption from ZGC - you really need a very large heap for it to play with. That was our discovery anyway.
2GB feels like a fairly small heap to me. The only thing I've noticed so far after going from JDK 11->17 but keeping all the GC settings the same is marginally higher CPU usage by the G1 collection (from 0.1-0.2% to 0.2-0.3%) and that G1 survivor space seems to stay lower, so I guess G1 is a bit more aggressive in 17?
Different GC algorithms have different strategies. ZGC may need a bigger heap to be more effective
Interesting. Thank you, both. We only have one app that normally runs with a large heap. Most of our apps have only a 1GB heap (a couple have 1.5-2GB).
I feel like this has come up for me before, but cannot remember if there was a worthy work-around. As it stands now a record with a protocol implemented satisfies?
but an arbitrary object with method implementations added via with-meta
does not. Is there a reasonable substitute implementation of a predicate that would return true for the object that implements all the methods of a protocol via meta-data? https://flexiana.com/2021/08/on-the-nature-of-clojure-protocols summarizes better than I:
> Indeed, extending via metadata will not satisfy a protocol, it only makes the protocol’s functions work on an object with the correct metadata.
At our company we use a library which ships a satisfies?
, it's on http://github.com/nedap/speced.def
@john493 https://clojure.atlassian.net/browse/CLJ-2426 -- part of the issue is that extension via metadata allows for just a subset of the protocol to be implemented so it's not clear whether a given object's metadata "satisfies" the protocol or whether it just provides enough implementation for a given use case. My feeling is that if you're calling satisfies?
you are probably doing something wrong (in the same way that instance?
should also make you question your choices). Sometimes, you do need to do explicit type checking but I'd say that it is mostly a "code smell".
I can understand this perspective. It was being used in a precondition just to make sure the component being passed in was actually the intended one. Easy to pass the wrong one when working with nested components. Definitely not in the logic of any app.
It might, for this application, be enough to just write a predicate on the metadata to see if the methods are there.
(Somewhat related; satisfies?
isn’t particularly fast, so you might want to be careful in using it often)
That's an understatement. It's astronomically slow in relation to any other core function
somewhat random question, not intending to nerd snipe anyone today, but just curious, what do you think is the most “normal”/idiomatic/clojure way of rounding numbers? I’ve seen (format ".%2f" val)
or (-> val (* 100) Math/round (/ 100) double)
, but curious what other people do
(not exactly equivalent.. the first one still needs to be coerced back to a number, which makes it not ideal)
that is what I would say, if it is for display format is fine, otherwise the latter (using 100.0 for the mult and divide and without the call to double)
oh that’s smart! nice one.
That's an understatement. It's astronomically slow in relation to any other core function
or; even more evil; give it an exponentially increasing 💤
anyone have some time to have a discussion about a few issue im running into with clojure: • how do i set up a productive development environment? • java interop is difficult, how do I avoid using java libraries? • im not used to dynamic typing, is there a solution? expanding on this more, im having a lot of fun with clojure but the lack of tooling (or my knowledge of tooling) is causing a lot of difficulties for me. im also running into issue with interop causing a lot of hacky code, and dynamic typing is difficult for me, Im looking at some of my old code and I dont seem to fully understand it until runtime
Happy to discuss, but #beginners might be a better channel for that. What sort of IDE are you already familiar with?
thanks, didn't notice that channel, might ask there as well. currently im using vscode with calva as the plugin for writing clojure and starting the repl
it seems to work pretty well, it has linting, syntax highlighting, and code completion. debugging doesn't seem well developed in calva though compared to these other features
I'm used to using a debugger to find and resolve some bugs, but ive heard some say a debugger is not needed and that repl + code structure + tests can solve the need for a debugger
So, you seem to be reasonably well set on the IDE front. Correct. A debugger is less important in a REPL driven environment.
As to Java interop, you should (re-)consider. There are many useful libraries in the java ecosystem. Some of them have Clojure wrappers. Even then, for many Java libs, direct interop is probably more common.
You might find spec or malli useful if dynamic typing is uncomfortable for you (I sympathize). They're not quite type systems, but they scratch many of the same itches.
thats a good point on dynamic typing, I am able to prototype much faster in the early stages. its later stages where I am getting a bit lost without the type system
Definitely endorse the use of spec - once you understand the problem you are trying to solve. REPLs and dynamic typing are great for exploratory programming - to help you understand the problem.
I'll look into spec, ive heard some mention it solves a lot of similar problems the type system solves
Good naming, good docstrings, and judicious use of specs should mostly solve the "wait what was this again?" problem. In practice I usually find I only really crave specs for really complex map structures.
Yes, both are also available on cljs
do you have Java experience?
docstrings are another thing I struggle with here, I'm used to using tools that analyze doc strings and generate documentation. docstrings in clojure tend to be less verbose and more to the point, but im used to verbose documentation and lots of comments
Ive programmed in java and javascript, so my comparisions with clojure and clojurescript are coming from that perspective
its seems clj/cljs is a completely different way of doing things and thinking than in java and js
nothing stopping you from being verbose in your docstrings
Aye. Much more verbose if you choose - https://martinklepsch.org/posts/writing-awesome-docstrings.html
this is exactly how I set up environment in cloud https://cto.wladyka.eu/posts/2021/best-devops-choices/
thanks, im looking into cljdoc and codox now, do you know if they support type annotations in docstrings? it seems in the clojure world types aren't typically documented in docstrings
@U029J729MUP im looking at malli now, this looks interesting and maybe what I need to tighten up some code. is malli only runtime checked, will there be a performance penalty?
I haven't looked into malli personally, I just know by reputation that it's well-regarded as an alternative to spec
spec's strategy is to turn checking off by default and let you opt in (for unit tests, REPL sessions, dev environments, etc.), and I would take an educated guess at malli following a similar strategy
any chance there is editor support to use spec or malli schemas to check for errors while developing?
Not sure what you mean by "type annotations", but there is some work to get specs into docs.
Generally you will only get feedback from spec (and probably also malli) when the code runs, whether in the repl or in a test. There's https://github.com/arohner/spectrum out there for trying to do some of this statically, but it's extremely experimental.
> thanks, im looking into cljdoc and codox now, do you know if they support type annotations in docstrings?
If you mean things like ^String
, I would be extremely surprised if they did. Type hints are meant for the compiler, not for the developer. If you're trying to use them like static types, you're going to be disappointed and you're going to run into some frustrating cases.
clj-kondo (leveraged by Calva) is also a good tool for pointing out errors that are often found by IDEs via static type analysis and linting.
For myself, a productive development environment is having tooling that supports my repl driven development approach. https://practical.li/clojure/repl-driven-devlopment.html I also use clojure.spec when data crosses boundaries (database, http, JSON, etc) or to document non-trivial data structures in Clojure.
I added gen-class
call to a ns. I expected that when my ns gets loaded this class is added to classloader, but that doesn’t seem to be the case. What am I missing?