Fork me on GitHub
#beginners
<
2020-11-05
>
valerauko14:11:04

I'm playing around with performance tuning and I don't know how to move forward. I'm load testing a script and it has a sensible min and mean. I don't really mind a high max because accidents happen. What I don't know how to deal with is a very high standard deviation. I'm seeing SD two to three times the mean, which considering these numbers are representing the script run time, doesn't sound good. How do I find out what part(s) of the code are prone to underperform in statistically significant ways like that? Clojure/JVM.

andy.fingerhut14:11:01

You can use a profiler, and/or appropriate Java APIs, to determine whether some of that variation is due to GC being run in some executions but not others.

valerauko16:11:42

can you recommend a profiler that can show me the "victims" of gc runs?

andy.fingerhut17:11:13

The YourKit Java profiler can show you in real time when GCs are occurring. I'm not sure what you mean by "victims".

andy.fingerhut17:11:11

There are Java API calls you can make to query how much total GC time has been spent up to that point in time. You could call that at times in your program's execution that you care to know whether GC occurred between those two times, and subtract the results.

andy.fingerhut17:11:51

This is a namespace I created for a personal project that contains a macro time that collects elapsed time and elapsed GC time before and after evaluating some expression: https://github.com/jafingerhut/cljol/blob/master/src/clj/cljol/performance.clj

valerauko02:11:36

I'm trying to figure out if the variation I see is because of GC or thread competition. Is there a way to see what calls get "paused" for GC?

andy.fingerhut03:11:36

Depending upon which GC algorithm you are using, it might be no calls, e.g. some GCs run in separate threads, without pausing any of your computation threads.

valerauko05:11:19

makes sense. I'll read more on the subject!

Jim Newton17:11:54

is there a syntax for explicitly prefixing a symbol with the current ns? for example. I have a ns named genus and I alias it to gns when I require it elsewhere. So outside of genus the symbols get prefixed as gns/foo. However, I would love sometimes to also use gns/foo INSIDE the file which defines the genus namespace. especially in rare cases when the project has two different functions of the same name but different namespaces. That would help when i'm navigating with M-. to avoid confusion.

dpsutton17:11:09

clj
Clojure 1.10.1
user=> (require '[clojure.set :as set])
nil
user=> [`set/bob `bob]
[clojure.set/bob user/bob]
user=>

dpsutton17:11:33

just addressing "is there a syntax for prefixing a symbol with the current ns?"

dpsutton17:11:50

but then you talked about aliases so not sure which you want but this seems to be both

andy.fingerhut17:11:41

You can use something like (alias 'alias-for-myself 'the.current.full.ns.name) after the ns form. At least that seemed to work with a quick REPL test -- I did not try it within a multi-file project. The doc string for alias does not say that it supports creating an alias to the current namespace, but it seems to work.

👍 1
andy.fingerhut17:11:56

Regarding the case when the current namespace has a function named foo, and another namespace has a function named foo, I think one workable solution used by some is never to use (:use ...), only (:require [full.ns :as alias]), so that whenever you want to call foo in another namespace, it must have either the full namespace or an alias qualifying it.

andy.fingerhut17:11:12

Then the local foo can always be named without a namespace or alias.

Jim Newton10:11:53

yes that's the problem I want to avoid.

Jim Newton10:11:09

if there is a low level function like *, such as when doing low level with with spec. then there are 3 * 's floating around. the normal clojure.core/**, spec/** , and my-local/* ... I'd really like to prefix all three of them within my code. That way when I popping around the different files with the editor, every time i see *, I know exactly which one it is. And this is especially true in the file which defines my-local/*

Jim Newton10:11:34

I'll try out the alias trick. is what you're claiming, @U0CMVHBL2, than whenever the ns is required elsewhere, then the alias will have no effect in the other ns?

Jim Newton10:11:21

seems to work. Great! thanks.

Jim Newton13:11:44

One small problem I've discovered with this approach is that the alias doesn't allow me to define a function using the / syntax.

(defn rte/match 
  "docstring"
  [a b c]
   ...)
the syntax checker with defn demands that the name be a simple symbol.
Syntax error macroexpanding clojure.core/defn at (rte_construct.clj:401:1).
rte/expand - failed: simple-symbol? at: [:fn-name] spec: :clojure.core.specs.alpha/defn-args

Jim Newton13:11:38

I wonder whether this is an over-ambitious syntax check, or wether it is really fundamentally important to prevent defn from working on qualified symbols. I've noticed that def doesn't have any such restriction. So I suppose I could use (def prefix/name ... (fn [a b c] ...)). I can live with the restriction, but just wondering whether there's really a good reason?

andy.fingerhut14:11:12

I am personally a bit surprised that def does not have the same restriction.

andy.fingerhut14:11:21

If you allow the alias or namespace qualifier for def or defn, but only allow it to be used for the current namespace, that is a strange thing to allow. If you allow it for other namespaces, then the idea of the "current namespace" is practically moot.

Jim Newton17:11:40

@U0CMVHBL2 i really think it is just an overzealous syntax check. that's my gut feel.

andy.fingerhut18:11:30

What is the behavior you would expect if a namespace qualifier were permitted?

andy.fingerhut18:11:54

One implementation detail: defn is a macro in Clojure, and thus can have specs implemented for it do implement these kinds of checks. def is not a macro, and so I believe cannot have Clojure specs implemented for this kind of checking, only checks implemented in the Java implementation of the Clojure compiler. The spec restrictions were added in Clojure 1.9, I believe, about 2 years ago, and were I suspect pretty carefully reviewed by the three main Clojure core developers (Rich, Stu, Alex) before release. (That is my guess -- I wasn't there writing specs with them).

andy.fingerhut18:11:33

But best way to find out if defn's check is more restrictive than intended is probably to ask in #clojure-dev or #clojure-spec

Daniel Östling19:11:01

Hello 🙂 I have a web page I wish to download, but it has a DOM-manipulating button for “give me all of the page” I need to “click” via code (clojure, not clojurescript), and I’m not sure how to do that best. Any suggestions?

Daniel Östling09:11:17

Thanks, I’ll take a look at that. I also played around a bit with https://github.com/tatut/clj-chrome-devtools a bit