This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2022-05-14
Channels
I would try removing the auth from the URL and using this instead, in addition to the proxy: https://github.com/dakrone/clj-http/blob/3ff32c3663c1f540076ec6ec16594e282a756c15/README.org#basic-auth
Also could I ask what is "Authorization: Basic <some string>" in the Header for? Is there some common standard for another layer of authorization?
Note however that even if that works, your credentials will be transferred over HTTP in a readable format. In order to use HTTPS, you'd have to add an implementation to this multimethod with some custom class: https://github.com/dakrone/clj-http/blob/5a5288319abab50141b8785625feca8de5a11dfc/src/clj_http/core.clj#L186
That's called basic auth, yes: https://en.wikipedia.org/wiki/Basic_access_authentication
Do folks use clojure.core/juxt
at all? I keep wanting to use it because it results in clean code, but whenever I run benchmarks, it’s noticeably slower than using a well-defined function.
Just to show that a superficial benchmark isn't in line with your statement:
=> (use 'criterium.core)
nil
=> (def f1 (juxt :a :b))
#'user/f1
=> (def f2 (fn [m] [(:a m) (:b m)]))
#'user/f2
=> (def m {:a 1 :b 2 :c 3 :d 4})
#'user/m
=> (quick-bench (f1 m))
Evaluation count : 35165796 in 6 samples of 5860966 calls.
Execution time mean : 17.930086 ns
Execution time std-deviation : 3.351104 ns
Execution time lower quantile : 15.312267 ns ( 2.5%)
Execution time upper quantile : 22.549222 ns (97.5%)
Overhead used : 1.723916 ns
nil
=> (quick-bench (f2 m))
Evaluation count : 19842672 in 6 samples of 3307112 calls.
Execution time mean : 24.931762 ns
Execution time std-deviation : 5.464756 ns
Execution time lower quantile : 20.791853 ns ( 2.5%)
Execution time upper quantile : 33.432507 ns (97.5%)
Overhead used : 1.723916 ns
nil
=>
And to answer your initial question - I personally do use it. :) Mostly when making id->field maps:
(into {}
(map (juxt :id :some-field))
coll-of-maps)
> How and what exactly did you benchmark?
I used criterium and quick-bench
in a similar fashion to your original example. But I’m mapping the function across a collection (similar to your second example) as opposed to just invoking the function once.
juxt results: 40.495425 µs
fn results: 17.620704 µs
This isn’t the first time I’ve seen differences like this though
I’m tempted to write a macro that basically writes that fn
form for me
Can you provide the actual benchmarking code? Because I can't reproduce that behavior even when using multiple keys with a collection of hash maps.
And your results aren't only surprising, but suspicious.
juxt
is basically that macro you're describing, only it's a function and not a macro. It shouldn't result in a perf drop.
(def xs (mapv (fn [x]
{:x x
:y (inc x)})
(range 100)))
(quick-bench
(->> xs
(map (juxt :x :y))
(doall)))
(quick-bench
(->> xs
(map (fn [m]
[(:x m) (:y m)]))
(doall)))
Results in: Evaluation count : 51444 in 6 samples of 8574 calls. Execution time mean : 8.226674 µs Execution time std-deviation : 3.430178 µs Execution time lower quantile : 4.137545 µs ( 2.5%) Execution time upper quantile : 11.415239 µs (97.5%) Overhead used : 1.984144 ns Evaluation count : 60156 in 6 samples of 10026 calls. Execution time mean : 4.447924 µs Execution time std-deviation : 162.639343 ns Execution time lower quantile : 4.263218 µs ( 2.5%) Execution time upper quantile : 4.620822 µs (97.5%) Overhead used : 1.984144 ns
Interesting. Can't reproduce it at all on Clojure 1.11.1 running on JVM 11 (that's a lot of 1s, heh).
Same exact picture if I use a more thorough bench
:
user=> (bench (->> xs (map (juxt :x :y)) (doall)))
Evaluation count : 22241040 in 60 samples of 370684 calls.
Execution time mean : 2.700029 µs
Execution time std-deviation : 22.607454 ns
Execution time lower quantile : 2.680236 µs ( 2.5%)
Execution time upper quantile : 2.771291 µs (97.5%)
Overhead used : 6.026732 ns
Found 6 outliers in 60 samples (10.0000 %)
low-severe 6 (10.0000 %)
Variance from outliers : 1.6389 % Variance is slightly inflated by outliers
nil
user=> (bench (->> xs (map (fn [m] [(:x m) (:y m)])) (doall)))
Evaluation count : 17696700 in 60 samples of 294945 calls.
Execution time mean : 3.361388 µs
Execution time std-deviation : 25.461269 ns
Execution time lower quantile : 3.341838 µs ( 2.5%)
Execution time upper quantile : 3.437842 µs (97.5%)
Overhead used : 6.026732 ns
Found 5 outliers in 60 samples (8.3333 %)
low-severe 2 (3.3333 %)
low-mild 3 (5.0000 %)
Variance from outliers : 1.6389 % Variance is slightly inflated by outliers
nil
Unless you're optimizing specifically for your setup - OS, JDK, Clojure version, whatever else might affect this - I wouldn't bother with replacing juxt
with anything else.Interesting. I’m also on JDK 17, Clojure 1.11.1. I have ZGC GC enabled, so maybe that’s impacting it?
Just tried with ZGC - same deal. There are also a few versions of JDK 17. I'm also running on Linux, which has its specific kernel versions and settings. There are many variables.
But suppose that criterium.core/bench
is indicative of real-life application performance (it might easily not be at all).
Why do you care that juxt
is twice as slow as fn
? Is it in a really tight spot and most of the time is eaten indeed by fn
/`juxt` and not by, say, iteration over a lazy collection created by map
?
Leiningen turns JIT off by default for a REPL. Any benchmarking done is effectively meaningless in that case
Heh, that's funny - on my end, clj
with -Xint
(which presumably disables JIT) starts slower than without it.
Yeah, sure. Which makes it even more weird that Lein decided to make it the default, when it can slow things down. Oh well, makes me even more content with deps.
> Why do you care that juxt
is twice as slow as fn
? Is it in a really tight spot and most of the time is eaten indeed by fn
/`juxt` and not by, say, iteration over a lazy collection created by map
?
The example I provided was a small, contrived dataset. My initial, local development use case was on a small subset of the production data. But for both cases, the juxt
version takes approximately twice as long as the fn
version.
> Iirc when I tested this assumption with Lein it held true So lein is doing JIT which causes some Clojure forms to be slower than they would otherwise be in a production runtime?
> But for both cases, the juxt
version takes approximately twice as long as the fn
version.
Right, but if it's 1% or 0.5% from the overall run time - why care? Or is it the choke point?
> So lein is doing JIT which
Seems like it's the opposite, at least potentially. JIT makes frequently running things faster.
Yeah, I’m not sure what was happening earlier, but I’m not able to reproduce the issue after starting a new REPL.
FWIW, I’m starting my REPL using Cursive in Intellij. This invokes Java which calls clojure.main. I re-ran the previous benchmark and performance is roughly equivalent between the two approaches.
However, just for fun, I started up a lein repl
and the performance was about 3x slower than java clojure.main
. So that’s fun.
Anyhow, I guess I’ll happily continue using juxt
then.
Thanks for the help!
@UK0810AQ2 I don't see an evidence for this claim when running lein repl
https://clojurians.slack.com/archives/C03S1KBA2/p1652538774360309?thread_ts=1652533347.157599&cid=C03S1KBA2
> Leiningen turns JIT off by default for a REPL
this shows nothing althought I'm running a few lein repl
s (JDK 8, JDK 11) using leiningen 2.9.8
ps aux | grep -E 'Xint'
When I try on a minimal deps.edn project and run clj
such a project shows up:
{:aliases {:dev {:jvm-opts ["-Xint" "-XX:+PrintCompilation"]}}}
ps aux | grep -E 'Xint'
java -Xint -XX:+PrintCompilation -Dclojure.basis=.cpcache/2459096959.basis -classpath src:/Users/jumar/.m2/repository/org/clojure/clojure/1.10.3/clojure-1.10.3.jar:/Users/jumar/.m2/repository/org/clojure/core.specs.alpha/0.2.56/core.specs.alpha-0.2.56.jar:/Users/jumar/.m2/repository/org/clojure/spec.alpha/0.2.194/spec.alpha-0.2.194.jar clojure.main
I really only see this
java -classpath ... -Dfile.encoding=UTF-8 -Xss8m -Djdk.attach.allowAttachSelf=true -Xmx4g -Dclojure.compile.path=.../target/classes -D...version=0.1.0-SNAPSHOT -Dclojure.debug=false clojure.main -i /private/var/folders/hn/tgwyrdmj1tb5pmmbdkd1g_qc0000gn/T/form-init18004889263590279266.clj
Of those, we specify -Xmx
, -Xss
and -Djdk.attach.allowAttachSelf
explicitly in our project.clj@U06BE1L6T starting a lein repl from the command line without any project:
-Dfile.encoding=UTF-8
-Dmaven.wagon.http.ssl.easy=false
-Dmaven.wagon.rto=10000
-Xbootclasspath/a:/home/bsless/.lein/self-installs/leiningen-2.9.5-standalone.jar
-Xverify:none
-XX:+TieredCompilation
-XX:TieredStopAtLevel=1
-Dleiningen.input-checksum=
-Dleiningen.original.pwd=/home/bsless
-Dleiningen.script=/home/bsless/.local/bin/lein
Aha, so it's not that it runs in the interpreter mode but that the tiered compilation stops at the first levep
That's client compiler - I don't understand all the details but it's definitely not the same thing as purely interpreted mode
Right, I was wrong; 0: Interpreted code 1: Simple C1 compiled code 2: Limited C1 compiled code 3: Full C1 compiled code 4: C2 compiled code https://www.oreilly.com/library/view/java-performance-the/9781449363512/ch04.html
I did a very naive experiment with both clj
and lein repl
and I cannot explain the difference.
UPDATE: I probably mixed them up - did another test and it seems it's clj
that is significantly faster
clj
user=> (System/getProperty "java.version")
"17.0.2"
user=> (time (dotimes [i 100000000] (* i i)))
"Elapsed time: 1211.51349 msecs"
user=> (time (reduce + (range 100000000)))
"Elapsed time: 2808.390173 msecs"
lein repl
user=> (System/getProperty "java.version")
"17.0.2"
user=> (time (dotimes [i 100000000] (* i i)))
"Elapsed time: 74.241762 msecs"
user=> (time (reduce + (range 100000000)))
"Elapsed time: 609.163893 msecs"
4999999950000000
So lein repl
looks much faster - can somebody see why it's so?Here's another example for the performance impact: https://github.com/bsless/clj-fast/issues/19
Hi guys, a quick question: When executing
lein jar
It gives the following warning:
Warning: The Main-Class specified does not exist within the jar. It may not be executable as expected. A gen-class directive may be missing in the
namespace which contains the main method, or the namespace has not been AOT-compiled
Which leads to a question i wanted to ask a while ago:
What is gen-class for? Do we always add it in each namespace we create? If not, when do we add it to a namespace?
Thanks a lot!> What is gen-class for?
To create a specifically named class with desired methods.
> Do we always add it in each namespace we create? If not, when do we add it to a namespace?
No, only when you need such a class.
To run a JAR by itself, you either have to rely on the clojure.main
class to run your code (https://clojure.org/reference/repl_and_main#_clojure_main_help) or to compile your own class with a proper main
Java method. That's where you need gen-class
.
Relevant docs: https://clojure.org/reference/compilation
Oh ok thanks for the info @U2FRKM4TW! On a different topic, I wanted to ask if you know any good tutorial for releasing lein project as clojars package?
I'm not saying how I did it is the best way, but you can refer to one of my projects as an example https://github.com/bsless/more.async/blob/master/.circleci/config.yml#L50
@UK0810AQ2 thanks for the tip i'll look into it! @U2FRKM4TW Out of curiosity, what do you use now instead of lein? You mean deps.edn?
Deps is pretty simple to get into. I need to migrate my projects to it (and github actions)
I don't think there's a convention for that, given that this ticket is still open: https://clojure.atlassian.net/browse/CLJ-250
I ask this question is, it might be possible that the java compiler could do some optimization like dead code elimination or something else I do not konw.
There's little sense in DCE on JDK, unless your JAR has really tight size constraints.
But straight up not executing some code in run time will of course be faster than executing it.
> But straight up not executing some code in run time will of course be faster than executing it. Yes. Not about size. But might help correct branch predication.
Overall, there are two main ways:
• Create a macro that returns its body when some "debug" system property is set and returns nil
otherwise
• Have a different classpath between prod and dev. E.g. prod version of my.ns/log
is a no-op but a dev version of it that's in a different file would be a proper implementation.
Not a toggle, but a common way to separate is to put development only code into (comment ,,,)
expressions
This code is evaluated by the developer and skipped when running the code
https://practical.li/clojure/repl-driven-development.html#rich-comment-blocks---living-documentation
dev/user.clj
is a file that defines a user namespace loaded at REPL startup, include in the classpath when developing but not during production
https://practical.li/clojure/clojure-cli/projects/configure-repl-startup.html
I’ve been looking for a simple Dockerfile for Clojure + ClojureScript (w/ npm, shadowcljs) but haven’t found one. Any pointers?
shadow-cljs is just an optional npm i
+ an entry in your depenencies.
Regarding NPM - I ended up just installing it within a CLJ container.
And after working with that for a week, I ended up building the bundle on the dev machine and then simply copying it into a plain Clojure container. :D Much less hassle that way - after all, I already use npm + shadow-cljs locally.
thanks @U2FRKM4TW the former is pretty much what I’m doing
whatup Dennis! 😎 circle ci has some nice images https://github.com/CircleCI-Public/cimg-clojure#nodejs
What are folks' Key-Value store of choice? Looking for something which has easy setup in a cloud and locally
With dynamo i can say "oh yeah i want a db please", but local setup requires docker + nonsense
For context I'm thinking of making use of the domain I own - mccue.link
- to make a tinyurl service
conceptually its a good fit for nosql - well defined data model, read heavy, no issues with consistency, etc
I’m using it for https://yam.wiki/
it’s basically self-hosted heroku for docker but it also supports heroku configs using herokuish
I can get away with $20
on digital ocean and it’s quite portable should you need to move to AWS
@U3JH98J4R What do you mean by "heroku is on fire"?
--------------
| BLANK NODE |
--------------
|
v
-----------------
| NODE W/ DOKKU |
-----------------
|
v
observe --> -----------------
| | NODE W/ DOKKU | -> Build docker image -> Deploy to ...
| -----------------
|
|
------------
| GIT REPO |
------------
so when it observes your git repo and manages nginix - what machine does that actually happen on
@U2FRKM4TW Their security leak
I have sites on heroku where now they wont auto deploy from github because github doesnt trust heroku anymore
I’m not sure deploying from GH is supported with dokku. I just push to the dokku remote instead
Btw I deleted < PERSONAL PROJECT > from Heroku and closed my account.
Given their numerous security failures recently (looks like their passwords got breached, probably the whole core db was hacked) I don't even remotely trust it. Also the fact that Salesforce owns them lol
I know you're a fan of it, but I'd do the same if I were you tbh
Seems like they have resolved it though. At least, judging by their own claims. And you can still deploy - although not from GitHub but from Git, by pushing to the Heroku's remote. Not that related, but personally I've switched to a deployment via Docker and the whole experience now is much nicer. > numerous security failures recently Actually, I haven't heard about any other issues before. All I could find was a vulnerability discovered and fixed almost 10 years ago. Are there any links to their other security failures? > Also the fact that Salesforce owns them lol What's wrong with that? Have there been some shady practices coming from Salesforce? I'm genuinely curious about all that as I've been using Heroku myself for quite some time now. But I'm still highly skeptical that a switch to anything else would be worth it. In part because an absence of a security hole report doesn't mean that there isn't a security hole. But there are of course multiple other factors.
the github deploys broke specifically an app i made for a friend where they edited a file in github's editor as their database
I'm working on a project where I'd like to build my code for different platforms (specifically node.js and google apps script). These platforms expect different things from my code (e.g. some names provided by one are not provided by the other). Therefore, I'd like to somehow use macros to change the way my code is built for each platform. For instance, I'd like to do something like:
((if built-for-node node-func apps-script-func) args)
Then, my final built js artifacts would look like (node-func args)
when my code is built for node and (apps-script-func args)
when my code is built for apps script.
Does something exist like this for clojure?
I realize my project is specific to CLJS, but this seems like the kind of thing that would be provided by the base language.Yep, so depends on how you are buildng your code exactly - but you can make a sort of reader macro-like thing fairly easily
(defmacro env-switch [{:keys [node app-script]}
(case (System/getenv "SOMETHING")
"NODE" node
"APP_SCRIPT" app-script))
(env-switch {:node 3
:app-script (do (something))})
ok cool, this looks like exactly what i'm looking for! I was just trying it out, and I'm still getting some name errors, specifically with this code:
(defn ^:dev/after-load refresh []
(env-switch
{:node
(do (prn "Hot code Remount")
(dev/start! {:report (pretty/reporter)})) ; Check all malli function schemas
:app-script nil}))
the dev/start!
macro is only available in the node environment. when I try to compile for app-script, I get this error:
clojure.lang.ExceptionInfo: failed compiling file:/home/kovas/cljs_clamp/src/autojournal/core.cljs {:file #object[java.io.File 0x504862b5 "/home/kovas/cljs_clamp/src/autojournal/core.cljs"], :clojure.error/phase :compilation}
at cljs.compiler$compile_file$fn__3895.invoke(compiler.cljc:1724)
...
at user$eval1000$iter__1048__1052$fn__1053$fn__1079.invoke(form-init10693697271746039342.clj:1)
at user$eval1000$iter__1048__1052$fn__1053.invoke(form-init10693697271746039342.clj:1)
...
at clojure.main.main(main.java:40)
Caused by: clojure.lang.ExceptionInfo: null #:clojure.error{:source "/home/kovas/cljs_clamp/src/autojournal/core.cljs", :line 30, :column 11, :phase :macroexpansion, :symbol malli.instrument.cljs/collect!}
at cljs.analyzer$macroexpand_1_STAR_$fn__2622.invoke(analyzer.cljc:3923)
... 55 more
Caused by: java.lang.IllegalArgumentException: no conversion to symbol
at clojure.core$symbol.invokeStatic(core.clj:596)
... 279 more
Subprocess failed
i'm just hardcoding the env var to "APP-SCRIPT" for now to test
is this what you meant? https://github.com/kovasap/autojournal-on-gas/tree/main/src/autojournal
i'm still getting errors with this code
Caused by: clojure.lang.ExceptionInfo: No such namespace: autojournal.env-switching, could not locate autojournal/env_switching.cljs, autojournal/env_switching.cljc, or JavaScript source providing "autojournal.env-switching" (Please check that namespaces with dashes use underscores in the ClojureScript file name) in file /home/kovas/cljs_clamp/src/autojournal/sheets.cljs {:tag :cljs/analysis-error}
try making it a cljc file and doing
#?(:clj (defmacro env-switch [{:keys [node app-script]}]
(case "APP_SCRIPT"
"NODE" node
"APP_SCRIPT" app-script)))
that seemed to compile without errors!
yeah follow up in #clojurescript with people who know more than me to understand why, but that is hopefully a start for you
ok sounds good, thanks
i did run into one issue with this solution, and that is if i want to do some repl-driven development and execute some code that uses the macro in cljs, it fails because the env-switch
name is not available in cljs
this suggests to me that a macro here wouldn't really work, because AFAIK all cljs macros are actually run in clj code (and my repl is running cljs)
i'm using a plan def now, passing it anonymous functions to execute for :node and :apps-script
Not sure if this is what you are looking for: https://clojure.org/guides/reader_conditionals
@U033YE56GCV both branches use cljs, so there wouldnt be a reader conditional to use
I just noticed after sent the message 😅
in tools.deps’ deps.edn
using git deps, is there a way to specify a branch and implicitly the latest commit?
I was thinking on the exact question as you’ve posted here. My use case was to track the latest add-lib3 branch of tools.deps.alpha. But checking the docs of deps, seems it is impossible.
@U064X3EF3 is this true?
https://clojurians.slack.com/archives/C03S1KBA2/p1652557974527479
no, ether full sha or tag+partial sha
branches are not stable versions so we don't support those
if you're doing active dev on something, it's usually better to use a :local/root dep to the project on disk instead
Hi, guys, I’m having a bit of trouble crafting a macro and was hoping someone could offer some pointers. I’m trying to construct a function name in the macro based on params and then executing it. This is the macro (defmacro log-msg [level & messages] `((symbol (str “clojure.tools.logging/” (name ~level))) “\n\t*------->>>” (apply str ~@messages))) (macroexpand ’(su/log-msg :info “foo”)) => ((clojure.core/symbol (clojure.core/str “clojure.tools.logging/” (clojure.core/name :info))) “\n\t*------->>>” (clojure.core/apply clojure.core/str “foo”)) Which looks kinda like what I was intending, but it doesn’t execute the log/info function
Any reason you can't call clojure.tools.logging/logf
directly? eg.
(require '[clojure.tools.logging :as l])
(l/logf :info "foo %s" 1)
That looks like it will do what I need, wish I asked about this a day or two ago! THanks
I think I tried to do the same thing early on and then just read the source: https://github.com/clojure/tools.logging/blob/master/src/main/clojure/clojure/tools/logging.clj#L272-L277
Is there a better way of doing this?:
(->> [{:id 1 :label "A"} {:id 2 :label "B"}]
(group-by :id)
(map (fn [[k [v]]] [k v]))
(into {}))
=> {1 {:id 1, :label "A"}, 2 {:id 2, :label "B"}}
Wow.. Nice one