This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2022-04-20
Channels
- # announcements (3)
- # babashka (7)
- # beginners (36)
- # calva (71)
- # cider (25)
- # clj-commons (5)
- # cljdoc (19)
- # cljs-dev (5)
- # clojure (223)
- # clojure-austin (2)
- # clojure-bay-area (1)
- # clojure-europe (31)
- # clojure-france (6)
- # clojure-nl (2)
- # clojure-norway (19)
- # clojure-spec (13)
- # clojure-uk (7)
- # clojurescript (127)
- # core-logic (2)
- # cursive (21)
- # datalevin (53)
- # datomic (9)
- # emacs (37)
- # events (1)
- # graphql (8)
- # jobs (12)
- # lsp (8)
- # off-topic (92)
- # pathom (49)
- # pedestal (1)
- # polylith (3)
- # re-frame (25)
- # releases (2)
- # sci (11)
- # shadow-cljs (13)
- # vim (10)
Hi! Is there an equivalent to C's IFDEF
preprocessor ? My goal is to have some chunks of code that I can turn on or off with a simple flag when I build the uberjar (or npm when using shadow-cljs) ?
Not in core, but you can very easily write one like this
(defmacro compile-when
[test & body]
(when (eval test)
(cons 'do body)))
(defmacro compile-if
[test then else]
(if (eval test)
then
else)))
for reference, both of these can be accomplished with just putting a regular if
form at the top level of your program.
Unlike many other languages, Clojure allows you to put anything as a top level form, it doesn't have to just be ns
, def
, defn
, and declare
.
by making that test fetch an environment variable or something. The test is just code.
right but I don't want to use an environment variable 😅 but thanks anyway, that can be useful in other situations
How do you mean then? You want to pass commandline arguments to what? To the clojure compiler? And have them make it all the way to your code?
there is the dynamic var *command-line-args*
that allows you to inspect the command line arguments, but the problem with that is the command line arguments are first going to be processed by the clojure compiler, which will probably detect them as invalid.
To the clojure compiler, yes (hence my reference to IFDEF
where you can just do gcc -D ...
).
*command-line-args*
this might be a solution though, because one could pass some edn, and then use this in the macro, right?
Yes, but the challenge will be determining where that edn was, and having it not be interpreted by the compiler as invalid arguments. I think it's a bad idea to attempt that.
One thing you could do is use tools.build
to allow you to pass some edn at the command line, and then that process will later start the clojure compiler in a different JVM and can set its environment variables based on your commandline arguments.
@U5NCUG8NR I'm highly sceptical of that eval
- anything you return from the macro is going to run at macro expansion time, which is when the IFDEF would operate
the equivalent of IFDEF using an env var would be using System/getenv
or System/getProperty
to check java startup conditions (environment variables or flags to the vm, respectively)
Yes, this is why I say the same thing could be done by just using if
at the top level
what I'm saying is the macro looks just plain wrong
how's that? It's not intended to be used inside a def form or anything, it's meant to wrap def forms etc.
right, but that eval
is weird, at best it's a no-op
It depends entirely on if you want both the then
and else
to be compiled into the bytecode of the result.
the point here is that the eval happens at macroexpand time
the deeper issue here is the initial problem statement isn't addressed yet: > My goal is to have some chunks of code that I can turn on or off with a simple flag when I build the uberjar there's nothing here so far that conditionally puts contents in the jar, so at best we have things that only work when AOT is on, I don't see any insurance the decision is 100% compile time
it absolutely does conditionally include code into an uberjar if aot compilation is turned on
and if you're not doing an uberjar then there's no difference between build time and runtime
@U5NCUG8NR sure, clojure already compiles whatever you return from you return from your macro, and if it isn't in a defn, runs it immediately, I still don't see how eval helps
so at that stage you can only use classpath flags
nothing in your code can control what gets put into the jar
unless there's aot
I didn't say "not doing an uberjar" - AOT and uberjar are not the same
Yes, I am aware, but most people building an uberjar will also aot
Hence the whole point of the macro, it ensures that code is only compiled if it's needed, and you can only tell that at build time, and if build time and runtime are indistinguishable then it does nothing.
just acts like regular (toplevel) if
ofc if you use it inside the body of a function then it'll act much closer to ifdef
because the test will not have access to function local stuff because the function won't have been run yet, and the two bodies will be used based on that compilation.
So this macro remains useful even if you're not doing AOT
OK - that's a good rationale for eval, I was forgetting that it didn't see locals
well yeah, the macro body is evaluated only when the code is compiled, not when it's run, local values don't exist yet
except in &env, but that's just for tracking what will be bound, not what is bound.
but it's also weird, because it means your calls are treated differently if quoted, and if not quoted the eval is a no-op
(ins)user=> (defmacro show-and-test [t] (doto (eval t) prn))
#'user/show-and-test
(ins)user=> (show-and-test (System/getenv "HOME"))
"/home/justin"
"/home/justin"
(ins)user=> (show-and-test '(System/getenv "HOME"))
(System/getenv "HOME")
"/home/justin"
sure, but be aware the result of the test is not being included in code anywhere
I'm not quoting the whole form
I'm just directly returning either then
or else
as the code to replace the macro body
but why quote anything? it's a macro
it's just weird
nothing needs to be quoted
that's my point
then eval is a no-op
aside from ignoring locals, right
the eval isn't a no-op, it's used to branch inside the macro!
I really don't get what you're trying to say
I'm saying that for all inputs the code does the same thing without the eval, except for two cases I can think of: • you used a local (error with eval) • you created a function literal (error with eval)
usage looks like this:
(compile-if (System/getenv "HOME")
(something with home)
(something with default behavior))
This system/getenv is done once, at compile time, saving runtime checks if this code is called repeatedly, and when you AOT results in only one of those something
forms being in the compiled bytecode.but eval has nothing to do with that compile-time operation
it absolutely does, there's no way to check the system/getenv at compile time without eval
that's the distinction between macroexpand time and runtime
maybe you mean inside def or defn? because otherwise, at the top level, there really is
you literally can't use the result of code passed as an argument to a macro to branch in the macro at macroexpand time without calling eval
Yeah, but #ifdef is used inside the bodies of functions (correctly!) all the time
the original question was to allow usage of something like #ifdef
OK clearly I'm not thinking straight today, I appologise for wasting your time
all good, sorry if I came off as aggressive or anything, I was just really confused by what you were saying.
@U5NCUG8NR did I do something wrong here? this test makes it look like the eval does nothing as I first suspected
(cmd)justin@abjection:~/clojure-experiments/ifdef$ cat src/ifdef/core.clj
(ns ifdef.core
(:gen-class))
(defmacro ifdef-1
[v & code]
(when (eval (System/getenv v))
`(do ~@code)))
(defmacro ifdef-2
[v & code]
(when (System/getenv v)
`(do ~@code)))
(defn -main
"I don't do a whole lot ... yet."
[& args]
(prn :ifdef-1 (ifdef-1 "FEATURE" "feature is activated"))
(prn :ifdef-2 (ifdef-2 "FEATURE" "feature is activated")))
(cmd)justin@abjection:~/clojure-experiments/ifdef$ lein do clean, uberjar
Compiling ifdef.core
Created /home/justin/clojure-experiments/ifdef/target/uberjar/ifdef-0.1.0-SNAPSHOT.jar
Created /home/justin/clojure-experiments/ifdef/target/uberjar/ifdef-0.1.0-SNAPSHOT-standalone.jar
(cmd)justin@abjection:~/clojure-experiments/ifdef$ java -jar target/uberjar/ifdef-0.1.0-SNAPSHOT-standalone.jar
:ifdef-1 nil
:ifdef-2 nil
(ins)justin@abjection:~/clojure-experiments/ifdef$ FEATURE=true java -jar target/uberjar/ifdef-0.1.0-SNAPSHOT-standalone.jar
:ifdef-1 nil
:ifdef-2 nil
(ins)justin@abjection:~/clojure-experiments/ifdef$ FEATURE=true lein do clean, uberjar
Compiling ifdef.core
Created /home/justin/clojure-experiments/ifdef/target/uberjar/ifdef-0.1.0-SNAPSHOT.jar
Created /home/justin/clojure-experiments/ifdef/target/uberjar/ifdef-0.1.0-SNAPSHOT-standalone.jar
(cmd)justin@abjection:~/clojure-experiments/ifdef$ java -jar target/uberjar/ifdef-0.1.0-SNAPSHOT-standalone.jar
:ifdef-1 "feature is activated"
:ifdef-2 "feature is activated"
both with and without the eval, the condition happens at macro expansion time, which is during compilation, which happens before the jar is made if you use AOT
so as I said above, the only thing eval does here is prevent using locals
yeah your eval did nothing there
because eval isn't a macro, it's a function
so my code is wrong? I was copying what the original example was doing
That (System/getenv ...)
is evaluated eagerly, returns a string, evaling a string returns the string, strings are truthy, it always includes the code.
no - see the original nil
because it returned nil when the var wasn't set
sure, eval of nil also returns nil
so how is my example different from what you suggested
the problem is that you're not evaluating code, you're evaluating the result of running some code
right, just like your macro above
no, my example above passed a value that was a macro argument, which is the same as if it were quoted code
that would appear to act identically at top level
and in this specific case of getenv it would appear to act correctly everywhere, just because of what getenv does
and the eval is pointless because all it's doing is evaluating constants (either the returned string or nil)
doing my original example allows you to have tests that are more complex than getenv, like reading a file, or whatever else, and have it function correctly.
Your version here works identically both ways because you've mandated that the only thing that argument is used for is checking an environment variable.
OK - this version lets me see what eval is doing here
(cmd)justin@abjection:~/clojure-experiments/ifdef$ cat src/ifdef/core.clj
(ns ifdef.core
(:gen-class))
(defmacro compile-when
[test & body]
(when (eval test)
(cons 'do body)))
(defmacro compile-when-2
[test & body]
(when test
(cons 'do body)))
(defn file-exists?
[f]
(.exists (java.io.File. f)))
(defn -main
"I don't do a whole lot ... yet."
[& args]
(prn :compile-when (compile-when (file-exists? "some-file") "some-file exists"))
(prn :compile-when-2 (compile-when-2 (file-exists? "some-file") "some-file exists")))
(cmd)justin@abjection:~/clojure-experiments/ifdef$ lein do clean, uberjar
Compiling ifdef.core
Created /home/justin/clojure-experiments/ifdef/target/uberjar/ifdef-0.1.0-SNAPSHOT.jar
Created /home/justin/clojure-experiments/ifdef/target/uberjar/ifdef-0.1.0-SNAPSHOT-standalone.jar
(cmd)justin@abjection:~/clojure-experiments/ifdef$ java -jar target/uberjar/ifdef-0.1.0-SNAPSHOT-standalone.jar
:compile-when nil
:compile-when-2 "some-file exists"
For example:
(defmacro compile-when [test & body]
(when (eval test)
`(do ~@body)))
(def debug? (System/getenv "DEBUG"))
(defn compute-stuff [& args]
(do stuff)
(compile-when debug?
(println "some logging"))
(more stuff))
(defn other-stuff [& args]
(compile-when (System/getenv "FEATURE")
(do stuff)))
yes, exactly
the eval allows you to actually have code written in the macro arguments that gets run at macroexpand time, rather than at runtime, and instead of having some constant data that's interpreted by the macro like you had with your System/getenv example.
right, thanks for your patience
no problem! Sorry if I explained this in a way that was confusing.
@archibald.pontier_clo In Clojure you can do a lot of this stuff using macros which look at something during macro-expansion
It can also be a simple if
- depends on what you want to put in the condition.
Or you can change the classpath. So e.g. build script A has some a.clj
in classpath and build script B has some other a.clj
with same vars but different implementation.
I asked on beginners but didn't get any answer https://clojurians.slack.com/archives/C053AK3F9/p1650439281103429
(btw, the best place to ask this is #tools-build)
the question makes me think you maybe don't have an interactive dev environment set up with a repl. really, you don't want your build process to be in your feedback loop (and then you don't care)
the build mostly takes a while because it forks a jvm for compilation and compiles a lot of code, so you could make it faster by not compiling or by compiling less, but it's hard to tell what you're getting on those dimensions without knowing more about your code or your deps
Hey Alex thank you for replying.... The compile is out of my feedback loop and is very fast anyway
However it bugs me that creating the uberjar is so slow
compilation is comparatively slow - that's why to do it "ahead of time" :)
Hmmm I'm not sure i still think something's fishy. Is there a way to debug that so i can understand what's taking so long?
You can read the code and see what it does 😄 It’s copying all of your project files and your dependencies files into a temp dir, and then it’s re-compressing those into a single jar file. That’s a lot of IO.
that's generally super fast
the slow part is compiling the Clojure code
My experience has been that it takes multiple seconds (up to 10-20 for a relatively small project). Doesn’t really account for the 1m+ I suppose.
No it takes a lot of time every time... I'll try to measure it
Also everything else is super fast
all the compile step is doing is essentially - starting a jvm with the classpath from the basis, then running compile
on all of the namespaces in your project
that will transitively compile all code in all deps used by your project
but perhaps you don't need to do that much work
OR something in your top level code is doing work (perhaps unnecessarily) at load time (which happens as part of compile)
OR you use a future or agent in your top level def and have background agent threads (which wait 1 minute to shut down)
I don't have a great suggestion about how to determine which one of those things it is, other than by doing what I described above - start a repl with your project classpath, compile each namespace, look for something slow
Hmmm that a good suggestion Alex
I'm starting a ring jetty server on my top level context
Hmmm actually no i don't, sorry
(def app
(logger/wrap-log-response
(reitit.ring/ring-handler
cljcrd.routes/router
(reitit.ring/create-default-handler))))
(def reloadable-app
(if (cfg/get-conf :debug)
(wrap-reload #'app)
app))
(def port (cfg/get-conf :cljcrd-port 3000))
(defn serve
([] (serve {}))
([opts]
(ring.adapter.jetty/run-jetty
reloadable-app
(merge {:port port} opts))))
This is what i have
well, all that code gets loaded and run during compilation, so you are
Yes but is something slow?
you're the one that can answer that :)
No it ain't it strts immediately
start your repl and compile
that namespace
but also, I think it's useful to know if anything there uses a future, which starts an agent thread, which takes 1 minute to time out and shut down
Hmmm i don't think so, i don't even know that a future is :thinking_face:
it runs code in a background thread
that thread comes from a pool, which caches threads for up to 1 minute in case they can be reused
Hm no i don't think so
if you load that namespace, you can take a thread dump (ctrl-\) and look for agent threads named "clojure-agent-..."
Ah nice thank you
There's a chance compilation will be faster if you change defs to defns, but I noticed requiring some reitit namespaces can be slow
Just in general, I wouldn't have defs top level that aren't constant. Make it all defn or wrap them in a delay or promise would be my recommendation.
Hm thank you
Thank you didibud I'll change it
Hello friends continuing with this thingie, I added some (time )
commands to calculate each of the commands of the uber building: Here's my output
Building uber file...
Time for copy-dir:
"Elapsed time: 118.6574 msecs"
Time for compile-clj:
"Elapsed time: 14141.1312 msecs"
Time for uber:
"Elapsed time: 65064.87 msecs"
Notice that the b/uber command takes like 65 seconds (!)You can attach visualvm or JFR to profile the process and figure out exactly where you're spending all this time
clojure -J-XX:+FlightRecorder -J-XX:StartFlightRecording=duration=20s,filename=myrecording.jfr -Sdeps '{:deps {org.clojure/core.async {:mvn/version "1.3.618"}}}' -M -e "(require '[clojure.core.async])"
ExampleThat seems unreasonably too much. How large is the resulting Uberjars? So you have really big resource files you package with it? That step should be equivalent to just zipping things, which suis generally be relatively fast for just code.
The resulting uberjar is like 10 mb it's a simple ring webapp
I agree that that does not match what I've seen in other similarly sized setups I've seen
all the jars should be local by the time you have the basis - this should be effectively just unzipping all the jars and rezipping into the uber. any reason disk ops might be slow? network fs, container, etc
No reason. these run on the filesystem,. Notice I use windows... Also notice I tried on two different pcs (both with ssd). On this one it's a little better
Time for copy-dir:
"Elapsed time: 261.3086 msecs"
Time for compile-clj:
"Elapsed time: 23923.3097 msecs"
Time for uber:
"Elapsed time: 31093.3577 msecs"
next try is to try on wsl i guess
uh oh. wsl is even worse
serafeim@DESKTOP-UEUH3G7:~/progr/clojure/cljcrd$ clj -T:build uber
Building uber file...
Time for copy-dir:
"Elapsed time: 502.2897 msecs"
Time for compile-clj:
"Elapsed time: 91868.1961 msecs"
Time for uber:
"Elapsed time: 161638.7026 msecs"
5 minutes for a 10 mb jar:
serafeim@DESKTOP-UEUH3G7:~/progr/clojure/cljcrd$ ls -lh target/cljcrd-0.0.1.jar
-rw-r--r-- 1 serafeim serafeim 9.8M Apr 21 20:21 target/cljcrd-0.0.1.jar
I also tried the flight recorder jfr mentioned before; it created a file but how to understand what's happening ?
ok I used the jmc.exe
just as a point of comparison, the Clojure CLI uses tools.build to make its uber jar and it takes 10s to run compile-clj and 26s to create the uberjar, which is about 20 MB
I am seeing this in the first page of jmc:
The program generated an average of 243 errors per minute during 2022-04-21 20:37:06.000 – 20:38:06.
243 errors were thrown in total. The most common error was ''clojure.lang.LockingTransaction$RetryEx'', which was thrown 223 times. Investigate the thrown errors to see if they can be avoided. Errors indicate that something went wrong with the code execution and should never be used for flow control.
is this relative ?I don't know anything about jfr, so not sure I can help you there
After you have created the recording, you have a log of all events that happened on the JVM. You can open it with visualvm's gui and go to the CPU profiling tab. There you can see where you spent CPU
i'm not familair with visualvm; i used the java mission control program
yeah, that's the outer jvm waiting for the compile
no, compile-clj forks a jvm that does the compile - this is the build waiting for that
I also opened it with visual vm... but this recording is lying 😞
it says that the uptime was 20 seconds but that ain't true ! the uptime was 5 minutes. maybe the recording stops abruptly ? i run it like this clj -J-XX:+FlightRecorder -J-XX:StartFlightRecording=duration=20s,filename=myrecording.jfr -T:build uber
the duration's there 🙂
i'll change it to 300s to see what happens
here's the correct recording; duration is 3 min 23 second
and the timing output I got from running the build in case it helps:
Building uber file...
Time for copy-dir:
"Elapsed time: 308.6562 msecs"
Time for compile-clj:
"Elapsed time: 62769.2298 msecs"
Time for uber:
"Elapsed time: 135928.1001 msecs"
it is slow on windows also
but not so slow 😐
i'll try again on windows
hmmm it's much faster now 😐
Building uber file...
Time for copy-dir:
"Elapsed time: 175.3734 msecs"
Time for compile-clj:
"Elapsed time: 11570.7838 msecs"
Time for uber:
"Elapsed time: 17938.6844 msecs"
how is this possible ?
yes i know that's why i avoid it. i tried wsl because it was slow when i was using cmd.exe
but now it's much faster it takes less than 30 seconds to build the jar
which is not that bad
so let's pretend this issue never happend... thank you for the help
If you use WSL and keep things on the WSL filesystem, I think it should still be fast. But maybe not, it's still virtualized after all.
i used wsl with a wsl filesystem. however it was wsl 1 so this could be the problem
I have a local app (clj) with a front-end (cljs) displayed in the browser. I am using Sente (websockets) to send data to the front-end. What do you think would be the best way to send pngs/svgs to the front-end? The front-end can't read my local disk.
I basically want to show images stored locally in the browser.
Just embed an <img>
tag with the right src
, that's it. No need to make things more complicated than they need to be.
But if the server is the local computer?
Is that when you use the file:// url? Should have tried before asking XD
How do you serve your whole frontend? Probably via
or something like that - just use that.
file://
should not work.
Thanks. This was a dumb q I now realize XD
Yeah, I would recommend what p-himik said. If you are working in a system which eventually is hosting images across a network rather than only localhost, then you will likely want to host the images in a cdn to reduce network traffic to your business logic server.
Oh, another complication: the images are created by the server and have an absolute path.
I guess the easiest thing to do is to create a symlink to the images in my resources/public/img/
folder.
Hmm, symlinking just resulted in a missing image symbol, but actually copying the files to the folder did the trick.
Perhaps the issue is permissions so the webserver is not allowed to read the original folders? I'll investigate.
Edit: no, it seems like the original folders are ok. Example: -rwxr-xr-x
is there a way to un-extend a protocol? Wanting something to fall back to the Object
implementation and wondering if we have to restart the repl or if there is something along the lines of remove-method
for protocols
Heh, I stumbled upon it just yesterday. Here's how Cheshire does it: https://github.com/dakrone/cheshire/blob/master/src/cheshire/generate.clj#L239-L257
@alexmiller I recall you offering the advice “Don’t do work in :or
clauses.” I was wondering if the following would be considered incorrect:
:or {bd (Duration/ofDays 5)
od Duration/ZERO
t (Instant/now)}
2 of those could basically be compile-time constants, but the (Instant/now)
is the bit I wasn’t sure about
FWIW my guess is that the advice was given against using something that's not supposed to be run when t
is present in the map. Because whatever you pass to :or
will be run unconditionally.
Alex gave the more general advice of “don’t do work,” but I failed to ask what exactly that meant 😂
Ah, then it's "don't depend on the order of map destructuring". :)
So you must not do :or {a 1, b (inc a)}
.
Or even :or {a 1 b a}
. There's no computation, but there's a dependency. This should not be done.
So I'd say your case with (Instant/now)
is perfectly fine, as long as you don't mind a few extra cycles and aren't willing to make the code a couple of lines longer to remove those cycles.
I'm looking for something that'll do some of what find
does. Recurse directories without crossing filesystem boundaries, specifically. And not follow symlinks. I couldn't find something quite like that, so thought I'd ask before building it.
It doesn't look like that knows about filesystem boundaries. But it does understand symlinks.
I'd try to search for a Java library that does that.
And if you can't find any, you can use File.list
, File.getCanonicalPath
, and Files.getFileStorage
to fit all the requirements.
Just as a small example:
jshell> Path p = Path.of("/")
p ==> /
jshell> Path p2 = Path.of("/home")
p2 ==> /home
jshell> Files.getFileStore(p)
$15 ==> / (/dev/nvme0n1p3)
jshell> Files.getFileStore(p2)
$16 ==> /home (/dev/nvme0n1p2)
The closest I could find was an Apache project that looks dead: https://commons.apache.org/sandbox/commons-finder/
Guess I'll have to build it
If all the glob
is missing is the filesystem boundaries, you might ask borkdude to add that to glob
.