This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2024-02-06
Channels
- # aleph (43)
- # announcements (11)
- # babashka (35)
- # beginners (70)
- # calva (4)
- # cider (8)
- # clerk (15)
- # clojure (192)
- # clojure-dev (7)
- # clojure-europe (44)
- # clojure-nl (2)
- # clojure-norway (65)
- # clojure-uk (4)
- # code-reviews (4)
- # conjure (1)
- # cursive (41)
- # data-science (1)
- # datomic (8)
- # emacs (7)
- # fulcro (13)
- # humbleui (17)
- # hyperfiddle (53)
- # kaocha (4)
- # malli (7)
- # missionary (17)
- # music (1)
- # obb (1)
- # off-topic (8)
- # polylith (1)
- # portal (3)
- # releases (11)
- # shadow-cljs (36)
- # squint (4)
- # tools-deps (4)
When I search for libraries to ease game development, much of what I run into hasn't been updated in years. Is anyone doing game dev in clojure today and if so, what are they using?
I am, sort of. Mostly focused on Clojurescript right now, but I want my stuff to target cljc. So far I just have quadtree-cljc for 2D games and I'm working on a 2D game/engine that ill publish the code for soon enough.
If you want raw OpenGL stuff, @U080181PF has a cool library called play-cljc and some accompanying libraries.
I ran into issues with this last year and gave up. I tried to use libgdx but without intellisense, it was very painful to work with all of the interop. The couple of wrapping libraries I found didn’t work with recent versions of osx (vulcan vs metal).
raylib has clj bindings, but those seem to be old as well and not for the current version
perhaps something like https://jank-lang.org jank where you could directly use the C stuff
Note that there's also #C066UV2MV and a recent discussion mentions another thing. FWIW, I myself use IDEA so interop with Java is not a problem at all, which is one of the reasons I decided to settle on LibGDX.
Thanks. That channel didn't appear in my list but a direct link to it works
Hi! We started lately getting NVD security check warnings on an old https://nvd.nist.gov/vuln/detail/CVE-2017-20189. While this has been fixed already in Clojure 1.9.0, there seems to be several Clojure core libraries whose newest build is built on pre 1.9.0 Clojure and thus cause this somewhat theoretical security alert. The affected Clojure core libraries and their versions are e.g. • Clojure's transitive dependencies core.specs.alpha (0.2.62) and spec.alpha (0.3.218) • core.match 1.0.1 • tools.logging 1.2.4 • tools.macro 0.1.5 Would it be possible to build and publish a newer version of these libraries that would be built on Clojure 1.9.0 or newer?
Personally I'd recommend to simply add :exclusions
to those dependencies such that older Clojures are excluded
As far as I understand, the bytecode that is compiled with old Clojure and distributed in the mentioned jars is vulnerable. This means that clojure-level exclusions will not help because the mentioned libraries are needed. I guess the only remedy would be to just rebuild the libraries with a newer Clojure that would produce non-vulnerable bytecode, right?
Good caution!
> In Clojure before 1.9.0, classes can be used to construct a serialized object that executes arbitrary code upon deserialization.
Personally I don't immediately understand the issue / how to exploit it and whether e.g. an AOTed (?) core.match
is vulnerable.
Devil seems in the details in this case.
In general, Clojure libraries are distributed as sources, even if they're put in a jar.
That's certainly true for all those libs that you mention, except for spec.alpha
. But, while I might be wrong, I'm pretty sure that you should use the version that your version of Clojure depends on, so they end up being compiled together.
Also, it's not an issue at all if you never use ObjectInputStream
(or similar things, if there are any) on untrusted data. Personally, I never use binary protocols to begin with, they're a PITA to work with on all accounts. Of course, one has to be careful to not let any used library to do something automatically here - like e.g. a web server automatically handling a particular custom mime type or something like that.
Finally, even if you update everything to fix that particular CVE, you still aren't out of the water yet as it's still not safe to deserialize untrusted data: https://ask.clojure.org/index.php/13617/security-problems-command-execution-clojure-deserialization
> In general, Clojure libraries are distributed as sources, even if they're put in a jar.
> ...
> you should use the version that your version of Clojure depends on, so they end up being compiled together.
Aha! I didn't know this. As far as I understand, this solves the whole problem.
> Also, it's not an issue at all if you never use ObjectInputStream
(or similar things, if there are any) on untrusted data.
This is a very good point. For us, the vulnerabilitys is indeed quite theoretical. Obviously our NVD checker does not know this and that's why we need to consider whether suppressing the warning makes sense.
Thanks a lot for the rubber ducking!
I don’t have much to add here, the info above is all correct. Because Java serialization of Clojure objects inherently serializes “functions”, I think the only way to categorically fix this is to stop making Clojure data Serializable (which honestly is worth considering). But really, you should just never unserialize objects from an untrusted stream - attackers can use all manner of tricks to craft malicious unserialized objects.
Also, I released core.match 1.1.0, tools.logging 1.3.0, and tools.macro 0.2.0. the others you're getting via the Clojure dep and already have newer versions. may take a bit before these versions are available in maven central
I'm seeing that older CVE reported against core.specs.alpha and spec.alpha:
org.clojure/clojure 1.12.0-alpha5
. org.clojure/spec.alpha 0.3.218
. org.clojure/core.specs.alpha 0.2.62
which seems like a false positive since those are the most recent versions (right, @U064X3EF3?)
But it seems like the updated CVE will apply to all versions of Clojure unless several changes are made to parts of Clojure's implementation?There were changes made in 1.9 for this specific cve
But that cve is one of a class of likely inexhaustible category
That's why I think the older CVE is a false positive (in this specific case). I'm going to configure that to not be reported for our repo -- and then I can document how to configure false-positives.xml
for clj-watson
which I have an outstanding ticket for 🙂
the older CVE is specifically about Clojure so I don't think this is a false positive in the traditional sense
My repo does not pull in any earlier Clojure versions.
a lot of the clojure contrib libs specify no or a much older version of Clojure because they do not rely on newer versions, so you could be seeing it through those deps even if you are using a newer version explicitly
I don't see the older versions anywhere in -X:deps tree
which is what puzzled me about this.
In case anyone else is getting these specific libraries/versions flagged with that 2017 CVE, here's a suppression file for DependencyCheck that will make them go away (caveat programmer etc):
<?xml version="1.0" encoding="UTF-8"?>
<suppressions xmlns="">
<suppress>
<notes><![CDATA[
Clojure deps are flagged even though they are post-1.9.0
]]></notes>
<packageUrl regex="true">^pkg:maven/org\.clojure/core\.specs\.alpha@0\.2\.62.*$</packageUrl>
<vulnerabilityName>CVE-2017-20189</vulnerabilityName>
</suppress>
<suppress>
<notes><![CDATA[
Clojure deps are flagged even though they are post-1.9.0
]]></notes>
<packageUrl regex="true">^pkg:maven/org\.clojure/spec\.alpha@0\.3\.218.*$</packageUrl>
<vulnerabilityName>CVE-2017-20189</vulnerabilityName>
</suppress>
</suppressions>
@U064X3EF3 Looking at https://github.com/clojure/core.specs.alpha/blob/master/pom.xml it seems that 1.10.1 is the Clojure version for the latest build of core.specs.alpha
but that parent pom (1.1.0) defaults to Clojure 1.8.0. Similarly for https://github.com/clojure/spec.alpha/blob/master/pom.xml (same parent pom, but overrides to 1.10.3).
yes, I'm aware
I'm mulling what to do about the parent pom. will probably update it but I need to do a little broader recon before I do
(seems like it would cause quite a bit of version churn, just to try to suppress these CVE matches?)
most contribs are already using a specific version of Clojure as a dep that is newer
and most applications specify their own version of Clojure that's newer than any of those
I'm not going to go like release every contrib lib
but I might bump the parent pom and some or most of the contribs for future releases
Agree with all of that -- which is why I'm saying this is a false positive (since these are the most recent versions and built with Clojure 1.10.x).
Our dependency check has started failing, referencing https://nvd.nist.gov/vuln/detail/CVE-2024-22871 (https://hackmd.io/@fe1w0/rymmJGida) This is linked above https://ask.clojure.org/index.php/13617/security-problems-command-execution-clojure-deserialization What do people do with this in practice? Do you thoroughly review all your codebase and 3rd party libs or simply suppress the warning?
I'm also wondering why our check failed in the first place - it reports
....standalone.jar/META-INF/maven/org.clojure/clojure/pom.xml: CVE-2024-22871(7.5)
While the website says NVD score not yet provided.
We use owasp/dependency-check-action
docker image (https://hub.docker.com/r/owasp/dependency-check-action)
owasp/dependency-check-action \
--noupdate --format HTML --failOnCVSS 5ub
We suppress the warning
(We are using clojure 1.9.0 or later of course)
We suppressed as well the security check alert regarding https://nvd.nist.gov/vuln/detail/CVE-2017-20189. However, jumar linked to a possible new vulnerability https://nvd.nist.gov/vuln/detail/CVE-2024-22871 which supposedly affects Clojure 1.20 (?). It has not yet been analysed (by NVD?) and we did not yet receive an alert regarding this one, so let's see how this will settle.
For both of these, the vulnerability requires deserializing objects from an untrusted source, which you should never do in any case (regardless of whether Clojure is involved).
The former one was fixed by a change in Clojure 1.9, the latter we have not specifically looked at yet.
I assume that is supposed to be 1.2.0
On the original disclosure it says > Under org.clojure:clojur (1.2.0 - 1.12.0-alpha5) so could that be a typo for 1.12.0 ?
I've found myself writing this helper function to make composing transducers with conditional steps easier:
(defn comp*
"Like clojure.core/comp, but any nil values are ignored."
[& fs]
(apply comp (filter some? fs)))
This lets me do something like:
(comp* (when x (filter ,,,)) (map ,,,))
Does anyone have any alternative ways around this problem?of course identity
was designed to work with transducers from the start in 2007
Yeah I wish I understood transducers well enough to know why identity
works as expected in this case
@UE72GJS7J it works because transducers are functions of a reducing function
Looking at the source for map
I think it's more like:
(fn [rf]
(fn
([] (rf))
,,,))
So each transducer step is composing the previous step (the value of rf
)? In that case it makes sense that identity
just returns the previous transducer unchanged> it works because transducers are functions of a reducing function is this Clojure's monad moment
Just plain old middleware
I'm a little bit confused about assert
in Clojure. How is it supposed to be used in libraries so its users don't pay the perf cost if they aren't setting *assert*
to false before loading any namespace? Asking because I see *assert*
defaults to true. Should't it default to false or to class desiredAssertionStatus()
so it can be enable only when running with java -ea
?
according to
https://github.com/clojure/clojure/blob/0cbf655d181c2de8cd30bf234e40488fc94dbdfa/src/jvm/clojure/lang/RT.java#L200
clojure.core/*assert*
is a dynamic var, so u can (binding [*assert* false] (your.lib/fun ...))
should disable assertions in library function calls.
yeah, I know, but as consumer of libraries you won't know if your libraries are using asserts, so is it your responsibility to always set that to false before loading any namespace just in case? I'm not even sure how to do it when you build a uber jar, because it needs to be set before loading any ns
u have to be careful though, when constructing functions to be called later (or how to say it), because at that time or on that thread *assert*
might be bound to true
.
to avoid this pitfall, u would need to use https://clojuredocs.org/clojure.core/bound-fn
I think that is why java itself disables them by default, and you need to enable in dev by providing -ea
> ... always set that to false before loading any namespace ... why are you talking about NS loading? it's not something which is controlled at compile time, but run time. that's what i was trying to show with my example.
this jira is ok'ed for 1.12 to improve the docstrings of the assert and assert btw https://clojure.atlassian.net/browse/CLJ-2225
should be going in imminently!
yeah, I have seen it @U064X3EF3, but my question is more about its default value being true and the implications for libraries authors and consumers
I guess it is also related to this https://clojure.atlassian.net/browse/CLJ-2554
clojure.core/<star>assert<star>> When set to logical false, 'assert' will omit assertion checks in
> compiled code. Defaults to true.
in other words, it doesn't matter what was the value of *assert*
at compile time (which is the time when you are loading the library), because its value only matters, when you are running those library functions.
what you just said reads wrong to me. truly it only matters at compile time (but compile time may also be run time)
@U086D6TBN maybe I'm not explaining myself. The thing is, if you are creating a library and you put asserts everywhere to help you at dev time, then users need to know about this and remember to disable asserts globaly in prod (some how) or they will pay the perf price without knowing
and it is also not easy to do, since you need to set that before requiring namespaces
because assertions are on by default, you should be judicious in your use of them. users generally can turn assertions off when say, compiling an app (their code + libs) for maximum performance
I think if the default value comes from desiredAssertionStatus()
then you don't even need a jvm property, you can just use -ea
at dev, and people can use assert without thinking
ah, i just looked at the source of clojure.core/assert
and indeed my understanding was incorrect.
what is desiredAssertionStatus()
?
a method in java.lang.Class that returns the -ea
flag status
oh, I thought that was something in Clojure you were referring to
I think Clojure assertions may have pre-dated Java assertions
which is why they are not as well-knitted as they could be, happy to take problems on Ask Clojure
I think having something like that would make clojure assert even better than java one, since as I see, java assert just emits bytecode, so you have the checks everywhere in the final bytecode
which isn't the case in Clojure assert
Can't you bind *assert*
to false in your compile options when you start clojure? That allows you to globally turn it off for your entire application without a code change.
https://ask.clojure.org/index.php/13666/why-does-clojure-lang-assert-defaults-to-true
> Can't you bind *assert*
to false in your compile options when you start clojure? That allows you to globally turn it off for your entire application without a code change.
how do you do that in a jar?
> I think Clojure assertions may have pre-dated Java assertions
I'm interested in learning more of the history on both sides. Looks like java assert
originates in 2002 with 1.4 https://en.m.wikipedia.org/wiki/Java_version_history#J2SE_1.4 ?
@U5NCUG8NR even if you do it, everybody needs to remember to do it every time
by telling the person running the jar to add extra flags to it. Trivial if you're distributing your application with jpackage (it can be added to runtime config), trivial if you're controlling the invocation with a docker image or container distributed to the cloud, etc.
also if you distribute your application to people as a jar and don't wish to use jpackage you can AOT your code
I mean, I'm not saying it is impossible, but isn't it just much better to default *assert*
to false
or the value of the -ea
flag?
Now you're just throwing who has to "always remember" around. Assertions are useful to developers, and if they're off by default a lot of code that violates assertions will be written.
but IMHO that is much easier to remember, is like spec instrumentation, will not be on by default. Can be added to the :dev profile, for example. And if you miss it, then it is just a dev thing, as soon as you realize you add it. As it is now, people can be running software in prod much slower depending if library creators are using asserts in hot loops, etc.
If someone's code is slower and it's important, they will notice and fix it. If someone's code has an assert that's disabled, that code might break stuff.
I disagree there. I think it is why you don't enable spec instrumentation by default. I see asserts the same as spec instrumentation. For most people it will not be easy to tell their project could go faster because of the asserts. And I think we haven't run into this because probably not many people use asserts, because it is not ergonomic, because of this aspects.
I think there is a reason why java disables it by default.
> if someone's code has an assert that's disabled, that code might break stuff.
As as it says here https://docs.oracle.com/javase/7/docs/technotes/guides/language/assert.html Do _not_ use assertions to do any work that your application requires for correct operation
That's not what's meant. Yes obviously the code shouldn't rely on the assertion existing for correct operation, but the assertion can reveal that the caller has an error in the way they use it which violates how the library is intended to be used. that is a valid use of assertions.
The library would function fine when called correctly without assertions, but assertions illustrate incorrect usage of the library.
> but the assertion can reveal that the caller has an error in the way they use it which violates how the library is intended to be used. that is a valid use of assertions. True, but that is also what spec fn is for, and it is disabled by default
Maybe it actually should be enabled by default. :)
It's not too dissimilar to the recent discussion and change w.r.t. OmitStackTraceInFastThrow
.
as default in dev I agree, as default in everything I don't 🙂
FWIW, I think asserts should be always on, even in production. If you really think some condition "can't happen" and you want code to "fail fast" instead of trying to continue to run with bad data and potential further corruption etc, then you should want your production code to "fail fast" too -- instead of risking corruption of valuable production data. I've never understood this idea of "turning asserts off in production" -- you want your code to corrupt data faster, instead of failing? That makes no sense to me.
I've never understood this idea of "turning asserts off in production"So in my mind asserts
always were a dev only tool. The whole purpose of it is that you can add a million invariants checks to help you at dev/test time without having to worry about perf. If we need to check values in our apps we should be doing (when-not x (throw (ex-info "..." {...})))
. I don't see assert being the same as a when/throw but syntactically shorter, they have a completely different purpose, much more aligned with function specs. If implemented correctly.
I think asserts in all languages I know have this purpose, that is why they are disabled, in the case of Java https://docs.oracle.com/javase/7/docs/technotes/guides/language/assert.html and c++ https://learn.microsoft.com/en-us/visualstudio/debugger/c-cpp-assertions?view=vs-2022#BKMK_Assertions_in_Debug_and_Release_builds unless you say you are in dev
if you have assertions enable by default, then you will not be adding as much as you think they can help you because you will be worried about perf
Trouble is, you can't rely on that semantics unless you never use third-party libraries. Different people use asserts and :pre
/`:post` in different ways.
Hell, some people catch Throwable
and treat it as a recoverable exception.
well, I can in Java, I don't need to worry about what others do, because I'll never run with -ea
in prod. I'll just run with -ea
in my tests and dev to have the extra help
I mean, some library might be relying on assertions to report e.g. invalid user input. It's not that hard to imagine that, especially given that assert
in Clojure behaves slightly differently from Java's.
Personally, I don't think that disabling asserts by default in Clojure is a good idea at this point. I'd consider it a breaking change.
here you have other people explaining https://stackoverflow.com/questions/2758224/what-are-assertions-in-java-and-when-should-they-be-used
If something leads to performance degradation and if it's important to you, you notice it and you fix it. That's it. There's no need to make the change for everyone when they don't have the same problems as you do.
> I'd consider it a breaking change. that I could agree, but I think it needs some thinking about the possible impact
And I'm not talking about asserts in a vacuum and how every developer should be using them. I'm talking about asserts in Clojure specifically.
yeah, but my original question was that I found it confusing that we were doing it differently, and that by doing it this way assertions don't make a lot of sense to me. As a library writer I have to think about perf when writing assertions, which I don't need in other langs. Which renders asserts as just syntactic sugar for when/throw, and more confusing makes people think they are the same thing, which is against best practices in other languages like what I linked before.
it even breaks compatibility with shadow-cljs ClojureScript https://github.com/thheller/shadow-cljs/blob/dfd3d54eab12c8d15d097b756b6d218732a6bb8d/src/main/shadow/build.clj#L391-L396 which will disable assertions on release
> As a library writer I have to think about perf when writing assertions
No, not at all.
If your users care about the impact of assertions on performance, they disable them. Just like in Java.
If they care about catching every single error, they don't disable them. Just like in Java.
The only difference is the default state.
You cannot guarantee how other developers use assertions. Regardless of the language.
In Clojure, a library author might always be disabling asserts in their projects and never care about how much time a particular assert takes.
In Java, a library author might always be enabling asserts in their projects and always care about Throwable
.
In both cases, you have little control over that except for enabling or disabling assertions.
> it even breaks compatibility with shadow-cljs ClojureScript
Nothing breaks anything, it's irrelevant.
Shadow-cljs is a tool that wraps and extends the CLJS compiler, a tool with some defaults. Thomas has decided that disabling asserts in CLJS by default is a good idea and put it there. Some other author of some other tool that deals with CLJ might decide the same, or the opposite.
If your users care about the impact of assertions on performance, they disable them. Just like in Java.it is not even clear how do I go about disabling it in a non AOT compiled jar
I guess we will have to agree we disagree, it also seems that most disagrees with me 😞 haha but that was my opinion 🙂
> it is not even clear how do I go about disabling it in a non AOT compiled jar Maybe there are instances where it doesn't work, but this seems to work just fine:
$ cat a.clj
(ns a)
(defn f [x]
(assert (pos? x))
(inc x))
$ clj -Sdeps '{:paths ["."]}' -M -e "(var-set #'*assert* false)" -r
false
user=> (require 'a)
nil
user=> (a/f -1)
0
user=>
but that was not a non-AOT compiled jar
A JAR that's not AOT compiled is not different from a collection of local files, at least AFAIK. So you should be able to use that with a JAR as well.
I stopped caring about uberjars a long time ago, so it's a bit hard for me to check - still no clue about tools.build
and whatnot.
I mean, if I'm running with java -jar , that I think a lot of people do
I'm not saying it is imposible, I said "it is not even clear..."
@UAEH11THP can you share how to set `*asserts*` false when building a jar? What I would like to have is a clear way of disabling assertions in prod. Most of the time I build a uberjar and then run it with java -jar my-app.jar
. I don't AOT compile it because I have seen issues around it
tools.build these days
But how exactly?
I assume you're asking about building an application JAR. I see two sections in the tools.deps
docs - one for library JARs with sources, another for AOT compiled JARs for apps.
If my assumption is correct, you need something in between.
I'm ok with any way, even using a different library or lein, the thing is to create a uber jar, not AOT, that starts without asserts
but also, leaving the "default value for *assert*
" discussion aside, I would argue that this should be something easy for everybody, not something that depends on the tools you are using and on having a deep understanding of Clojure
I would agree if assertions in Clojure were the same as in Java. But they are not. Different behavior, different defaults, different expectations. Only the name and the thrown error type are the same.
I'm not talking about java only, I'm talking about c++, python (when you run with -O (optimized) they are disabled), rust (they call it debug_assert, will not be included in release), I'm talking about having a tool to make possible for people to add as many invariant checks as they like, even expensive ones, and a easy way for people running a system to not worry about how the lib creators are using the asserts and not paying for perf. Because some asserts could be very expensive. And what are the options after your profiling? just discard the libraries that choose to use expensive asserts for their dev and testing?
> I'm not talking about java only
The main point is that Clojure behaves in a way Clojure behaves.
We can list all the languages we want - no matter what behavior they've adopted and how mainstream they are, Clojure is Clojure. It's not any of those languages.
> I'm talking about having a tool
And assert
, given its current implementation, is not that tool. Even if you add a very convenient toggle, it's still not that tool because the implementation is very different and allows different things.
There might be a different tool that would behave exactly as assert
behaves in some other languages, sure - something like dev-assert
.
> just discard the libraries that choose to use expensive asserts for their dev and testing?
Just set *assert*
to false
when using those libraries. Assuming you fully recognize that it's not just "let me improve the performance with this little trick" but "let me disable a runtime check to improve performance".
I haven't tried this but assert is used as an example in the sample lein project.clj so I assume it works? I'd add it to the uberjar profile https://github.com/technomancy/leiningen/blob/d5647f26332358ff80edf28f81714fe898af0809/sample.project.clj#L297-L298 Then again this isn't relevant if you use clojure cli toolchain
Just tested, it works.
In order to set *assert*
to false
with tools.build
and have it work with a source JAR, you need an extra compile step that compiles just a single namespace that loads your main namespace dynamically but uses alter-var-root
before that. 7 extra lines of code compared to a plain source-based JAR with clojure.main
as an entry point.
> We can list all the languages we want - no matter what behavior they've adopted and how mainstream they are, Clojure is Clojure. It's not any of those languages.
I would say it is weird but I can accept that.
> There might be a different tool that would behave exactly as assert
behaves in some other languages, sure - something like dev-assert
I'm all in for something like that, on my ask clojure I added that even if it is called debug-assert
like in rust it would be fine, the thing is having a tool for that
> Just set *assert*
to false
when using those libraries
For this I think we need to find and document easy ways to do it for all the ways we can run systems in prod
> Assuming you fully recognize that it's not just "let me improve the performance with this little trick"
I don't think that is what it means in most languages, I don't think people coming to clojure will assume that
> In order to set *assert*
to false
with tools.build
and have it work with a ...
I guess I'll just wait for this https://clojure.atlassian.net/browse/CLJ-2554
maybe easier, which can work with any solution would be to add a main with a (gen-class :impl-ns main-impl :load-impl-ns false)
, so you first put assert in false and then require and run your "real main", called main-impl
> For this I think we need to find and document easy ways to do it for all the ways we can run systems in prod
"When using those libraries" meant using binding
around the functionality that uses asserts that you want to get rid of.
> I don't think that is what it means in most languages
You can't disable console.assert
in JavaScript (at least, not without monkey-patching, which can be precluded by anyone storing the original value of console.assert
)
You can't disable assert
in R.
You can't disable panic
in Go.
The list probably goes on.
> I guess I'll just wait for this https://clojure.atlassian.net/browse/CLJ-2554
By itself it won't help since there will still have to be something that reads those properties. If you have a custom entry point that forgets to read do that, assertions won't be disabled.
So there will still be an extra step.
> maybe easier [...]
But that's almost exactly what I did, where I said "Just tested, it works" above?..
Admittedly, I'm not well versed in all things compilation, but :gen-class
doesn't sound like it's harder.
Correction to the above "By itself it won't help" - it will of course help if the initial *assert*
value is set based on the prop.
A few general thoughts, without going into the implementation details – I believe there absolutely needs to be a generic, reliable way to add asserts/checks/validations to a library with the absolute certainty that they will not end up in anyone's production build by default under any circumstances, except if the user explicitly opts in. Some checks are clearly desireable during development and testing, but way too expensive to have in production by default, others may be essential in production to avoid the potential for data corruption already mentioned in this thread. Wherever one chooses to draw that line – there's a line, and library authors should have the ability to make that distinction. It should not be an all or nothing proposition, and authors shouldn't be restricted in how thoroughly they validate stuff during development/testing because they have to worry about potentially dumping overzealous, expensive checks on the consumer in a "you deal with it" manner by default – that strikes me as user-unfriendly in the extreme. CLJ-2554 sounds great.
Here are https://github.com/fulcrologic/guardrails?tab=readme-ov-file#static-exclusions-special-attention-library-authors on the approach we ended up settling on in the new Guardrails release.
@U9S6X97KQ My ideal situation would be to be able to keep assertions in my own app code enabled in production but disable assertions in one or more third-party libraries that I depend on. I definitely want my assertions to be enabled in production: I want my code to "fail fast" rather than attempt to continue with bad data and, in particular, I want my dev/test/CI/QA/production systems to be running the same version of my application code. What makes this more challenging than in most languages is that Clojure libraries are usually distributed as source -- so library developers do not control the "compile-time" behavior of assertions in their code: only application developers control that.
That sort of "elide assert" behavior could only really come as an extension to require
, since that is typically when third-party library code is compiled for a Clojure app.
I think if the name assert
is already taken in Clojure with different semantics (and then different expectations) than say java assert, then we need a new tool and call it differently, maybe debug-assert
(like Rust's) or whatever. But imho a tool that you can use to check invariants in your library code that you are sure no one will use unless they explicitly opt in (like -ea in java) is very useful to have.
But again, I'll argue that it is confusing, because people will use Clojure assert
to do application logic, and it will fail in the scenarios where people disable assertions via *assert*
which is already possible
My thinking for library code that wants to check arguments are valid in dev/test (but not "production") is to use Spec and instrumentation of fdef
. If you really want to guard against misuse of your library, you should have explicit checks that throw specific exceptions.
But, yeah, because of the blurred lines of "compile time" in Clojure, and libraries distributed as source, we really need two different mechanisms.
@U04V70XH6
YMMV of course, but in my experience even in my own code there are often assertions I absolutely want to keep in production and those I definitely do not. If you want all of your assertions enabled in prod, then that implies to me that you're never using assertions which are prohibitively expensive. That may of course be the case, but I'm arguing that those are very much used by others (myself included), are not at all uncommon, and IMHO there should be a way to turn them on/off separately from the essential stuff.
EDIT: When I say assertions I'm referring ot all kinds of checks/validations, not just assert
@U0739PUFQ Yup, a separate assert – elided by default – would make sense to me. I agree that the semantics are a bit mixed up ATM and it's not immediately obvious what would be the "proper" way to untangle them.
@U04V70XH6 arguments is one thing that fspec already covers, but assert in languages like Java can be used in any place in your code, not just args
The reason I mentioned the Guardrails documentation above, is because it pretty much hits the sweet spot for me: • When you're working on your own library, every function call is validated, including hot internal loops, and that's expensive • When consumers are using your library to work on theirs, only the public-facing calls are validated so they see when they're calling your library wrong. • In production none of it is validated. • The user can turn any part of that on/off incl. in production, but the defaults are sensible. Just as a general note on how I think stuff like this should be approached, not suggesting all of it is perfectly applicable here 1:1.
@U0739PUFQ Right, I'm well aware of that. I did Java for many years and C++ for a decade before that. I'm used to assertions. I'm pointing out that Clojure provides specific unusual concerns around the use of assertions -- especially given that they are on by default, even after AOT compilation, unless you specifically turn them off. As a library developer, I can't safely use assertions for "prohibitively expensive" checks in Clojure because that's likely to impact downstream users. As an application developer, I like that I can leave my own assertions on. I agree with you that we really need two different pieces of machinery here (since we don't want to break backward compatibility for existing assertion uses).
I strongly suspect the majority of Clojure application developers leave assertions enabled in production simply because doing otherwise requires work and most people just don't think about it.
So, to summarize:
• assert
for more essential stuff, on by default, can be disabled for library calls with binding
• debug-assert
for more expensive stuff, off by default, can be enabled selectively with binding
as well
Is that the emerging consensus...?
That would be my preference, yes.
yeah, that makes sense to me, I think it is what Rust also has without the binding disabling oc
Someone should post a proposal on http://ask.clojure.org 🙂
The law dictates that in absence of a volunteer that honour falls to @U0739PUFQ as the OP. 😄
the same day I started this thread I created this https://ask.clojure.org/index.php/13666/why-does-clojure-lang-assert-defaults-to-true
maybe someone else can comment on it if it is not clear enough
Upvoted, and I'd suggest that anyone else here who agrees do the same – you can login via GitHub and get it done in a second.
It's pretty clear overall, however I would suggest that you elevate the edit at the bottom to be the core suggestion and/or maybe just change the title accordingly, because right now at first glance it seems that what's being suggested is to change the default value of *assert*
, which I suspect we also agree is not the way to go, and doesn't have much hope of being considered anyway due to being much too invasive and breaking a change.
My suggestion would be something like 'Add debug-assert
as a separate, off-by-default operation for expensive dev/test-only assertions'
Just changed the title and edited the description with the recommendations here
I find it tempting to turn vanilla defns from 'util' namespaces into macros such that loggers will log the ns name of the caller, not the ns name of the ns that may offer the util. In other words, the first log line can be considered more useful than the second one:
INFO myapp.models.invoice - SELECT * FROM users WHERE id = ? 0ms - 1 records
INFO myapp.util.sql - SELECT * FROM users WHERE id = ? 0ms - 1 records
The difference is more evident if you consider than in a typical request/response cycle, many different namespaces may be responsible for the emitted SQL. The util
ns is the messenger, not the culprit :)
Now, favoring macros over functions just for this concern feels a bit dubious. Is there an alternative?I have resorted to macros for exactly this purpose in the past so you’re definitely not alone. In terms of an alternative, the only thing that comes to mind is passing down *ns*
at call site to each log function call.
You could walk the stack to find the frame before the one that represents the util function
https://docs.oracle.com/javase%2F9%2Fdocs%2Fapi%2F%2F/java/lang/StackWalker.html
hmm stack walker is lazy and in theory you'd only ever be going up two frames?
also you'll probably want https://clojuredocs.org/clojure.main/demunge
luke-stackwalker
- good name for a Clojure wrapper 😄
Sounds worth checking out but a small cost repeated N times with logging indeed could add up
sure, would be curious to see benchmarks if you try
we dump stack frames like this in a few places in core for the same reason
From memory, runtime (as opposed to macroexpansion time) *ns*
often points to user
or myapp.main
- unfortunately it doesn't mean "the ns of the caller"
I guess the kind of solution I want is a generailzed macro e.g. (log/preserving-ns ...)
which I could add in each callsite.
(A bit tedious, but possibly better if the alternative is having a lot of macros or macro-writing-macros which is what would happen otherwise for my specific scenario)
clojure.spec.test.alpha has some stuff that drops top frames that are plumbing. I think clojure.test and pst both do similar things.
As a wild alternative in some contexts, perhaps the queries could be named. "invoices by user" could potentially be more informative/specific than a namespace name, with the obvious trade-off of naming things.
yeah in clojure.core
(defn ^:private elide-top-frames
[^Throwable ex class-name]
(let [tr (.getStackTrace ex)]
(doto ex
(.setStackTrace
(when tr
(into-array StackTraceElement
(drop-while #(= class-name (.getClassName ^StackTraceElement %1)) tr)))))))
I’ve seen logging performance cause some serious degradations. We logged in a hot loop in snowflake or bigquery data results that caused an issue. And it was traced back to an issue with multi-release jars.
I wrote an article where logging in a loop with bad c.t.l config could absolutely tank performance by just changing the classpath, no change in actual code.
That’s why i’m nervous about stackwalking on each log message versus in a point where a little extra performance penalty doesn’t hurt.we have a couple macros at nubank that capture caller context and push an explicit argument downstairs
In theory getStackTrace
has to allocate an array of all the frames but StackWalker only has to allocate two and should perform better (you can allocate one walker and reuse). I'm not suggesting it's free, but vemv also cited no performance constraints. I'll try to benchmark later if nobody else does
You could also just add an inline definition to the function
@U45T93RA6 could you please share your macro here?
I don't have a macro atm :) the SQL logging is real though, it implements https://github.com/seancorfield/next-jdbc/blob/ee1511f097931f29e0838960d873ba16312be53c/doc/getting-started.md#naive-logging-with-timing Thanks all for the input! Qs usual, one can get unexpected and diverse ideas from #C03S1KBA2
Our dependency check has started failing, referencing https://nvd.nist.gov/vuln/detail/CVE-2024-22871 (https://hackmd.io/@fe1w0/rymmJGida) This is linked above https://ask.clojure.org/index.php/13617/security-problems-command-execution-clojure-deserialization What do people do with this in practice? Do you thoroughly review all your codebase and 3rd party libs or simply suppress the warning?