This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2021-09-17
Channels
- # announcements (17)
- # aws (2)
- # babashka (21)
- # beginners (67)
- # calva (19)
- # cider (29)
- # clara (3)
- # clj-kondo (6)
- # cljsrn (10)
- # clojure (140)
- # clojure-europe (164)
- # clojure-nl (3)
- # clojure-uk (8)
- # clojurescript (62)
- # conjure (7)
- # core-async (24)
- # cursive (21)
- # datomic (5)
- # docker (40)
- # emacs (14)
- # fulcro (25)
- # gratitude (1)
- # honeysql (6)
- # introduce-yourself (1)
- # jobs (1)
- # jobs-discuss (32)
- # juxt (7)
- # lsp (13)
- # minecraft (2)
- # off-topic (49)
- # pathom (24)
- # practicalli (8)
- # re-frame (18)
- # react (23)
- # remote-jobs (6)
- # reveal (2)
- # shadow-cljs (75)
- # tools-deps (7)
Exceptions on a future that isn’t deref’ed don’t end up in the default uncaught exception handler, right. What would I use if I do want to ‘fire and forget’ a task, but do want raised exceptions to propagate to my default uncaught exception handler?
Is this something I should want even
Maybe this post is helpful: https://stuartsierra.com/2015/05/27/clojure-uncaught-exceptions
Right yeah I had found that, so that would suggest that I have a different executor service somewhere and manually .execute
(not! .submit
) my task?
I don’t want the current thread to be blocked on this task, but I do want to log failure of this task (asynchronously)
Could you do something like this?
(require '[clojure.core.async :as a])
(let [c (a/chan 10)]
(defn submit-future [f]
(a/put! c (future (f)))
(println "future submitted"))
(a/go
(loop []
(when-let [val (a/<! c)]
(try
@val
(catch java.util.concurrent.ExecutionException e
(println (.getCause e))))
(recur)))))
Right, but I can also put a try catch inside my future call to do that right. I was wondering whether there would be a way to not do logging at this point, but instead rely on the default uncaught exception handler I have set up anyway. Does that make sense?
(Missed a negation, edited)
It seems that the ExecutorService running the futures catch exceptions before the default uncaught exception handler has a chance of dealing with them. I don't know if this is recommended usage, but have you looked at clojure.core.async/thread
?
(Thread/setDefaultUncaughtExceptionHandler
(reify Thread$UncaughtExceptionHandler
(uncaughtException [_ thread ex]
(println "Uncaught exception on" (.getName thread)))))
(a/thread
(/ 1 0))
=> Uncaught exception on async-thread-macro-9
Oh interesting, didn’t know that core async threads didn’t have this behavior
If you want to write your own Executor instead of depending on clojure.core.async
, you can check out clojure.core.async/thread-call
for inspiration
Nice q! A related thing to wonder / research is why the executorservice has this design. Futures are a bit awkward in this regard and play against another Java mechanism (the Thread default handlers) Maybe it is for a good reason?
The best reason I can immediately think of is that fire-and-forget isn't really a thing to be fostered, if you wanted to you could as well spawn Threads by hand (which is super easy) and skip the ExService machinery altogether i.e. either you're doing things quick-and-dirty or not, no in-between
I have an api question: I want to provide a new reference type in a library. It does not have atomic semantics because it's backed by a native pointer. What I had wanted to do is to use proxy
to extend clojure.lang.Volatile
and use the existing vreset!
and vswap!
operations, however Volatile
is a final class, and therefore I can't proxy it.
I see two paths forward, either A) extend IAtom
and IAtom2
to provide the methods that are already provided on atoms, or B) move it to a deftype
and roll my own reset and swap functions.
At the moment I'm leaning towards B, but I like the idea of providing the same interface as one of the core references. I'd like some feedback on which would be preferable.
IRef
actually is the only part of the general reference api I don't want to support.
Because I can't add watches onto native memory.
And the idea is to be able to both read and write, just not atomically.
Alright, then I'll go with that. Probably just call it freset!
and fswap!
for "foreign swap"
since there's no question of atomicity or retries, why not just use set!
which actually has the semantics you are implementing?
offering a swap with no atomicity or retries seems misleading
@U051SS2EU is there an interface you can use with set!
?
it appears to be clojure.lang.Settable - finding it now
https://www.javadoc.io/static/org.clojure/clojure/1.10.0-alpha6/clojure/lang/Settable.html
I'm not sure what the difference between doSet and doReset is supposed to be
it looks as if compiler.java expands set!
to a doSet
call and there are no matches for doReset in the file
yeah, we have nicer ways to update a state, set! is pretty hazardous in comparison
but it seems to match the semantics of what we are discussing here - sets the right expectations
Oh, I wasn't aware that set!
was extensible
I agree this does seem to fit with the semantics pretty well, but I will admit that I don't know how I feel about using set!
on something that's not directly mutable but is rather a reference to something mutable
Like to get the value you have to use deref
@U5NCUG8NR This is why I asked if the value is read-only or also mutable
looks like it doesn't actually work..
(ins)user=> (set! (reify clojure.lang.Settable (doSet [this v] "OK")) :foo)
Syntax error (IllegalArgumentException) compiling set! at (REPL:1:1).
Invalid assignment target
Well it's both. I have IDeref
, but the value behind it is any type. Additionally, there is a way to mutate the value behind it, but it itself isn't mutable.
What this maps to by the way is a static global variable in a native library.
user=> (set! (reify clojure.lang.Settable (doSet [this v] (prn :hello))) 1)
Syntax error (IllegalArgumentException) compiling set! at (REPL:1:1).
Invalid assignment target
doing a deref in clojure will dereference the pointer and deserialize what it finds into a clojure data structure (and it's on the user to ensure that you do that at a time when its state won't be updating to see an invalid state)
doing a set will also dereference the pointer, and then serialize a clojure data structure into the memory segment it finds there.
I'm generating a kafka streams serde with deftype
. it works in dev, but in prod (jib built docker image, no AOT) the class cannot be found when kafka streams tries to load it, even though I explictly import
it before kafka streams is initialized. how do I go about debugging this?
import
in Java doesn't make anything compile, you need to AOT that type before using it in Java
so I actually solved it without understanding why.. you can pass to the kafka streams config either a string representing the full path of the class (only works in dev) or the class instance itself (works in prod). This is AOT related? not sure why it would work in dev, when is the type being compiled there?
Given the symptom, perhaps there are two classes with the same name. Using a string loads it from some other place, so that x.y.Z
is not the same as the class instance x.y.Z
.
$ mkdir -p src/foo
$ echo '(ns foo.core) (deftype Foo [])' > src/foo/core.clj
$ clj -M -e "(compile 'foo.core)"
foo.core
$ ls classes/foo/Foo.class
classes/foo/Foo.class
You need to actually see the Foo class there as a result of compiling the deftypeIf that class isn't there (or in the jar, or however you package) then Java cannot find it
it's probably that the kafka streams' classloader cannot resolve the class from the name
if the clojure classloader is a child of the kafka streams classloader, it will not see it
anyone know is the repo for clojure cli the same as the clojure repo I get the feeling they are different but I cant find a link to the clojure cli repo
oh so that's for linux as well, I assumed that was mac only
Yep. This is the main executable: https://github.com/clojure/brew-install/blob/1.10.3/src/main/resources/clojure/install/clojure
It's just that there was some shuffling at some point and I remember trying to understand an outdated script at some point.
okay great thanks
@U02DXJUS5JA brew
works great on Linux too BTW -- that's what I use on both macOS and Ubuntu for my Clojure-releated deps.
on a related note I notice https://github.com/clojure/brew-install/releases/tag/1.10.3.981 is different to the download it seems to be missing a comple of files in particular clojure-tools-${project.version}.jar is refereed to in the install but does not exist in the sources, It also does not look like there is a build to create this, anyone know how this works, I am trying to build a package if your wondering why I am after these details. It works when I built I as requested to make it build from the source which is why I am hitting a few issues.
@U04V15CAJ not sure what you mean I get that project.version is replaced, but where does clojure-tools.jar come from its not in the release do you build it does it come from some where else ? when you download from the clojure site it does exist in the archive
brew-install
is the source repo and the tags are on that source; the downloads that GitHub creates there are of the source; but the "download" -- the deployed artifact is not those, it is "built" into the appropriate shape. You can't expect to download GitHub's automated source ZIPs and have them work for libraries/tools that include build steps.
yeah I thought it might be part of the build, I guess the question kind of becomes where is the source for clojure-tools-1.10.3.967.jar which contains the files below as they are not in the brew-install script repo
@U02DXJUS5JA If you are creating a package for linux, I think you can re-use those ingredients (tools jar, bash script), without knowing how they were built? I'm doing a similar thing for deps.clj.
The tools jar is an uberjar which packages tools.deps.alpha with a main function that is being invoked from the bash script.
okay thanks that makes sense, I think the main issue is that the package maintainers ideally want to build from source and sha check the files to avoid files being injected at a later date
If i can build the jar and include it that may work, I kind of understand why just makes packaging it more difficult 😕
but thanks for the tip I will look into that when I get a chance
Someone else recently went through the whole "compile from source" thing to build this (Clojure CLI) and it was pretty gnarly because you'll have to build Clojure itself from source, right? And you have to specify some flags to build it without dependencies on Spec (since that's a separate lib) and without dependencies on core.specs
(again, a separate lib). Then you can build Spec, then core.specs
, then you can rebuild Clojure with those Spec dependencies...
...and clojure-tools
as an uberjar depends on a huge number of other Java libraries that you'll also have to build from source?
ah that sucks that its not simpler 😕
I know this has been answered multiple times but how am I supposed to run a Clojure app in production? clojure -X app.core/-main
?
those are independent questions
at production time, you need to a) have a classpath will all your code and deps on it and b) ultimately run some compiled class with a main - those things are true of the JVM, independent of Clojure
Perhaps. I looked for some guidance on http://Clojure.org but nothing interesting and it seems like on Stackoverflow everybody has his own opinion on the matter 🙂
a classpath can be made up of your code + dependency jars, or an uberjar of the same, or a compiled uberjar of the same
the last one has the benefit of being "one thing" and the fastest Clojure path to startup
as far as the main class, you can:
• use clojure.main and call a Clojure source file or AOT compiled Clojure namespace
• AOT compile a Clojure namespace with a -main entry point
• use clojure -M
to invoke a Clojure namespace -main
• use clojure -X
to invoke a Clojure function that takes a map
(not an exhaustive list)
there are tradeoffs and which is best for you may depend on where you're deploying, how you want to work on things in dev etc
If speed of startup is the only advantage I (personally) do not feel compelled doing uberjar
That's not the only advantage. The other one is deployment. Generally, you'd be better not relying on public repositories for your production deployments, if you use the Clojure CLI and tools.deps for that, it will pull down the defined versions of your dependencies from the public repos they are hosted in, most likely Clojars, Maven Central and Github. Each machine you deploy too, the machine will re-download them from those public repos. Some issues you can face with this is that the repo goes down while you are deploying, thus leaving your fleet in a weird state. Someone can override a version with a different code base without changing the version, so now two hosts might actually have pulled different code, even though they are marked as the same version. If you do an Uberjar, you basically download all dependencies from those public repos once, and bundle it all in a single Jar file. You then just upload that Jar to each host, so they are guaranteed to have the exact same artifact, and there's less things that can go wrong while you are deploying.
If you have an internal repository replica, this might not be as big of an advantage, like if you setup clojure cli to pull dependencies only from your internal Artifactory repo or something like that, which mirrors public repos, and is configured to never overwrite an existing versioned artifact, etc.
@U0K064KQV I do not really buy the argument. The dependency download is happening while building the docker image; so if something goes wrong during that, the image is not built and the deployment does not happen (no weird state happening). You have good points though 🙂
I'm not familiar with docker, but if you create a docker image with all dependencies pulled down and the app ready to go, and then that docker image is stored and simply copied over to each host with everything already available in the image you're right, then you don't really need an Uberjar.
You could use the Clojure CLI -P
option then when creating the docker image to have it prepare and download and cache everything, so you can later exec it.
Another way to look at it, is that an uberjar is essentially a lightweight docker. Obviously their are various advantages and disadvantages to each — but just as you can avoid using an uberjar because you’re using dockers, you can often avoid using docker because you’re using an uberjar.
@vincenz.chianese For ease of deployment, it can be worth doing an uberjar (without AOT) and then using it as your entire classpath:
java -cp the.jar clojure.main -m your.main.ns
That will use Clojure's (compiled) -main
entry point to invoke your.main.ns/-main
-- we used to do that in production for a long time.There are substantial benefits to having a single, bundled JAR file with all dependencies included as something easy to ship around/deploy, and then you only need a JDK on the target systems, be they bare metal, VMs, Docker, whatever.
(we do also have the Clojure CLI on all our servers just for the convenience of running small command-line tasks, starting a REPL, etc)
If you have an uberjar, you can run it with just the java
command -- you don't need the Clojure CLI. But there are conveniences for having the CLI deployed as well.
A threading macro/reduce style question … I have a series of validations to run on params from a ring request. For example, let’s imagine a request representing a new user signup. Let’s say I’m doing backend form validation. The first thing I want is to make sure the password and password confirmation match. The next thing is that the password should pass a min strength score check. There’s no point in checking the strength if the passwords in the form don’t match. My first guess is to do something like:
(-> params (check-passwords-match) (check-password-strength-ok) ,,,)
and each check is map->map, and a result builds up like a series of sieves.
I’m interested if there’s a good idiomatic pattern that can act the way reduced
would, to just stop and return what I’ve got when one fails. I can imagine doing it by setting a :done?
bool on the map getting passed along and only having every check fn proceed if :done?
is false, but this seems like it tangles the short-circuiting concern with the actual check in question.I guess maybe
(cond-> []
(check-passwords-match params) (conj "don't match")
(check-strength params) (conj "not strong enough")
,,,)
is close, but that doesn’t short; it’ll run all of them.maybe an actual reduce
where the collection being reduced over is a coll of the check functions, and then it’s actually possible to use (reduced ,,,)
on a failure.
For a chain of validations that you want to stop at the first failure, I would reach for reduce
I think.
For a chain of validations where I wanted all of them to run and accumulate errors, I'd add :errors []
to params
and just thread that through all of them.
@U08BW7V1V maybe something like:
(defmacro none->
"When expr is not nil, threads it into the first form (via ->)
and short-circuits. When it is nil, threads into the next
test, etc"
[expr & forms]
(assert (even? (count forms)))
(let [g (gensym)
steps (map (fn [[test step]] `(if (not (~test ~step)) (-> ~g ~step) ~g))
(partition 2 forms))]
`(let [~g ~expr
~@(interleave (repeat g) (butlast steps))]
~(if (empty? steps)
g
(last steps)))))
As a corollary to some->
. And then you could do like:
(none-> []
check-passwords-match (conj "don't match")
check-strength (conj "not strong enough"))
If you implement it with nonblocking middlewares you can always break out of the chain by way of respond and raise
Using exceptions for an expected control flow is an anti-pattern, as far as I'm concerned.
ofc now I’m pondering how much work there is in setting up an interceptor chain since those are pretty straightforward to short-circuit and maybe I’m greatly over-solving the problem.
If it's a pattern that's appearing all over your code, it's worth building an abstraction. I'll be honest, for the specific case you outlined, I'd probably just have a validate-passwords
function and have it handle both checks (conditionally) and return a vector of zero or more errors.
How often are you finding sequences of checks where you always want the first failure to exit the sequence?
For nearly all of my checking sequences, I want them all to run and give me as many errors as possible in a single pass.
In your specific case, I'd probably want to complain about poor password strength and mismatched passwords rather than make either conditional on the success of the other.
I suggest to use clojure spcec and your particular example is a job for a frontend, not a backend. You send to BE only password
without password-repeat
. You need this field only on FE.
With clojure spec you can use the same code on BE and FE to validate. At least clojure spec is what I prefer for validation over other solutions.
Here you can read a little about validating with spec. You can see here how I am doing this. There is also a library with an example in github repo.
https://clojure.wladyka.eu/posts/form-validation/
On BE I also use spec with similar concept, but of course with different libraries, because BE doesn’t visualise things on screen.
there are a bunch of libraries in this space that you can check out, I haven’t used them so I can’t endorse any in particular: • https://github.com/fmnoise/flow • https://github.com/kumarshantanu/promenade • https://github.com/adambard/failjure
but, see also https://grishaev.me/en/no-monads/
i'm in the same camp as ivan in the above linked article. monads were invented because haskell and alike have a strict type system. it solves a problem "over there".
@U0WL6FA77 in this particular example I have a constraint that there’s no frontend here; just doing it all with an old-school form POST and :flash
, but yeah, point taken.
@U3L6TFEJF thanks, my google-fu didn’t get me to Ivan’s article, which I had a dim memory of.
password repeat and checking password strength without frontend? Whatever constraint you have I still recommend clojure spec. I like it more, than whatever libraries for that purpose. But of course this is my preference. PS Article which I linked to you show how to give human readable errors for validation purpose in clojure spec. It is in ClojureScript, but in Clojure it is the same.
thanks, everybody … it’s a (very slightly) made up example so there probably some obvious ways to bypass the specific problem I outlined entirely. Great discussion and thanks for the pointers.
(also one of the things I’m loving about Clojure and FP generally … the building blocks are so simple and general the creativity is in the assembly of them, not generally in making new custom blocks all the time. One of the things that drives me a bit nuts in old-school-2x4-brick versus new-school-specialized-parts-everywhere LEGO).
for validation I often use this pattern:
(or
(user-not-found-error request)
(not-authorized-error request)
(invalid-unicorn-error request)
(the-real-response request))
where the -error
functions return something like {:success false :error ...}
and the -response
functions return something like {:success true ...}