Fork me on GitHub
#clojure
<
2021-09-17
>
Lennart Buit14:09:13

Exceptions on a future that isn’t deref’ed don’t end up in the default uncaught exception handler, right. What would I use if I do want to ‘fire and forget’ a task, but do want raised exceptions to propagate to my default uncaught exception handler?

Lennart Buit14:09:13

Is this something I should want even

Lennart Buit14:09:05

Right yeah I had found that, so that would suggest that I have a different executor service somewhere and manually .execute (not! .submit) my task?

Fredrik16:09:22

Do you want the exception to propagate to the thread that called the future?

Lennart Buit16:09:42

I don’t want the current thread to be blocked on this task, but I do want to log failure of this task (asynchronously)

Fredrik16:09:47

Could you do something like this?

(require '[clojure.core.async :as a])

(let [c (a/chan 10)]

  (defn submit-future [f]
    (a/put! c (future (f)))
    (println "future submitted"))

  (a/go
    (loop []
      (when-let [val (a/<! c)]
        (try
          @val
          (catch java.util.concurrent.ExecutionException e
            (println (.getCause e))))
        (recur)))))

Fredrik16:09:53

(submit-future (fn [] (/ 1 0)))
=> #error {:cause Divide by zero
           ...}

Lennart Buit16:09:21

Right, but I can also put a try catch inside my future call to do that right. I was wondering whether there would be a way to not do logging at this point, but instead rely on the default uncaught exception handler I have set up anyway. Does that make sense?

Lennart Buit16:09:20

(Missed a negation, edited)

Fredrik16:09:41

It seems that the ExecutorService running the futures catch exceptions before the default uncaught exception handler has a chance of dealing with them. I don't know if this is recommended usage, but have you looked at clojure.core.async/thread?

Fredrik16:09:42

(Thread/setDefaultUncaughtExceptionHandler
 (reify Thread$UncaughtExceptionHandler
   (uncaughtException [_ thread ex]
     (println "Uncaught exception on" (.getName thread)))))

(a/thread
  (/ 1 0))

=> Uncaught exception on async-thread-macro-9

Lennart Buit16:09:57

Oh interesting, didn’t know that core async threads didn’t have this behavior

Lennart Buit16:09:43

Having the same hesitation you have… but thanks for thinking along :)!

😀 2
Fredrik16:09:42

If you want to write your own Executor instead of depending on clojure.core.async, you can check out clojure.core.async/thread-call for inspiration

Fredrik17:09:54

You can see it uses .execute instead of .submit

vemv20:09:18

Nice q! A related thing to wonder / research is why the executorservice has this design. Futures are a bit awkward in this regard and play against another Java mechanism (the Thread default handlers) Maybe it is for a good reason?

vemv20:09:50

The best reason I can immediately think of is that fire-and-forget isn't really a thing to be fostered, if you wanted to you could as well spawn Threads by hand (which is super easy) and skip the ExService machinery altogether i.e. either you're doing things quick-and-dirty or not, no in-between

Joshua Suskalo14:09:38

I have an api question: I want to provide a new reference type in a library. It does not have atomic semantics because it's backed by a native pointer. What I had wanted to do is to use proxy to extend clojure.lang.Volatile and use the existing vreset! and vswap! operations, however Volatile is a final class, and therefore I can't proxy it. I see two paths forward, either A) extend IAtom and IAtom2 to provide the methods that are already provided on atoms, or B) move it to a deftype and roll my own reset and swap functions. At the moment I'm leaning towards B, but I like the idea of providing the same interface as one of the core references. I'd like some feedback on which would be preferable.

borkdude15:09:50

Would IRef work?

borkdude15:09:10

should people be able to set the value or only read from it?

Joshua Suskalo15:09:55

IRef actually is the only part of the general reference api I don't want to support.

Joshua Suskalo15:09:07

Because I can't add watches onto native memory.

Joshua Suskalo15:09:38

And the idea is to be able to both read and write, just not atomically.

borkdude15:09:13

Having a separate API is not a bad idea I think

2
Joshua Suskalo15:09:26

Alright, then I'll go with that. Probably just call it freset! and fswap! for "foreign swap"

👍 2
noisesmith17:09:14

since there's no question of atomicity or retries, why not just use set! which actually has the semantics you are implementing?

noisesmith17:09:29

offering a swap with no atomicity or retries seems misleading

borkdude17:09:28

@U051SS2EU is there an interface you can use with set! ?

noisesmith17:09:51

it appears to be clojure.lang.Settable - finding it now

noisesmith17:09:29

I'm not sure what the difference between doSet and doReset is supposed to be

noisesmith17:09:09

it looks as if compiler.java expands set! to a doSet call and there are no matches for doReset in the file

borkdude17:09:11

nice, I didn't know that

noisesmith17:09:37

yeah, we have nicer ways to update a state, set! is pretty hazardous in comparison

noisesmith17:09:06

but it seems to match the semantics of what we are discussing here - sets the right expectations

Joshua Suskalo18:09:11

Oh, I wasn't aware that set! was extensible

Joshua Suskalo18:09:43

I agree this does seem to fit with the semantics pretty well, but I will admit that I don't know how I feel about using set! on something that's not directly mutable but is rather a reference to something mutable

Joshua Suskalo18:09:00

Like to get the value you have to use deref

borkdude18:09:14

@U5NCUG8NR This is why I asked if the value is read-only or also mutable

borkdude18:09:25

if it's only readable by the users, then I would implement IDeref

borkdude18:09:42

(or your own API, still a good choice as well)

noisesmith18:09:50

looks like it doesn't actually work..

(ins)user=> (set! (reify clojure.lang.Settable (doSet [this v] "OK")) :foo)
Syntax error (IllegalArgumentException) compiling set! at (REPL:1:1).
Invalid assignment target

Joshua Suskalo18:09:43

Well it's both. I have IDeref, but the value behind it is any type. Additionally, there is a way to mutate the value behind it, but it itself isn't mutable.

Joshua Suskalo18:09:58

What this maps to by the way is a static global variable in a native library.

borkdude18:09:38

user=> (set! (reify clojure.lang.Settable (doSet [this v] (prn :hello))) 1)
Syntax error (IllegalArgumentException) compiling set! at (REPL:1:1).
Invalid assignment target

Joshua Suskalo18:09:50

doing a deref in clojure will dereference the pointer and deserialize what it finds into a clojure data structure (and it's on the user to ensure that you do that at a time when its state won't be updating to see an invalid state)

borkdude18:09:53

oh. you did the same :)

Joshua Suskalo18:09:22

doing a set will also dereference the pointer, and then serialize a clojure data structure into the memory segment it finds there.

nivekuil18:09:38

I'm generating a kafka streams serde with deftype. it works in dev, but in prod (jib built docker image, no AOT) the class cannot be found when kafka streams tries to load it, even though I explictly import it before kafka streams is initialized. how do I go about debugging this?

borkdude18:09:09

import in Java doesn't make anything compile, you need to AOT that type before using it in Java

nivekuil18:09:06

so I actually solved it without understanding why.. you can pass to the kafka streams config either a string representing the full path of the class (only works in dev) or the class instance itself (works in prod). This is AOT related? not sure why it would work in dev, when is the type being compiled there?

p-himik18:09:35

Given the symptom, perhaps there are two classes with the same name. Using a string loads it from some other place, so that x.y.Z is not the same as the class instance x.y.Z.

borkdude18:09:45

$ mkdir -p src/foo
$ echo '(ns foo.core) (deftype Foo [])' > src/foo/core.clj
$ clj -M -e "(compile 'foo.core)"
foo.core
$ ls classes/foo/Foo.class
classes/foo/Foo.class
You need to actually see the Foo class there as a result of compiling the deftype

borkdude18:09:59

If that class isn't there (or in the jar, or however you package) then Java cannot find it

ghadi18:09:14

it's probably that the kafka streams' classloader cannot resolve the class from the name

ghadi18:09:41

if the clojure classloader is a child of the kafka streams classloader, it will not see it

nivekuil19:09:29

thanks for the help, I didn't realize a classloader was a whole thing in itself

Oliver Marks19:09:28

anyone know is the repo for clojure cli the same as the clojure repo I get the feeling they are different but I cant find a link to the clojure cli repo

p-himik19:09:41

Don't trust the name.

Oliver Marks19:09:08

oh so that's for linux as well, I assumed that was mac only

borkdude19:09:07

correct, that is the bash script

p-himik19:09:39

It's just that there was some shuffling at some point and I remember trying to understand an outdated script at some point.

Oliver Marks19:09:57

okay great thanks

seancorfield20:09:01

@U02DXJUS5JA brew works great on Linux too BTW -- that's what I use on both macOS and Ubuntu for my Clojure-releated deps.

Oliver Marks08:09:26

on a related note I notice https://github.com/clojure/brew-install/releases/tag/1.10.3.981 is different to the download it seems to be missing a comple of files in particular clojure-tools-${project.version}.jar is refereed to in the install but does not exist in the sources, It also does not look like there is a build to create this, anyone know how this works, I am trying to build a package if your wondering why I am after these details. It works when I built I as requested to make it build from the source which is why I am hitting a few issues.

borkdude08:09:20

yeah, it's a template

Oliver Marks11:09:16

@U04V15CAJ not sure what you mean I get that project.version is replaced, but where does clojure-tools.jar come from its not in the release do you build it does it come from some where else ? when you download from the clojure site it does exist in the archive

seancorfield17:09:09

brew-install is the source repo and the tags are on that source; the downloads that GitHub creates there are of the source; but the "download" -- the deployed artifact is not those, it is "built" into the appropriate shape. You can't expect to download GitHub's automated source ZIPs and have them work for libraries/tools that include build steps.

Oliver Marks07:09:48

yeah I thought it might be part of the build, I guess the question kind of becomes where is the source for clojure-tools-1.10.3.967.jar which contains the files below as they are not in the brew-install script repo

borkdude07:09:09

@U02DXJUS5JA If you are creating a package for linux, I think you can re-use those ingredients (tools jar, bash script), without knowing how they were built? I'm doing a similar thing for deps.clj.

borkdude07:09:20

The tools jar is an uberjar which packages tools.deps.alpha with a main function that is being invoked from the bash script.

borkdude07:09:38

This jar is probably built with code from the brew-install repo.

Oliver Marks09:09:04

okay thanks that makes sense, I think the main issue is that the package maintainers ideally want to build from source and sha check the files to avoid files being injected at a later date

Oliver Marks09:09:53

If i can build the jar and include it that may work, I kind of understand why just makes packaging it more difficult 😕

Oliver Marks09:09:09

but thanks for the tip I will look into that when I get a chance

seancorfield18:09:55

Someone else recently went through the whole "compile from source" thing to build this (Clojure CLI) and it was pretty gnarly because you'll have to build Clojure itself from source, right? And you have to specify some flags to build it without dependencies on Spec (since that's a separate lib) and without dependencies on core.specs (again, a separate lib). Then you can build Spec, then core.specs, then you can rebuild Clojure with those Spec dependencies...

seancorfield18:09:29

...and clojure-tools as an uberjar depends on a huge number of other Java libraries that you'll also have to build from source?

Oliver Marks20:09:22

ah that sucks that its not simpler 😕

vncz20:09:12

I know this has been answered multiple times but how am I supposed to run a Clojure app in production? clojure -X app.core/-main?

vncz20:09:23

When shall I jar?

Alex Miller (Clojure team)20:09:07

those are independent questions

Alex Miller (Clojure team)20:09:15

at production time, you need to a) have a classpath will all your code and deps on it and b) ultimately run some compiled class with a main - those things are true of the JVM, independent of Clojure

vncz20:09:26

Perhaps. I looked for some guidance on http://Clojure.org but nothing interesting and it seems like on Stackoverflow everybody has his own opinion on the matter 🙂

Alex Miller (Clojure team)20:09:54

a classpath can be made up of your code + dependency jars, or an uberjar of the same, or a compiled uberjar of the same

Alex Miller (Clojure team)20:09:19

the last one has the benefit of being "one thing" and the fastest Clojure path to startup

Alex Miller (Clojure team)20:09:51

as far as the main class, you can: • use clojure.main and call a Clojure source file or AOT compiled Clojure namespace • AOT compile a Clojure namespace with a -main entry point • use clojure -M to invoke a Clojure namespace -main • use clojure -X to invoke a Clojure function that takes a map

Alex Miller (Clojure team)20:09:00

(not an exhaustive list)

Alex Miller (Clojure team)20:09:56

there are tradeoffs and which is best for you may depend on where you're deploying, how you want to work on things in dev etc

vncz20:09:20

If speed of startup is the only advantage I (personally) do not feel compelled doing uberjar

didibus03:09:47

That's not the only advantage. The other one is deployment. Generally, you'd be better not relying on public repositories for your production deployments, if you use the Clojure CLI and tools.deps for that, it will pull down the defined versions of your dependencies from the public repos they are hosted in, most likely Clojars, Maven Central and Github. Each machine you deploy too, the machine will re-download them from those public repos. Some issues you can face with this is that the repo goes down while you are deploying, thus leaving your fleet in a weird state. Someone can override a version with a different code base without changing the version, so now two hosts might actually have pulled different code, even though they are marked as the same version. If you do an Uberjar, you basically download all dependencies from those public repos once, and bundle it all in a single Jar file. You then just upload that Jar to each host, so they are guaranteed to have the exact same artifact, and there's less things that can go wrong while you are deploying.

didibus03:09:03

If you have an internal repository replica, this might not be as big of an advantage, like if you setup clojure cli to pull dependencies only from your internal Artifactory repo or something like that, which mirrors public repos, and is configured to never overwrite an existing versioned artifact, etc.

vncz16:09:56

@U0K064KQV I do not really buy the argument. The dependency download is happening while building the docker image; so if something goes wrong during that, the image is not built and the deployment does not happen (no weird state happening). You have good points though 🙂

didibus22:09:37

I'm not familiar with docker, but if you create a docker image with all dependencies pulled down and the app ready to go, and then that docker image is stored and simply copied over to each host with everything already available in the image you're right, then you don't really need an Uberjar. You could use the Clojure CLI -P option then when creating the docker image to have it prepare and download and cache everything, so you can later exec it.

rickmoynihan08:09:00

Another way to look at it, is that an uberjar is essentially a lightweight docker. Obviously their are various advantages and disadvantages to each — but just as you can avoid using an uberjar because you’re using dockers, you can often avoid using docker because you’re using an uberjar.

seancorfield20:09:06

@vincenz.chianese For ease of deployment, it can be worth doing an uberjar (without AOT) and then using it as your entire classpath:

java -cp the.jar clojure.main -m your.main.ns
That will use Clojure's (compiled) -main entry point to invoke your.main.ns/-main -- we used to do that in production for a long time.

seancorfield20:09:23

There are substantial benefits to having a single, bundled JAR file with all dependencies included as something easy to ship around/deploy, and then you only need a JDK on the target systems, be they bare metal, VMs, Docker, whatever.

vncz20:09:51

What do you mean by “I only need a JDK on the target system”?

seancorfield20:09:12

(we do also have the Clojure CLI on all our servers just for the convenience of running small command-line tasks, starting a REPL, etc)

vncz20:09:24

Yeah indeed, so I guess I would still use the clojure image anyway

seancorfield20:09:48

If you have an uberjar, you can run it with just the java command -- you don't need the Clojure CLI. But there are conveniences for having the CLI deployed as well.

rgm23:09:07

A threading macro/reduce style question … I have a series of validations to run on params from a ring request. For example, let’s imagine a request representing a new user signup. Let’s say I’m doing backend form validation. The first thing I want is to make sure the password and password confirmation match. The next thing is that the password should pass a min strength score check. There’s no point in checking the strength if the passwords in the form don’t match. My first guess is to do something like:

(-> params (check-passwords-match) (check-password-strength-ok) ,,,)
and each check is map->map, and a result builds up like a series of sieves. I’m interested if there’s a good idiomatic pattern that can act the way reduced would, to just stop and return what I’ve got when one fails. I can imagine doing it by setting a :done? bool on the map getting passed along and only having every check fn proceed if :done? is false, but this seems like it tangles the short-circuiting concern with the actual check in question.

rgm23:09:36

If I squint it starts to look like Maybe or something similar.

rgm23:09:10

I guess maybe

(cond-> []
   (check-passwords-match params) (conj "don't match")
   (check-strength params) (conj "not strong enough")
   ,,,) 
is close, but that doesn’t short; it’ll run all of them.

rgm23:09:38

maybe an actual reduce where the collection being reduced over is a coll of the check functions, and then it’s actually possible to use (reduced ,,,) on a failure.

seancorfield23:09:52

For a chain of validations that you want to stop at the first failure, I would reach for reduce I think.

seancorfield23:09:30

For a chain of validations where I wanted all of them to run and accumulate errors, I'd add :errors [] to params and just thread that through all of them.

rgm23:09:42

thanks, yeah … I think you’ve talked me into reduce and reduced.

rgm23:09:23

seems like the cleanest way to make the individual checks testable.

rgm23:09:56

then they’re all check :: params -> [error-string]

rgm00:09:11

I guess I don’t hate this

john00:09:12

@U08BW7V1V maybe something like:

(defmacro none->
  "When expr is not nil, threads it into the first form (via ->) 
   and short-circuits. When it is nil, threads into the next 
   test, etc"
  [expr & forms]
  (assert (even? (count forms)))
  (let [g (gensym)
        steps (map (fn [[test step]] `(if (not (~test ~step)) (-> ~g ~step) ~g))
                   (partition 2 forms))]
    `(let [~g ~expr
           ~@(interleave (repeat g) (butlast steps))]
       ~(if (empty? steps)
          g
          (last steps)))))
As a corollary to some->. And then you could do like:
(none-> []
        check-passwords-match (conj "don't match")
        check-strength (conj "not strong enough"))

👍 2
john00:09:03

Sorry, there was a bug in the above impl and I updated it

🙏 2
Ben Sless05:09:41

If you implement it with nonblocking middlewares you can always break out of the chain by way of respond and raise

rgm06:09:04

hm … a homebrew try-catch middleware pattern is an interesting idea

seancorfield06:09:04

Using exceptions for an expected control flow is an anti-pattern, as far as I'm concerned.

rgm06:09:46

yeah, not great from that point of view.

rgm06:09:01

ofc now I’m pondering how much work there is in setting up an interceptor chain since those are pretty straightforward to short-circuit and maybe I’m greatly over-solving the problem.

seancorfield06:09:08

If it's a pattern that's appearing all over your code, it's worth building an abstraction. I'll be honest, for the specific case you outlined, I'd probably just have a validate-passwords function and have it handle both checks (conditionally) and return a vector of zero or more errors.

seancorfield06:09:33

How often are you finding sequences of checks where you always want the first failure to exit the sequence?

seancorfield06:09:12

For nearly all of my checking sequences, I want them all to run and give me as many errors as possible in a single pass.

seancorfield06:09:51

In your specific case, I'd probably want to complain about poor password strength and mismatched passwords rather than make either conditional on the success of the other.

kwladyka08:09:05

I suggest to use clojure spcec and your particular example is a job for a frontend, not a backend. You send to BE only password without password-repeat. You need this field only on FE. With clojure spec you can use the same code on BE and FE to validate. At least clojure spec is what I prefer for validation over other solutions. Here you can read a little about validating with spec. You can see here how I am doing this. There is also a library with an example in github repo. https://clojure.wladyka.eu/posts/form-validation/ On BE I also use spec with similar concept, but of course with different libraries, because BE doesn’t visualise things on screen.

schmee10:09:00

there are a bunch of libraries in this space that you can check out, I haven’t used them so I can’t endorse any in particular: • https://github.com/fmnoise/flowhttps://github.com/kumarshantanu/promenadehttps://github.com/adambard/failjure

kulminaator13:09:16

i'm in the same camp as ivan in the above linked article. monads were invented because haskell and alike have a strict type system. it solves a problem "over there".

☝️ 2
👍 2
Ben Sless13:09:55

some-> is usually good enough

rgm16:09:30

:man-facepalming: oh yeah, I guess some-> would do it.

rgm16:09:50

@U0WL6FA77 in this particular example I have a constraint that there’s no frontend here; just doing it all with an old-school form POST and :flash, but yeah, point taken.

👍 2
rgm16:09:11

@U3L6TFEJF thanks, my google-fu didn’t get me to Ivan’s article, which I had a dim memory of.

👍 2
kwladyka16:09:30

password repeat and checking password strength without frontend? Whatever constraint you have I still recommend clojure spec. I like it more, than whatever libraries for that purpose. But of course this is my preference. PS Article which I linked to you show how to give human readable errors for validation purpose in clojure spec. It is in ClojureScript, but in Clojure it is the same.

🙏 2
rgm16:09:32

thanks, everybody … it’s a (very slightly) made up example so there probably some obvious ways to bypass the specific problem I outlined entirely. Great discussion and thanks for the pointers.

🍻 2
rgm16:09:01

(also one of the things I’m loving about Clojure and FP generally … the building blocks are so simple and general the creativity is in the assembly of them, not generally in making new custom blocks all the time. One of the things that drives me a bit nuts in old-school-2x4-brick versus new-school-specialized-parts-everywhere LEGO).

opqdonut04:09:08

for validation I often use this pattern:

(or
 (user-not-found-error request)
 (not-authorized-error request)
 (invalid-unicorn-error request)
 (the-real-response request))
where the -error functions return something like {:success false :error ...} and the -response functions return something like {:success true ...}

opqdonut04:09:42

That's for when you can't / don't want to collate all errors. If you want to do that, you can write a custom function merge-errors and then do something like (or (merge-errors (foo-error req) (bar-error req)) (resp req))

rgm16:09:17

oh for the love of. Yep. or has exactly the right behaviour here. Thanks.