Clojurians Log v2
beginners
Hi, I need some pointers w.r.t. stubbing in a multi-threaded environment. I have a test which should stub a long running function with just "skipped long running function" . Before my code used multiple threads, this worked:
(defn- stubbing-fixture [f]
(with-redefs [long-running-fn (fn [& _] "skipped long running function")]
(f)))
(use-fixtures :each stubbing-fixture)with-redefs is not thread-safe, it does not work anymore in my multi-threaded code, where long-running-fn get called in a new thread. The long running function gets executed as is. How can I solve this problem?
By restructuring the code in a way that lets you override long-running-fn without with-redefs.
Or by making the tests less granular, if it's applicable here, so that you don't even care about long-running-fn.
1. I assume you mean passing the functions as arguments? 2. That would work, but is not desired for my use case. Basically it's an integration test of code that sends out emails at various points, which I want to avoid during the test run
1. That would be the most reasonable option, yes. Either explicitly or by using something like Integrant and turning that function into a separate component. Another option is a dynamic var, but in the long run it's a strictly worse option 2. In that case, a potential alternative is to have that function respect some external configuration and disable sending out emails via that configuration. But that then automatically becomes something for which I'd use Integrant from the get go
Lucky me, I have Integrant already set up (with an aero-based system.edn where I can switch on #profile -> prod/dev/test). I guess I just did not yet wire it up correctly for this. I am not entirely sure what a good way would be here. For example I am currently setting up my AWS-SES (email sending service) client like so:
(defmethod ig/init-key :aws/ses-client [_ {:keys [credentials-provider]}]
(make-client-fn credentials-provider)):env key to make a dummy client function when the env is :test ?
This is how I would try to do it but it seems like it might not be the best way...Either the :aws/ses-client component itself can decide what to do when some particular config is received, or the config can decide so externally.
To expand on the latter - you can have your tests adjust the system config so that :aws/ses-client points to a mock component.
The latter idea sounds brilliant, thanks. I'll give that a go and may report back if I run into problems 🙂
Just in case - I had some misunderstanding the way Integrant helps you with this, but the author of Integrant helped me a bit here: https://old.reddit.com/r/Clojure/comments/1iiz6j1/open_source_diary_cv4/mba539b/
What do people typically use in clojure to read/write to a daemon process' stdin/stdout? do we typically just call the java API for e.g. ProcessBuilder with the java ...Writer and ...Reader classes?
oh it seems that the very last clojure stable version introduced clojure.java.process for this purpose, correct?
Yes
neat, thank you!
In Linux, what is the underlying mechanism through which BufferedWriter is able to write to another process' stdin? Is this done by means of spawning a linux pipe?
I'm a bit confused about what exactly the input streams and output streams are, for a Process that's been started with ProcessBuilder
from: https://docs.oracle.com/javase/8/docs/api/java/lang/ProcessBuilder.html
> • a source of standard input. By default, the subprocess reads input from a pipe. Java code can access this pipe via the output stream returned by Process.getOutputStream(). However, standard input may be redirected to another source using redirectInput. In this case, Process.getOutputStream() will return a null output stream, for which:
> ◦ the write methods always throw IOException
> ◦ the close method does nothing
> • a destination for standard output and standard error. By default, the subprocess writes standard output and standard error to pipes. Java code can access these pipes via the input streams returned by Process.getInputStream() and Process.getErrorStream(). However, standard output and standard error may be redirected to other destinations using redirectOutput and redirectError. In this case, Process.getInputStream() and/or Process.getErrorStream() will return a null input stream, for which:
> ◦ the read methods always return -1
> ◦ the available method always returns 0
> ◦ the close method does nothing
>
ah thank you very much! I got a bit lost in the java docs trying to figure out which class is doing what, I completely missed this, ty!
Hello. I'm trying to figure out why the JVM process exists when I stop the http-kit server
I read about threads and daemon threads and seems to me as long as the nREPL server thread keeps running, the JVM process should too but I think I'm missing something.
> This small example is part of my effort to use clj-reload and mount in a similar situation - to stop parts of an application (e.g. http-kit web server), reload namespaces and then start the parts up again. All this while maintaining the JVM process and nREPL server running.
Minimal example for reproducing is here:
src/main.clj:
(ns main
(:require
[org.httpkit.server :as http]
[nrepl.server :as nrepl]))
(defonce nrepl-server (nrepl/start-server :port 7888))
(defn stop-nrepl [] (nrepl/stop-server nrepl-server))
(def http-server-stopper (atom nil))
(defn handler [_] {:status 200 :headers {"Content-Type" "text/plain"} :body "Hello, World!"})
(defn -main [& _]
(reset! http-server-stopper (http/run-server handler {:port 8080})))deps.edn:
{:paths ["src"]
:deps {org.clojure/clojure {:mvn/version "1.12.1"}
http-kit/http-kit {:mvn/version "2.8.0"}
nrepl/nrepl {:mvn/version "1.3.1"}}}clojure -M -m main
2. Sanity check for web server: curl
3. Stop the http-kit server in REPL
lein repl :connect localhost:7888
user=> (in-ns 'main)
#object[clojure.lang.Namespace 0x5c61ee82 "main"]
main=> (@http-server-stopper)
SocketException The transport's socket appears to have lost its connection to the nREPL server
nrepl.transport/bencode/fn--10334/fn--10335 (transport.clj:122)
nrepl.transport/bencode/fn--10334 (transport.clj:122)
nrepl.transport/fn-transport/fn--10300 (transport.clj:50)
clojure.core/binding-conveyor-fn/fn--5823 (core.clj:2047)
java.util.concurrent.FutureTask.run (FutureTask.java:317)
java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1144)
java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:642)
java.lang.Thread.run (Thread.java:1583)
Bye for now!@hiredman I think I got to the bottom of it.
My (wrong) assumption was that nREPL server when binding to a port & listening will use a non-daemon thread for it. After digging in the source I found it not to be the case. So indeed nREPL binds and listens in a daemon thread, so when HTTP server is gone, there's nothing left to wait upon.
1. Starts listening with threading/run-with https://github.com/nrepl/nrepl/blob/3cc751aa78e822de8ca38d4959477c89aec6d741/src/clojure/nrepl/server.clj#L235
2. listen-executor https://github.com/nrepl/nrepl/blob/3cc751aa78e822de8ca38d4959477c89aec6d741/src/clojure/nrepl/util/threading.clj#L33
3. thread-factory uses DaemonThreadFactory https://github.com/nrepl/nrepl/blob/3cc751aa78e822de8ca38d4959477c89aec6d741/src/clojure/nrepl/util/threading.clj#L17
4. t.setDaemon(true) 🙂 https://github.com/nrepl/nrepl/blob/3cc751aa78e822de8ca38d4959477c89aec6d741/src/java/nrepl/DaemonThreadFactory.java#L26
Do you have clojure-idiomatic approaches to recommend how to keep the process from exiting? I remember having seen clojure.core.future used for it or maybe reading from an "exit" channel (that when value gets put into it causes main process to terminate)? Or something else?
Went for (future (async/<!! stop-ch)) in main/-main
I'll check back here for more suggestions. Thank you for your responses hiredman & practically-johnny!
The way I usually do this is I first start the repl (`lein repl` or clj -M <long options for cider et al>), and the within that long-running repl I start the application.
You seem to be doing it the other way around: starting the application and then running an nrepl server within it. With that approach, it's hard to see how you can easily make the nrepl server to survive the app itself.
Using clj I'd do clj -M:cider-clj using this definition: https://github.com/gaverhae/dotfiles/blob/380d8b562ded7485b5531f4fb338de651a0feaa9/home/dot/clojure/deps.edn
A more complete example, unfortunately using lein: https://github.com/gaverhae/vendom/blob/main/dev/repl.clj
I'd start my repl in that ("repl-only") namespace and do (go) to (re)start the server.
putting a @(promise) at the end of -main is not unheard of, it will stop the main thread from finishing
The nrepl client exiting and the jvm exiting are not the same thing
But actually in this case both are happening. It is because the jvm exits when there are no running or blocked threads that are not marked as daemon threads
Your main thread starts the http server and then finishes running, the http server is running some non-daemon thread somewhere, you stop the http server, no more threads and the jvm exits
I run the REPL (with an nrepl server) and then start the server from the REPL session. Stopping or restarting the server does not terminate the REPL