io/copy operates on input/output streams that, in turn, operate on bytes but no characters. Why would you care about lines?

I was trying to find out if there is a standard way to copy one line stream to another line stream

I am now using a loop/recur with .readLine and .write from one stream to another

it looks like you're using readers but not stream. You can turn them into streams and then just io/copy

I'm actually trying to interact with a java.lang.Process's (a bash shell) output and input stream, by reading/writing to them my self, thats why the line by line requirement

More explicitly: I'm spawning a bash shell that calls a script, and I want to interact with it up to a point that I get a specific output, from then on I'll be handling the input / output programmatically

well, to me, there is no need to split both input and output by lines. Just write/read from/into them, and that's it

say, if the process reads a line, it will just wait until it faces the \n character

I suppose you mean I must change this:

(let [reader (io/reader (:err @streams))
      (loop []
        (let [line (.readLine reader)]
          (.write *err* line)
          (recur)))])

(loop []
  (.write *err* (.read (:err @streams)))
  (recur))

Maybe simpler: (io/copy (.getErrorStream p) *err*) where .getErrorStream is a method which returns the error stream of the process

Thats what I initially did for *out* but it did not work/blocked because I assume it buffers by default 1024 bytes first?

I don't think the buffer size does matter. The copy call should hang until the source is closed. It will be closed once the process is terminated.

But I want to stream the process output to my terminal output

It is for interactive processes not one shot ones

A trivial solution would be to redirect the err/output channels into a file. The ProcessBuilder class has redirectError and redirectOutput methods that accepts http://java.io.File instances

I want to keep everything in memory as much as possible, and the problem is that If I call any redirect() method on Process, the out/err/in streams get replaced by NullStreams

Sorry, I've got interrupted. Here is what you can use:

(def -pb (new ProcessBuilder ["ls" "-l" "/"]))
(def -p (.start -pb))
(def -out (.getInputStream -p))
( -out *out*)

total 9
drwxrwxr-x+ 63 root  admin  2016 May  1 09:13 Applications
drwxr-xr-x  93 root  wheel  2976 Nov 30 15:52 Library
drwxr-xr-x@  8 root  wheel   256 May 28  2020 System
drwxr-xr-x   5 root  admin   160 May 28  2020 Users
drwxr-xr-x   3 root  wheel    96 May 10 10:16 Volumes
drwxr-xr-x@ 38 root  wheel  1216 Dec 30  2020 bin
...

there is a sort of mixture of terms: the getInputStream method returns the output channel, and the getOutputStream returns the input

as you see, there is no need to split the output by lines

Yes, that is exactly what I did in the first place, but it works only for oneshot (non-interactive) processes! Try making an interactive bash instance with (def -pb (ProcessBuilder ["bash"]) and you'll see that io/copy blocks indefinitely because it waits for bash to exit first, while we want to show everything that bash spits in its stdout (and subsequently provide the relevant stdin to interact with it)

This doesn't work due to buffering. It might work with this solution: https://twitter.com/borkdude/status/1628786679447269377 Also more info here: https://github.com/babashka/process/discussions/102

Ah yes, I am reading about pty/tty s the last hour, will check this one too

You can io copy a string. So you can loop on the lines and just use copy on the string to whatever destination

Is this a limitation of clojure.test fixtures?

yes. Fixtures are attached to namespaces

And fixtures don't take arguments

I'm not sure if it answers your question, but in still: in our case, we bootstrap docker compose only once in the background, then run tests, then stop the compose. It works fine. E.g.

docker-compose up -d ;; spawns pg/redis/kafka in background
lein test :integration
docker-compose down

during the development, I also have docker compose running in the background, so it's easy to reach all the data sources from repl

This is what we do as well

I wanted to do it from inside the REPL, I just need a single image

I usually use cli as well (either compose or plain docker) but if I wanted to use repl only I would use https://github.com/lispyclouds/contajners. There is https://github.com/javahippie/clj-test-containers as well (never used)

@UK0810AQ2 Try this approach: https://stuartsierra.com/2016/05/19/fixtures-as-caches

I've done that in the past by checking to see if the port that I'm going to try and connect to is available, and if it isn't, try and start the process

What's wrong with the global delay?

Feels icky, although that's what I ended up going with for now

AWS DMS will help you with exactly that. https://aws.amazon.com/dms/