Fork me on GitHub
#babashka
<
2023-09-13
>
Mateusz Mazurczak10:09:09

Hi, I'm trying to process streaming output from the docker build command. But for some reason my read-line is always nil, even though I get output to the console from the command itself. So I have fn like this

(let [proc (p/process command
                            {:dir dir
                             :shutdown p/destroy-tree
                             :out :inherit
                             :err :inherit
                             })]
        (with-open [rdr (io/reader (:out proc))]
          (binding [*in* rdr]
            (loop []
              (when-let [line (read-line)]
                (log/trace line))
              (when (or (.ready rdr) (.isAlive (:proc proc)))
                (recur))))))
where command is: ["docker" "build" "--platform" "linux/amd64" "-t" my_image "."] And generally, it works, as the output of the docker build is properly logged to the console, but streaming doesn't work. As I want to take that output to the file and for some reason, I always get nil in
(when-let [line (read-line)]
I'm using newest versions of bb.
babashka/fs #:mvn{:version "0.4.19"},
babashka/process #:mvn{:version "0.5.21"},
Any idea how to read the output?

borkdude11:09:11

If you do :out :inherit , then the output is sent to the OS stdout

borkdude11:09:31

so you have to omit that, if you want to read the output yourself

Mateusz Mazurczak11:09:37

I've tried that already, ommiting :out in a process map, changes nothing, it acts the same. Output to console, line is nil

borkdude11:09:49

with :out :inherit it wil surely not work

๐Ÿ‘ 1
Mateusz Mazurczak11:09:40

Okay, I've removed it, but any idea why it still is not working?

borkdude11:09:32

Try this:

(require '[babashka.process :as p] '[ :as io])

(let [proc (p/process "bb -e '(loop [] (println \"Hello\") (Thread/sleep 3000) (recur))'"
                      {:shutdown p/destroy-tree
                       :err :inheri})]
  (with-open [rdr (io/reader (:out proc))]
    (binding [*in* rdr]
      (loop []
        (when-let [line (read-line)]
          (prn :line line))
        (when (or (.ready rdr) (.isAlive (:proc proc)))
          (recur))))))

borkdude11:09:44

This example works for me

Mateusz Mazurczak11:09:31

Yes, this example also works for me

Mateusz Mazurczak11:09:47

But when it's replaced with docker command it's not working

borkdude11:09:24

The logical explanation could be that docker is writing to stderr and not stdout?

borkdude11:09:18

no, that's not it, I'm testing locally now

โค๏ธ 1
borkdude11:09:01

When you do:

docker build . > /tmp/log.txt
you see it won't redirect to the log file either, at least not on my system

๐Ÿ‘ 1
borkdude11:09:13

so maybe first investigate that

borkdude11:09:43

This does work:

docker build . 2> /tmp/log.txt

๐Ÿ‘ 1
borkdude11:09:19

Ah got it:

(require '[babashka.process :as p] '[ :as io])

(let [proc (p/process "docker build .'"
                      {:shutdown p/destroy-tree})]
  (future (with-open [rdr (io/reader (:out proc))]
            (binding [*in* rdr]
              (loop []
                (when-let [line (read-line)]
                  (prn :stdout line))
                (when (or (.ready rdr) (.isAlive (:proc proc)))
                  (recur))))))
  (with-open [rdr (io/reader (:err proc))]
    (binding [*in* rdr]
      (loop []
        (when-let [line (read-line)]
          (prn :stderr line))
        (when (or (.ready rdr) (.isAlive (:proc proc)))
          (recur))))))

borkdude11:09:48

When you read from stdout it will block, but when you also want to read from stderr, you'll have to do the first in a future or so, to not block that

borkdude11:09:35

So tl;dr, docker logs to stderr

๐Ÿ’ก 2
๐Ÿ˜„ 2
Mateusz Mazurczak12:09:08

Thanks borkdude! You are a hero!

๐Ÿ˜† 1
jackrusher14:09:43

periodic reminder that you cannot purchase a support contract this good

๐Ÿ‘ 20
โค๏ธ 21
gratitude 9
1
โ˜๏ธ 7
โž• 6
clojure-spin 1
elken14:09:50

Appreciate you and borkdude ๐Ÿ™‚ And everyone else here who so frequently helps others

๐Ÿ–ค 7
โž• 3
vemv20:09:25

Does babashka (-nrepl) have the notion of a classpath? I see that bb-nrepl does not implement the classpath op (it's not in nrepl proper to be fair, but a cider-nrepl op) CIDER can fall back to (thread-first "(seq (.split (System/getProperty \"java.class.path\") \":\"))" (cider-sync-tooling-eval) (nrepl-dict-get "value") read) - should that work?

vemv20:09:48

anyway I'll start playing with this stuff tomorrow, but probably the discussion has at least some value

borkdude20:09:11

Afk now brb

borkdude20:09:29

$ bb -cp 'foo' -e '(System/getProperty "java.class.path")'
"foo"

vemv20:09:36

Thanks! Will investigate tomorrow why multiple things in the cider <-> bb-nrepl integration aren't doing the right thing

borkdude20:09:20

this requires a relatively new version of bb, the sync between System/getProperty of the classpath is a recent thing I believe

borkdude20:09:40

other way of getting the bb classpath: (babashka.classpath/get-classpath) , this has worked for years

๐Ÿ‘ 2
borkdude11:09:41

@U45T93RA6 I made this commit on a flight to strangeloop yesterday: https://github.com/babashka/babashka.nrepl/tree/classpath-op does that look alright? perhaps you have a suggestion on how to test this?

borkdude11:09:32

(except for the failing test, this is just a minor problem ;))

vemv11:09:47

If you can include a unit test, I'll be able to confirm that its 'schema' matches cider-nrepl's

๐Ÿ‘ 1
borkdude11:09:32

basically it would look like {"classpath" ["src" "test" "/Users/borkdude/.m2/foobar/foobar.jar"]}

vemv11:09:50

Thanks, I've set a reminder for tomorrow

borkdude11:09:23

I'm using cljfmt from clojure-lsp whenever I hit a keystroke in emacs that does formatting. No complaints here.

vemv11:09:53

wrong ๐Ÿงต

๐Ÿ˜‚ 1
borkdude12:09:22

jetlag ;)

๐ŸŒ€ 1
vemv12:09:58

(Have a great one!)

borkdude12:09:09

thank you :)

borkdude12:09:30

(I had to move this to babashka proper since babashka.nrepl can be re-used for other non-bb projects)

vemv22:09:22

You can exercise nrepl's classpath op with:

(cider-sync-request:classpath)
which is reflected in *nrepl-messages* as e.g.:
(<--
  id         "7"
  session    "5c300c9d-19d2-43d4-aa68-fc5db6a61328"
  time-stamp "2023-09-23 15:00:28.293704000"
  classpath  ("/Users/vemv/orchard/test-cljs" "/Users/vemv/orchard/test" "/Users/vemv/orchard/test-newer-jdks" "/Users/vemv/.lein/scripts" "/Users/vemv/formatting-stack/src" "/Users/vemv/formatting-stack/worker" "/Users/vemv/formatting-stack.alias-rewriter/src" "/Users/vemv/parallel-reload/src" "/Users/vemv/eastwood/src" "/Users/vemv/eastwood/copied-deps" "/Users/vemv/tufte.auto/src" "/Users/vemv/orchard/src-spec-alpha-2/src/main/clojure" "/Users/vemv/orchard/dev" "/Users/vemv/orchard/src" "/Users/vemv/orchard/src-jdk8" "/Users/vemv/orchard/src-newer-jdks" "/Users/vemv/formatting-stack/resources" "/Users/vemv/eastwood/resource" "/Users/vemv/eastwood/resources" "/Users/vemv/.lein/resources/rebl.jar" "/Users/vemv/orchard/test-resources" "/Users/vemv/orchard/not-a.jar" "/Users/vemv/orchard/does-not-exist.jar" "/Users/vemv/orchard/dev-resources" "/Users/vemv/orchard/resources" "/Users/vemv/orchard/target/classes" "/Users/vemv/.m2/repository/com/cognitect/aws/api/0.8.612/api-0.8.612.jar" "/Users/vemv/.m2/repository/com/stuartsierra/component.repl/0.2.0/component.repl-0.2.0.jar" "/Users/vemv/.m2/repository/joda-time/joda-time/2.10/joda-time-2.10.jar" "/Users/vemv/.m2/repository/org/apache/maven/shared/maven-shared-utils/3.3.4/maven-shared-utils-3.3.4.jar" "/Users/vemv/.m2/repository/com/gfredericks/test.chuck/0.2.13/test.chuck-0.2.13.jar" "/Users/vemv/.m2/repository/org/apache/maven/resolver/maven-resolver-util/1.8.2/maven-resolver-util-1.8.2.jar" "/Users/vemv/.cache/mx.cider/enrich-classpath/1601/2147402929/1797031119.jar" "/Users/vemv/orchard/java" "/Users/vemv/orchard/test-java" "/Users/vemv/.cache/mx.cider/unzipped-jdk-sources/1601" "/Library/Java/JavaVirtualMachines/adoptopenjdk-16.jdk/Contents/Home/lib/src.zip")
  status     ("done")
)

vemv22:09:31

So, yes, the format has little room for error, it's a vector of strings You may want to verify it matches a few nuances here https://github.com/clojure-emacs/cider-nrepl/blob/master/src/cider/nrepl/middleware/classpath.clj and that it passes https://github.com/clojure-emacs/cider-nrepl/blob/master/test/clj/cider/nrepl/middleware/classpath_test.clj

vemv11:09:49

Hey Michiel ๐Ÿ‘‹ just catching up Is there an issue I can subscribe for this one? I'll want to try out this in CIDER again once it's fixed

borkdude11:09:50

Hi. Just got back from Strange Loop and test positive for COVID so taking it easy for a bit ... Just created the issue, also have a branch, no PR yet: https://github.com/babashka/babashka.nrepl/issues/66

borkdude11:09:16

Will get back to this soon

vemv11:09:40

Thanks, subscribed! Please take it clj-easy indeed

๐Ÿ˜† 1