Clojurians Log v2

#!/usr/bin/env bb

(require '[babashka.process :as proc]
         '[clojure.java.io :as io])

(def path "./repro")

(let [p (proc/process (str "tail -f -n +1 " path))]
  (with-open [reader (io/reader (:out p))]
    ;; this future runs forever:
    (future
      (doseq [l (line-seq reader)]
        :do-nothing))

    (println "sleeping to let the future catch up")
    (Thread/sleep 1000)
    (println "done sleepin")
    ;; At this point, if we don't manually kill the process, we'll block forever:
    #_(.destroy (:proc p))))

This script is named "repro", and I'm just having it read from the script file itself since I just needed the child process to tail -f any old file forever.

This is what I'm seeing:

borkdude@MBP25-3 /tmp $ bb /tmp/foo.bb
sleeping to let the future catch up
done sleepin
borkdude@MBP25-3 /tmp $

Did you update the path variable to be ./foo.bb ?

Because if it can't open the file specified by path, it will behave as you describe.

no, let me try

The path could just as well be "/etc/hosts" or something but that looked scarier - I just need to tail from SOME valid file.

ah yes, I can repro it now

Cool!

Can you make a github issue?

perhaps it's a feature, not a bug, but I'd have to check

Yes, I will try.

i think that is expected, java/graalvm will not end the program if there is a non daemon thread still running, the future runs forever because you're doing a "tail -f"

It even occurs with the —force-exit flag

The behavior also is different depending on whether the future gets to the end of the input and blocks on more lines or not.

That inconsistency alone suggests to me that there is a problem to be addressed.

doesn't that make sense since the future will end if there's no more input?

it seems there is some deadlock/racing condition with the fact that the reader is closed in a different thread than is being read, if you move the with-open to inside the future, the program ends

👍

To answer your question asked at 2:36pm, no, it doesn't make sense, because the future is (intentionally) written in such a way that it will never end - it will block forever waiting for another line. What I was pointing out is that if you happen to reach the end of the program before the future has reached the end of the available input, the program will exit. The difference in this case is not whether the future is running or not - it's always running in both examples I'm talking about. The behavior with not exiting appears to happen when the future is blocking waiting for a line when all lines have been exhausted.

I'll try to file a github issue later and I'll include this info and you can decide if you think it qualifies as a bug or not.

@rfhayashi is right. consider this:

(let [p (proc/process (str "tail -f -n +1 " path))]
  (with-open [reader (io/reader (:out p))]
    ;; this future runs forever:
    (future
      (doseq [l (line-seq reader)]
        :do-nothing))

    (println "sleeping to let the future catch up")
    (Thread/sleep 1000)
    (println "done sleepin"))
  (prn :dude))

Interesting - so is your conclusion that there's nothing that can be done?

not on bb's side. clojure JVM has exactly the same behavior, so I don't consider it a bb bug

Ok - I won't file the issue then! It's still interesting - and it's not hard to work around it.

Thanks for looking into it, and thanks for providing insights @rfhayashi

This came up organically while writing a program to parse logs and visualize them with charm.

I needed to force the sequence using future so that charm's update function wouldn't get stuck (forever) when there wasn't any more input.

I wonder if there's a way to do that, without using future ... I tried also using core/async threads, with no better luck.

I suppose by reimplementing line-seq to not use .readLine on BufferedReader, but rather something that only read when the reader was ready, and otherwise just periodically slept. Inefficient, but I'm reasonably sure it would work and be interruptible.

I think I'd rather just live with the workarounds - either cause the child process to exit, or just call System/exit.

Using Claude I was able to find out more. Both BufferedRead.readLine and BufferedReader.close synchronize on the same lock. So it's kind of a race condition. If the thread aqcuired the lock and is waiting for input, then close is waiting for the lock and will never execute. If close goes first, then the thread will just die quickly without reading input. So it's kind of indeterministic too. • https://github.com/openjdk/jdk/blob/8444fdae4afd99369a321bda95d1ee034f1835f6/src/java.base/share/classes/java/io/Reader.java#L271 • https://github.com/openjdk/jdk/blob/8444fdae4afd99369a321bda95d1ee034f1835f6/src/java.base/share/classes/java/io/BufferedReader.java#L321 • https://github.com/openjdk/jdk/blob/8444fdae4afd99369a321bda95d1ee034f1835f6/src/java.base/share/classes/java/io/BufferedReader.java#L525

This is an even simpler repro.

(require '[babashka.process :refer [process]])
(let [reader ( (:out (process "sleep 10")))]
  (future (.readLine reader))
  (.close reader))

You forgot to clone the submodules?

https://github.com/babashka/babashka/blob/master/doc/dev.md#clone-repository

That's a long way down the instructions... ...if I've already cloned it, is there a way to "clone the submodules", or do I have to start over? (I've never used git submodules)

yes, the link describes how :)

here it's also written: https://github.com/babashka/babashka/blob/master/doc/build.md#clone-repository there seems to be a tiny difference, --init. not sure what does does

perhaps you need that if you haven't cloned them

Neither of those commands work, so I guess it's delete and start again.

Tests are running now. You might want to mention the submodule stuff *above* the Test section 🙂

Is this error expected while running tests? It doesn't seem to fail any tests:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x4f7946]

goroutine 1 [running]:
, {0x6d2a60, 0xc000012060})
        /home/circleci/project/babashka/ops.go:72 +0x86
main.main()
        /home/circleci/project/main.go:207 +0x25

(that was just before running flaky tests BTW)

yeah you can ignore that one, it's a pod that's killed

PR submitted.

Thanks and congrats on your first bb PR!

Thank you for being so receptive to my little foible 🤣