I thought it was more clj-kondo shaped today!

I found this snippet gives the wrong error message in babashka

(do
  (defn rf [a b] (conj a b))
  (reduce rf []))
;; Wrong number of args (0) passed to: clojure.core/reduce

Wrong number of args (0) passed to: user/rf

I'm running a command with very long output (~5mb of text in total) and I need to process each line. I'm using a similar approach as described here: https://github.com/babashka/process#processing-output but it's still a bit slow. Any general advice for processing large outputs?

@alex.sheluchin do you need to process the output as you go or can you do it first by writing a file or in-memory string and then process the file?

@borkdude I don't have any strict requirements about that.

Then I recommend {:out :string}and then process the output string

This approach might help with not holding everything in memory at once https://blog.michielborkent.nl/transducing-text.html

if you need that

@borkdude I've tried to output to string, split lines, and then do some processing, but it's still pretty slow. Like, with 10K lines of output:

(time                                        
 (doall                                      
  (doseq [part (->> (bp/process cmd          
                     {:dir dir               
                      :err :inherit          
                      :out :string           
                      :shutdown bp/destroy}) 
                    deref                    
                    :out                     
                    string/split-lines)])))) 

"Elapsed time: 11573.721905 msecs"

yeah, maybe it makes sense to write to a file first with :out (io/file ...)

and then read from the file

$ bb -e '(-> (babashka.process/process ["cat" "/tmp/lines.txt"] {:out (io/file "/tmp/out.txt")}) deref)'

if that part is slow, then there might be another solution, but try that first

you might first have to measure which part is causing the slowness

(prn "processing")                                           
@(bp/process cmd                                             
  {:dir dir                                                  
   :err :inherit                                             
   :out (io/file out-file)                                   
   :shutdown bp/destroy})                                    
; (prn "sleeping")                                           
; (Thread/sleep 2000)                                        
(prn "opening")                                              
(with-open [rdr ( out-file)]           
  (def all-data (doall (->> (line-seq rdr)                   
                            (partition 2)                    
                            (map #(string/join " " %)))))))) 

Maybe the process which delivers the output is just slow? Can that be the case?

I tested a file with 100k lines which happens in milliseconds for me.

Looks like a lot of it is just io, I guess as expected. Does this seem like there are options to reduce processing time or am I just running into my machine's limits here?

heh, it's just git log. You are right, it's slow itself:

real    0m12.475s
user    0m10.921s
sys     0m1.028s

Hmm, maybe not...

No, that is right, it's the command itself. With --no-pager flag it takes just as long.

try git log > /tmp/output.log

and then measure the time

ah ok then

you could try direct Java interop like this:

(require '[ :as io])

(-> (doto (java.lang.ProcessBuilder. ["git" "log" "--no-pager"])
      (.redirectOutput (java.lang.ProcessBuilder$Redirect/to (io/file "/tmp/output.txt"))))
    (.start)
    (.waitFor))

Do you know how to set dir with the interop by any chance?

I think since it's git log itself that's slow here, this won't make much of a difference, but I'm happy to try it.

@alex.sheluchin with dir:

(require '[ :as io])

(-> (doto (java.lang.ProcessBuilder. ["cat" "/tmp/lines.txt"])
      (.redirectOutput (java.lang.ProcessBuilder$Redirect/to (io/file "/tmp/output.txt")))
      (.directory (io/file "subdir")))
    (.start)
    (.waitFor))

@borkdude "Elapsed time: 11457.470212 msecs" Not much of a change there I'm afraid.

Thanks for the help. I'll have to either pull the data I need from the reflog or something or just deal with the delay. Should be alright.

alright then