Fork me on GitHub
#beginners
<
2020-10-20
>
Mitch15:10:30

We are using clojure.java.shell/sh for a long running command. Is there a way to stream the output of the command to stdout as it is running? We are hoping to get visual feedback of the command's progress instead of having to wait for it to complete to check the return map's :out key.

walterl15:10:18

I've not used it myself, but maybe @U04V15CAJ's https://github.com/babashka/process can help

borkdude15:10:13

@UF9E03ZEX Yes. Use (babashka.process/process ["your-program" "arg"] {:inherit true})

Mitch15:10:20

you are the best! tyvm everyone

Mitch16:10:36

So if I wanted to capture the output to be processed by my script while also outputting it to stdout, I would have to open a reader which prints to stdout and also appends to a string

borkdude16:10:56

you could just use a loop to read from the out stream

borkdude16:10:15

and read lines from that and append to some string writer + stdout

borkdude16:10:01

Something like:

(binding [*in*  (io/reader (:out tail))
          *out* (io/writer (:in cat-and-grep))]
  (loop []
    (when-let [x (read-line)]
      (println x)
      (recur))))

borkdude16:10:21

(this is from the README)

noisesmith16:10:16

this is also easy in normal clojure via interop:

hammurabi$ clj
Clojure 1.10.1
(cmd)user=> (-> ["ls"] (ProcessBuilder.) (.inheritIO) (.start) (.waitFor))
CHANGELOG.md  examples	project.clj  resources	target	todo.otl
doc	      LICENSE	README.md    src	test
0
user=> 

noisesmith16:10:12

the .waitFor makes the repl wait until the process exits before printing the next prompt

borkdude17:10:29

yes, but if you want to read from the process's stdout you should not use inheritIO

borkdude17:10:48

he wants to get the string, but also write to out

noisesmith17:10:40

ahh, that's different, yes

noisesmith17:10:59

definitely more complex

(let [pb (ProcessBuilder. ["ls"])
      p (.start pb)
      in (.getInputStream p)
      result (StringBuilder.)]
  (loop []
    (let [b (.read in)]
      (if-not (pos? b)
        (str result)
        (let [c (char b)]
          (.append result c)
          (print c)
          (recur))))))

borkdude17:10:42

(require '[babashka.process :refer [pipeline pb]]
         '[ :as io])

(let [[catp teep] (pipeline (pb ["cat"]) (pb ["tee" "log.txt"] {:out :inherit}))]
  ;; write to cat every 100 milliseconds 10 times
  (future
    (with-open [writer (io/writer (:in catp))]
      (loop [i 0]
        (when (< i 10)
          (binding [*out* writer]
            (println i)
            (flush))
          (Thread/sleep 100)
          (recur (inc i))))))

  @teep ;; wait for the tee process to finish
  ;; finally we print the file's content
  (println (slurp "log.txt")))

(shutdown-agents)

borkdude17:10:13

This works if you have tee on your system

borkdude17:10:06

the first process is cat, to which we write 10 times (just to simulate some output)

borkdude17:10:17

and this output is captured by tee, which writes it to stdout AND to a file

borkdude17:10:28

I think it could be useful to have this built into the process lib

William Skinner16:10:20

What's the idiomatic way to say (if (str/blank? dir) nil dir)

Darin Douglass16:10:06

Seems fine to me. There’s also:

(when-not (str/blank? dir) dir)

noisesmith16:10:36

there's also (not-empty dir)

(ins)user=> (not-empty "")
nil
(ins)user=> (not-empty "foo")
"foo"

noisesmith16:10:22

I'd argue not-empty is most idiomatic as it is specifically designed for this use case (nil on empty, without changing input type to seq)

adityaathalye16:10:25

not-empty won't defend against a blank string, which dir could be

noisesmith16:10:56

did you look at my example?

noisesmith16:10:08

it explicitly returns nil for empty string

noisesmith16:10:19

oh - you mean whitespace?

noisesmith16:10:54

right, if whitespace is a concern not-empty? doesn't help

adityaathalye16:10:30

yes, I assumed they wanted to defend against a "blank" string as opposed to an empty one

clojuregeek17:10:12

Hi .. i want to change the log level on the command line when i run tests (sometimes) I tried JVM_OPTS='-Dlog.console.threshold=INFO" lein test and also using `LEIN_JAVA_OPTS instead of JVM_OPTS`

noisesmith17:10:58

@clojuregeek I would usually handle this by making a separate lein profile with :quiet {:jvm-opts ["-Dlog.console.threshold=INFO"]} and use with-profile to add that profile lein with-profile +quiet test

clojuregeek17:10:40

oh thats a good idea

noisesmith17:10:06

that way it combines correctly with other additions you might want to make to the java command line

j18:10:51

I'm trying to setup kaocha, and I tried running the following line in the very beginning of the kaocha install docs and got this error:

❯ clojure -Sdeps '{:deps {lambdaisland/kaocha {:mvn/version "1.0.700"}}}' -m kaocha.runner --test-help
Execution error (FileNotFoundException) at kaocha.specs/fn (specs.clj:82).
Could not locate clojure/test/check/generators__init.class, clojure/test/check/generators.clj or clojure/test/check/generators.cljc on classpath.
Can some one give me some tips? I tried adding the test alias in the docs to deps.edn, and it give me the same error

noisesmith18:10:54

you need to pull in the generative deps sepearately to use that feature of spec

noisesmith18:10:30

org.clojure/test.check {:mvn/version "1.1.0"}

j18:10:00

That worked! Thanks @U051SS2EU! I didn't see that library dependency anywhere in the kaocha docs. It looks like clojure/test/check is a reference to the test.check library? Can I assume that future error messages that has that kind of path will help me identify missing dependencies?

noisesmith18:10:06

right - the deal is that clojure.spec contains code that dynamically looks for that code

noisesmith18:10:29

it's not a "hard dependency" because if you don't want to use those features you don't need to include the extra dep

noisesmith18:10:01

it's a gotcha because most libs don't work this way

j18:10:23

Got it! I just started playing with kaocha, and I didn't know that kaocha.runner needed spec 😕

noisesmith18:10:47

IMHO spec should intercept the error and include a message like "to use this optional spec feature you need the following package ..."

noisesmith18:10:57

spec comes with clojure

noisesmith18:10:09

certain spec features require this other optional dep that doesn't come with clojure

j18:10:39

oh! my bad! I guess kaocha.runner calls spec, then spec in turns calls test.check?

j18:10:00

cool, understood! Thanks for the tip! 😄

noisesmith19:10:46

most clojure libraries change API very little, even as versions are updated. The main version problems come from poorly designed deps coming from java (eg. jackson, a recurring problem for many projects). If fast startup of your server is a goal I'd switch to another language, graal native image comes with a huge set of problems for clojure usage

noisesmith19:10:21

My experience with REPL state is that I change dependencies very rarely compared to the amount of development I do within one REPL with one dependency set.

noisesmith19:10:02

NB about server startup: my baseline assumption is that you are using an uberjar, and running your server via the java command or a java process runner, if you use eg. deps.edn or lein on your server itself you are doing it wrong

noisesmith20:10:23

one more about the microservices: reduced memory usage is not a feature under development (both the vm itself and clojure have multiple design decisions implemented that trade more memory usage for speed)

Day Bobby20:10:27

thank you for taking the time to answer, these are valuable insights

Day Bobby19:10:43

How big of a problem is dependency conflict in Clojure? I've seen a lot of libraries that have not been updated for 3 - 4 years. I'd imagine in a scenario where use one such library which depends on an older version of another library (library-x for example), and I also use a more updated library which depends on a recent version of library-x, would that have high probability of causing troubles?

I've found that in Clojure, I always want to make the perfect API upfront because being a dynamic typed language, it's harder to refactor after. I come from a background of typescript where I just write function that work first then come back to change the interface, maybe even much later along the road, because it's very easy. What have been your experience?

I corrupt the REPL state a lot, especially on newer projects when everything is not so figured out (dependencies, etc...). Can you share tips on working with REPL so that it doesn't hinder productivity.

What need to happen so that a typical clojure web project (jdbc, ring api) can be run under graalvm? I know in Java land, there are Quarkus and Micronaut which support graal pretty well. Fast startup and low memory usage are sought after features these days because of microservices. Do you think we Clojurians will have these in the future?

Thank you very much~

Lennart Buit20:10:32

To answer the first, clojure library authors tend to prefer stability. Meaning that a library gets very little breaking changes, and versions that are years old will still work perfectly fine now. Some libraries go as far as just starting a new namespace when they introduce breaking changes, meaning you can use either the ‘old’ api, or the new

Lennart Buit20:10:18

Compared to the JS/TS communities, Clojure libraries tend to include less breaking changes (if any), and, I’d personally say thats also part of the … how do you say … style? culture? of Clojure

Lennart Buit20:10:29

Its like the Python community has their zen of Python right, part of the zen of Clojure is to write slow moving software 🙂 If that makes sense

Day Bobby20:10:51

That makes total sense. And I think that also answers my second question: it's normal to want to put more thoughts into the initial API, in the spirit of writing stable software. I am gonna have to be much more disciplined when writing Clojure code and ditch that "just gonna put this thing here for now" attitude that I'm so used to. Thank you!

noisesmith21:10:27

you can get very far with using immutable data structures with descriptive keys for nearly everything, the spec system ignores unknown keys so it's automatically "future proof" in that respect

noisesmith21:10:47

overuse of deftype / defrecord is something to look out for coming from ts

kennytilton03:10:21

“being a dynamic typed language, it’s harder to refactor after” How so @U01BMAKUNEP? This sounds like an experience report, not a surmise. Could it be that, growing up on static typing, your code ends up somehow reliant on static typing, even when it is not available? I have certainly coded in statically typed languages and I get “if it compiles it is prolly right” as an excellent crutch, but perhaps that crutch inhibits development of the ability to design at a deeper clarity where dynamic typing presents nothing but wins.

kennytilton03:10:34

As for “corrupt the REPL state”, I just do not rely on REPL state, nor do I program “in the REPL”. AS soon as my code gets past (+ 2 2) I have a test function that, yes, gets evaluated ad hoc “in the repl”, but the test function starts from scratch each time. No long-lived state to keep straight.

kennytilton03:10:41

“I am gonna have to be much more disciplined when writing Clojure code” Yer doin’ it wrong. 🙂

seancorfield04:10:05

Yeah, these seem like weird questions to me... I've been doing primarily Clojure for a decade now in production. Dependency conflict just isn't a thing: Clojure folks value stability/backward-compatibility; when I have problems, it's nearly always a Java library that isn't even paying lip service to SemVer (which is broken anyway). No idea why you find refactoring in Clojure harder: I find that Clojure is great for allowing a design to evolve, starting in REPL experiments and then expanding in terms of provide more/require less. I almost never corrupt my REPL state: I eval every source file change as I make it (and I never type into the REPL); I occasionally remove a ns because I want to change aliases or something but I have REPLs running for a week or more with no problem. As for Graal, I neither know nor care: I write backend software that runs for a long time so the JVM is perfectly optimized for that. @U01BMAKUNEP

ordnungswidrig07:10:20

I only ever run into dependency conflicts down deep in the java library ecosystem, typically some apache commons library or one of the inevitable logging libs. The only time I remember where I could not easily resolve this by pinning either version was for some dependency in AWS java SDK.

Day Bobby08:10:33

Thank you all for your answers. To clarify some of my points earlier: I've never had a problem with dependency conflicts, though the programs I'm writing to learn Clojure are very small and don't have a lot of dependencies. From my experience in other languages, issues with dependencies are often huge timesinks so I just want to know before fully committing to writing my next side project in Clojure. It's reassuring to know most of the issues come from Java packages, I will be on the look out for those. On the REPL issue, I have always been suspecting that I'm doing something wrong. Rarely goes a day when I don't have to restart it. Sometimes its not a state issue though: a library that I use dumps its error logs (a massive map that contains its debug values) onto the REPL rendering cider near unresponsive -- But now that I think more about it, clearing the cider output instead of restarting would probably work too. On the refactor issue, for example in TS I can rename the keys of a map a function expects, and my editor would automatically fix the keys at every callsite. In Clojure, what I usually do is to do a search and fix them manually. In TS, if I change the order of a function arguments, my editor will tell me that action results in errors in these different places, and I just follow the red squiggles to fix them all.

Day Bobby09:10:24

Guess what I'm trying to say is that I feel more confident evolving my API in a static typed language knowing that the existing parts of my application will be kept intact. I understand that in dynamic languages we have the freedom to change our APIs much more easily due to lack of constraints. The only constraint is test which is a luxury in some cases (i.e early startups), and maybe spec too but specs are often not strict. In my case, I haven't utilized that freedom due to fear of breaking stuff. Also refactoring with tests and specs does not have the same editor support as with static types.

Day Bobby09:10:12

Thanks everyone for taking the time to answer so far. I have to admit I used to have strong opinion on static typing. I used ruby from 2008 to 2015, jumped on the static type bandwagon and never looked back, or so I thought 😛 Since then I been doing frontend stuff using React in TS which can be coded in a very functional way (a bit Clojure-y if you can even believe it). I turned to Clojure because it is a functional language with sane data structures, not because it's a dynamic language. I believe the benefits will outweight the mental shift. I guess I just need some guidance and a lot more time building serious stuff in Clojure.

kennytilton11:10:28

My big gripe with static is that it makes me propagate a change throughout the entire app before it will even compile. In a dynamic language I can start a refactoring on a small wodge of code and see how it goes, change course, maybe even abandon altogether. I know the rest of the app is broken now, but I defer the overhaul until I know where I am going.

simongray11:10:07

I think most people taking on Clojure will have had some experience with static typing, so they probably expect some kind of mess now that the guard rails are off. There is no single thing in Clojure that replaces what you get with static typing, but I would say it's a combination of functional programming patterns typically resulting in much less coupled code (and therefore much less to refactor), spec or some other validation method for your external inputs, a tight feedback loop provided by REPL-driven development, a preference for commonsense plain data literals over complex blobs of code (when you've already reduced your data to the bare essentials what is there to refactor?), and perhaps the higher maturity level of the average Clojure programmer, a culture that Rich Hickey definitely helped instill in this young language. Using namespace-qualified keywords is also helpful.

simongray11:10:56

Tangentially relevant Rich Hickey rant: https://youtu.be/aSEQfqNYNAc

Day Bobby13:10:00

@U0PUGPSFR You made very interesting/valid point. Thats a clear win for dynamic languages. Guess I just never thought like that before. @U4P4NREBY To be fair, with JS/TS you can do stuff in a functional way using only simple data structures: arrays, objects which are like maps,... There are things like Set and Map which have a class based API, but you can get very far ignoring those and just resort to arrays and objects (of course they are mutable so most of the times you will have to use spread operator (https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Spread_syntax) to manipulate them in a non destructive way 😥). I know those are hacky measures and that was one reason why I turned to Clojure. In my most recent project which contains almost 100k LOCs of TS, there is only one class (wrapper of an external HTTP client). I feel it's more because of the way I develop, not because of the complexity of the data structures I use. I tend to move stuff around a lot. Some of the changes that I often make are the opposite of "provide more/require less" @U04V70XH6 mentioned: things like "remove this default behavior of this component and make user specifically ask for it" or "make this component a headless component by taking in a render function instead of an element so the user has total responsibility of how it looks". I haven't incorporated spec into my workflow yet but I will do, and right now I'm also looking at ghostwheel (from what I gathered, a lot of large teams use tools like this). I don't mind changing the way I do things now to write more stable APIs but that might take some time to get used to. Thank you for the tip on ns qualifed keywords.

simongray13:10:21

@U01BMAKUNEP The idioms of the language and the norms the ecosystem you find yourself in matter a lot more than what is technically possible to do in the language. I don’t really have much experience with TypeScript, but I’ve had to debug some in som third party code and it just looked like JavaScript with more boilerplate 😉

noisesmith16:10:04

I actually agree that refactoring in a strongly typed language (in my case OCaml) allows a freedom that you don't have without static checks - the compiler literally tells you which things you need to change to make your code work again after your change.

noisesmith16:10:34

so I can do drastic changes in an OCaml codebase that would be foolish to do in Clojure

seancorfield16:10:53

@U051SS2EU But are they the same sorts of changes you would need/want to make in a similar project in Clojure?

seancorfield16:10:06

I ask, because I don't find myself doing a lot of refactoring-in-the-large in Clojure: my refactorings tend to be very localized. I do sometimes change a function signature but I might do that initially via an additional arity so I don't have to update every call. And I do rely heavily on a linter (`clj-kondo`) to pick up incorrect numbers of arguments and some other things.

noisesmith16:10:17

they are general purpose data model / algorithm changes (like changing arg list to add more data, or using a different collection type with a different set of functions)

noisesmith16:10:00

things I do frequently in Clojure, but need to be very cautious with once a code base reaches a certain size (or I need extensive test coverage which I simply don't need in the OCaml case)

noisesmith16:10:41

and yes, linters add static checking to a language that lacks it, almost :D

noisesmith16:10:10

I still prefer Clojure all told, but I'd be lying if I said the refactoring experience wasn't comparatively painful

noisesmith16:10:02

big picture, being more careful about your initial API has a lot of benefits, and Clojure pushes you in that direction...