Fork me on GitHub
#babashka
<
2020-04-12
>
bherrmann14:04:16

Happy Easter! Is there a way to tell if the output is going terminal or to a pipe? I'm indirectly asking if babashka can invoke a "isatty" on graalvm? My use case is piping JSON output though "jq" if not used in a pipe. (my fall back is to add command line option to enable/disable jq) by way of example, I'd like this kind of behavior. (Since I'm no master of written clarity.)

borkdude14:04:25

so your question is to be able to tell if bb is invoked as part of a pipe?

borkdude14:04:30

as far as bb is concerned, input on stdin could come from a pipe or user input, I'm not sure how you would check that

bherrmann14:04:44

I know c programs can look at the attached try. but I don't know if that is exposed in the graalvm, and/or bb.

bherrmann14:04:59

ok, command line option for now. Thanks.

borkdude14:04:23

are you using jq for the colored output?

bherrmann14:04:13

I'm taking some bash scripts and moving them to bb, and the bash scripts would pipe to jq for the colorization and formatting (the sorting is handy too.)

borkdude14:04:42

ok. cheshire.core/generate-json can also pprint btw, no coloring

bherrmann14:04:20

(println (json/generate-string (json/parse-string (:body resp) {:pretty true})))

bherrmann14:04:49

yea, although it is odd to take json convert it to edn, then convert it back to json and pretty print it.

borkdude15:04:18

babashka v0.0.83: several fixes, mostly around nREPL https://github.com/borkdude/babashka/releases/tag/v0.0.83

Olical15:04:35

Nice, thanks a lot! 🎉

dharrigan15:04:58

arch updated 🙂

👌 4
victorb15:04:22

hiello! Any http server (as a lib) available in babashka? Saw the example(s) in the documentation, but would feel a bit waste of time for me to use and maintain my own http implementation if something were already to be provided out-of-the-box. Feels like a pretty standard use-case. If not, would a contributed one be welcomed as a PR to the curated babashka libs repo?

victorb15:04:59

btw, awesome work on babashka, enables all kind of crazy new use cases where clojure was not appropriate before, so many cheers in y'alls general direction

borkdude15:04:18

@victorbjelkholm429 Community contributions in the form of libraries is always welcome. I'm not sure what it would entail to add a built-in http server

👌 4
borkdude15:04:59

well yeah, maybe ring + httpkit could be added somehow. that seems like a massive undertaking, but some experiments in that direction would be interesting. I can't guarantee that it will make into bb, but it's always nice to see how far one can get

borkdude15:04:25

( I think httpkit was the http server that was compatible with GraalVM, maybe there are more )

victorb15:04:53

hm, interesting, I've remember I tried to get Jetty (and Ring) to play nicely with GraalVM in the past (maybe 6-7 months ago) and didn't get very far, sounds promising if httpkit works

victorb15:04:57

thanks for the links!

borkdude15:04:58

maybe a fork of bb which is specialized in making small web-apps with instant startup could also be a direction. as bb is intended for scripting, but another tool may be focused on web apps?

borkdude15:04:13

for a list of tools made with sci, look here: https://github.com/borkdude/sci/#why babashka is one example of what you can do with sci, but there's also one that does static html generation, some ansible-like tool, etc.

victorb15:04:12

ah, look at that, that's a interesting collection of tools. Thanks for sharing that, I just started looking into bb but sci looks what makes bb really powerful (and GraalVM of course)

borkdude15:04:50

yep, I'd love to see more bb-like tools popping up.

🎉 4
bherrmann15:04:43

Question? When using cider with babashka....

$ bb --nrepl-server
then in emacs, cider-connect to the port it emits.... but I get this chatter.... is this normal? or am I doing it wrong?
WARNING: Can't determine nREPL's version.
Please, update nREPL to 0.6.0 (or newer).
         More information.WARNING: Can't determine Clojure's version. CIDER requires Clojure 1.8.0 (or newer).
         More information.WARNING: CIDER requires cider-nrepl to be fully functional. Some features will not be available without it!
         More information.clojure.lang.ExceptionInfo: Cannot call print-curl with 0 arguments
I can evaluate in cider, and everything seems ok....

borkdude15:04:01

That's probably normal yes

👍 4
borkdude16:04:51

well, not normal, but nothing problematic and also nothing bb should fix, because a client might assume too much about an nREPL server

nate20:04:44

This is too perfect. I need this for a script in writing.

ivangalbans19:04:26

hello @borkdude I would like to know if this is an expected behaviour:

$ bb '#{#(-> %) #(-> %)}'
clojure.lang.ExceptionInfo: Set literal contains duplicate key: (fn* [%1] (-> %1)) [at line 1, column 2]
despite:
$ clojure
user=> #{#(-> %) #(-> %)}
#{#object[user$eval5$fn__6 0x29e6eb25 "user$eval5$fn__6@29e6eb25"] #object[user$eval5$fn__143 0x62435e70 "user$eval5$fn__143@62435e70"]}

borkdude19:04:32

that's an interesting edge case

borkdude19:04:07

@ivangalbans in Clojure:

user=> #{#() #()}
Syntax error reading source at (REPL:6:11).
Duplicate key: (fn* [] ())

Mno19:04:14

Hey borkdude I finally took the time to make an extremely rough version of the native-binary compile script

👀 4
Mno19:04:33

I'm probably going to make a bunch of issues so I don't forget what I need to fix

borkdude19:04:06

@hobosarefriends Wow, exciting! I'm going to try in a bit

borkdude19:04:24

@hobosarefriends I quick try:

$ cat which.clj
#!/usr/bin/env bb

(require '[ :as io])

(defn where [executable]
  (let [path (System/getenv "PATH")
        paths (.split path (System/getProperty "path.separator"))]
    (loop [paths paths]
      (when-first [p paths]
        (let [f (io/file p executable)]
          (if (and (.isFile f)
                   (.canExecute f))
            (.getCanonicalPath f)
            (recur (rest paths))))))))

(when-let [executable (first *command-line-args*)]
  (println (where executable)))
$ bb nativity/nativity.clj which.clj
reading file
making directory
making modified script-file
making deps file
compiling native image
{:exit 1, :out "Compiling which\n", :err "Cloning: \nDownloading: org/clojure/tools.cli/1.0.194/tools.cli-1.0.194.pom from central\nDownloading: org/clojure/core.async/1.1.587/core.async-1.1.587.pom from central\nDownloading: org/clojure/pom.contrib/1.0.0/pom.contrib-1.0.0.pom from central\nDownloading: bencode/bencode/0.2.5/bencode-0.2.5.pom from clojars\nChecking out:  at 7708e7fd4572459c81f6a6b8e44c96f41cdd92d4\nDownloading: org/clojure/tools.analyzer.jvm/1.0.0/tools.analyzer.jvm-1.0.0.pom from central\nDownloading: commons-codec/commons-codec/1.8/commons-codec-1.8.pom from central\nDownloading: org/clojure/tools.analyzer/1.0.0/tools.analyzer-1.0.0.pom from central\nDownloading: org/clojure/tools.analyzer/1.0.0/tools.analyzer-1.0.0.jar from central\nDownloading: bencode/bencode/0.2.5/bencode-0.2.5.jar from clojars\nDownloading: org/clojure/tools.analyzer.jvm/1.0.0/tools.analyzer.jvm-1.0.0.jar from central\nDownloading: org/clojure/tools.cli/1.0.194/tools.cli-1.0.194.jar from central\nDownloading: org/clojure/core.async/1.1.587/core.async-1.1.587.jar from central\nExecution error (NullPointerException) at java.lang.ProcessBuilder/start (ProcessBuilder.java:1090).\nnull\n\nFull report at:\n/var/folders/yc/31nmj9057jscr2wlf846n_g80000gp/T/clojure-2743258849994375315.edn\n"}

Mno19:04:41

So it was downloading dependencies and then it got a null pointer somehow.

borkdude19:04:40

@hobosarefriends I think it succeeded until the compile step

borkdude19:04:50

$ "clj" "-A:native-image"
Compiling which
Execution error (NullPointerException) at java.lang.ProcessBuilder/start (ProcessBuilder.java:1090).
null

Full report at:
/var/folders/yc/31nmj9057jscr2wlf846n_g80000gp/T/clojure-2594253467581970811.edn

Mno19:04:55

oddly enough it worked on my machine (with a warning)

borkdude19:04:07

maybe it expects native-image to be on the path

Mno19:04:01

Sounds reasonable

Mno19:04:34

I'll look into that! thanks

borkdude19:04:00

ah my bad, my $GRAALVM_HOME variable was set wrong

Mno19:04:38

Oh thank god, because I was under the impression that having it as a dependency in :extra-deps was good enough

Mno19:04:52

if not this was going to be a bit trickier because the script would have to install it before compiling

borkdude19:04:31

I think one nice addition over using clojure.java.shell is using ProcessBuilder directly so you can just stream stdout and the user can see what's happening, instead of waiting a long time

Mno19:04:50

That's literally the first Issue I created

Mno19:04:06

I remember reading a few days ago on this channel how that can be done

borkdude19:04:49

another useful one: don't use the implicit requires in a script, that's mostly for one-liners, not so much for full blown scripts

borkdude19:04:40

and also: don't generate all these requires into the generated script, that will just make the resulting binary bigger

borkdude19:04:09

my current example doesn't even compile, while it just uses a small bit of clojure. > Warning: Image 'which' is a fallback image that requires a JDK for execution (use --no-fallback to suppress fallback image generation).

borkdude19:04:37

using the --no-fallback option should prevent generating fallback images which are just images which pack a JVM and have no good startup

Mno19:04:59

implicit requires?

Mno19:04:31

I don't know what the person will use from bb, so I don't really have a choice?

borkdude19:04:53

@hobosarefriends Yes, you should rely on them using require, not putting in those requires yourself

Mno19:04:35

I'll add that as an issue later

Mno19:04:53

actually I'll just copy paste everything and put that into an issue

borkdude19:04:01

yeah. so if people only use http://clojure.java.io like I did in the example, I must require http://clojure.java.io and you don't have to add anything. this makes it way easier on graalvm

Mno19:04:49

but then the script would require people adding those explicit requires instead of just working with anything?

borkdude19:04:11

yes, but that is good practice. follow the hints that clj-kondo gives you

borkdude19:04:57

I don't think it's a small price to pay, it's a good price to pay, both for compatibility with clojure and also for the resulting binary

Mno19:04:57

Thanks dude!

borkdude19:04:42

but all in all, very nice! I'm going to give it a try again once you fix those requires. I think the require of clojure.test is not compatible with GraalVM

borkdude19:04:54

I had to work around some issues in clojure.test to make it compatible

borkdude20:04:30

@hobosarefriends I fixed the requires manually and it compiled within 30s now instead of blowing up:

$ ./which rg
/usr/local/Cellar/ripgrep/11.0.1/bin/rg

borkdude20:04:57

@hobosarefriends It might make sense to scan for top-level requires or ns-forms and put those on top and not inside the -main function. Somehow.

borkdude20:04:43

Also top-level defns you don't want to throw inside the -main as now the functions will be defined every time you run -main

borkdude20:04:17

and defmacro

borkdude20:04:25

so maybe everything, unless ...

borkdude20:04:37

ns, require, def, defonce, defn, defn- and defmacro, those are probably the things you don't want to wrap inside -main.

nate20:04:40

Wow, nativity looks amazing. Can't wait to try it out!

Mno20:04:03

cool, I'll add options to make select these modes

Mno20:04:25

that way you have the "rough quick way" and the "efficient ways"

borkdude20:04:27

@hobosarefriends This may require some advanced parsing. Maybe it's not unreasonable to expect from people to put some extra effort in their script instead of relying on nativity to transform it into a thing that can be fed into native-image. E.g. if I write my script like this:

;; #!/usr/bin/env bb

(ns which
  (:require [ :as io])
  (:gen-class))

(defn where [executable]
  (let [path (System/getenv "PATH")
        paths (.split path (System/getProperty "path.separator"))]
    (loop [paths paths]
      (when-first [p paths]
        (let [f (io/file p executable)]
          (if (and (.isFile f)
                   (.canExecute f))
            (.getCanonicalPath f)
            (recur (rest paths))))))))

;; main invocation:

(if (resolve 'babashka.classpath/add-classpath)
  (when-let [executable (first *command-line-args*)]
    (println (where executable)))
  (defn -main [& args]
    (when-let [executable (first args)]
      (println (where executable)))))
it can be run both with bb and with your tool without changing a line of code.

Mno20:04:30

and I want to do all of this so I'll add them all as issues

Mno20:04:59

learning to use the homoiconicity of clojure to leverage these kinds of changes is great part of the language

Mno20:04:50

so I'll add an option which compiles the script differently by specifying which one to use as the main function

Mno20:04:30

and another option which parses and tries to understand which libraries you used and only require those.

borkdude20:04:15

yeah, but note that requiring everything doesn't even compile

Mno20:04:29

I thought it compiled but it had a warning

Mno20:04:34

There's so much here I don't understand

borkdude20:04:36

it doesn't compile into a binary.

Mno20:04:51

I kinda jumped into a very deep pool

borkdude20:04:52

the warning says it's generating a fallback image which is basically a thing with an entire JVM in it

borkdude20:04:13

so you should use --no-fallback

borkdude20:04:25

as a setting for native-image

Mno20:04:56

hmmm okidoke

borkdude20:04:58

@hobosarefriends If you want to go the parsing route, I would personally use rewrite-clj for rewriting clojure code. But that doesn't work as a babashka script. Since you already need a JVM, it might not be so bad to do this project all in clojure, if it's going to be that advanced. But still I think rewriting scripts will be brittle and relying on people to set up their scripts so that it doesn't need any rewriting might be the way to go.

borkdude20:04:04

Heck, just generating the deps.edn file is already useful 😉

Mno21:04:51

Yeah I think I'll keep it as a brute script for now

Mno21:04:01

but I do want to get into rewrite eventually

nate21:04:43

Would starting from an uberscript make it easier? No need to chase down deps.

borkdude21:04:54

hmm... good point!

nate21:04:29

Which brings me to another question. Is it possible to carve an uberscript?

Mno21:04:50

That's what I was trying to do originally

Mno21:04:05

but I didn't understand how it worked and then I was told that it wasn't meant for that

borkdude21:04:02

well, the uberscript is meant to create a single script from all required namespaces, into a single file. but you still don't solve the -main problem with that

borkdude21:04:18

I made a better version which is compatible with both bb and graalvm without any changes:

#!/usr/bin/env bb
(ns which
  (:require [ :as io])
  (:gen-class))

(defn where [executable]
  (let [path (System/getenv "PATH")
        paths (.split path (System/getProperty "path.separator"))]
    (loop [paths paths]
      (when-first [p paths]
        (let [f (io/file p executable)]
          (if (and (.isFile f)
                   (.canExecute f))
            (.getCanonicalPath f)
            (recur (rest paths))))))))

(defn -main [& args]
  (when-first [executable args]
    (println (where executable))))

(when (find-ns 'babashka.classpath) ;; we're running as a script
  (apply -main *command-line-args*))

nate21:04:56

This is nifty. Reminds me of a bit of code you can put at the end of a python file to trigger running a function if it's invoked directly.

borkdude21:04:40

yeah, I don't think it's unreasonable to shape your script like this to make it work with nativity

borkdude21:04:32

when I copy this script over the src/which.clj that nativity generated, I can compile it with:

clojure -A:native-image
and it just works:
$ ./which rg
/usr/local/Cellar/ripgrep/12.0.0/bin/rg

Mno21:04:43

It seems I have plenty of work to do

Mno21:04:52

luckily I have monday off so I'm probably going to focus on this

borkdude21:04:15

it's a pretty awesome idea 🙂

borkdude21:04:52

a similar idea might work for producing docker images

Mno21:04:54

One project at a time for me, I'm not as good as you.

borkdude22:04:33

I wonder how one does this with normal clojure in case you want to run a script with that.

(when ... ;; we're running as a script
  (apply -main *command-line-args*))