Fork me on GitHub
#babashka
<
2020-04-13
>
Roman Tsopin04:04:27

‪Hi there, what’s the best way to use something like JSch with babashka? Ssh is often needed for scripting tasks‬ and native clients are not always friendly (e.g. requires keychain on osx)

borkdude07:04:48

@romantsopin babashka doesn’t have Jsch, but you can either shell out or use #spire, which is similar to bb but specialized in provisioning

Roman Tsopin10:04:49

Thanks! I have separately compiled jsch tool, but thought it would be nice to somehow combine bb and jsch into single binary

borkdude10:04:21

If you can explain more about this I would be open to it

borkdude10:04:39

E.g. a link to your repo

Roman Tsopin10:04:48

It’s not open source now, but basically I connect to remote host by ssh (just for forwarding) but before that I check that 22 port is open and do some basic scripting. It’s a good fit for bb, but I don’t want to use system ssh, so I compiled jsch

Roman Tsopin10:04:29

Actually problem could be formulated more broadly, If I know that some tool could be graal compiled maybe there is a generic way to use it within bb?

borkdude10:04:32

@romantsopin Yes, classes can be added in babashka/impl/classes.clj.

borkdude10:04:53

But you cannot use it with an existing version of bb, it has to be compiled together with it

Roman Tsopin10:04:13

Thank you! Will try it out

borkdude10:04:52

check out https://github.com/borkdude/babashka/blob/master/doc/dev.md for some pointers. Note that you have to use git clone --recursive to clone the repo.

👍 4
borkdude10:04:49

made some tweaks to the deps.clj babashka script: it now no longer shells out to calculate checksums or find executables on the path 🙂 so as pure Clojure as can be. https://github.com/borkdude/deps.clj

sogaiu10:04:47

nice! btw, in https://github.com/borkdude/deps.clj#rationale perhaps another reason for its existence could be if you wanted to know more-or-less what the bash script does, it might be easier to study your implementation. wdyt?

borkdude10:04:52

makes sense

borkdude10:04:46

@sogaiu Do you perhaps have a working Windows environment where you can build graalvm binaries? I can't build anymore in mine and don't know what's going on. I was trying to reproduce an issue I'm having with babashka.curl. It now writes to a tempfile, but apparently the path to a tmp dir is hardcoded in graal and when that dir doesn't exist, curl doesn't work anymore. This is the repro:

public class TempFile {
    public static void main(String [] args) throws .IOException {
        .File f = .File.createTempFile("foo", "bar");
        System.out.println(f.exists());
        System.out.println(f.getPath());
    }
}

borkdude10:04:10

compile with javac Tempfile.java and then native-image.cmd Tempfile and then run it

borkdude10:04:48

in babashka it seems the tmp dir is hardcoded to c:\\Temp

borkdude10:04:11

and also files that curl creates there, babashka can not remove. it's weird.

sogaiu10:04:27

i can take a look in a bit -- dinner time 🙂

borkdude11:04:25

I made an issue for it here: https://github.com/borkdude/babashka/issues/352 - have a good dinner

bherrmann11:04:00

So my script is getting big enough that I want to break it into parts... so I have some related questions, 1. Is there a way to get to bash's $0 - aka the full path to the executing bb script? (this is so I can do an add-class to a subdirectory which has utils code) (I'm aware of the Preloads option, but I'm ignoring that atm) 2. is :all not an option with require? aka (require '[utils :refer :all))

borkdude12:04:43

@bherrmann I think you want to use user.dir maybe?

$ bb -e '(System/getProperty "user.dir")'
"/Users/borkdude/Dropbox/dev/clojure/deps.clj"

borkdude12:04:06

yeah *file* also works probably

borkdude12:04:19

2. no, not implemented yet, also not endorsed 😉

borkdude12:04:51

I think I will implement it at one point, but it kind of goes against most commonly accepted style guides. Using explicit refers is preferred and if that becomes too elaborate, use an alias.

borkdude12:04:23

I'm going to add :refer :all now:

$ lein run "(require '[clojure.walk :refer :all]) (postwalk identity {:a 1})"
{:a 1}

Mno12:04:57

ok I made a new version of the script

Mno12:04:03

changed most of the things you I could do quickly, added command line options changed the readme a bit, removed all implicit requires (i think) and gave better logging

Mno12:04:03

it works with your which.clj script if you remove the require

Mno12:04:59

also now you manually select the dependencies you want to require, or just choose all of them if you're super lazy

borkdude12:04:16

I'll try it

Mno12:04:27

Also do you mind me talking about it here?

Mno12:04:03

it might be considered off topic for this channel

borkdude12:04:05

Fine with me. We can also do a thread.

borkdude12:04:10

Syntax error compiling at (which.clj:12:17).
No such namespace: io

Full report at:
/var/folders/2m/h3cvrr1x4296p315vbk7m32c0000gp/T/clojure-2111109642317177985.edn

Mno12:04:22

use the flag

Mno12:04:42

you now select which ones you want, if you want to use all of them because you're lazy you can use -d all

Mno12:04:52

if you want to use multiple -d io,str,json

borkdude12:04:13

why don't I just get an option to compile my script unmodified? that seems less to think about?

borkdude12:04:31

[which:99014]     analysis:  14,457.98 ms
Error: com.oracle.svm.hosted.substitute.DeletedElementException: Unsupported method java.lang.ClassLoader.defineClass(String, byte[], int, int) is reachable: The declaring class of this element has been substituted, but this element is not present in the substitution class

Mno12:04:34

I'll make that an option now

Mno12:04:04

which one are you compiling? the last one you sent?

Mno12:04:35

I was working with this one

borkdude12:04:26

Can you add H:+ReportExceptionStackTraces to the graalvm options?

Mno12:04:53

oh that just doesn't seem like the way you'd normally write a script, is it?

Mno12:04:23

I'm not sure where the graalvm options go, lemme check

Mno12:04:48

in the jvm-opts of the deps.edn I suppose

borkdude12:04:03

no in the options passed to clj native-image

Mno12:04:56

oooh I see, like the other -H with the anme

Mno12:04:37

done, I hope

borkdude12:04:50

I don't think wrapping all of the code in a large -main is a great idea, but we already discussed that. I think you should leave most things on the top level of the script. I tend to believe that most scripts require a few minor tweaks to make them work with graalvm. E.g. adding type hints to prevent reflection. It will be a major headache on your side to deal with these sorts of issues. Just a project which generates a skeleton in which you can paste your script code and take it from there is what I would go for. This can just be a template for instance on https://clj-templates.com/

borkdude12:04:29

I'll try again

borkdude12:04:36

Hmm, I'm getting:

Unrecognized option: -H:+ReportExceptionStackTraces
Error: Could not create the Java Virtual Machine.

Mno12:04:43

Yeah I wanna have this quick and dirty version running sorta decent, then I'll make other packaging solutions

borkdude12:04:43

maybe it doesn't like the leading space?

Mno12:04:14

I accidentally left another option in the jvm options

Mno12:04:22

already pushed the fix I hope

borkdude12:04:59

ok, trying again

Mno12:04:48

so this version is if you just made a bb script with inline stuff really quick and dirty and you have to pass it off to someone else next I'll make the more skeleton version which will leave everything except 1 function out of the main

borkdude13:04:40

I keep getting the same exception. I think it doesn't like functions being defined in the -main. When I do this, it works:

borkdude@MBP2019 /tmp $ cp which.clj src/which.clj
borkdude@MBP2019 /tmp $ clj -A:native-image

borkdude13:04:59

$ ./which rg
/usr/local/Cellar/ripgrep/12.0.0/bin/rg

Mno13:04:08

yeah it's not meant to use scripts formatted this way

Mno13:04:34

I normally wouldn't add a namespace to a bb script, because, why?

borkdude13:04:49

that is pretty common. why not.

Mno13:04:51

I'll make another processing option which just grabs the script as si

Mno13:04:22

You don't need to, since it has the implicit requires..

Mno13:04:43

there's no real point unless you plan on compiling it, or at least I haven't found one

Mno13:04:51

most of your gallery doesn't have a ns, so I assumed that was the common pattern

borkdude13:04:56

like I said, the implicit requires are only used for one-liners. e.g. cat foo.json | bb '(json/parse-stream *in*)'. Not for scripts. Although that works, you should NOT rely on it.

borkdude13:04:53

for namespaces: you don't have to, but you can use a namespace. and bb libraries will most certainly have them

Mno13:04:50

hmmm okidoke then I'll leave this mode as the one-liner mode, and make another one that leaves it completely untouched

borkdude13:04:27

makes sense

borkdude13:04:11

There will be other issues, but people can figure those out once they have their generated skeleton, which is the strength of this project imho

Mno13:04:59

oh in that case should I make it keep the source files by default? right now it deletes the generated clj and deps.edn

borkdude13:04:30

I think so yes

Mno13:04:52

and new version! that compiles your script by not fucking it up!

borkdude13:04:07

Awesome. Maybe we can add some hints to what people can do to make their script compatible with both native-image and babashka itself. Using the which script as an example maybe? - use a namespace form and explicit requires - add (:gen-class) to the namespace form (bb will just ignore this) - add a -main function and call it using a conditional (like the which.clj script).

borkdude13:04:48

also adding (set! *warn-on-reflection* true) (which bb also ignores)

borkdude13:04:57

I think that will help scripts a lot to be compatible with graalvm

Mno13:04:29

okidoke! I'll add a Hints section to the readme along with examples for each mode

borkdude13:04:14

cool! did I miss something or did you miss the clean flag here? https://github.com/MnRA/nativity#clean-files

Mno13:04:01

ah, good catch yeah I forgot to update that

Mno13:04:40

I'm considering making clean remove the .classes and .cpcache directories

borkdude13:04:37

@bherrmann

$ bb
Babashka v0.0.84-SNAPSHOT REPL.
Use :repl/quit or :repl/exit to quit the REPL.
Clojure rocks, Bash reaches.

user=> (require '[clojure.set :refer :all])
nil
user=> (union #{1 2 3} #{4 5 6})
#{1 4 6 3 2 5}
See #babashka_circleci_builds for new binaries 🙂

borkdude13:04:59

(it could use some testing)

bherrmann13:04:36

ok, thanks, I agree that :all is for the extremely lazy... or for the transition process of breaking something into pieces... and *file* does seem to hold the path to my script!! I really appreciate the error messages from babashka are concise, relevant and helpful. Makes using it fun! (I can stay in the zone and not get sidetracked with mysterious errors)

borkdude13:04:53

I'm curious what errors you mean specifically 🙂

bherrmann13:04:21

Ha! (assuming you arent joking) when anything is not defined, or when missing a '}" or ')' - It could in part be the speed too, it is very immediate.

borkdude13:04:04

ah right. no I wasn't joking.

$ bb "("
clojure.lang.ExceptionInfo: EOF while reading, expected ) to match ( at [1,1] [at line 1, column 1]

bherrmann13:04:27

I tested the :all change locally, it worked. I used *file* to locate a library of common code, also worked great. Thanks.

borkdude13:04:16

I wonder if it makes sense if babashka sets a java property when you use a -main function like bb -cp . -m foo

(System/getProperty "babashka.main")
"foo"
so then you can write:
(defn -main [& args]
  (when-first [executable args]
    (println (where executable))))

(when-not (System/getProperty "babashka.main")
  (apply -main *command-line-args*))
? I don't think Clojure itself has a way of detecting if you're running with a main arg or just as a script

nate14:04:51

A property sounds straightforward.

borkdude14:04:18

And also compatible with Clojure, so no weird errors from clj-kondo 😉

nate14:04:50

Haha. Too true.

borkdude14:04:52

I mean, when we would introduce custom variables for this like joker has, *main-file* or something, this would break clj-kondo

nate14:04:51

I really appreciate the clojure compatibility as I use clojure to write bb scripts.

borkdude14:04:07

I noticed when reading your script repo!

borkdude14:04:19

Although bb itself now also has nREPL, that's still a sane approach

nate14:04:39

I do want to try out bb's nrepl stuff when I give the new conjure a try.

bherrmann14:04:53

for my single script, I simply invoke main at the end... ala...

(docopt/docopt usage *command-line-args* main)

nil

nate14:04:27

I like it better than checking a namespace existence.

bherrmann14:04:33

I include the nil to silence any additional displaying of the last eval.

borkdude14:04:59

@bherrmann That's OK, almost all scripts do that (not having a -main I mean)

Mno14:04:24

The readme is updated, If you guys have any other suggestions lemme know. I'm going to go eat something since I forgot to eat lunch

borkdude14:04:14

have a good lunch 🙂

Mno18:04:22

Thanks got back and continude

Mno18:04:03

New version with directed mode, which wraps a slice of your script with a main call and leaves the rest as is.

Mno18:04:14

and I'm done for a while.

borkdude19:04:26

Your project is now the top post on Reddit Clojure 😉

Mno07:04:43

Oh wow, how did that happen?

yonatanel18:04:59

Hi, I'm trying to use sci in my own code. When I use it as part of the malli library it works (they use sci 0.0.11-alpha.2). When I use that same dependency it works. When I use sci 0.0.13-alpha.14 I get this error immediately when requiring the namespace in repl:

nREPL server started on port 39131 on host 127.0.0.1 - 
REPL-y 0.3.7, nREPL 0.2.12
Clojure 1.10.1
OpenJDK 64-Bit Server VM 1.8.0_242-8u242-b08-0ubuntu3~18.04-b08
...
user=> (require '[sci.core :as sci])

Execution error (IllegalStateException) at sci.impl.parser__init/load (REPL:13).
Attempting to call unbound fn: #'edamame.impl.parser/normalize-opts

borkdude18:04:31

Can you give me the deps.edn you are using so I can try locally?

borkdude18:04:31

it could be that you should also bump edamame

yonatanel18:04:48

project.clj:

(defproject playing "0.1.0-SNAPSHOT"
  :dependencies [[org.clojure/clojure "1.10.1"]
                 [org.clojure/core.async "1.1.587"]
                 [org.clojure/spec.alpha "0.2.187"]
                 [metosin/malli "0.0.1-SNAPSHOT"]
                 [borkdude/sci "0.0.13-alpha.14"]]
  :profiles {:dev {:source-paths ["dev"]
                   :dependencies [[org.clojure/tools.namespace "1.0.0"]]}}
  :global-vars {*warn-on-reflection* true})

yonatanel18:04:28

Do I need to add edamame?

yonatanel18:04:04

I've now included [borkdude/edamame "0.0.11-alpha.6"] and it works

borkdude18:04:10

Yes, I think malli explicitly depends on an older version. When I add [borkdude/edamame "0.0.11-alpha.6"] it works:

user=> (require '[sci.core :as sci])
nil
user=> (sci/eval-string "(+ 1 2 3)")
6

yonatanel18:04:51

Oh, so malli won the transitive dependency

borkdude19:04:08

also apparently the order matters:

[borkdude/sci "0.0.13-alpha.14"]
                 [metosin/malli "0.0.1-SNAPSHOT"]
this also works

borkdude19:04:19

in maven it's first come first serve apparently

yonatanel19:04:58

Yes. I'll just add the latest edamame as a top level dependency.