Fork me on GitHub
#babashka
<
2021-04-29
>
kokada03:04:34

Probably a super niche usage, but maybe useful for someone: https://gist.github.com/thiagokokada/854e6be42fe80afb946324c62d299ce8 An equivalent of Python's os.isatty() in Babashka, to check if the stdin/`stdout`/`stderr` is connected to a TTY or not (useful to check if the script is being redirected) This uses test (the same as if [ ... ] used in shell) to check it. I don't know if it is portable, but I think it should be considering that test is POSIX.

kokada03:04:37

BTW, process is really great compared to other subprocess libraries that I used in Clojure in the past. Without it, this kind of feature would be impossible (since AFAIK, even Java doesn't offer anything to do it)

borkdude09:04:55

Cool, I will test this on Windows

borkdude09:04:16

On Windows cmd.exe I get: Message: Cannot run program "test": CreateProcess error=2, The system cannot find the file specified

borkdude09:04:34

Same in powershell

kokada14:04:17

Yeah, I think on Windows will not work unless in WSL

kokada14:04:28

Since test is POSIX

kokada14:04:44

(Portable I mean, anything except Windows)

borkdude14:04:05

is this so you are wrapping rlwrap?

kokada14:04:23

I don't even know if Windows has the concept of stdin/stderr/stdout, I think they have something similar but the implementation is different

borkdude14:04:52

They do have that, but the tty stuff isn't there, it's totally different

kokada14:04:43

> is this so you are wrapping rlwrap? No, actually it is "kinda" common to have a script that needs to check if its connect to a TTY to offer a better user experience

borkdude14:04:18

Maybe you can contribute that function/script to $BABASHKA_REPO/examples

馃憤 1
kokada14:04:58

In my particular case, I have a command that when called by another command, it shouldn't throw (since the commands that call it doesn't expect a stack trace in the output of this command)

kokada14:04:20

But I still want stack trace when I use this command directly, since it makes it easier to debug

borkdude14:04:52

Is this a command in bb.edn or just a script?

kokada14:04:02

It is a script

borkdude14:04:20

makes sense

kokada14:04:22

(Actually it is kinda of a full Clojure project CLI, Babashka is very powerful to makes sense for this case too 馃檪 )

borkdude14:04:09

I just gave a presentation at a company titled "Clj-kondo and babashka: keeping your Clojure projects sane." ;)

borkdude14:04:15

Where is the actual script?

kokada14:04:23

Oh, sorry, forgot to commit it

borkdude09:04:56

Included :enter, :leave and current-task in a branch. It works like this:

$ cat bb.edn
{:tasks {:enter (println ">>" (:name (current-task)))
         :leave (println "<<" (:name (current-task)))
         a (+ 1 2 3)
         b {:depends [a]
            :task (+ 2 3 a)}
         c {:enter (println "My beautiful message")
            :leave nil
            :depends [b]
            :task b}}}
$ bb run --prn c
>> a
<< a
>> b
<< b
My beautiful message
11

馃憤 2
borkdude09:04:17

I wonder if :enter should be run before the dependencies of the task or like it is now (it was easier to implement like it is now).

borkdude10:04:52

current-task gives more info, e.g. when I do :leave (clojure.pprint/pprint (current-task)) in c, I get:

{:enter (println "My beautiful message"),
 :leave (clojure.pprint/pprint (current-task)),
 :name c,
 :before [a b],
 :depends [b],
 :task b}

borkdude10:04:22

:before are the tasks that were scheduled before the current task

borkdude10:04:56

you can stick any data in the task and it will also be available in current-task

borkdude10:04:45

Binaries in #babashka-circleci-builds if you want to play around with this. Perhaps @karol.wojcik this is useful for logging somehow.

Karol W贸jcik11:04:28

I'm totally satisfied with current printing 馃檪 Here are some screenshots:

borkdude11:04:49

You mean, you don't need any hooks for this, you do it manually, right?

Karol W贸jcik11:04:13

I'm using

(def TASK_NAME (or (resolve 'babashka.tasks/*-task-name*)
                   (resolve 'babashka.tasks/*task-name*)))

(alter-var-root #'babashka.tasks/-log-info
                (fn [f]
                  (fn [& strs]
                    (hpr (str "Command " (red "<") (accent @TASK_NAME) (red ">"))))))
As long as this API stays I'm very happy 馃檪

Karol W贸jcik11:04:38

Anything else I'm doing manually

borkdude11:04:52

This API will slightly change to (:name (current-task))

borkdude11:04:16

also -log-info won't be there anymore

borkdude11:04:29

(everything starting with -foo is private, you should not be using it, only for experimentation)

Karol W贸jcik11:04:04

Ok got it. Next release of babashka and I will have to use hooks

borkdude11:04:59

How did you get the "required" commands, by parsing the EDN manually?

Karol W贸jcik11:04:59

I just know what holy-lambda needs. I'm trying to check whether tool exists via which. If it does not exists then I'm returning a warning.

borkdude11:04:10

ah, I thought it was about the required tasks. in the next version one could print:

:enter (println "Already completed tasks" (:depends (current-task)))
for example

馃憤 1
pithyless10:04:37

for what it's worth, I consider enter/leave wrapping the specific task, so I agree it should work as it does now; i.e. [enter-dep dep leave-dep enter-task task leave-task] :thumbsup:

馃槄 1
borkdude10:04:54

If I put :enter (clojure.pprint/pprint (current-task)) in b you will also get:

:dependees #{c}
so you can answer the question "why am I running" in the task. Am I the main task, or am I a dependency?

pithyless10:04:00

Cool, but is it a set to future-proof with parallel tasks, etc? ie. is it smart enough to parse a graph and understand all potential dependees? Or is it basically "who called me"?

borkdude10:04:51

In the --parallel setting, if c depends on the current task, c will always wait before the current task is finished.

borkdude10:04:00

Same is true for the non-parallel setting

borkdude10:04:17

oh, you mean, why is it a set?

鈽濓笍 1
borkdude10:04:46

because there is not really an ordering of who depends on "me" , I guess

borkdude10:04:56

you will also get :after [c] which is the exact ordering (scheduling) of tasks that are run after you, even if they don't depend on you

borkdude10:04:58

the algorithm is deterministic (but may change over versions, not likely)

pithyless10:04:52

Makes sense and it's nice to have this metadata available during runtime; I also like the move to the function current-task and that it returns a map with all this context!

pithyless10:04:00

I think there is some unwritten rule that any data-driven Clojure library slowly grows into an ad hoc, informally specified DAG (and basically just differs with the amount of work that is put into the implementation of graph traversals and inspection tooling). 馃槄

borkdude10:04:47

hehe. maybe I should base tasks on EQL / Pathom or something? ;)

borkdude10:04:40

Btw, maybe it's better to make :after, :before and :dependees functions of the current-task rather than putting it in the map, since there might be some work needed to calculate this (although it's pretty fast right now)

borkdude10:04:55

:after and :before are basically free because I'm adding them as I go through a loop, they are readily available, but :dependees is calculated once at the start by basically reversing :depends of all the relevant tasks

pithyless10:04:29

How exactly would that work? you would still need to pass all the info needed to generate that context info, right? I can't imagine a lot of extra work is going into actually generating the sets and maps. (And it's not like we're spending a lot of memory, either)

pithyless10:04:40

yeah, so you build an index once and use it; I suppose you could wrap it in a function/delay so it won't run if nobody ever needs it, but 炉\(銉)/炉

pithyless10:04:18

and so then :dependees can be a function so it can be lazy-calculated

borkdude10:04:49

(dependees (current-task))

borkdude10:04:09

(dependees (current-task) {:transitive true})

borkdude10:04:45

but anyway, yeah, it's pretty cheap to add it in a field right now.

borkdude10:04:18

even with 100 tasks it's below 1ms:

user=> (def test-case (zipmap (range 0 100) (repeatedly #(do {:depends (take (rand-int 5) (repeatedly (fn [] (rand-int 100))))}))))
#'user/test-case
user=> (time (tasks->dependees (range 100) test-case))
"Elapsed time: 0.769574 msecs"

borkdude11:04:30

heck, I guess it can even be lazily calculated when people call current-task. I guess that's the nice thing about exposing a function rather than a dynamic var

jkrasnay12:04:29

BTW if someone depends on me they are my dependent, so perhaps :dependees should be :dependents

borkdude12:04:16

Ah, is that how you call it

borkdude13:04:02

I will rename it

馃憤 1
borkdude13:04:27

For the auto-completion stuff, I was also playing around with the idea in my head to just generate stub scripts:

bb tasks --gen-stubs run
so you could get
run/task-a
run/task-a.bat
run/task-b
run/task-b.bat
(for windows and unix) so you can just do run/<TAB> and get task- autocompletions, where run/task-a just contains #!/usr/bin/env bb\n bb run task-a For subcommands this can become run/task-a/subcmd-1. Maybe a dumb idea, but might work? ;) At least it's robust enough to work crossplatform.

jeroenvandijk16:04:54

Yeah i think this is a valid approach as well :)

jeroenvandijk16:04:06

Regarding subcommands, that鈥檚 not a thing in bb.edn yet, right?

borkdude16:04:07

arg parsing is the job of a task, bb doesn't really get in the way but doesn't help with it either

borkdude16:04:29

it just provides the raw args

jeroenvandijk16:04:42

yeah exactly, I鈥檝e used this property during experimentation. It鈥檚 useful

borkdude16:04:47

Here is a version of generating stubs:

(require '[clojure.edn :as edn])

(def tasks (->> (slurp "bb.edn")
                (edn/read-string)
                :tasks
                keys
                (filter simple-symbol?)))

(require '[babashka.fs :as fs])

(fs/create-dirs "run")

(doseq [t tasks
        :let [f (fs/file "run" (str t))]]
  (spit f (str "#!/usr/bin/env bash\nbb run " t))
  (.setExecutable f true))

borkdude16:04:04

And then run/a, run/b

jeroenvandijk17:04:11

I鈥檓 not sure it is common practise, but I鈥檓 used to use script/ (I guess I copied this from Rails). Is using run/ as directory also common practise?

borkdude17:04:38

No, I did this because bb run <task> is the command, but script would also work. This should be configurable.

馃憤 1
borkdude17:04:45

I personally like script

jeroenvandijk17:04:14

Cool. Yeah it seems many people use it

jeroenvandijk17:04:51

Many old Rails users maybe I don鈥檛 know

wilkerlucio17:04:24

I understand this is a safe path, but having completion over bb ... would be much cooler IMO, I love that now with bb my scripts directory is gone, and I hope for good 馃檪 I guess would be nice to have both approaches, so people/teams that need more compatible solutions could use that, and others could avoid the extra files

1
wilkerlucio17:04:12

I dont know anything about how shells do auto-complete, is the issue that different shells (bash, zsh鈥) have different ways to do it?

borkdude17:04:10

yeah, I think it can be supported using these bash/zsh scripts

jeroenvandijk18:04:15

I鈥檓 trying to understand https://github.com/remkop/picocli/blob/master/src/test/resources/picocompletion-demo-help_completion.bash as we speak 馃槄 on a second look it doesn鈥檛 look that complicated. I鈥檒l let you know

jeroenvandijk20:04:44

This semi works. Needs proper unit tests for all the weird edge cases https://gist.github.com/jeroenvandijk/42a570415a45c68989f4f506977050c4

borkdude20:04:03

how must one use it?

jeroenvandijk20:04:06

this should do it https://gist.github.com/jeroenvandijk/42a570415a45c68989f4f506977050c4#file-autocomplete-clj-L6-L19 although it can be a bit funky. There seems to be some state involved

jeroenvandijk20:04:56

so in this case bb-complete is not a command but it will be autocompleted. If you type bb-complete and press TAB it should do something

jeroenvandijk20:04:24

very vague sorry. I need to test it more thoroughly and several times to make sure it works in any state

jeroenvandijk20:04:13

I鈥檒l look into it later again, but i鈥檓 just posting it here in case someone wants to try it

cldwalker13:05:30

In case anyone is interested in zsh autocompletion for bb tasks:

_bb_tasks() {
    local matches=(`bb tasks |tail -n +3 |cut -f1 -d ' '`)
    compadd -a matches
    _files # autocomplete filenames as well
}
compdef _bb_tasks bb
If we want to make this work across more platforms i.e. don't rely on tail and cut, then bb could just print completions with an option e.g. bb --completions

馃檶 1
鉂わ笍 2
wilkerlucio14:05:22

@U08ALHZ2N thank you very much!!

1
borkdude14:05:04

@U066U8JQJ I'd be happy to merge this as is, but please read this: https://github.com/babashka/book/blob/master/CONTRIBUTING.md Also @U08ALHZ2N should give his permissions to use the code :)

cldwalker14:05:29

Gave permission 馃檪

wilkerlucio14:05:23

I guess @U08ALHZ2N you may need to create PR similar to 鈽濓笍 , @U04V15CAJ can confirm if that's the case

borkdude14:05:18

merged

馃帀 2
borkdude14:05:43

republishing book now

borkdude14:04:47

Interestingly this library (which has been suggested to me to use for bb itself) supports generating an auto-completion script: https://picocli.info/autocomplete.html

馃憣 2
jeroenvandijk16:04:23

Did you find examples of generated scripts? I鈥檓 curious what they will look like

borkdude16:04:31

no, I guess you could try it

jeroenvandijk16:04:39

Or am i misunderstanding and it鈥檚 not generating a bash script

jeroenvandijk16:04:50

yeah i鈥檒l try it eventually 馃檪

jeroenvandijk16:04:06

So the magic happens in bash鈥 I think we can do better 馃檪

jeroenvandijk16:04:38

But good for inspiration

borkdude16:04:20

port to bb ;)

jeroenvandijk16:04:40

Maybe we can use our Dutch charm here. I believe the creator of PicoCli is Dutch

borkdude17:04:21

I mean for our usage, not for picocli, I don't think picocli will make users require bb ;)

jeroenvandijk17:04:48

haha maybe not

borkdude17:04:46

A script that dumps scripts in script so you can invoke a task using auto-completion based on filename:

#!/usr/bin/env bb

(require '[clojure.edn :as edn]
         '[clojure.string :as str])

(def tasks (->> (slurp "bb.edn")
                (edn/read-string)
                :tasks
                keys
                (filter simple-symbol?)))

(require '[babashka.fs :as fs])

(def script-dir "script")
(fs/create-dirs script-dir)

(doseq [t tasks
        :let [f (fs/file script-dir (str t))]
        :when (not (str/starts-with? t "-"))]
  (spit f (str "#!/usr/bin/env bash\n# generated by stubs.clj\nbb run " t))
  (.setExecutable f true))

borkdude17:04:04

Kind of an happy accident that forward slashes don't work in task names ;)

eigenhombre17:04:18

Hi! A friend of mine showed me using docopt with Babashka, loaded using deps/add-deps:

(deps/add-deps
 '{:deps {nubank/docopt {:git/url ""
                         :sha "12b997548381b607ddb246e4f4c54c01906e70aa"}}})

(require '[docopt.core :as docopt])
With some experimentation I was able to load local dependencies
(deps/add-deps
 '{:deps {
          ;; A local dependency:
          eigenhombre/example {:local/root "."}}})
and even Maven dependencies:
(deps/add-deps
 '{:deps {eigenhombre/namejen {:mvn/version "0.1.14"})
My question is, how does this work?! I thought Babashka used sci under the hood to provide a sort of interpreted Clojure, taking advantage of GraalVM鈥檚 native compilation. (1) What is it actually doing when it loads these dependencies? How are they executing within the Babashka process? (2) Can this be used to make Babashka more stripped down, so fewer libraries need to be built in? (3) What are the tradeoffs of loading libraries this way, as opposed to building them into Babashka? Thanks!

jeroenvandijk17:04:29

Hopefully I鈥檓 answering this right, I鈥檓 happy to be corrected 馃檪

jeroenvandijk17:04:32

1) Code is loaded from the classpath by shelling out to the clj command line tool

jeroenvandijk17:04:35

2) Babashka has many dependencies built in because a) some cannot be dynamically loaded b) are big and would add extra latency because of extra parsing. So yes, you could strip down babashka if that鈥檚 important to you (see the different versions of Babashka based on feature flags)

jeroenvandijk17:04:12

An alternative to libraries are Pods. These are binaries themselves, often Clojure programmes compiled by Graalvm itself (not a requirement), and connect with Babashka via a socket

borkdude18:04:30

A few additions. 1) You can set the BABASHKA_CLASSPATH or --classpath manually using your favorite build tool. This became tedious so now babashka also supports a bb.edn similar to deps.edn (but with less features, no aliases yet for example). 2) When bb must calculate the classpath using bb.edn or babashka.deps/add-deps it will shell out to java and uses a tools.deps uberjar to download deps and calc the classpath, This is effectively what the clojure bash script also does. 3) So this happens similar to how the clojure CLI does it, but it does not use or need the installed clojure CLI, because this is done using https://github.com/borkdude/deps.clj as a built-in library 4) Interpreted libs are slower than built-in ones and not all of the code is compatible with babashka, so not all libs can be run from source. Having some things built-in, even if they are source-compatible with bb, sometimes makes sense, like clojure.tools.cli. 5) Babashka has feature flags that can be used to trim the binary down, if you don't need all of the built-in libs. It also has feature flags to enable more built-in libs. 6) Most notably libs relying on non-standard Java classes, deftype or definterface are not source-compatible with babashka 7) Some non-compatible libs can be made compatible by introducing :bb reader conditionals

馃憤 1
kokada20:04:59

I have some other questions: 3) This still needs java installed right? If I want a fully "static" (something that only depends on Babashka) script I need either to use uberscript or uberjar right? 4) If I add the built-in libraries to babashka in my bb.edn, does it load the built-in ones or the ones defined in bb.edn. I am asking this because I include some builtin libraries in my bb.edn because my Babashka's project also have a project.clj with lein-tools-deps to make development easier (e.g.: at least for now, CIDER doesn't support Babashka REPL very well), but I wouldn't want to inccur an additional load for this. If this is an issue I may move all bb.edn dependencies to project.clj. 7) Does Babashka's interprets :clj reader macro? If yes, what it does in case of having both :clj and :bb?

borkdude20:04:40

3) correct, or take care that the deps have been already downloaded and provide the classpath manually 4) it loads the built-in ones, unless you use (:require [foo.bar] :reload) 7) Yes, it does, for compatibility with existing libraries. If you have both, then put :bb as the first branch

馃憤 1
kokada20:04:51

> If this is an issue I may move all聽`bb.edn`聽dependencies to聽`project.clj`. Actually, this seems to work fine, and seems to be a better idea regardless so I did it Anyway, I am still curious if Babashka will load the dependencies from bb.edn or the builtin ones

eigenhombre20:04:21

Thank you @U04V15CAJ. So, when Babashka pulls in a dependency from, say, Maven, and runs it, it is literally interpreting the source code it gets out of the downloaded jar file?

eigenhombre20:04:05

Wow, cool. It is so great to see Babashka continuing to evolve in new and useful ways.

kokada20:04:19

> it is literally interpreting the source code it gets out of the downloaded jar file? And this only works for Clojure jars right? Since sci can't interpret Java files

borkdude20:04:20

True. It only works for Clojure source

馃憤 1
鉁旓笍 1
kokada20:04:33

It is really amazing how well eveything works, thanks for the explanation @U04V15CAJ

馃憤 1
borkdude21:04:48

@pithyless and others. I have one other naming bikeshed:

:after [update-project-clj -uberjar build-server -jar lsp-jar]
:before [java1.8]
If you see this in the output of (current-task), how do you interpret this? What I want to express: these jobs were scheduled before, these other jobs were scheduled after. But I imagine one could also read it as: this job comes before these and after these others.

Tommy DeVito22:04:30

:past :future or :previous :next?

馃憤 1
pithyless07:04:32

I think the question is why you want this info. The depends/dependents is used to identify dependency relationships between tasks. What you're proposing is how those dependencies (multiple potential orderings) were actually flattened to a single concrete ordering. So this may be interesting for auditing (the past ones) and planning (the future ones). I think the names should reflect that - perhaps one of #{:processed :executed :audit :completed :past} for the previous and #{:queued :planned :plan :upcoming :future} for the future ones?

borkdude07:04:29

I was contemplating on adding just the :plan [a b c d e] but if you are task c it might be awkward to see where you are in the plan.

borkdude07:04:00

but this info doesn't necessarily mean that a and b were already completed if you use --parallel, unless you :depends on them (or you depend on them transitively)

borkdude07:04:52

so maybe I'll just leave it out for now

pithyless07:04:14

> but this info doesn't necessarily mean that a and b were already completed if you use聽`--parallel`, unless you聽`:depends`聽on them (or you depend on them transitively)聽(edited) Are you referring to the proposed before/after or to the aforementioned [:plan a b c d e]?

borkdude07:04:18

both are the same, but before/after is just a split of the whole sequence based on the current task

pithyless07:04:50

OK, so this doesn't convey that the previous tasks were completed, but the previous ones have definitely been already started, correct? And in that order. (Even if running in parallel, so may not complete in that order)

borkdude07:04:58

Maybe just a transitive depends and transitive dependents is more useful, since you can rely on those being completed / not yet started

pithyless07:04:59

OK, so the names were not that obvious to me; perhaps something more vague ala :started / :planned ?

pithyless07:04:54

And if we don't have this, what do we lose? The ability to understand all tasks that would be run in this session (without processing all the transitive deps)?

borkdude07:04:23

OK, so to back up a bit: The primary reason for me to add :enter and :leave and (current-task) + :name is so you can decide on your own logging, when the job begins and ends. You may not want to log certain stuff if you are not the main invoked task (:dependents is empty) or maybe you do if you are invoked by some main wrapping task (the only task in dependents is main-task which does some setup and then calls you)

borkdude07:04:36

so it seems we only need :name and :dependents for this

馃憤 2
pithyless07:04:51

The only use-case I can think of for a global :plan is if we want some logic in a task that is conditional on whether some arbitrary task will ever run during this session.

pithyless07:04:05

Another use-case may be an initial logger that says: "This is everything that will run now: ..." (but that seems a bit of a stretch)

pithyless07:04:29

But I reserve the right to not be particularly creative with use-cases 馃槈

borkdude08:04:00

It might be best to not include anything at this point and just wait for the proper use cases to arise

borkdude08:04:35

You can fairly easy get the already running tasks like this:

{:tasks {:init (def ctx (atom []))
         :enter (swap! ctx conj (:name (current-task)))
         bar []
         foo {:depends [bar]
              :task @ctx}}}

borkdude08:04:37

or capture the first running task by putting it in some state and if this state exists, not overwrite it

borkdude08:04:11

I mean, the main task, but hmm, this is not true, since they won't get to :enter until after the deps

borkdude08:04:38

therefore having the entire plan might still be useful since you can see "who" invoked the entire chain

borkdude08:04:22

Maybe :started and :pending are useful names

borkdude08:04:16

Just :pending perhaps

borkdude08:04:09

The use case I have in mind for this: you might have a deploy task. Subtasks may only do want to do certain work if a certain environment variable is set. So uberjar might only want to do work if it's not started because of deploy or if the CLOJARS_USERNAME is set. But this might be getting too complicated if you write tasks like this.

borkdude08:04:14

I'll leave out these hypothetical keys and wait until later

borkdude21:04:31

I could also just leave this information out completely, and only add :dependents #{-jar update-project-clj -uberjar}. I guess that info is useful in the sense of answering the question "why am I being invoked", whereas the exact order of tasks isn't very useful information and can even be confusing (with respect to parallel).

kokada22:04:52

I think this kind of information maybe interesting to debugging purposes, but not if it is showed always

cldwalker13:05:30

In case anyone is interested in zsh autocompletion for bb tasks:

_bb_tasks() {
    local matches=(`bb tasks |tail -n +3 |cut -f1 -d ' '`)
    compadd -a matches
    _files # autocomplete filenames as well
}
compdef _bb_tasks bb
If we want to make this work across more platforms i.e. don't rely on tail and cut, then bb could just print completions with an option e.g. bb --completions

馃檶 1
鉂わ笍 2