Fork me on GitHub
#babashka
<
2021-04-11
>
grazfather02:04:20

is it looking like :when won’t be a thing?

grazfather02:04:35

tasks are looking pretty damn good to me

borkdude08:04:27

@grazfather Thanks. I had trouble figuring out what the behavior of :when in dependent tasks should be. E.g. invoke foo which depends on bar which depends on baz. But :when in bar returns false, then what?

grazfather14:04:52

If foo depends on bar which depends on baz, then if bar’s :when says not to run, then it implies that baz is done as well.

grazfather14:04:39

I don’t think it’s very critical that you have this feature, but if you do, then I would liteirally just copy the behaviour from make. I was using a makefile I wrote myself to compare my bb task

borkdude14:04:20

does Make have something like when?

grazfather14:04:42

only for non-phony targets, it uses the time stamp of the target vs any of its dependencies

a: b c
basically does:
a {:when (and (modified-since a b) (modified-since a c)}

grazfather14:04:03

but

a: b
b: c
only checks the time of b vs a, and if b is newer it runs a. it doesn’t care about c.

grazfather14:04:45

i might have gotten the order of the args for modified-since wrong 🙂

borkdude14:04:24

one example I had in mind was the above:

{:tasks {a {:depends [b]} {b {:when (System/getenv "CLOJARS_USER_NAME)}}}
` so if the condition of b didn't pass, I might want to skip as well

borkdude14:04:42

let's say that b is deploying something based on that env var and a depends on the thing having been deployed

grazfather14:04:12

oh, Make isn’t smart enough for that, the way you’d do it in make is you could put the env var ITSELF as an uinbuildable dependency

grazfather14:04:39

so a and b depend on env and env doesn’t set anything, but just fails if the end var is not set

borkdude14:04:35

right. we could encode some special return value like :tasks/invalid which would yield the entire dependency tree invalid, for example.

grazfather14:04:37

that makes me wonder: Do you capture failure from tasks and propagate then?

grazfather14:04:48

exactly 🙂

grazfather14:04:12

again I stress that we should not make up new semantics as best we can

borkdude14:04:25

right now this is done using System/exit. So if a shell command doesn't return with 0 the entire process quits with that exit code

borkdude14:04:46

that might be the least surprising way indeed

grazfather14:04:52

yep that is perfect to me

borkdude11:04:16

@grazfather I think you can have a creative workaround like this:

{:tasks {clojars-user-name
         (when-not (System/getenv "CLOJARS_USER_NAME")
           (println "Needs clojars username")
           (System/exit 1))
         deploy {:depends [clojars-user-name]
                 :task (println "Deploying...")}}}

grazfather14:04:48

Yep I was using when/when-not instead,no big deal :)

grazfather14:04:56

nice. hidden tasks!

borkdude14:04:32

yep. - will make it hidden, or :private true.

borkdude14:04:14

You can still call it from the command line if you want, it just won't show up in the tasks list

grazfather14:04:13

I am still pretty slow writing lisps and edn but I think I will soon use this to write all my ‘test targets’. a lot of then depend on certain docker containers to be running, and some need to run inside the container. We do a lot with makefiles but it gets hacky do determine if a container is running and if you’re in a container

borkdude14:04:52

great! keep me posted as we have still room to improve and break things if we need to

grazfather14:04:25

Yep I will let you know if I hit any snags

pithyless11:04:18

@borkdude at first I was skeptical, but I managed to convert some code to the new tasks format:

{:init
  (do
    (ns my.tasks
      (:require
       [proj.repo.task.version :as task.version]
       [proj.repo.ednopt :as ednopt]))

    (def default-args
      {:min-version/clojure  "1.10.3.814"
       :min-version/babashka "0.3.3"})

    (def args (merge default-args
                     (ednopt/parse *command-line-args*))))

  check-versions
  {:doc "Check runtime versions are supported"
   :task (task.version/check-versions args)}}
1. if dependent tasks do not allow passing args to each other (as it is currently implemented), they are consistent, in that, they act the same whether called by a different task or directly from the command line. :thumbsup: 2. I thought not passing args around would be a problem for my monorepo (where the reason I moved from Makefile to Justfile was better support for arguments in tasks); but it turns out if I only allow global arguments (via parsing kvs in :init and removing any conflicts via namespaced-keywords), I still get all the benefits of task parameterization, while also having the benefits of (1) and the nice task aliasing that a common bb tasks runtime convention provides 3. I think :when and :depends DSL is completely optional, since this all can be done via when and function calls in the task itself; but it raises some points: should the semantics be (when <guard> (run-dependent-tasks) (run-task)) or (do (run-dependent-tasks) (when <guard> (run task))) - all of which would be mute, if we took more care when writing dependent tasks to be idempotent (is that feasible?)

pithyless11:04:25

(3) is kind of rambling, I'm just wondering if the current notions of task dependencies and guards in Makefile/Justfile/etc are just a poor-man's approach to idempotent functions (and maybe we can leverage clojure idioms more here)

borkdude11:04:08

@pithyless co-incidentally, I just posted an example of :depends that replaces the need for :when possibly, around the same millisecond you posted your long message

😅 3
borkdude11:04:54

I'll take a look at your other comments later today. Thanks for the feedback

borkdude12:04:03

@pithyless 1+2, cool. 3 :when doesn't exist anymore. :depends is there to topologically sort tasks, so when two tasks depend on a third task and you depend on these two tasks, the third task is only executed once, which is a common feature in build systems.

pithyless12:04:57

ad 3. Yeah, that makes sense, without :when there is no question about the semantics of how it interacts with :depends.

borkdude12:04:10

right that was the reason I removed it (`:when`)

borkdude12:04:50

you can just short-circuit using System/exit if needed

pithyless12:04:00

I'm going to try to build up something more substantial with the existing bb tasks. For me the missing feature was an easy way to parameterize tasks, and I think I found it via combining the existing :init with that previous idea I talked about inspired by passing EDN kw-args on the command line.

pithyless12:04:19

Thanks for all the work @borkdude! 🙂

borkdude12:04:44

Thanks for trying and looking forward to more feedback

grazfather14:04:41

I think it’s pretty safe to not rush :when and let :depend ( which is more important) to shake out any issues first

👍 3
borkdude14:04:43

If you have any topics for a talk at a virtual Clojure conf, feel free to respond on Twitter or here (in a thread): https://twitter.com/borkdude/status/1380950518567145473

grazfather15:04:28

posted without comment 😈

grazfather15:04:37

bash
$ time make coffee
measuring beans
grinding beans
pouring water
heating water
getting filter
getting mug
brewing coffee

real    0m15.121s
user    0m0.016s
sys     0m0.023s
$ time make -j4 coffee
pouring water
measuring beans
getting mug
getting filter
grinding beans
heating water
brewing coffee

real    0m9.039s
user    0m0.016s
sys     0m0.024s

borkdude15:04:45

Can you post a comment anyway? What am I looking at? What's -j4 ?

grazfather15:04:44

parallelizes (4 threads)

borkdude15:04:59

zomg, yeah we could do that

grazfather15:04:05

the topo sort can find independent chains

grazfather15:04:12

see how it’s 15s vs 9 seconds?

grazfather15:04:45

first first heats water then grinds beans. the second does both at the same time. I use it as an example always to explain makefiles 🙂

borkdude15:04:14

in tasks you can have futures or core.async and do this yourself as well

grazfather15:04:01

yeah I am going to play with it right now. I was just brewing coffee and literally thought “might as well start the kettle while I grind” 😉

borkdude15:04:20

keep the Java flowing

👌 3
clj 3
grazfather15:04:50

so I am not actually sure how/if you could share futures between tasks, I just made this dummy task that acts like a parallelized make coffee but it doesn’t properly specify dependencies

(defn measure-beans []
 (println "measuring beans")
 (Thread/sleep 1000)) 

(defn grind-beans []
  @(future (measure-beans))
  (println "grinding beans")
  (Thread/sleep 2000))

(defn pour-water []
 (println "pouring water")
 (Thread/sleep 1000)) 

(defn heat-water []
  @(future (pour-water))
  (println "heating water")
  (Thread/sleep 2000))

(defn get-filter []
  (println "getting filter")
  (Thread/sleep 1000))

(defn get-mug []
  (println "getting-mug")
  (Thread/sleep 1000))

(defn make-coffee []
 (let [grounds (future (grind-beans))
       hot-water (future (heat-water))
       filter (future (get-filter))
       mug (future (get-mug))]
   (println "brewing coffee")
   @grounds
   @hot-water
   @filter
   @mug
   (Thread/sleep 3000)))

(defn- main []
 (make-coffee)) 

(main)

borkdude15:04:08

@(future (pour-water))
this won't really help, since you're immediately waiting for the thread

borkdude15:04:09

How does make parallelization work when one of the tasks returns with a non-zero exit code?

grazfather15:04:23

Yeah I know about the @(future thing 🙂 it’s just there so that I show that i am doing it in another thread, not real code

grazfather15:04:22

if anything in the dependency chain fails the failures percolate upward

grazfather15:04:27

play with it 🙂

grazfather15:04:35

.PHONY: a b c

coffee: grounds hot_water filter mug
	@echo "brewing coffee"
	@sleep 3

grounds: beans
	@echo "grinding beans"
	@sleep 2
	exit 1

hot_water: water
	@echo "heating water"
	@sleep 2

filter:
	@echo "getting filter"
	@sleep 1

mug:
	@echo "getting mug"
	@sleep 1

water:
	@echo "pouring water"
	@sleep 1

beans:
	@echo "measuring beans"
	@sleep 1

grazfather15:04:23

I am not sure if/how Make stops ‘parallel’ tasks. I know it doesn’t interrupt them

grazfather15:04:14

but say that a depends on b and c, and b depends on b2 and c on c2, if b2 and c2 are running and c2 fails, I am pretty sure that make will stop b from running (after b2)

grazfather15:04:26

but also I think that that’s less important

grazfather15:04:40

make: *** [beans] Error 1
make: *** Waiting for unfinished jobs....

borkdude15:04:30

how tasks are currently implemented, bb would just quit while the others are still running (because it has no concept of parallel jobs yet)

borkdude16:04:12

maybe we could have a special (quit! 1) or so function that would delay the real exit while other tasks are still running

borkdude16:04:55

(set-exit-code! 1) something in this vain

grazfather16:04:32

yeah, it makes sense that we could have an implicit check before starting a task to see if the whole task tree has been abandoned. I think, actually, that the user shouldn’t have to see/use this. Just implicitly set the global fair when a task fails. would you support kicking off more than one task at a time? Then this failure signal would have to be local to that task chain

borkdude16:04:41

I guess throwing an exception would be the better, more supported alternative

grazfather16:04:18

Yeah I am thinking abstractly, not really how it would be implemented (I don’t really know Java)

grazfather16:04:02

I am just thinking about the semantics and about whether the ‘ui’ would have people set the error and check it.. I think that they should not. It definitely can be considered ugly/less pure but it makes sense to me that if A is kicked off and b2 fails while c2 is running, c2 can finish (`Waiting for jobs to finish`) but c never starts since it implicitly checks if the job (which is A) is still running before starting

borkdude17:04:19

yeah, I'm thinking how people can signal the flow to stop and which exit code to use, you could do (throw (ex-info "" {:babashka/exit 1})) which would get caught by the consumer of your future and this exit could would then be handled by babashka

borkdude17:04:00

this is already how futures work, once you deref them and an exception occurred, then the exception is re-thrown

borkdude17:04:43

so for the above you could have:

(def b2 (future ....))
(def c2 (future ....))
(def b (future (do @b ....)))
(def c (future (do @c2 ...)))
(def a (do @b @c ...))
and it would work

borkdude17:04:56

we could make this work relatively easy by letting people just write future around their task and handle the rest automatically. this would give some control about which tasks you want to run in a future and which you don't

borkdude19:04:41

@grazfather On master. Binary in #babashka-circleci-builds for trying.

grazfather19:04:34

you’re a machine!

grazfather19:04:05

Just looked at the commit, very cool. I am learning a lot

mike_ananev15:04:09

I'm trying to use Cursive as IDE to write bb scripts. What library should I include to get babashka.deps resolved?

(require '[babashka.deps :as deps])
Execution error (FileNotFoundException) at tasks/eval17985 (REPL:1).

borkdude15:04:08

unfortunately that lib isn't available from the JVM as this is very bb specific and not generally applicable

mike_ananev16:04:16

@borkdude, thanks. I'm trying to rewrite my app and lib templates using babashka only. I want to throw away just or make utilities from templates.

borkdude17:04:16

sounds cool :)

dabrazhe19:04:41

Hi all. Is there possibility to have the autocomplete on the BB repl? I am running it on the Mac OS shell as rlwrap bb (I might have missed it in the previous discussions).

borkdude19:04:08

@dennisa not with the terminal REPL, but with the nrepl server you will get more functionality like that

dabrazhe19:04:37

Great, thanks will give it a try

dabrazhe17:04:14

@borkdude I gave the bb nrepl a go. It connects fine. The autocomplete fails on pressing the tab clojure.lang.ExceptionInfo: Could not resolve symbol: complete.core/completions

dabrazhe17:04:47

It is my setup or bb?

borkdude17:04:37

@dennisa it seems to work for me. cc @U051BLM8F

dabrazhe18:04:35

It's mine then. Where does bb take it's dependencies from, when connected to nrepl?

borkdude18:04:58

what do you mean?

bozhidar19:04:46

I’m puzzled by this error - to my knowledge bb doesn’t use clojure-complete (and neither does CIDER), so I don’t get where this error is coming from.

borkdude19:04:46

yeah, I'm puzzled too :)

dabrazhe10:04:40

I mean that perhaps I need to configure something similar like .clojure/deps.edn for bb to pick the cljoure-comlete dependency.

borkdude10:04:50

@dennisa No, bb nrepl-server doesn't use this dependency. Also cider.el doesn't use this dependency. So I wonder how you got this error and so is @U051BLM8F

borkdude10:04:49

@dennisa it could be that your nrepl client is trying to evaluate this code in the bb nrepl-server

borkdude10:04:56

but why it's doing that is the question

borkdude10:04:24

what editor are you using

dabrazhe10:04:47

Mac OS iterm terminal application. I'll summarise my steps to reproduce it in a bit.

dabrazhe11:04:14

babashka v0.3.3 Mac OS iTerm2 app Start the nrepl server bb nrepl-server

Started nREPL server at 127.0.0.1:1667
connect with lein lein repl :connect localhost:1667
OpenJDK 64-Bit Server VM warning: Options -Xverify:none and -noverify were deprecated in JDK 13 and will likely be removed in a future release.
Connecting to nREPL at localhost:1667
clojure.lang.ExceptionInfo: Could not resolve symbol: nrepl.core/version
nil
Error loading namespace; falling back to user
nil
user=> (inc 1)
2
(f
press tab after (f, or any other symbol, fails with the exception:
user=> (fclojure.lang.ExceptionInfo: Could not resolve symbol: complete.core/completions

borkdude11:04:35

Ah, I see that. Yeah, it's not something bb can fix, but should be fixed in lein repl to work with nREPL servers that do not have complete.core/completions

borkdude11:04:43

lein repl assumes too much

borkdude11:04:39

completions do work in bb nrepl-server, but using the standard completions op: https://nrepl.org/nrepl/ops.html

borkdude11:04:09

and an nrepl client should check the capabilities of an nrepl server and behave as such

dabrazhe11:04:28

is there's smth I can fix myself? eg connect to the bb nrepl via another cli, ie non-lein?

borkdude11:04:46

Maybe @U051BLM8F knows some other client. Calva has pretty good support for talking to the bb nrepl-server

borkdude11:04:22

but this is not a command line tool, more a full fledged editor

dabrazhe11:04:52

Yes, I checked Calva in VSC, no issues with auto-complete. But... there's no (doc) function : (

(doc +)
; clojure.lang.ExceptionInfo: Could not resolve symbol: doc

borkdude11:04:21

In calva it should be sufficient to hover on the symbol to see the docstring

👍 3
borkdude11:04:50

but doc is available as (require '[clojure.repl :refer [doc]])

dabrazhe11:04:02

Yes, indeed, thank you!

grazfather19:04:19

I love parallelized tasks!

$ time ./bb coffee
measuring beans
grinding beans
pouring water
heating water
getting filter
getting mug
brewing coffee

real    0m11.294s
user    0m0.023s
sys     0m0.042s

$ time ./bb coffeep
measuring beans
grinding beans
pouring water
heating water
getting filter
getting mug
brewing coffee

real    0m5.052s
user    0m0.018s
sys     0m0.018s

sh5420:04:59

Any chance that java.nio.charset.Charset will be supported in the future? I was hoping to use https://github.com/helins/binf.cljc via babashka. That class is the current blocker. I have a custom format that I decode via that library and given I am supporting clojurescript too its nice to call something thats got the bit wrangling for both clojure/script sorted already. It would be great if a little babashka script could just call on my existing regular library clojure/script code to do the decoding rather than having to write a separate path.

sh5420:04:07

to reproduce:

sh5420:04:29

(require '[babashka.deps :as deps]) (deps/add-deps '{:deps {io.helins/binf {:mvn/version "1.0.0-beta1"}}}) (require '[helins.binf :as binf])

borkdude20:04:15

@slack1003 Please make an issue and I'll investigate

sh5420:04:41

k will do

borkdude22:04:02

Implemented the `bb run --parallel <task>` option now so tasks run in parallel when they can.

🎉 10