Fork me on GitHub
#babashka
<
2021-04-25
>
Karol Wójcik08:04:36

@borkdude how can I point to remote scripts? This is what I have now:

{:paths [""]

 :holy-lambda/options {:build-tool    :lein ;; or :deps
                       :runtime       {:name :babashka} ;; Possible runtime :babashka, :native, :java
                       }

 :infrastructure {:bucket-prefix  "prefix"
                  :bucket-name    "bucket"
                  :region         "region"}

 :tasks {:requires                ([holy-lambda.tasks])

         docker:build:ee          holy-lambda.tasks/docker-build-ee
         sync:update              holy-lambda.tasks/sync:update
         version                  holy-lambda.tasks/version
         doctor                   holy-lambda.tasks/doctor
         purge                    holy-lambda.tasks/purge}}
When running the task the script is not found.

borkdude08:04:00

@karol.wojcik Where did you get the idea that "remote scripts" were supported :-D?

borkdude08:04:20

You can make a git library from it and point to it in :deps same as with deps.edn

borkdude08:04:13

Oh, I misinterpreted that. I thought you meant: is it ok if I share the link to a library on github in this channel :)

Karol Wójcik08:04:36

Will try the deps.edn approach then 😄

borkdude08:04:05

@karol.wojcik What was the other bug you found?

Karol Wójcik08:04:55

{:paths ["script"
         ;; ""
         ]

 :holy-lambda/options {:build-tool    :lein ;; or :deps
                       :runtime       {:name :babashka} ;; Possible runtime :babashka, :native, :java
                       }

 :infrastructure {:bucket-prefix  "prefix"
                  :bucket-name    "bucket"
                  :region         "region"}

 :tasks {:requires                ([holy-lambda.tasks :as tasks])

         docker:build:ee          tasks/docker-build-ee <-- needs fully qualified symbol otherwise it does not work.
         sync:update              tasks/sync:update
         version                  tasks/version
         doctor                   tasks/doctor
         purge                    tasks/purge
         }}

Karol Wójcik08:04:40

Using :as does not work in :requires when script is from :paths

borkdude08:04:33

oh cool, I'll fix that

borkdude09:04:54

Fixed on master

borkdude09:04:41

@karol.wojcik Was this bug only related to bb tasks docstrings or also to execution of these tasks?

borkdude09:04:20

I guess also to execution, will also fix that

borkdude09:04:17

Fixed on master

borkdude09:04:40

I am contemplating letting tasks not print the return value by default (like scripts always have done so far which might have been a mistake, but accidentally happened) But still it is convenient to view the result of a (helper) task to debug.

borkdude09:04:07

So maybe bb run --prn foo could be used to explicitly print and bb foo doesn't print

borkdude10:04:41

Or maybe it should be a property of a task if it will print or not?

borkdude10:04:15

{:tasks {foo {:print true :task (System/getenv "FOOBAR")}}}

borkdude12:04:04

@mike1452 shell now supports :continue true to ignore the exit code (so it won't exit bb in that case)

borkdude12:04:01

Also tasks do not print their return value anymore (unless explicitly done or using the run --prn command line opts)

borkdude12:04:05

It's also possible to use the return value of shell and clojure now

borkdude12:04:37

And there is some logging around task names and shell/clojure arguments when setting :log-level :info in the task(s)

borkdude12:04:01

There is also a :min-bb-version to print a warning in case people use an older one (top level in bb.edn)

borkdude13:04:10

Please do try out your bb with {:tasks {:log-level :info ...}} to see if it's useful

mike_ananev13:04:37

@borkdude cool! I'll check new version and change my templates. You are doing awesome work!

Karol Wójcik15:04:53

@borkdude can I pass my own decoration fn for log-level?

borkdude15:04:24

no not yet. I think I might revise the entire log thing based on some feedback

borkdude15:04:25

to experiment: there is babashka.tasks/*-task-name* , babashka.tasks/-log-info and babashka.tasks/-log-error and you can probably roll your own thing with this

👍 2
borkdude15:04:46

all implementation details which will change, but feel free to play around and provide feedback

Karol Wójcik16:04:04

Did I ask how to run babashka tasks like in makefile? make clean pack deploy? 😄

Karol Wójcik16:04:13

Will experiment with it

borkdude16:04:07

@karol.wojcik bb doesn't yet have a way to do this, other than bb clean && bb pack && bb deploy or do that in a task using three shell calls

borkdude16:04:51

I am considering doing something like bb run clean, pack, deploy like comma separated values like lein do has

borkdude16:04:06

so you can still pass args to the tasks

Karol Wójcik16:04:33

Exactly that would work, since I'm parsing the arguments for holy-lambda.

borkdude16:04:32

Made an issue: https://github.com/babashka/babashka/issues/804 But still I'm not yet certain how much needed this is, since it's not much more typing than just using && and like @jeroenvandijk has said: lein do is also about startup time of the JVM, a problem that bb doesn't have.

👍 2
jeroenvandijk07:04:48

There is one thing that might be interesting at some point. When I have nested babashka calls for simple tasks I noticed some extra delay (every bb call is 30ms). So it might be handy to be able to call babashka in process. I’m guessing this would easily beat the 30ms penalty and could have the same isolation (starting with empty context, copy env vars etc)

jeroenvandijk07:04:38

I’m also wondering if you can do multiple clojure calls in one. This would be interesting for commands that shell out to clojure multiple times (i see some examples here https://clojurians.slack.com/archives/CLX41ASCS/p1619387075084200)

borkdude07:04:56

For bb this is certainly possible. Another possibility is to have an explicity run function that runs another task within another task. I'm guessing this is what you are doing in your task? For clojure this is not possible, since that lives in a different process and different calls are going to have different classpaths, etc.

jeroenvandijk07:04:17

Yeah run sounds interesting. Another thing I ran into was that i wanted to have my own uberjar command, but using the bb uberjar task. Not sure if this is smart, but I noticed that I couldn’t call the overrided task. Not sure if run would solve this as well

jeroenvandijk07:04:31

I’m also experimenting with creating a module system. I’m loading babashka files dynamically based on the command that’s given. I’m not sure yet where this is going, but it’s an example of where I’m calling babashka from babashka.

borkdude08:04:44

@jeroenvandijk Could you share your bb.edn perhaps (in secret?) so I can see that is needed with respect to this run function?

borkdude08:04:28

The priority in which things are run is file > task > subcommand. So if you have a file or task named uberjar this will hide the uberjar subcommand.

👍 2
jeroenvandijk08:04:09

I don’t really have one proper bb.edn . I’m working on multiple projects and I’m still figuring things out mostly. (maybe i missed something, but one central bb.edn is not a thing yet, right?)

borkdude08:04:50

bb.edn is supported in the current working directory (of a project)

jeroenvandijk08:04:27

ah ok indeed. Yeah i was trying to figure out how to reuse the same tasks over different projects.

jeroenvandijk08:04:21

I’m trying to move away from leiningen and boot, but I miss some standards. Hence my idea for some kind of universal module system

jeroenvandijk08:04:26

E.g. I think this is nice https://github.com/practicalli/clojure-deps-edn But I don’t want to lookup all the commands I need to use. I was thinking of wrapping (some of) these commands in babashka tasks. I would like to do things like bb help -> list everything, bb help <cmd> -> list all sub commands. A bit like the cli tool of AWS

borkdude08:04:26

We've already moved away from boot to deps.edn at work now. The benefit of lein is that it has a standard way of doing things indeed. bb.edn is more like Make / npm scripts: it contains project-specific things. But perhaps some standards will evolve...

borkdude09:04:57

Btw maybe there is a nice workaround for file / task / subcommand conflicts. As of now, each bb subcommand is also accessible by --uberjar instead of uberjar. A local file is also accessible by bb ./uberjar And a task is also accessible by bb run uberjar. So if we maintain the --uberjar CLI option in the future, there's always a way out...

borkdude16:04:13

This might also conflict with passing JSON or EDN values on the command line. Not sure how well lein do copes with that.

Karol Wójcik16:04:29

Does it look cool?

👍 4
Karol Wójcik16:04:06

There is one thing I would love to see. Ordering of bb tasks

Karol Wójcik16:04:43

:tasks {:requires                ([holy-lambda.tasks])
         :log-level               :info

         docker:build:ee          holy-lambda.tasks/docker-build-ee
         bucket:create            holy-lambda.tasks/create-bucket
         bucket:remove            holy-lambda.tasks/remove-bucket
         stack:destroy            holy-lambda.tasks/stack-destroy
         stack:deploy             holy-lambda.tasks/stack-deploy
         version                  holy-lambda.tasks/version
         doctor                   holy-lambda.tasks/doctor
         purge                    holy-lambda.tasks/purge
         logs                     holy-lambda.tasks/logs}}
Here is the preferred order

Karol Wójcik16:04:37

Maybe it could be taken from the metadata of the function

borkdude16:04:17

Hmm, we could support sorting if we supported the notation:

{:tasks
  [[:requires ...]
   [:log-level ...]
   [docker-build foobar]]
}

👍 2
borkdude16:04:44

but because it's a map, we cannot preserve the order

👍 2
Karol Wójcik16:04:11

Are you interested in preserving the order or you just don't care?

borkdude16:04:53

I think it would be nice, we could support a vector notation like this:

{:tasks [
  :requires ...
  :log-level ...
   docker-build foobar
]
}
so same as a map but within a vector to indicate the sort order

👍 2
Karol Wójcik16:04:25

Yep. Btw how can I overwrite babashka.tasks/log-info? I'm not familiar with sci.

borkdude16:04:06

Same as in clojure, just use def. The name is -log-info (indicating that it is an impl detail)

borkdude16:04:41

oh you want to intern it into the babashka.tasks namespace

Karol Wójcik16:04:13

Yep. tried to use alter-var-root but it doesn't work.

borkdude16:04:35

show me what you did then

Karol Wójcik16:04:08

(alter-var-root #'babashka.tasks/-log-info
                  (fn [& strs]
                    (println "xx" strs)))

Karol Wójcik16:04:31

~/Workspace/Work/holy-lambda/modules/holy-lambda-babashka-tasks master*
❯ bb doctor
xx (#object[babashka.impl.tasks$log_info 0x38037185 babashka.impl.tasks$log_info@38037185])
----- Error --------------------------------------------------------------------
Type:     java.lang.NullPointerException

borkdude16:04:09

you need to pass a function which returns a function to alter-var-root, same as in clojure ;)

borkdude16:04:50

@karol.wojcik Use constantly around the function, for example

borkdude16:04:54

afk for dinner now

Karol Wójcik16:04:04

Ahh you're right. Thanks!

pithyless18:04:21

Any chance we could add https://github.com/clj-commons/ordered to babashka? Then perhaps we could use the #ordered/map literal when defining tasks in bb.edn ? Having the ordered literal around could be useful in a couple of places when working with bb in general :]

👍 2
borkdude18:04:02

@pithyless afaik that is already available:

bb -e "(use 'flatland.ordered.map)"
(but not the .set namespace I noticed)

borkdude18:04:33

but you're right, that could be a nice solution

borkdude18:04:47

@dharrigan btw your suggestion of having the argument order (copy-into dir f1) has been suggested by other people as well + the suggestion of making it varargs:

(copy-into dir f1 f2 f3)
If this comes up more often, I'll re-consider it. For now copy just accepts the unix semantics as well

👍 2
borkdude18:04:51

(one of the difficulties in that case is: where to put the options)

pithyless18:04:37

oh, I didn't realize the literal syntax is #ordered/map ([:a 1] [:d 4] [:b 7]) ; I was hoping it still looked mostly like a map

borkdude18:04:07

ah crap, yeah, I guess it can't be a map literal because by the time the lib sees it the order has already been messed up

☝️ 2
borkdude18:04:38

so we might as well go with [:a 1 :b 2 :c 3] as the alternative notation?

borkdude18:04:16

and then use apply hash-map for map semantics, but use the order for printing

pithyless18:04:22

uhh, I'd vote for [[:a 1] [:b 2]] (I guess that's the compromise malli ordered-maps also settled on)

👍 2
borkdude18:04:09

{:tasks [[:requires ([babashka.fs :as fs])]
         [ls (shell "ls")]
         [clear {:doc "blabla"
                 :task (shell "rm -rf targt")}]]}
vs
{:tasks [:requires ([babashka.fs :as fs])
         ls (shell "ls")
         clear {:doc "blabla"
                :task (shell "rm -rf targt")}]}

borkdude18:04:01

yeah ok, the first one might be more "idiomatic", although (require '[foo :as x :refer [1 2 3]) is similar to the first one

borkdude18:04:22

as in, it squashes basically a map in a vector

pithyless18:04:02

I'm still wondering if it's worth it; the map syntax is nice and perhaps the order of task printing is a separate concern? i.e. it can be a separate thing in bb.edn (either a vector or even a function that sorts -- or even groups -- tasks by some logic)?

borkdude18:04:08

alternatively, we could support something like a :print-order function or literal, but this might be duplication of names

borkdude18:04:05

yeah, you are right, I was also thinking a function which can sort the tasks topologically, so you get the "main" tasks first for example

👍 2
pithyless18:04:51

having a :print-order function or literal sounds like the better approach; I'm wondering if the default implementation couldn't even be something crazy like slurping in bb.edn and generating the list of tasks by parsing the edn "by hand"?

Karol Wójcik18:04:54

I would like to hide as much of tasks implementation from the user as it is possible. If I could hook to some kind of babashka.tasks/print-order function from tasks impl I would be most satisfied.

borkdude18:04:55

I think babashka could cheat

borkdude18:04:17

by determining the position of the first key in the map and then read the tokens one by one until it encounters }

👍 4
Karol Wójcik18:04:37

Seems like a hack, but I like it

pithyless18:04:59

that's kind of what I had in mind

borkdude18:04:39

it's actually pretty easy since edamame can read the begin and end position of the tasks map, then just strip all of that stuff away, read it as a vector (by wrapping) and done

pithyless18:04:55

I forgot you had the power of edamame to wield freely

borkdude18:04:10

I don't use it currently to read the edn (I want to make sure the .edn is compatible with the regular EDN reader) but for the printing of the tasks, I will 👿

borkdude18:04:49

Maybe two blank lines means one empty line will be printed in the output as well, for some grouping?

borkdude18:04:09

Heck, I could even use rewrite-clj for this since it’s bundled with bb now

borkdude21:04:15

The printing of namespaces conform their occurrence should work now on master (binary available in #babashka-circleci-builds)

Karol Wójcik06:04:02

Thank you so much! Will give it a spin.