Fork me on GitHub
#babashka
<
2021-05-27
>
arthur06:05:04

I'm looking for a task scheduler / cron-like-thing that's happy in babashka, any suggestions?

borkdude06:05:08

Probably just use cron?

borkdude07:05:27

What's the use case? How would you do this in normal Clojure?

Steven Katz14:05:36

Hi, I would like to confirm my understanding of how to run the aws cli from a babashka file. I would use “https://book.babashka.org/#_clojure_java_shell” as described in the book? Is this the prefered way?

borkdude14:05:39

@steven.katz it depends. if you would like to capture output, or only like to cause a side effect and don't want to use the output, but see the output as you execute it

Steven Katz14:05:36

hmm, not sure, right now I guess I don’t really need to capture the output, just look at it as it executes

borkdude15:05:51

In that case babashka.process offers a better alternative:

(require '[babashka.process :refer [process]])
@(process ["aws-cli" "your-argument1" "your-argument2"] {:inherit true})

borkdude15:05:04

The shell function from babashka tasks can also be used, it basically the same with a slightly different API, but with defaults for running it as a task:

(require '[babashka.tasks :refer [shell]])
(shell "aws-cli your-arg1 your-arg2")

Steven Katz15:05:25

Great, thanks!

skuro15:05:48

I'm trying to use babashka in my CI environment, currently trying to just install the binary and make it work on a "vanilla" alpine image. here's my trouble:

bash-5.0# wget -O /tmp/bb-install 
Connecting to  (185.199.111.133:443)
bb-install           100% |***********************************************|  3504  0:00:00 ETA
bash-5.0# bash /tmp/bb-install
Downloading  to /tmp
Successfully installed bb in /usr/local/bin
bash-5.0# which bb
/usr/local/bin/bb
bash-5.0# bb
bash: /usr/local/bin/bb: No such file or directory

skuro15:05:55

not sure what's going wrong tbh

borkdude15:05:28

@skuro for Alpine you need the --static argument of the installer script

borkdude15:05:36

perhaps we can switch to this as the default on linux at one point, @rahul080327 @thiagokokada

lispyclouds15:05:46

At this point I'm much more confident in making it static everywhere should be safe

lispyclouds15:05:57

Maybe we can do an A/B test somehow 😛

borkdude16:05:37

hehe rand-int

kokada17:05:55

Maybe, but something to keep in mind is that maybe musl may have some difficults doing requests in some configurations (thanks to the fact that it embeds its own DNS resolver instead of using NSS)

kokada17:05:48

The cases where (and if) this will cause issues should be rare, so IMO it is safe to switch, but I think it is important for people to understand the implications of static binaries

borkdude17:05:49

ok, we'll just leave it for now

skuro15:05:45

bash-5.0# /tmp/bb-install --static
Downloading  to /tmp
mkdir: unrecognized option: static

borkdude15:05:24

hmm, a bug in the script?

skuro15:05:33

install_dir seems to become --static

borkdude15:05:14

ah here's the bug:

if [[ $# -eq 1 ]]; then
   install_dir=${1:-}
fi

borkdude15:05:38

we can remove that part I guess, just use explicit --dir ...

borkdude15:05:48

lemme do that

borkdude15:05:19

fixed, but github caches it for a while, so try again in some minutes

borkdude15:05:29

@skuro the issue should be fixed now I think

skuro06:05:58

worked like a charm, thanks!

skuro16:05:20

Thanks I'll try again later

borkdude16:05:44

@skuro there is also a bb alpine image btw:

babashka/babashka:0.4.3-alpine

skuro17:05:53

Oh ok, missed that completely, i blindly tried borkdude/babashka:alpine

borkdude17:05:56

I deleted borkdude/babashka and borkdude/clj-kondo on Dockerhub now. Let's see how many questions I will get and how many CIs will break... tl;dr: use babashka/babashka and cljkondo/clj-kondo

kokada17:05:43

Let me check nixpkgs. I think I migrated both, but not sure

borkdude18:05:31

Maybe I should have been more careful? On the other hand, it's been 6 months now

kokada18:05:12

Well, at least for nixpkgs seems ok

kokada18:05:43

> Maybe I should have been more careful? On the other hand, it's been 6 months now If you really want people to fix their CIs breaking it is the only way 🤷 Those group renamings on GitHub are really bad on UX side (since it only shows when pulling/pushing)

kokada18:05:28

Ah, just saw that you're referring to the Dockerhub (I missed it and thought it was the GitHub repo facepalm )

BuddhiLW17:05:51

Hello! I would like to load a babashka file, which only contains "helper-functions" for what I would like to script. In the end, I'm loading this file on a closh shell, and running a pipe. 1. First problem I have, when I run bb -f partitioning.clj It returns, as defined, only the last-defined function in the file, #'devibrary.partitioning/create-populate-dir The integral code can be found in, https://github.com/BuddhiLW/CloshBashika/blob/main/devibrary/src/devibrary/partitioning.clj Q: How could I load all the functions properly?

borkdude17:05:12

What you see is just the printed result.

borkdude17:05:25

Babashka prints the last return value

borkdude17:05:02

so the file is loaded.

BuddhiLW18:05:17

Seems like I'm unable to use the functions inside the namespace

borkdude18:05:21

Can someone help Pedro out? I'm afk for an hour

BuddhiLW18:05:41

Also, if I manually load the function on the closh environment, it works fine.

BuddhiLW18:05:15

I thank you for your time, @U04V15CAJ. I'm not really in a rush. If anyone knows what is happening would be great. But, for now, loading things in the shell has been working fine, for my scope/work.

jeroenvandijk18:05:48

@U022GV4MVTQ I’m guessing the problem is that you are not calling a “main” function. Like @U04V15CAJ said “Babashka prints the last return value”. So the last value is a function

jeroenvandijk18:05:31

If you put (your-function) and the end of the file it should run

jeroenvandijk18:05:01

A common pattern is

#?(:bb (apply your-main-function *command-line-args*))
This will run only in babashka. If you want to only run when it is invoked from the command line (and not when requiring it from another ns) you can add an additional check

jeroenvandijk18:05:52

@U022GV4MVTQ I’m rereading the thread and I’m not sure if I actually understand your problem. Is it your goal to invoke this function from the command line? https://github.com/BuddhiLW/CloshBashika/blob/main/devibrary/src/devibrary/partitioning.clj#L68

BuddhiLW19:05:21

Yes, actually, I wish to use it as a command in a pipe. In the repository I prepared a demonstration. Let's say you have shifted-ordered files. So, text5.txt, text6.txt, ..., text10.txt. You can rearrange them removing this shif-value of 5. And, you can change their regex "text", as well as the extension ".txt". All this with this one-liner which is a defcmd-macro (from closh). Then, let's say you want to partition these files in sub-folders with determined regex base-name. You can then use another simple pipe, twoliners. (to reproduce the results, you can go to ./sh-scripting/, run sh lupo-setup.sh , then, cd ../test/ , and run the following:

$rename-test "test" ".txt" 5 "demo" ".org"

$ls |> (map #(file-ids % "demo" ".org")) |> #(populate-key-map (create-range [0 3 7 10]) % 1) |> #(create-populate-dir % "dir-demo" "demo" ".org")
This rename files from text5.txt, ... , text15.txt to http://demo0.org, ... , http://demo10.org. Then, move these demo[0,3].org to dir-demo1; [4,7] to dir-demo2; and, [8,10] to dir-demo3.

BuddhiLW20:05:15

As it is, I need to load all the functions in closh repl, to use these one-liners - which defeats the purpose a bit. I could make some tweaks and write it in "pure babashka" syntax, and leave closh aside. Currently, I'm maintaining both versions in parallel. I guess I could, then, write the commands to execute inside the script - as I noticed that was the purpose, as you spoke. But, going further in this problematic, do you see an easy way I could load these namespaces and use in a pipe | bb -e (ns-function) fashion?

jeroenvandijk20:05:10

It’s hard to understand the exact problem you are trying to solve. Can you try to isolate the issue? Is this specific to closh or is this also happening in the bash shell?

jeroenvandijk20:05:23

For piping process it’s good to have a look here https://book.babashka.org/#_input_and_output_flags

BuddhiLW20:05:53

I eval a babashka file. I want to call, now, the functions which were loaded in this eval. I can't. That's basically the isolated issue there.

BuddhiLW20:05:10

(irrespective of what shell I'm in)

jeroenvandijk20:05:04

ok so first thing is that bb loses it’s state between 2 calls. So the first one is unrelated to the second one

jeroenvandijk20:05:24

Let me see if I can give you a working version

jeroenvandijk20:05:23

You are missing a bb.edn file in the devibrary dir

jeroenvandijk20:05:51

bb -m devibrary.core/foo 2
2 Hello, World!

jeroenvandijk20:05:17

That should work when you add a bb.edn file like this one

{:paths ["src"]}

jeroenvandijk20:05:55

The other function can be called like bb -m devibrary.partitioning/create-populate-dir 1 2 3 4

jeroenvandijk20:05:02

I have it working here

jeroenvandijk20:05:13

➜  CloshBashika git:(main) ✗ git status
On branch main
Your branch is up to date with 'origin/main'.

Untracked files:
  (use "git add <file>..." to include in what will be committed)
	devibrary/bb.edn

nothing added to commit but untracked files present (use "git add" to track)

jeroenvandijk20:05:25

@U022GV4MVTQ Hope this is what you needed

BuddhiLW20:05:35

Yes, it is! I will dive more deeply in the documentation, also. Because, certainty I will be using it in daily use. Thank you @U0FT7SRLP and @U04V15CAJ

jeroenvandijk20:05:37

np @U022GV4MVTQ. Hope you are having fun 🙂 Also cool to see you are using Closh

🎉 3
kokada19:05:49

Is there some way in Babashka that I can set the directory where a script/project is running, without needing to change the directory :thinking_face: ? Context in 🧵 .

kokada19:05:57

I have a Babashka project (has a bb.edn and src, test directories) that needs to be accessed from anywhere (so I added a bin directory from this project to my PATH)

lispyclouds19:05:34

Maybe this helps: https://github.com/bob-cd/bob/blob/main/bb.edn#L30 the way im using tasks in this mono-repo?

kokada19:05:10

Nowadays what I do to load the bb.edn (that includes this project deps, classpath, etc) is to have a hybrid shell script/babashaka script that cd into the script directory and runs the program

kokada19:05:32

@rahul080327 I don't think this will work because of the next reason

kokada19:05:08

The approach above worked fine, but today I discovered that I can't change to the project directory for one reason: I will lose the directory where the user is running the project

kokada19:05:46

So I have another command that tries to download some things on the current directory, but now it is always downloading it to the directory that my project is hosted (since I changed the directory before running the project)

kokada19:05:17

I can't change this command because it is not called directly by my project (it is called indirectly in a fallback flow)

kokada19:05:08

The major issue that I am facing now is that while bb -f /path/to/script.clj works, it "ignores" the bb.edn file

kokada19:05:56

So if we could have something like bb -f /path/to/script.clj --bb-file "/path/to/bb.edn, I think this would solve the problem

kokada19:05:59

(BTW, this works in Python, since a venv can be called anywhere if you, just by calling your script with path/to/venv/bin/python)

kokada19:05:59

> So if we could have something like bb -f /path/to/script.clj --bb-file "/path/to/bb.edn, I think this would solve the problem Actually, this would need to be a --context /path/to/project, since the :path (and probably other things) on the bb.edn file are relative

lispyclouds19:05:56

maybe launch bb with the --classpath set to the place with the src etc and launch it from the place bb.edn is? Im not sure if the bb.edn path could be specified :thinking_face:

kokada19:05:45

I think this could work, however I would need to load my dependencies manually somehow

lispyclouds19:05:54

i think the bb.edn is also picked up from the --classpath afaik

kokada19:05:06

Let me try

lispyclouds19:05:40

could be wrong 😅

kokada20:05:21

Well, I can set my --classpath to path/to/project/src, but this will not load the dependencies

kokada20:05:50

Using --classpath path/to/project will not find my project namespaces

kokada20:05:50

Passing both (`--classpath path/to/project:path/to/project/src`) also doesn't work

lispyclouds20:05:03

how are you invoking bb? providing a particular ns with the classpath?

kokada20:05:49

I actually call the script itself, that loads a namespace from the project, that them loads the -main function 😅

kokada20:05:15

Because I need to do some setup (and I try to avoid as much bash as possible)

kokada20:05:37

But well, I think parsing the :deps from bb.edn should work

lispyclouds20:05:59

yeah i could imagine shoving the entry point into bb.edn and calling the code from that instead of the scripts?

lispyclouds20:05:37

invoke bb with a classpath and a task name maybe

kokada20:05:55

I don't think this would work anyway in my case since to read bb.edn, AFAIK you need to be in the same directory as the bb.edn is created

lispyclouds20:05:09

right, im not sure of this, @U04V15CAJ should help better 🙏:skin-tone-3:

kokada20:05:50

This works 😄

#_(
   export PROJECT_DIR="$(cd "$(dirname "$(dirname "$0")")"; pwd -P)"
  
   exec bb -f "$0" -- "$@"
  )

;; Add debugging helpers and dependencies
(require '[babashka.fs :as fs]
         '[babashka.deps :as deps]
         '[babashka.classpath :as cp]
         '[clojure.edn :as edn])

(defn- file-from-project-dir [filename]
  (str (fs/path (System/getenv "PROJECT_DIR") filename)))

(deps/add-deps (edn/read-string (slurp (file-from-project-dir "bb.edn"))))
(cp/add-classpath (file-from-project-dir "src"))

kokada20:05:38

Kinda hacky? Maybe. But not bad (better than bash)

lispyclouds20:05:15

yeah i guess having the ability to specify the bb.edn path should be the best option

borkdude20:05:15

@thiagokokada I don't know if this solves your problem, since I didn't read the entire thread, but: > bb -f /path/to/script.clj --bb-file "/path/to/bb.edn you can do this:

BABASHKA_EDN=/path/to/bb.edn bb -f path/to/script.clj

borkdude20:05:34

but note that BABASHKA_EDN is not documented and only used for testing right now

lispyclouds20:05:08

we were missing the --bb-edn flag and missed it in the help too

borkdude20:05:40

there is no --bb-edn flag

borkdude20:05:51

I was citing kokada

lispyclouds20:05:58

ah sorry i misread

lispyclouds20:05:22

yeah i see the env var 😅

lispyclouds20:05:26

would it be nice to add a cli opt for the edn file? most tools have it for a non standard path?

borkdude20:05:15

I think there is a reason that deps.edn doesn't support this. Since you create ambiguity about :paths. Are they relative to the bb.edn file or the local dir?

borkdude20:05:39

So I'd like to keep it under the radar for now maybe

kokada20:05:46

Huh... It doesn't seem to work for me It does seems to load the :deps, but not the :paths

lispyclouds20:05:17

we can model it like docker? it uses -f for a non standard path to the Dockerfile and takes a context path

kokada20:05:53

Well, I am quite happy with my solution if this is not possible

borkdude20:05:04

I'm not even sure what the problem is, so right now: no :)

kokada20:05:46

> we can model it like docker? it uses -f for a non standard path to the Dockerfile and takes a context path Yeah, this would be ideal for me

kokada20:05:55

But if it isn't possible it seems fine too

borkdude20:05:25

Perhaps you can lay out the issue in clear words in a Github discussion

borkdude20:05:37

I am kind of overflown with Slack and Twitter threads right now

kokada20:05:37

At least it is possible to manipulate the classpath on those cases

borkdude20:05:44

and I will take a look at this later

lispyclouds20:05:51

I can start it, maybe you can add to it @thiagokokada?

borkdude20:05:17

Note: please use discussions, not issues, so we can have some back and forth first

3
👍 3
lispyclouds20:05:53

Should i add it to the bb tasks thread?

borkdude20:05:03

Please start a new one

borkdude21:05:42

Can you remove the reference to BABASHKA_EDN? It's an internal detail, I think it's better if nobody finds it there and start using it

3
borkdude21:05:40

So the way I'm reading that issue now. My initial response is: just write a bash wrapper script that cds into the right directory and then calls bb

lispyclouds21:05:44

Makes sense. I'm looking at it from the lens of a task runner like Gradle, make etc where we can specify the path to the conf, when not in pwd

arthur19:05:14

sorry if this is a common qustion, i'm new to this is there an existing time expiring cache that is happy in babashka?

(def devices (cache/ttl-cache-factory #{} :ttl 5000))
this seems to be missing the sorted protocol

borkdude21:05:00

@arthur currently we don't have core.cache / core.memoize in babashka. not sure what the vanilla Clojure / Java answer to this should be

borkdude21:05:14

perhaps just an atom with maps with timestamps

Buidler22:05:11

There's a tiny error in the docs here https://babashka.org/fs/babashka.fs.html#var-delete. Think that should be (delete f) instead of dir.