Can we use some kind of middleware/interceptor pattern for tasks (similar to :init)? That way we could globally define and customize things like printing task names, logging responses, etc.

hmm, interesting!

:pre-task (fn [...] ...) :post-task (fn [...] ...) ?

I guess if you want to save some state, you can accumulate it in some atom you define in :init

nice idea

:interceptors [{:name "optional"
                :pre-task (fn [..] ...)
                :post-task (fn [..] ...)}]

^ perhaps something like that, and you can probably ship with a couple default interceptors built-in

so the previous discussions would boil down to:

:interceptors [log-task-names, log-task-outputs]

https://github.com/babashka/babashka/issues/805

Please allow binding interceptors from task script as well.

What do you mean exactly?

I want to add interceptors from script not via bb.edn:

(ns holy-lambda.tasks
  "This namespace contains tasks!"
  (:require
   [clojure.string :as s]
   [clojure.pprint :as pprint]
   [clojure.java.shell :as csh]
   [clojure.edn :as edn]
   [babashka.tasks :as tasks]
   [babashka.deps :as deps]
   [babashka.fs :as fs]
   [babashka.curl :as curl]
   [babashka.process :as p]
   [ :as io]))

(tasks/add-interceptors [{:name "Wrap task"
                          :enter (fn [task-name] (println task-name))
                          :leave (fn [task-name] (println task-name))}])

What is the reason you want this in addition to declaring them in bb.edn?

Hmm, we could add the value of each task execution into the middleware map as it executes

and then each middleware fn gets to see the middleware map

and can add/remove to it

> Hmm, we could add the value of each task execution into the middleware map as it executes hmm, but this is already supported by :depends as well, but perhaps it doesn't hurt to have some more flexibility

baz {:enter (fn [m1] (assoc m :foobar 2)) :task 66 :leave (fn [m2] ...)}

Interceptors is an implementation detail for me. I don't want the user to change them.

What is "the user"

The one who interacts with holy-lambda via executing provided bb tasks.

Is holy lambda generating a template project for the user with a bb.edn in it?

or how does it work?

When I'll be done with tasks I will just reference to holy-lambda-babashka-tasks/bb.edn from template. Running

lein new holy-lambda <some-project>

bb version

ok but note that a user of the template is as much a "power" user of the bb.edn as you are, a template is only a starting point of a project after which you make your own modifications

so whatever changes they want to make to the bb.edn, that should be allowed

Actually that's a good point

and btw, version is also a subcommand of babashka which you are overriding

but that shouldn't be a problem

I will use tasks:version then

Thanks!

Instead of calling it interceptors maybe a more basic "hook" system works better (also in combination with parallel tasks). You can define a global :enter and :leave hook (which gets to mutate the global context map) and you can override those per task. If you want to call the global hook, just call it manually in the override.

This way you don't get any weird API edge cases with interceptor ordering/insertion/appending.

Pretty much the same as (sync) Ring but two hooks, one before and one after

or can we get away with just one?

I think it's more flexible to have one before and one after hook probably

possibly you can also decide to abort/skip the task in the :enter hook based on some condition

yeah, even logic like should a "failing" task continue quietly or Sys/exit could be customized by a :leave hook

Should the :enter hook be executed before or after dependencies... :thinking_face:

I think I'll keep it more basic, in the same fashion :tasks are just expressions:

Re-considering, I think this is all that's needed to support logging of task names:

add *task-name* dynamic var
global :enter (println :> *task-name*)
global :leave (println :< *task-name*)
able to replace :enter and :leave on the task level
This is all that's needed, even to support some kind of middleware pattern, since you can define an atom in :init where :enter and :leave can mutate something to.

https://github.com/babashka/babashka/blob/bbf144fbce66c6986253119eb81392a440cb17c6/src/babashka/impl/cheshire.clj

the answer is no

😅

indeed. I'm not planning to since I'm not certain if cheshire is the right json tool in 2021 now that jsonista and clojure.data.json are also gaining popularity

so I'd like to keep the surface area somewhat minimal so it might be easier to port to some other solution in the future

could you convince me of the usefulness of providing your own object-mapper? examples?

Options when parsing json. Right now I'm getting: ; clojure.lang.ExceptionInfo: [line 1, col 51] Unsupported escape character: \U. This would not be a problem if I configure the underlying object-mapper. e.g with jsonista:

(def object-mapper
  (-> jsonista/keyword-keys-object-mapper
      (.configure (.mappedFeature JsonReadFeature/ALLOW_UNESCAPED_CONTROL_CHARS) true)))

so now, when using jsonista I can do: (jsonista/read-value json-string my-object-mapper)

so basically this blocks me from using babashka from our ci-pipeline, but im going to figure out if I can use jsonista some way.

you can't currently use jsonista in babashka. Is there a way to fix the json first with str/replace?

Gonna see, the json we receive is malformed from a system which we do not control. This option is kind of hacky indeed 🙂, I'll check it out. We still use BB for other parts of our CI and it's a pleasure to work with!

its basically a nested json structure containing a json string as a value of a key which we do not want to parse directly.

Perhaps you can use a tool like jq to fix the JSON (if that tool supports it...)

Can you give a small example of such JSON?

CLOJURE_TOOLS_DIR is the thing! 🙂

What is CLOJURE_TOOLS_CP ?

Before I started reading deps.clj source code I tried my luck with this env var 😄

Have you ever seen babashka load/pod to write to '?' folder?

This environment variable is also documented in the README

no

Ok. It's really weird. Here is repro: https://github.com/FieryCod/holy-lambda/commit/22e18f5f434cfff484db5dd019a541ac04705062 git clone [email protected]:FieryCod/holy-lambda.git && make build-docker && cd modules/holy-lambda-babashka-tasks && bb stack:sync After command finishes you will see '?' folder 😮

could it be related about the order of these commands?

RUN mkdir -p /.babashka
RUN chmod -R 777 /clojure
RUN chmod -R 777 /.m2
RUN chmod -R 777 /.babashka

WORKDIR /project

shouldn't WORKDIR go before mkdir?

I'm just guessing, I don't know your project well enough

Nope. It's something with resolver. I got folder .holy-lambda/.babashka/pods/repository/org.babashka/aws/0.0.5/, but bin is downloaded to '?' ?/.babashka/pods/repository/org.babashka/aws/0.0.5/

$ bb stack:sync
----- Error --------------------------------------------------------------------
Type:     java.lang.Exception
Message:  File does not exist: stack:sync

In which folder are you?

oh right

> After command finishes you will see '?' folder how do I see this folder?

oh yes, I see it

Uff. It's good that I'm not crazy 😄

I don't have time to debug your program, so can you make a small repro?

Sure

@borkdude Here it is: https://github.com/FieryCod/babashka-docker-path-repro

Just run make

Do you have a more minimal one? There is a lot happening in this Dockerfile

Oh I see you have stripped it already

It's stripped.

I can use alpine if you want me to to make it even more minimal

But I guess you have graalvm-ce in cache so build should not take long

I don't use docker locally for graalvm, but I'm going to try. I do see the question mark dir, but how do I know this is not an issue with your docker config vs pods?

😄 I'm already using the same strategy for mountig .aws directory no question mark there

FWIW when I make a .babashka directory inside your repo, then I don't get the question mark dir

Hmm isn't it proof that it's something wrong with babashka resolver then?

I don't say there isn't but right now I'm debugging your docker issues while I have another job to do, so I'd say this is not a clear repro yet

Sure. I fixed the repro. It should be very minimal right now. You can spot '?' in error:

----- Context ------------------------------------------------------------------
 7: 
 8: (def PODS {'org.babashka/aws "0.0.5"})
 9: 
10: (doseq [[s v] PODS]
11:   (pods/load-pod s v))
      ^--- ?/.babashka/pods/repository/org.babashka/aws/0.0.5/manifest.edn (No such file or directory)

----- Locals -------------------------------------------------------------------
seq_2:   ([org.babashka/aws "0.0.5"])
chunk_3: nil
count_4: 0
i_5:     0
vec__9:  [org.babashka/aws "0.0.5"]
s:       org.babashka/aws
v:       "0.0.5"

----- Stack trace --------------------------------------------------------------
user - /usr/bin/download_pods:11:3

@karol.wojcik When I print in the script:

(prn (or
            (System/getenv "XDG_CACHE_HOME")
            (System/getProperty "user.home")))

When you run make you will see "?"

yes. it seems your XDG_CACHE_HOME env variable isn't properly set in the script. so it picks the value of user.home which is a question mark.

so it turns out I have been debugging your Docker issues instead of pods. :/

Hmm.. With this it still outputs question mark:

FROM 
MAINTAINER Karol Wójcik <Karol Wójcik>

WORKDIR /

RUN microdnf install wget

RUN mkdir -p /project
ENV _JAVA_OPTIONS=-Duser.home=/project
ENV XDG_CACHE_HOME=/project
ENV XDG_CONFIG_HOME=/project
ENV CLJ_CACHE=/project
ENV CLOJURE_TOOLS_DIR=/project
ENV HOME=/project
ARG BABASHKA_VERSION=0.3.6

RUN wget -c  -O - | tar -xz
RUN chmod +x bb
RUN mv bb /bin/bb

COPY download_pods .
RUN chmod +x download_pods
RUN mv download_pods /bin/download_pods

I think I have provided you with enough debugging now. I won't look at this anymore unless you can provide a repro without Docker, etc.

😕

I don't want to sound like an asshole, but I what I mean is: this is a contextual problem between your Docker setup and bb. You can do more diagnosis before you poke me about it. E.g. bash into the container and inspect:

bash-4.4$ bb
Babashka v0.3.6 REPL.
Use :repl/quit or :repl/exit to quit the REPL.
Clojure rocks, Bash reaches.

user=> (System/getProperty "user.home")
"?"
user=> (System/getenv "XDG_CACHE_HOME")
nil

My first repro lacked one env variable which is: XDG_DATA_HOME

It sounded like you being asshole, but no worries I deserved it 😄

I should have studied resolver code more. I did not spot XDG_DATA_HOME I'm really sorry!

no worries ;)

I'm deeply sorry. I will spend more time on reading babashka code! 🙂 No more pokes today 🙂 Have a great day!

I don't mind the pokes, I think you're making a very interesting project. It's just that in case the bugs aren't directly related to bb a more clear cut repro could be made. Feel free to poke me anytime after you've done your part of the research, so it will save me time.

and if I think you should make it more clear cut, I will just say so (hopefully without sounding like an asshole :-D)

I also enjoy your feedback on the tasks, so thanks for that

at least if you run as a user, which doesn't have a /etc/passwdentry, then you get ? for (System/getProperty "user.home") (just came to my mind from some previous docker adventure)

Huh. I didn't know that. Thanks @U06QSF3BK

You can force this property in bb by doing bb -Duser.home=/foobar

@karol.wojcik Btw, have you tested if the aws pod works in your docker container / lambda? If it's a musl based container (alpine, busybox) often pods don't work (because they depend on libc++) and have to be compiled statically, so that would be good to test.

aws pod?

org.babashka/aws "0.0.5"

I will not run babashka/pod on my docker image. 😄

but will it work in your target environment?

that is the question right

oh I guess it's the same for bb itself, if the normal bb works then the pod will also work

Yep exactly.

:thumbsup:

https://clojurians.slack.com/archives/CLX41ASCS/p1619463021130800?thread_ts=1619443902.117900&cid=CLX41ASCS This was super helpful. Thank you so much @borkdude!

Can I set a system property for clojure command as well? Don't see anything such option

@karol.wojcik How are you invoking Clojure?

clojure -P

without bb that is, right?

Yep

clojure -J-Duser.home=/foo

Ahh..

Checking

May I set it for deps.clj which babashka uses as well?

> without bb that is, right? > yes ?

so are you or are you not using bb to invoke clojure?

if not, where does deps.clj come in?

Ok. You're right. I think bb tasks needs deps.clj since I'm using a :deps keyword in bb.edn. To conclude I use both.

bb.edn doesn't forward any Java properties to its call to deps.clj

Maybe we should support this, but currently it doesn't

is this about setting the local .m2 folder?

Yes

you can put :mvn/local-repo "/tmp/.m3" in your bb.edn

it's a bit of a workaround probably

It's a little bit complicated, but what I want to achieve is to run clojure -P in docker container and sync it with .holy-lambda directory on host. Now I cannot set user.home for Clojure. I just tried what you've sent and user.home is not set.

With _JAVA_OPTIONS it kinda worked, but it prints a low of "Picked _JAVA_OPTIONS etc.

> but it prints a low of "Picked JAVAOPTIONS etc. I have no idea what this means, sorry =)

what is _JAVA_OPTIONS even?

https://unix.stackexchange.com/questions/495469/how-to-fix-picked-up-java-options-no-options-listed-on-linux It's environment variable to which you can pass some java options.

😄

Hmm. Ok it seems that I messed something with my docker again 😄

clojure -J-Duser.home=/foo -e "(println (System/getProperty \"user.home\"))"

@borkdude You're some sort of wizard. :mvn/local-repo works like a charm! Thank you so much! You rock man!

cool. Perhaps bb should have a separate argument to pass through to deps.clj.

bb --deps-args '-J-Duser.home=/foo'

Yep. It would be useful

It is printing the correct line but it always print the namespace user

Weird, the gist still seems to work for me where it prints the ns bar

I am using the -m flag

Since it is a full Clojure project (with src, etc)

ah!

Maybe because of this :thinking_face:?

(ns user (:require [foo.core])) (apply foo.core/-main *command-line-args*)

I was trying to create a new project to demonstrate, but this got to my attention

(It is an exception btw)

Can you try this instead?

(defmacro log [& msgs]
  (let [m (meta &form)
        file *file*]
    `(binding [*out* *err*] ;; or bind to (io/writer log-file)
       (println (str ~file ":"
                     ~(:line m) ":"
                     ~(:column m))
                ~@msgs))))

pulling *ns* outside of the macro body might also work

Yep, the one with *file* works

BTW, if you want to reproduce it

Run bb -m foo.core

But this alternative with *file* seems even more useful to me

ok, you like file better than ns?

(ns logger)

(defmacro log [& msgs]
  (let [m (meta &form)
        ns *ns*]
    `(binding [*out* *err*] ;; or bind to (io/writer log-file)
       (println (str ~ns ":"
                     ~(:line m) ":"
                     ~(:column m))
                ~@msgs))))

Yeah, now that I think about it maybe ns is better (the printed line is smaller)

Don't know yet, I will do a experiment here

But thanks anyway

ok, adapted the gist

if you mean to add variables when calling shell, you can use:

(shell {:extra-env {"SOME" "VAR"}} "your command")

No, in this case I really need to overwrite the environment variables while the script is running

Like doing a export ENV on shell

For now my workaround is to use the #_(<shell code here>) trick described on Babashka's book (https://book.babashka.org/#_running_a_script) to edit the environment variables, but this is ugly

Specially because I can't define a shell function, since this would cause an error because Clojure will try to interpret anything inside {} as a map (and fail to parse it

BTW, it is also possible to change an environment variable in Python I found it curious that even the Java solutions involved doing some magic since it seems that Java itself doesn't have a method for it

AFAIK you cannot change env vars in a Java runtime

Here is another hack: https://stackoverflow.com/a/64804398/6264