Fork me on GitHub

clojure-lsp is now available in the pod registry:

(ns clojure-lsp
   [babashka.pods :as pods]))

(pods/load-pod 'com.github.clojure-lsp/clojure-lsp "2022.01.03-19.46.10")

(require '[clojure-lsp.api :as api])
;; see 

(api/clean-ns! {}) ;; see settings in .lsp/config.edn

;; @ericdallo: somehow this isn't working, even when there are lint warnings:
(defn foo [unused])
(comment unresolved-symbol)
(api/diagnostics {})
/cc @ericdallo: somehow the diagnostics aren't working, but clean-ns! is!


I'll double check!


@U04V15CAJ I tested with clj-kondo and the diagnostics does seem to work:

(ns clojure-lsp
   [babashka.pods :as pods]))

(pods/load-pod 'com.github.clojure-lsp/clojure-lsp "2022.01.03-19.46.10")

(require '[clojure-lsp.api :as api])

(api/diagnostics {})


LMK if you are doing something different

Dimitar Uzunov12:01:16

hello! how do I do globbing with babashka.tasks/shell? It seems to interpret expressions like dir/* like a single dir

Dimitar Uzunov12:01:37

i.e. (babashka.tasks/shell "ls clj/*")
ls: clj/*: No such file or directory


@dimitar.ouzounoff shell isn't a bash invocation. for globbing you can use (babashka.fs/glob "." "clj/*")

☝️ 1

or if you want to invoke bash, you can do (shell "bash -c ls clj/*") but this isn't recommended as this won't be portable

Dimitar Uzunov12:01:56

seems like I’ll be using the fs library to copy files around instead of shelling out


that's the way to go in bb

Dimitar Uzunov15:01:06

what is the proper way to copy a directory? I looked at copy-tree but that copies a single file it seems


@dimitar.ouzounoff fs/copy-tree is indeed the function you want for coping a directory

Dimitar Uzunov15:01:10

what tripped me up is that it when I invoked it like this:

(fs/copy-tree "scripts/" "bundle/scripts")

Dimitar Uzunov15:01:18

and I expected it to create bundle

Dimitar Uzunov15:01:24

but it doesn’t hmm

Dimitar Uzunov15:01:19

and it gave the no such file exception which I interpreted as it thinking scripts/ is a dir

Dimitar Uzunov15:01:36

its just me being asleep at the wheel I guess


yeah it doesn't do that, you should first do that with fs/create-dirs


which is similar to mkdir -p

Dimitar Uzunov15:01:55

hmm the docs seem to suggest it calls create-dirs for the user


oh does it. let me check...

Dimitar Uzunov15:01:32

copy-tree (copy-tree src dest) (copy-tree src dest {:keys [:replace-existing :copy-attributes :nofollow-links]})

Copies entire file tree from src to dest. Creates dest if needed
using create-dirs, passing it the :posix-file-permissions
option. Supports same options as copy.


you're right. which version of bb are you using?


I think this create-dirs was added later


hmm, should be good probably

Dimitar Uzunov15:01:39

i will try with the latest


also, try without the trailing /

Dimitar Uzunov15:01:14

same without the trailing /

Dimitar Uzunov15:01:45

same with v0.7.3

Dimitar Uzunov15:01:10

----- Stack trace -------------------------------------------------------------- babashka.fs/copy-tree/fn--9542        - <built-in> babashka.fs/walk-file-tree/reify--9491 - <built-in> babashka.fs/walk-file-tree            - <built-in> babashka.fs/copy-tree                 - <built-in>


Let's go in a thread.

Dimitar Uzunov15:01:53

I can log a ticket


Let's first see if this is a bug or not. I don't even understand the problem yet. Can you make a repro with a few files?

mkdir -p foo/bar
touch foo/bar/baz.txt
bb -e '(fs/copy-tree ...)'

Dimitar Uzunov15:01:58

I’m copying a dir inside another dir

Dimitar Uzunov15:01:15

➜  ~ mkdir foo
➜  ~ touch foo/bar1
➜  ~ touch foo/bar2
➜  ~ bb -e '(fs/copy-tree "foo/" "foo2/foo")'
----- Error --------------------------------------------------------------------
Type:     java.nio.file.NoSuchFileException
Message:  /Users/dimitaruzunov/foo2/foo
Location: <expr>:1:1

Dimitar Uzunov15:01:28

this is what I’m trying to do


$ mkdir -p foo/bar
$ touch foo/bar/baz.txt
$ mkdir dest
$ bb -e '(fs/copy-tree "foo/bar" "dest")'
$ ls dest

Dimitar Uzunov15:01:41

I can work around it by creating foo2

Dimitar Uzunov15:01:47

yeah my case is different in that the dest is nested

Dimitar Uzunov15:01:58

this doesn’t really create a nested dir (like mkdir -p would), rather it walks a nested dir into another dir


yeah, it copies whatever is inside the source dir into the destination dir

Dimitar Uzunov15:01:30

should it be able to create foo2?


let me check


do you have a repro?


just make a repro and I'll try it locally

Dimitar Uzunov15:01:38

Sorry I showed something else earlier and deleted it

Dimitar Uzunov15:01:44

this is the repro


ah ok I'll try

Dimitar Uzunov15:01:31

brb in 20 minutes going out to pick up the kid


ok, you're right, it does not create the nested dir, although the docstring says so. weird.


feel free to create an issue with the above repro. workaround for now is to make it yourself

👍 1

Thanks! If you want to do a PR + test, this is ok too, but also ok if you don't

👍 1
Dimitar Uzunov08:01:15

Hello @U04V15CAJ so we checkout the source with @U0509NKGK and it looks like a call to create-dirs like this solves the issue:

(defn copy-tree
  "Copies entire file tree from src to dest. Creates dest if needed
  using create-dirs, passing it the :posix-file-permissions
  option. Supports same options as copy."
  ([src dest] (copy-tree src dest nil))
  ([src dest {:keys [:replace-existing
   (create-dirs dest) ;;; <<<<<<<<<<<<<<<<<<<<<< Only change here 
   (let [copy-options (->copy-opts replace-existing copy-attributes false nofollow-links)
         link-options (->link-opts nofollow-links)
         from (real-path src {:nofollow-links nofollow-links})
         ;; using canonicalize here because real-path requires the path to exist
         to (canonicalize dest {:nofollow-links nofollow-links})]
     (walk-file-tree from {:pre-visit-dir (fn [dir _attrs]
                                            (let [rel (relativize from dir)
                                                  to-dir (path to rel)]
                                              (when-not (Files/exists to-dir link-options)
                                                (Files/copy ^Path dir to-dir
                           :visit-file (fn [from-path _attrs]
                                         (let [rel (relativize from from-path)
                                               to-file (path to rel)]
                                           (Files/copy ^Path from-path to-file

Dimitar Uzunov08:01:50

is the repro in a test enough or should I think of other cases as well?


Yes, that is sufficient. The repro then has to be turned into a test

Dimitar Uzunov08:01:16

cool will send a PR soon


Thanks for publishing the Docker images for babashka - I'm finding them super useful already. To take a closer look at its dependencies and their vulnerabilities, I added it to atomist's database (, but we can only see those dependencies brought in by the base image because first babashka is uberjared and then made into a graal native-image, which squishes things even more. Scanners such as grype ( and trivvy obviously don't see anything. I've been playing around with a lein plugin that generates what I'm calling a meta-bom-jar, that basically just contains the group, artifact and version of all a project's dependencies ( If we included this in the babashka Docker images, we could give vulnerability scanners visibility into the bill of materials in use by uberjars, graal native images and so on. I tried on a fork and it seems to work (in that grype can now see all the packages in the babashka images). There are probably other approaches to improving traceability, such as adding a bill-of-materials to an image label, adding metadata to native-image (and adding support to grype), but this seemed more immediately tractable in the short term. FWIW - grype didn't find any additional vulnerabilities over those introduced by the base image - mostly likely because babashka is keeping pretty up to date with regard to its dependencies! 🙂 What do you think?


Any ideas on this one @U7ERLH6JX?


I'll try to have a think about it by today hopefully, quite caught up in post holidays workload 😕 but looks like a worthy thing to consider


Thanks. Yeah, I know how it is! The other thing I'm thinking about a lot is repeatable image builds. With git/maven style deps, we've got used to pinning to specific versions of things, and explicitly updating things in project.clj/deps.edn or whatever. In docker land, we (the industry) have been using a SNAPSHOT style uptake of the base image FROM ubuntu:latest and to install packages like apt-get update && apt install curl . We tend to be much more rigorous when it comes to installing the things we control or are more tightly coupled to (e.g. Linting tools are beginning to pick up on these things, and suggesting installing specific versions of packages and pinning to specific base image digests - unfortunately there aren't many tools (such as lein ancient or whatever) to make this practical in most cases. 😕


Hey, so just to understand, the effect of running meta-bom is to have another jar in the docker image we ship right? How exactly do the scanners work? If they see a jar they can look into it is it?


Yeah for the second case unless we have a apt repo for graalvm versions its quite difficult to install in a specific way. the only other system dependency we have is curl which we pretty much install latest


looking at the vulnerabilities reported by atomist, should we update the base packages like glibc etc? run apt update && apt upgrade -y for every docker image build @U04V15CAJ?


I would be fine with that :)


> Hey, so just to understand, the effect of running meta-bom is to have another jar in the docker image we ship right? How exactly do the scanners work? If they see a jar they can look into it is it? That's right. In an ideal world, all our build/packaging tools would output standard bills of materials (cylconedx/spdx etc), and we'd put that in package metadata somehow (labels for Docker images, META-INF for jars etc). The scanners currently work by interrogating the OS package databases e.g. for apt and by looking inside jar's for pom.xml/ to find packages and their dependencies. The details vary by scanning tool and package index type of course.


Going off topic a little here, but the other related thing here is traceability. I think for babashka the git tags line up well with the docker tags, but it's not really possible to look at the docker image on its own and figure out the git repo, commit, Dockerfile and so on. There's a spec for image labels that folks are adopting these days. I think GitHub actions has ways of adding this automatically during the build, as well as signing and so on. To me it feels like we (our industry) are moving more towards a delivery model where the supply chain and attestation are more rigorous and provable (like if you're building an iOS app). It feels super weird that we have artifact signing, semantic versioning, commit signing, mature dependency descriptors (pom.xml/deps.edn etc) and so on, but when we repackage and/or deploy our apps downstream , we throw it all away and cross our fingers 😄


@U06NZ3HFX adding the meta bom jar looks good for me from the ease perspective! Would you raise a PR from your fork? Also we need to add the CI step to call the lein command. I can have a look at it then! Also we could add a apt update && apt upgrade -y in the final docker image when adding curl to hopefully take care of the OS deps


Also if you want you can add in the commit sha attribute to the docker image too! 😄


As for repeatable/reproducible builds i think one of the bigger blockers apart from the apt installs is the native image spit out by Graal, not sure if that ones reproducable? :thinking_face:


Yeah - even building class files from clojure isn't repeatable, let alone jars or native images. I've had a JIRA/patch open to fix that for a decade now 🙂


I'll try to make a PR to bring some of this stuff together! Thanks.

🙏 1

It looks like there are different solutions for different CI's to populate the opencontainers labels, but I'm a little confused about the multi-ci setup. It looks like GitHub actions is used for master stuff, but circle-ci for other stuff, like PR's? What's the plan here? Maybe we should start with the meta-bom 🙂


@U06NZ3HFX github actions isn't used, it's just a backup for when circleci doesn't work


redundant CI


so just look at CircleCI, you can ignore actions


oh good - that helps. thanks.


Thanks a lot for this @U06NZ3HFX!


Thanks you both!


If we would ever migrate from lein to deps for the build, is this plugin also available?


taking a look at the build failure


@U04V15CAJ is it some weird bash string expansion im failing to see?


looked okay by the looks of it :thinking_face:


--label defintely is a valid flag


of build. should be docker buildx build --label ...


its taking --label "org.opencontainers.image.description as a flag i think


why are these things escaped?

\"org.opencontainers.image.description=Native, fast starting Clojure interpreter for scripting\" \


oh because it's within a string, I see


perhaps it's better to use a bash array here


and then splice the array?


would be good to try this locally first


only if a thing was there to allow me to write clojure instead of bash for scripts

🤪 1

we have to have one place to remind ourselves why babashka exists


which is the babashka build ;)


well lesser painful reminders are nicer


well at some point in the build we can assume that the binary babashka is around and then use that to bootstrap the rest of the build, I would be ok with that I guess


this seems to work:


label_args=("--label" "org.opencontainers.image.description=Native, fast starting Clojure interpreter for scripting"
            "--label" "org.opencontainers.image.title=Babashka"
            "--label" "org.opencontainers.image.created=`date -Iseconds`"
            "--label" "org.opencontainers.image.url=${CIRCLE_REPOSITORY_URL}"
            "--label" "org.opencontainers.image.documentation=${CIRCLE_REPOSITORY_URL}"
            "--label" "org.opencontainers.image.source=${CIRCLE_REPOSITORY_URL}"
            "--label" "org.opencontainers.image.revision=${CIRCLE_SHA1}"
            "--label" "${CIRCLE_TAG:${CIRCLE_BRANCH}}"
            "--label" "org.opencontainers.image.version=${image_tag}")

echo "${label_args[@]}"
shall i commit on master?


i think it should also be ${CIRCLE_TAG}:${CIRCLE_BRANCH}" not ${CIRCLE_TAG:${CIRCLE_BRANCH}}" right?


yeah otherwise its a weird expansion. making the change. should i directly push?


Fine but it would be faster if you can test that Docker stuff locally first on a dummy image


right, dont have the buildx setup here yet but giving it a quick try


ok, then push to master ;)


well almost set it up now, more bash weirdness


seems to start the build, pushing 🤞:skin-tone-3:


cool! maybe also verify if the docker images contains all the good stuff now?


well we missed out the important bit, the copying of the jar was in Dockerfile and not and Dockerfile.alpine which actually pushes the images to hub 😛 we have the labels though!


probably need to change the release script


i can try to give it a shot sometime tomorrow hopefully, please do have a go at it if any one you have some time too! 😄 im thinking of copying the metabom jar along with the binary tar in /tmp/releases in the release step and then copy it into the docker image in the docker script by making sure its in the context


also better approaches welcome!


Is it better that we revert the commits for now?


I'd rather not have unfinished stuff on master


the labels part is complete though, the docker things could be reverted


ok, please do


can revisit when either of you has the time properly


with a new PR


yep, this needs some thought on the CI


@U06NZ3HFX if youre able to look at this again, have a look at ./circleci/script/release, we copy the tar.gz of the binary into /tmp/releases which is then persisted in the circleci workspace and is picked up in the docker step. we need to copy the metabom jar too in the same way for and Dockerfile.alpine to pick up


oh dear - sorry, was afk for a bit. I can have a look at those change for sure. What's the plain Dockerfile for then?


@U06NZ3HFX Dockerfile is more for people who want to build it locally

👍 1

we decided to let it sit there and not break changes to it and make for our own ci


it might still be good to have the bom stuff in there too, but ultimately it should also go into the .ci and .alpine one


Got it. FWIW - the weird variable expansion was a copy/paste from the circle-ci docker orb: - guess they have the same bug - not even sure that's valid shell script is it? 🙂