Fork me on GitHub
#babashka
<
2022-01-05
>
borkdude10:01:41

clojure-lsp is now available in the pod registry:

(ns clojure-lsp
  (:require
   [babashka.pods :as pods]))

(pods/load-pod 'com.github.clojure-lsp/clojure-lsp "2022.01.03-19.46.10")

(require '[clojure-lsp.api :as api])
;; see 

(api/clean-ns! {}) ;; see settings in .lsp/config.edn

;; @ericdallo: somehow this isn't working, even when there are lint warnings:
(defn foo [unused])
(comment unresolved-symbol)
(api/diagnostics {})
/cc @ericdallo: somehow the diagnostics aren't working, but clean-ns! is!

ericdallo12:01:06

I'll double check!

ericdallo13:01:43

@U04V15CAJ I tested with clj-kondo and the diagnostics does seem to work:

(ns clojure-lsp
  (:require
   [babashka.pods :as pods]))

(pods/load-pod 'com.github.clojure-lsp/clojure-lsp "2022.01.03-19.46.10")

(require '[clojure-lsp.api :as api])

(api/diagnostics {})

ericdallo13:01:59

LMK if you are doing something different

Dimitar Uzunov12:01:16

hello! how do I do globbing with babashka.tasks/shell? It seems to interpret expressions like dir/* like a single dir

Dimitar Uzunov12:01:37

i.e. (babashka.tasks/shell "ls clj/*")
ls: clj/*: No such file or directory

borkdude12:01:53

@dimitar.ouzounoff shell isn't a bash invocation. for globbing you can use (babashka.fs/glob "." "clj/*")

☝️ 1
borkdude12:01:19

or if you want to invoke bash, you can do (shell "bash -c ls clj/*") but this isn't recommended as this won't be portable

Dimitar Uzunov12:01:56

seems like I’ll be using the fs library to copy files around instead of shelling out

borkdude12:01:08

that's the way to go in bb

Dimitar Uzunov15:01:06

what is the proper way to copy a directory? I looked at copy-tree but that copies a single file it seems

borkdude15:01:28

@dimitar.ouzounoff fs/copy-tree is indeed the function you want for coping a directory

Dimitar Uzunov15:01:10

what tripped me up is that it when I invoked it like this:

(fs/copy-tree "scripts/" "bundle/scripts")

Dimitar Uzunov15:01:18

and I expected it to create bundle

Dimitar Uzunov15:01:24

but it doesn’t hmm

Dimitar Uzunov15:01:19

and it gave the no such file exception which I interpreted as it thinking scripts/ is a dir

Dimitar Uzunov15:01:36

its just me being asleep at the wheel I guess

borkdude15:01:17

yeah it doesn't do that, you should first do that with fs/create-dirs

borkdude15:01:36

which is similar to mkdir -p

Dimitar Uzunov15:01:55

hmm the docs seem to suggest it calls create-dirs for the user

borkdude15:01:29

oh does it. let me check...

Dimitar Uzunov15:01:32

copy-tree (copy-tree src dest) (copy-tree src dest {:keys [:replace-existing :copy-attributes :nofollow-links]})

Copies entire file tree from src to dest. Creates dest if needed
using create-dirs, passing it the :posix-file-permissions
option. Supports same options as copy.

borkdude15:01:01

you're right. which version of bb are you using?

borkdude15:01:34

I think this create-dirs was added later

borkdude15:01:45

hmm, should be good probably

Dimitar Uzunov15:01:39

i will try with the latest

borkdude15:01:53

also, try without the trailing /

Dimitar Uzunov15:01:14

same without the trailing /

Dimitar Uzunov15:01:45

same with v0.7.3

Dimitar Uzunov15:01:10

----- Stack trace -------------------------------------------------------------- babashka.fs/copy-tree/fn--9542        - <built-in> babashka.fs/walk-file-tree/reify--9491 - <built-in> babashka.fs/walk-file-tree            - <built-in> babashka.fs/copy-tree                 - <built-in>

borkdude15:01:29

Let's go in a thread.

Dimitar Uzunov15:01:53

I can log a ticket

borkdude15:01:29

Let's first see if this is a bug or not. I don't even understand the problem yet. Can you make a repro with a few files?

mkdir -p foo/bar
touch foo/bar/baz.txt
bb -e '(fs/copy-tree ...)'

Dimitar Uzunov15:01:58

I’m copying a dir inside another dir

Dimitar Uzunov15:01:15

➜  ~ mkdir foo
➜  ~ touch foo/bar1
➜  ~ touch foo/bar2
➜  ~ bb -e '(fs/copy-tree "foo/" "foo2/foo")'
----- Error --------------------------------------------------------------------
Type:     java.nio.file.NoSuchFileException
Message:  /Users/dimitaruzunov/foo2/foo
Location: <expr>:1:1

Dimitar Uzunov15:01:28

this is what I’m trying to do

borkdude15:01:40

$ mkdir -p foo/bar
$ touch foo/bar/baz.txt
$ mkdir dest
$ bb -e '(fs/copy-tree "foo/bar" "dest")'
$ ls dest
baz.txt

Dimitar Uzunov15:01:41

I can work around it by creating foo2

Dimitar Uzunov15:01:47

yeah my case is different in that the dest is nested

Dimitar Uzunov15:01:58

this doesn’t really create a nested dir (like mkdir -p would), rather it walks a nested dir into another dir

borkdude15:01:26

yeah, it copies whatever is inside the source dir into the destination dir

Dimitar Uzunov15:01:30

should it be able to create foo2?

borkdude15:01:39

let me check

borkdude15:01:44

do you have a repro?

borkdude15:01:53

just make a repro and I'll try it locally

Dimitar Uzunov15:01:38

Sorry I showed something else earlier and deleted it

Dimitar Uzunov15:01:44

this is the repro

borkdude15:01:52

ah ok I'll try

Dimitar Uzunov15:01:31

brb in 20 minutes going out to pick up the kid

borkdude15:01:00

ok, you're right, it does not create the nested dir, although the docstring says so. weird.

borkdude16:01:51

feel free to create an issue with the above repro. workaround for now is to make it yourself

👍 1
borkdude17:01:51

Thanks! If you want to do a PR + test, this is ok too, but also ok if you don't

👍 1
Dimitar Uzunov08:01:15

Hello @U04V15CAJ so we checkout the source with @U0509NKGK and it looks like a call to create-dirs like this solves the issue:

(defn copy-tree
  "Copies entire file tree from src to dest. Creates dest if needed
  using create-dirs, passing it the :posix-file-permissions
  option. Supports same options as copy."
  ([src dest] (copy-tree src dest nil))
  ([src dest {:keys [:replace-existing
                     :copy-attributes
                     :nofollow-links]}]
   (create-dirs dest) ;;; <<<<<<<<<<<<<<<<<<<<<< Only change here 
   (let [copy-options (->copy-opts replace-existing copy-attributes false nofollow-links)
         link-options (->link-opts nofollow-links)
         from (real-path src {:nofollow-links nofollow-links})
         ;; using canonicalize here because real-path requires the path to exist
         to (canonicalize dest {:nofollow-links nofollow-links})]
     (walk-file-tree from {:pre-visit-dir (fn [dir _attrs]
                                            (let [rel (relativize from dir)
                                                  to-dir (path to rel)]
                                              (when-not (Files/exists to-dir link-options)
                                                (Files/copy ^Path dir to-dir
                                                            ^"[Ljava.nio.file.CopyOption;"
                                                            copy-options)))
                                            :continue)
                           :visit-file (fn [from-path _attrs]
                                         (let [rel (relativize from from-path)
                                               to-file (path to rel)]
                                           (Files/copy ^Path from-path to-file
                                                       ^"[Ljava.nio.file.CopyOption;"
                                                       copy-options)
                                           :continue)
                                         :continue)}))))

Dimitar Uzunov08:01:50

is the repro in a test enough or should I think of other cases as well?

borkdude08:01:47

Yes, that is sufficient. The repro then has to be turned into a test

Dimitar Uzunov08:01:16

cool will send a PR soon

kipz15:01:09

Thanks for publishing the Docker images for babashka - I'm finding them super useful already. To take a closer look at its dependencies and their vulnerabilities, I added it to atomist's database (https://dso.atomist.com/images/babashka/babashka/digests/sha256%3A5fd551a5602ccce77a505fefed39f1ca2f76db3acc99b860ca1b876ea22a97bc), but we can only see those dependencies brought in by the base image because first babashka is uberjared and then made into a graal native-image, which squishes things even more. Scanners such as grype (https://github.com/anchore/grype) and trivvy obviously don't see anything. I've been playing around with a lein plugin that generates what I'm calling a meta-bom-jar, that basically just contains the group, artifact and version of all a project's dependencies (https://github.com/kipz/lein-meta-bom). If we included this in the babashka Docker images, we could give vulnerability scanners visibility into the bill of materials in use by uberjars, graal native images and so on. I tried on a fork https://github.com/kipz/babashka/tree/metabom and it seems to work (in that grype can now see all the packages in the babashka images). There are probably other approaches to improving traceability, such as adding a bill-of-materials to an image label, adding metadata to native-image (and adding support to grype), but this seemed more immediately tractable in the short term. FWIW - grype didn't find any additional vulnerabilities over those introduced by the base image - mostly likely because babashka is keeping pretty up to date with regard to its dependencies! 🙂 What do you think?

borkdude16:01:24

Any ideas on this one @U7ERLH6JX?

lispyclouds08:01:29

I'll try to have a think about it by today hopefully, quite caught up in post holidays workload 😕 but looks like a worthy thing to consider

kipz09:01:38

Thanks. Yeah, I know how it is! The other thing I'm thinking about a lot is repeatable image builds. With git/maven style deps, we've got used to pinning to specific versions of things, and explicitly updating things in project.clj/deps.edn or whatever. In docker land, we (the industry) have been using a SNAPSHOT style uptake of the base image FROM ubuntu:latest and to install packages like apt-get update && apt install curl . We tend to be much more rigorous when it comes to installing the things we control or are more tightly coupled to (e.g. https://github.com/babashka/babashka/blob/master/Dockerfile#L18). Linting tools are beginning to pick up on these things, and suggesting installing specific versions of packages and pinning to specific base image digests - unfortunately there aren't many tools (such as lein ancient or whatever) to make this practical in most cases. 😕

lispyclouds09:01:35

Hey, so just to understand, the effect of running meta-bom is to have another jar in the docker image we ship right? How exactly do the scanners work? If they see a jar they can look into it is it?

lispyclouds09:01:13

Yeah for the second case unless we have a apt repo for graalvm versions its quite difficult to install in a specific way. the only other system dependency we have is curl which we pretty much install latest

lispyclouds09:01:32

looking at the vulnerabilities reported by atomist, should we update the base packages like glibc etc? run apt update && apt upgrade -y for every docker image build @U04V15CAJ?

borkdude10:01:42

I would be fine with that :)

kipz11:01:14

> Hey, so just to understand, the effect of running meta-bom is to have another jar in the docker image we ship right? How exactly do the scanners work? If they see a jar they can look into it is it? That's right. In an ideal world, all our build/packaging tools would output standard bills of materials (cylconedx/spdx etc), and we'd put that in package metadata somehow (labels for Docker images, META-INF for jars etc). The scanners currently work by interrogating the OS package databases e.g. for apt and by looking inside jar's for pom.xml/pom.properties to find packages and their dependencies. The details vary by scanning tool and package index type of course.

kipz11:01:16

Going off topic a little here, but the other related thing here is traceability. I think for babashka the git tags line up well with the docker tags, but it's not really possible to look at the docker image on its own and figure out the git repo, commit, Dockerfile and so on. There's a spec for image labels https://github.com/opencontainers/image-spec/blob/main/annotations.md that folks are adopting these days. I think GitHub actions has ways of adding this automatically during the build, as well as signing and so on. To me it feels like we (our industry) are moving more towards a delivery model where the supply chain and attestation are more rigorous and provable (like if you're building an iOS app). It feels super weird that we have artifact signing, semantic versioning, commit signing, mature dependency descriptors (pom.xml/deps.edn etc) and so on, but when we repackage and/or deploy our apps downstream , we throw it all away and cross our fingers 😄

lispyclouds10:01:35

@U06NZ3HFX adding the meta bom jar looks good for me from the ease perspective! Would you raise a PR from your fork? Also we need to add the CI step to call the lein command. I can have a look at it then! Also we could add a apt update && apt upgrade -y in the final docker image when adding curl to hopefully take care of the OS deps

lispyclouds10:01:32

Also if you want you can add in the commit sha attribute to the docker image too! 😄

lispyclouds10:01:47

As for repeatable/reproducible builds i think one of the bigger blockers apart from the apt installs is the native image spit out by Graal, not sure if that ones reproducable? :thinking_face:

kipz09:01:14

Yeah - even building class files from clojure isn't repeatable, let alone jars or native images. I've had a JIRA/patch open to fix that for a decade now 🙂

kipz09:01:49

I'll try to make a PR to bring some of this stuff together! Thanks.

1
🙏 1
kipz10:01:52

It looks like there are different solutions for different CI's to populate the opencontainers labels, but I'm a little confused about the multi-ci setup. It looks like GitHub actions is used for master stuff, but circle-ci for other stuff, like PR's? What's the plan here? Maybe we should start with the meta-bom 🙂

borkdude10:01:22

@U06NZ3HFX github actions isn't used, it's just a backup for when circleci doesn't work

borkdude10:01:42

redundant CI

borkdude10:01:24

so just look at CircleCI, you can ignore actions

kipz10:01:10

oh good - that helps. thanks.

lispyclouds19:01:31

Thanks a lot for this @U06NZ3HFX!

borkdude19:01:42

Thanks you both!

borkdude19:01:00

If we would ever migrate from lein to deps for the build, is this plugin also available?

lispyclouds19:01:18

taking a look at the build failure

lispyclouds19:01:09

@U04V15CAJ is it some weird bash string expansion im failing to see?

lispyclouds19:01:22

looked okay by the looks of it :thinking_face:

lispyclouds19:01:07

--label defintely is a valid flag

lispyclouds19:01:17

of build. should be docker buildx build --label ...

lispyclouds19:01:03

its taking --label "org.opencontainers.image.description as a flag i think

borkdude19:01:21

why are these things escaped?

\"org.opencontainers.image.description=Native, fast starting Clojure interpreter for scripting\" \

borkdude19:01:44

oh because it's within a string, I see

borkdude19:01:15

perhaps it's better to use a bash array here

borkdude19:01:26

and then splice the array?

borkdude19:01:41

would be good to try this locally first

lispyclouds19:01:50

only if a thing was there to allow me to write clojure instead of bash for scripts

🤪 1
borkdude19:01:15

we have to have one place to remind ourselves why babashka exists

borkdude19:01:24

which is the babashka build ;)

lispyclouds19:01:26

well lesser painful reminders are nicer

borkdude19:01:10

well at some point in the build we can assume that the binary babashka is around and then use that to bootstrap the rest of the build, I would be ok with that I guess

lispyclouds20:01:32

this seems to work:

CIRCLE_REPOSITORY_URL=foo
CIRCLE_SHA1=shafoo
CIRCLE_TAG=foo_tag
CIRCLE_BRANCH=main
image_tag=foo_tag

label_args=("--label" "org.opencontainers.image.description=Native, fast starting Clojure interpreter for scripting"
            "--label" "org.opencontainers.image.title=Babashka"
            "--label" "org.opencontainers.image.created=`date -Iseconds`"
            "--label" "org.opencontainers.image.url=${CIRCLE_REPOSITORY_URL}"
            "--label" "org.opencontainers.image.documentation=${CIRCLE_REPOSITORY_URL}"
            "--label" "org.opencontainers.image.source=${CIRCLE_REPOSITORY_URL}"
            "--label" "org.opencontainers.image.revision=${CIRCLE_SHA1}"
            "--label" "org.opencontainers.image.ref.name=${CIRCLE_TAG:${CIRCLE_BRANCH}}"
            "--label" "org.opencontainers.image.version=${image_tag}")

echo "${label_args[@]}"
shall i commit on master?

lispyclouds20:01:17

i think it should also be ${CIRCLE_TAG}:${CIRCLE_BRANCH}" not ${CIRCLE_TAG:${CIRCLE_BRANCH}}" right?

lispyclouds20:01:58

yeah otherwise its a weird expansion. making the change. should i directly push?

borkdude20:01:47

Fine but it would be faster if you can test that Docker stuff locally first on a dummy image

lispyclouds20:01:23

right, dont have the buildx setup here yet but giving it a quick try

borkdude20:01:24

ok, then push to master ;)

lispyclouds20:01:21

well almost set it up now, more bash weirdness

lispyclouds20:01:39

seems to start the build, pushing 🤞:skin-tone-3:

borkdude20:01:18

cool! maybe also verify if the docker images contains all the good stuff now?

lispyclouds21:01:41

well we missed out the important bit, the copying of the jar was in Dockerfile and not http://Dockerfle.ci and Dockerfile.alpine which actually pushes the images to hub 😛 we have the labels though!

lispyclouds21:01:23

probably need to change the release script

lispyclouds21:01:24

i can try to give it a shot sometime tomorrow hopefully, please do have a go at it if any one you have some time too! 😄 im thinking of copying the metabom jar along with the binary tar in /tmp/releases in the release step and then copy it into the docker image in the docker script by making sure its in the context

lispyclouds21:01:36

also better approaches welcome!

borkdude21:01:10

Is it better that we revert the commits for now?

borkdude21:01:35

I'd rather not have unfinished stuff on master

lispyclouds21:01:58

the labels part is complete though, the docker things could be reverted

borkdude21:01:14

ok, please do

borkdude21:01:18

can revisit when either of you has the time properly

borkdude21:01:26

with a new PR

lispyclouds21:01:49

yep, this needs some thought on the CI

lispyclouds21:01:50

@U06NZ3HFX if youre able to look at this again, have a look at ./circleci/script/release, we copy the tar.gz of the binary into /tmp/releases which is then persisted in the circleci workspace and is picked up in the docker step. we need to copy the metabom jar too in the same way for http://Dockerfile.ci and Dockerfile.alpine to pick up

kipz22:01:06

oh dear - sorry, was afk for a bit. I can have a look at those change for sure. What's the plain Dockerfile for then?

borkdude22:01:59

@U06NZ3HFX Dockerfile is more for people who want to build it locally

👍 1
borkdude22:01:28

we decided to let it sit there and not break changes to it and make http://Dockerfile.ci for our own ci

borkdude22:01:02

it might still be good to have the bom stuff in there too, but ultimately it should also go into the .ci and .alpine one

kipz22:01:34

Got it. FWIW - the weird variable expansion was a copy/paste from the circle-ci docker orb: https://circleci.com/developer/orbs/orb/ledger/docker - guess they have the same bug - not even sure that's valid shell script is it? 🙂