This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2022-01-05
Channels
- # aleph (1)
- # announcements (18)
- # babashka (145)
- # beginners (70)
- # calva (34)
- # cider (3)
- # clj-kondo (98)
- # cljdoc (5)
- # cljs-dev (13)
- # clojure (134)
- # clojure-europe (57)
- # clojure-nl (4)
- # clojure-uk (4)
- # clojurescript (40)
- # code-reviews (3)
- # conjure (1)
- # core-async (5)
- # data-science (3)
- # datomic (8)
- # fulcro (9)
- # google-cloud (2)
- # inf-clojure (9)
- # jobs (1)
- # lsp (9)
- # malli (25)
- # polylith (4)
- # reitit (4)
- # releases (2)
- # remote-jobs (3)
- # rewrite-clj (8)
- # shadow-cljs (34)
- # tools-build (1)
- # tools-deps (67)
clojure-lsp is now available in the pod registry:
(ns clojure-lsp
(:require
[babashka.pods :as pods]))
(pods/load-pod 'com.github.clojure-lsp/clojure-lsp "2022.01.03-19.46.10")
(require '[clojure-lsp.api :as api])
;; see
(api/clean-ns! {}) ;; see settings in .lsp/config.edn
;; @ericdallo: somehow this isn't working, even when there are lint warnings:
(defn foo [unused])
(comment unresolved-symbol)
(api/diagnostics {})
/cc @ericdallo: somehow the diagnostics aren't working, but clean-ns! is!@U04V15CAJ I tested with clj-kondo and the diagnostics does seem to work:
(ns clojure-lsp
(:require
[babashka.pods :as pods]))
(pods/load-pod 'com.github.clojure-lsp/clojure-lsp "2022.01.03-19.46.10")
(require '[clojure-lsp.api :as api])
(api/diagnostics {})
hello! how do I do globbing with babashka.tasks/shell? It seems to interpret expressions like dir/* like a single dir
i.e. (babashka.tasks/shell "ls clj/*")
ls: clj/*: No such file or directory
@dimitar.ouzounoff shell
isn't a bash invocation. for globbing you can use (babashka.fs/glob "." "clj/*")
or if you want to invoke bash, you can do (shell "bash -c ls clj/*")
but this isn't recommended as this won't be portable
gotcha!
seems like I’ll be using the fs library to copy files around instead of shelling out
what is the proper way to copy a directory? I looked at copy-tree but that copies a single file it seems
@dimitar.ouzounoff fs/copy-tree
is indeed the function you want for coping a directory
ooops sorry
yeah it works
what tripped me up is that it when I invoked it like this:
(fs/copy-tree "scripts/" "bundle/scripts")
and I expected it to create bundle
but it doesn’t hmm
and it gave the no such file exception which I interpreted as it thinking scripts/ is a dir
its just me being asleep at the wheel I guess
hmm the docs seem to suggest it calls create-dirs for the user
copy-tree
(copy-tree src dest)
(copy-tree src dest {:keys [:replace-existing :copy-attributes :nofollow-links]})
Copies entire file tree from src to dest. Creates dest if needed
using create-dirs, passing it the :posix-file-permissions
option. Supports same options as copy.
v0.7.0
i will try with the latest
same without the trailing /
same with v0.7.3
----- Stack trace -------------------------------------------------------------- babashka.fs/copy-tree/fn--9542 - <built-in> babashka.fs/walk-file-tree/reify--9491 - <built-in> babashka.fs/walk-file-tree - <built-in> babashka.fs/copy-tree - <built-in>
I can log a ticket
Let's first see if this is a bug or not. I don't even understand the problem yet. Can you make a repro with a few files?
mkdir -p foo/bar
touch foo/bar/baz.txt
bb -e '(fs/copy-tree ...)'
I’m copying a dir inside another dir
oops sorry
wrong scenario
➜ ~ mkdir foo
➜ ~ touch foo/bar1
➜ ~ touch foo/bar2
➜ ~ bb -e '(fs/copy-tree "foo/" "foo2/foo")'
----- Error --------------------------------------------------------------------
Type: java.nio.file.NoSuchFileException
Message: /Users/dimitaruzunov/foo2/foo
Location: <expr>:1:1
this is what I’m trying to do
$ mkdir -p foo/bar
$ touch foo/bar/baz.txt
$ mkdir dest
$ bb -e '(fs/copy-tree "foo/bar" "dest")'
$ ls dest
baz.txt
I can work around it by creating foo2
yeah my case is different in that the dest is nested
this doesn’t really create a nested dir (like mkdir -p would), rather it walks a nested dir into another dir
should it be able to create foo2?
Sorry I showed something else earlier and deleted it
this is the repro
brb in 20 minutes going out to pick up the kid
ok, you're right, it does not create the nested dir, although the docstring says so. weird.
feel free to create an issue with the above repro. workaround for now is to make it yourself
Hello @U04V15CAJ so we checkout the source with @U0509NKGK and it looks like a call to create-dirs like this solves the issue:
(defn copy-tree
"Copies entire file tree from src to dest. Creates dest if needed
using create-dirs, passing it the :posix-file-permissions
option. Supports same options as copy."
([src dest] (copy-tree src dest nil))
([src dest {:keys [:replace-existing
:copy-attributes
:nofollow-links]}]
(create-dirs dest) ;;; <<<<<<<<<<<<<<<<<<<<<< Only change here
(let [copy-options (->copy-opts replace-existing copy-attributes false nofollow-links)
link-options (->link-opts nofollow-links)
from (real-path src {:nofollow-links nofollow-links})
;; using canonicalize here because real-path requires the path to exist
to (canonicalize dest {:nofollow-links nofollow-links})]
(walk-file-tree from {:pre-visit-dir (fn [dir _attrs]
(let [rel (relativize from dir)
to-dir (path to rel)]
(when-not (Files/exists to-dir link-options)
(Files/copy ^Path dir to-dir
^"[Ljava.nio.file.CopyOption;"
copy-options)))
:continue)
:visit-file (fn [from-path _attrs]
(let [rel (relativize from from-path)
to-file (path to rel)]
(Files/copy ^Path from-path to-file
^"[Ljava.nio.file.CopyOption;"
copy-options)
:continue)
:continue)}))))
is the repro in a test enough or should I think of other cases as well?
cool will send a PR soon
Thanks for publishing the Docker images for babashka - I'm finding them super useful already. To take a closer look at its dependencies and their vulnerabilities, I added it to atomist's database (https://dso.atomist.com/images/babashka/babashka/digests/sha256%3A5fd551a5602ccce77a505fefed39f1ca2f76db3acc99b860ca1b876ea22a97bc), but we can only see those dependencies brought in by the base image because first babashka is uberjared and then made into a graal native-image, which squishes things even more. Scanners such as grype (https://github.com/anchore/grype) and trivvy obviously don't see anything. I've been playing around with a lein plugin that generates what I'm calling a meta-bom-jar, that basically just contains the group, artifact and version of all a project's dependencies (https://github.com/kipz/lein-meta-bom). If we included this in the babashka Docker images, we could give vulnerability scanners visibility into the bill of materials in use by uberjars, graal native images and so on. I tried on a fork https://github.com/kipz/babashka/tree/metabom and it seems to work (in that grype can now see all the packages in the babashka images). There are probably other approaches to improving traceability, such as adding a bill-of-materials to an image label, adding metadata to native-image (and adding support to grype), but this seemed more immediately tractable in the short term. FWIW - grype didn't find any additional vulnerabilities over those introduced by the base image - mostly likely because babashka is keeping pretty up to date with regard to its dependencies! 🙂 What do you think?
Any ideas on this one @U7ERLH6JX?
I'll try to have a think about it by today hopefully, quite caught up in post holidays workload 😕 but looks like a worthy thing to consider
Thanks. Yeah, I know how it is! The other thing I'm thinking about a lot is repeatable image builds. With git/maven style deps, we've got used to pinning to specific versions of things, and explicitly updating things in project.clj/deps.edn or whatever. In docker land, we (the industry) have been using a SNAPSHOT style uptake of the base image FROM ubuntu:latest
and to install packages like apt-get update && apt install curl
. We tend to be much more rigorous when it comes to installing the things we control or are more tightly coupled to (e.g. https://github.com/babashka/babashka/blob/master/Dockerfile#L18). Linting tools are beginning to pick up on these things, and suggesting installing specific versions of packages and pinning to specific base image digests - unfortunately there aren't many tools (such as lein ancient
or whatever) to make this practical in most cases. 😕
Hey, so just to understand, the effect of running meta-bom is to have another jar in the docker image we ship right? How exactly do the scanners work? If they see a jar they can look into it is it?
Yeah for the second case unless we have a apt repo for graalvm versions its quite difficult to install in a specific way. the only other system dependency we have is curl which we pretty much install latest
looking at the vulnerabilities reported by atomist, should we update the base packages like glibc etc? run apt update && apt upgrade -y
for every docker image build @U04V15CAJ?
> Hey, so just to understand, the effect of running meta-bom is to have another jar in the docker image we ship right? How exactly do the scanners work? If they see a jar they can look into it is it? That's right. In an ideal world, all our build/packaging tools would output standard bills of materials (cylconedx/spdx etc), and we'd put that in package metadata somehow (labels for Docker images, META-INF for jars etc). The scanners currently work by interrogating the OS package databases e.g. for apt and by looking inside jar's for pom.xml/pom.properties to find packages and their dependencies. The details vary by scanning tool and package index type of course.
Going off topic a little here, but the other related thing here is traceability. I think for babashka the git tags line up well with the docker tags, but it's not really possible to look at the docker image on its own and figure out the git repo, commit, Dockerfile and so on. There's a spec for image labels https://github.com/opencontainers/image-spec/blob/main/annotations.md that folks are adopting these days. I think GitHub actions has ways of adding this automatically during the build, as well as signing and so on. To me it feels like we (our industry) are moving more towards a delivery model where the supply chain and attestation are more rigorous and provable (like if you're building an iOS app). It feels super weird that we have artifact signing, semantic versioning, commit signing, mature dependency descriptors (pom.xml/deps.edn etc) and so on, but when we repackage and/or deploy our apps downstream , we throw it all away and cross our fingers 😄
@U06NZ3HFX adding the meta bom jar looks good for me from the ease perspective! Would you raise a PR from your fork? Also we need to add the CI step to call the lein command. I can have a look at it then! Also we could add a apt update && apt upgrade -y in the final docker image when adding curl to hopefully take care of the OS deps
Also if you want you can add in the commit sha attribute to the docker image too! 😄
As for repeatable/reproducible builds i think one of the bigger blockers apart from the apt installs is the native image spit out by Graal, not sure if that ones reproducable? :thinking_face:
Yeah - even building class files from clojure isn't repeatable, let alone jars or native images. I've had a JIRA/patch open to fix that for a decade now 🙂
It looks like there are different solutions for different CI's to populate the opencontainers labels, but I'm a little confused about the multi-ci setup. It looks like GitHub actions is used for master stuff, but circle-ci for other stuff, like PR's? What's the plan here? Maybe we should start with the meta-bom 🙂
@U06NZ3HFX github actions isn't used, it's just a backup for when circleci doesn't work
Thanks a lot for this @U06NZ3HFX!
If we would ever migrate from lein to deps for the build, is this plugin also available?
taking a look at the build failure
@U04V15CAJ is it some weird bash string expansion im failing to see?
looked okay by the looks of it :thinking_face:
--label defintely is a valid flag
of build. should be docker buildx build --label ...
its taking --label "org.opencontainers.image.description
as a flag i think
why are these things escaped?
\"org.opencontainers.image.description=Native, fast starting Clojure interpreter for scripting\" \
https://docs.docker.com/engine/reference/commandline/buildx_build/ this does mention --label
right
similar to this I mean: https://github.com/babashka/babashka/blob/6e2c32ff84fc826fc9c10ecee29694a53258273e/script/compile#L42
only if a thing was there to allow me to write clojure instead of bash for scripts
got it
well lesser painful reminders are nicer
well at some point in the build we can assume that the binary babashka is around and then use that to bootstrap the rest of the build, I would be ok with that I guess
this seems to work:
CIRCLE_REPOSITORY_URL=foo
CIRCLE_SHA1=shafoo
CIRCLE_TAG=foo_tag
CIRCLE_BRANCH=main
image_tag=foo_tag
label_args=("--label" "org.opencontainers.image.description=Native, fast starting Clojure interpreter for scripting"
"--label" "org.opencontainers.image.title=Babashka"
"--label" "org.opencontainers.image.created=`date -Iseconds`"
"--label" "org.opencontainers.image.url=${CIRCLE_REPOSITORY_URL}"
"--label" "org.opencontainers.image.documentation=${CIRCLE_REPOSITORY_URL}"
"--label" "org.opencontainers.image.source=${CIRCLE_REPOSITORY_URL}"
"--label" "org.opencontainers.image.revision=${CIRCLE_SHA1}"
"--label" "org.opencontainers.image.ref.name=${CIRCLE_TAG:${CIRCLE_BRANCH}}"
"--label" "org.opencontainers.image.version=${image_tag}")
echo "${label_args[@]}"
shall i commit on master?i think it should also be ${CIRCLE_TAG}:${CIRCLE_BRANCH}"
not ${CIRCLE_TAG:${CIRCLE_BRANCH}}"
right?
yeah otherwise its a weird expansion. making the change. should i directly push?
Fine but it would be faster if you can test that Docker stuff locally first on a dummy image
right, dont have the buildx setup here yet but giving it a quick try
well almost set it up now, more bash weirdness
seems to start the build, pushing 🤞:skin-tone-3:
Finally
well we missed out the important bit, the copying of the jar was in Dockerfile and not http://Dockerfle.ci and Dockerfile.alpine which actually pushes the images to hub 😛 we have the labels though!
probably need to change the release script
i can try to give it a shot sometime tomorrow hopefully, please do have a go at it if any one you have some time too! 😄 im thinking of copying the metabom jar along with the binary tar in /tmp/releases in the release step and then copy it into the docker image in the docker script by making sure its in the context
also better approaches welcome!
the labels part is complete though, the docker things could be reverted
okay reverting https://github.com/babashka/babashka/commit/9cf6a1ae7129a187f1a12d090724515e6a7c722d
yep, this needs some thought on the CI
@U06NZ3HFX if youre able to look at this again, have a look at ./circleci/script/release
, we copy the tar.gz of the binary into /tmp/releases
which is then persisted in the circleci workspace and is picked up in the docker step. we need to copy the metabom jar too in the same way for http://Dockerfile.ci and Dockerfile.alpine to pick up
oh dear - sorry, was afk for a bit. I can have a look at those change for sure. What's the plain Dockerfile
for then?
we decided to let it sit there and not break changes to it and make http://Dockerfile.ci for our own ci
it might still be good to have the bom stuff in there too, but ultimately it should also go into the .ci and .alpine one
Got it. FWIW - the weird variable expansion was a copy/paste from the circle-ci docker orb: https://circleci.com/developer/orbs/orb/ledger/docker - guess they have the same bug - not even sure that's valid shell script is it? 🙂