Fork me on GitHub
#babashka
<
2021-11-06
>
hugod02:11:23

Would it be possible to get tools.build to run on bb?

hugod02:11:53

Looks like the first thing it complains about is clojure.tools.deps.alpha.util.concurrent

hugod02:11:55

It imports java.util.concurrent Callable Future ThreadFactory ExecutorService Executors TimeUnit

borkdude08:11:37

@U0HFRSY0M Some parts of tools.build can run with babashka. But tools.deps.alpha itself doesn't because of the dependency on all kinds of maven and aws stuff. Often functions in tools.build receive a basis as argument which means it doesn't have to call tools.deps.alpha anymore since all the info is already in the basis. If we can set things up in such a way that for the basis, we can shell out to a JVM and for the rest we can use babashka, then it can work, but tools.build itself isn't organized in this way.

borkdude08:11:27

Last time I tried there were a couple of classes missing in bb. I'd be happy to add those if this approach is promising.

hugod12:11:12

My aim was to speed up running tools.build invocations. If we’re starting the JVM to get the basis, then its probably just as fast to run everything in the jvm. One possibility might be to use the cached classpath, if available, and only invoke the JVM to recompute it. Not sure if that is feasible or not.

borkdude12:11:03

> If we’re starting the JVM to get the basis, then its probably just as fast to run everything in the jvm. Indeed. This is what I figured too.

borkdude12:11:01

I think it might be possible to build a tools.build pod at one point though. I have an experimental version of tools.deps.alpha compiled to native.

borkdude12:11:30

it's very very fast

borkdude12:11:27

@U0HFRSY0M I did use fs/modified-since in some projects to not invoke a JVM when not necessary

borkdude12:11:36

this can speed up things if there's nothing to do

borkdude12:11:27

I think it is possible to make a fork of tools.build to strip out tools.deps.alpha and only include JVM standard lib classes to make it bb compatible

borkdude12:11:42

and then the basis has to come from shelling out which can be cached.

borkdude12:11:35

but using this native tools deps alpha thing, you could then calculate the basis using that

borkdude12:11:40

which is extremely fast

borkdude12:11:50

I suggest you try it out that experimental tools deps alpha binary

hugod12:11:04

A tools.deps.alpha pod would be awesome, and would probably greatly simplify getting tools,build to work.

borkdude12:11:39

you would run into issues when you try to include more libraries like https://github.com/slipset/deps-deploy/blob/master/src/deps_deploy/deps_deploy.clj though

hugod13:11:32

Isn’t that just a case of needing to build the relevant message parser in tools-deps-native?

borkdude13:11:07

@U0HFRSY0M you mean, to turn it into a pod?

hugod13:11:50

Yes. I’m trying to understand your comment on deps-deploy.

borkdude13:11:34

I mean, if you would use deps-deploy in your build scripts, to deploy to clojars. this wouldn't work with babashka. so in general you couldn't replace all of your build needs with the pod probably is what I'm saying.

hugod13:11:50

Ah, indeed, but there may be ways of getting round that (e.g. I have a deploy task that only use the maven metadata dependency, and works over http)

borkdude13:11:28

I think the shortest path to a POC is to alter the tools-deps-native binary to receive the args of create-basis as EDN and print out a basis as EDN instead of a classpath.

borkdude13:11:43

Then this can be used to shell out to in a function create-basis which you can use within bb

borkdude13:11:54

and then the rest of tools.build compatibility will follow from that

borkdude13:11:22

we don't even have to create a pod to get this started

hugod13:11:07

turning it into a pod looks fairly straightforward

borkdude13:11:38

yeah it isn't hard. PR welcome :)

borkdude13:11:07

bb sets an env var BABASHKA_POD to true if you invoke the binary as a pod

borkdude13:11:23

so this can be used to use one binary both as a CLI and as a pod

hugod13:11:41

we’ll see how it goes this week

borkdude11:11:47

@U0HFRSY0M I'm trying to use the pod + making bb more compatible with tools.build. This script now works with the tools-build branch of bb.

#!/usr/bin/env bb

(ns foo)
(require '[babashka.pods :as pods])
(pods/load-pod "./tools-deps-native")

(require '[babashka.deps :as deps])
(deps/add-deps '{:deps {borkdude/spartan.spec {:git/url ""
                                               :sha "12947185b4f8b8ff8ee3bc0f19c98dbde54d4c90"}
                        io.github.clojure/tools.build {:tag "v0.6.2"  :sha "226fb52"}}})

(require '[spartan.spec])
(require '[clojure.tools.build.api :as b])

(def lib 'my/lib1)
(def version (format "1.2.%s" (b/git-count-revs nil)))
(def class-dir "target/classes")
(def basis (b/create-basis {:project "deps.edn"}))
(def jar-file (format "target/%s-%s.jar" (name lib) version))

(defn clean [_]
  (b/delete {:path "target"}))

(require 'clojure.tools.build.tasks.copy)

(defn jar [_]
  #_(b/write-pom {:class-dir class-dir
                :lib lib
                :version version
                :basis basis
                :src-dirs ["src"]})
  (b/copy-dir {:src-dirs ["src"]
               :target-dir class-dir})
  (b/jar {:class-dir class-dir
          :jar-file jar-file}))

(jar nil)

borkdude11:11:23

The write-pom needs more work since a couple of things from clojure.data.xml aren't there in bb yet. But we can iterate on this.

borkdude11:11:56

Another option would be to make the tools-deps-alpha pod a tools-build pod so we move everything there.

hugod12:11:59

I don’t think anything in these namespaces is going to be used in a tight loop, so may not be worth getting into bb itself. We could make it a clojure tools pod.

borkdude12:11:46

@U0HFRSY0M I added some stuff in tools-build branch and so far it caused only very minor increase in binary size.

borkdude12:11:20

I have yet to add the extra clojure.data.xml namespaces to make write-pom compatible

hugod12:11:55

build tasks are now so much faster 🙂

borkdude12:11:50

yeah... you got it working?

hugod13:11:56

I’ve not played any more with the pod - just trying out what already works

borkdude13:11:50

you can try bb from #babashka-circleci-builds

borkdude13:11:05

from the tools-build branch

hugod13:11:57

will give it a try - just doing some refactoring to switch to babashka.fs

hugod14:11:43

With the tools-build https://github.com/babashka/babashka/commit/bc0943f45dd06308099eb0f3a5b2a86598c6175d bb I get , Unable to resolve classname: javax.xml.namespace.NamespaceContext when running the jar task

borkdude14:11:07

yeah, this is because it tries to load clojure.data.xml namespaces from source

borkdude14:11:17

which should actually be in bb but they aren't in there yet

borkdude14:11:00

feel free to make PRs agains the tools-build branch for this

hugod14:11:17

I’m stepping away for the rest of the day. It’s great to see this coming to fruition.

borkdude14:11:26

alright, I'll see if I can fix those nss later if you don't beat me to it

djblue04:11:25

Would it be worth adding a real babashka.deps namespace? It would only implement what actually works in the jvm version of clojure, so no add-deps . I ask because it would be kinda dope to use my bb tasks with my existing jvm nrepl connection. I have a https://github.com/djblue/portal/commit/dc0c6ae9f0339e79b08072207e511da27e55ea5b#diff-8b9e9b3023c4ac2fa9d74f705f9b7e6ec82fce4552b9456e31e7d2ec563a00f0 for my project, but was wondering if something like this could live in bb.

borkdude08:11:46

so basically with only the clojure function?

borkdude08:11:57

so instead of babashka.impl.deps we would rename it to babashka.deps?

borkdude12:11:03

@U1G869VNV Done on master. Use 0.6.5-SNAPSHOT to test

djblue05:11:20

Awesome, that was really quick, thanks!

schmee15:11:46

I’m so, so happy that I don’t have to write stuff like this in bash anymore, thank you @borkdude! 🙏 https://github.com/schmee/dotfiles/blob/master/install_zig.bb

👍 1
👏 1