tools-build 2022-05-25 | Slack Archive

ozimos10:05:51

Are there any tools based on tools.build to support parallel builds for a monorepo similar to lein modules https://github.com/jcrossley3/lein-modules For a monorepo of lein projects one can run lein modules -p 4 build and up to 4 sub projects can be built in parallel I'm thinking of modifying b/compile-clj to run in parallel. Are there any existing tools that already do this

delaguardo11:05:21

with tools build it is easy to make such tool: build.clj

(ns build
  (:require [clojure.tools.build.api :as b]))

(def basis (b/create-basis {:project "deps.edn"}))

(defn clean [_]
  (b/delete {:path "target"}))

(defn compile-module [module]
  (let [class-dir (format "target/%s/classes" module)
        src-dir (format "%s/src" module)]
    (b/copy-dir {:src-dirs [src-dir]
                 :target-dir class-dir})
    (b/compile-clj {:basis basis
                    :src-dirs [(format "%s/src" module)]
                    :class-dir class-dir})))

(defn compile-clj [_]
  (clean nil)
  (doall (pmap compile-module ["module1" "module2"])))

deps.edn

{:paths []
 :deps {}

 :aliases
 {;; Run with clj -T:build function-in-build
  :build {:deps {io.github.clojure/tools.build {:git/tag "v0.8.2" :git/sha "ba1a2bf"}}
          :ns-default build}}}

files before:

❯ tree              
.
├── build.clj
├── deps.edn
├── module1
│   └── src
│       └── foo
│           └── core.clj
└── module2
    └── src
        └── bar
            └── core.clj

6 directories, 4 files

run compile-clj command:

clj -T:build compile-clj

files after:

❯ tree
.
├── build.clj
├── deps.edn
├── module1
│   └── src
│       └── foo
│           └── core.clj
├── module2
│   └── src
│       └── bar
│           └── core.clj
└── target
    ├── module1
    │   └── classes
    │       └── foo
    │           ├── core$fn__142.class
    │           ├── core$loading__6789__auto____140.class
    │           ├── core.clj
    │           └── core__init.class
    └── module2
        └── classes
            └── bar
                ├── core$fn__142.class
                ├── core$loading__6789__auto____140.class
                ├── core.clj
                └── core__init.class

13 directories, 12 files

by doing that you have more control over how to parallelise compilation

delaguardo11:05:25

so instead of changing internal b/compile-clj - build you own compile process on top

Alex Miller (Clojure team)13:05:38

this is the whole benefit of builds being programs in a capable programming language

👍 2

ozimos16:05:23

Thanks @U04V4KLKC This is a nice starting point. But there are more requirements • The modules in my monorepo depend on each other. Automatic ordering of module builds based on dependencies. For a parallel build with 4 threads for the example below, only 3 threads will be occupied for the first round. So a simple pmap will not suffice

[;; module dependencies
    1       [2 3]
    2        [4 5]
    3        []
    4        [5 8]
    5        [10]
    6        [8 10]
    7        [8]
    8        [9]
    9        []
   10        []]

• I also want separate jars for each module, so I'll build module 1 and all the others as well. (Perhaps separate jars not needed with tools.deps. Will need to investigate further) • maybe share namespace compilation output across all modules to remove duplication of compilation but each modules jars contains only its own classes So not so straightforward to put together in a build.clj but I think common enough use case that one can expect to find existing tooling

delaguardo17:05:18

It definitely not straightforward, my example is just to illustrate the idea 🙂 dependencies have nothing to do with the way you build modules. You might have for each module its own deps.edn listing all module dependencies where you can refer to another module using :local/root coordinate. here is how I change compile-module function from my example

(defn compile-module [module]
  (let [class-dir (format "target/%s/classes" module)
        src-dir (format "%s/src" module)
        basis (b/create-basis {:project (format "%s/deps.edn" module)})]
    (b/copy-dir {:src-dirs [src-dir]
                 :target-dir class-dir})
    (b/compile-clj {:basis basis
                    :src-dirs [(format "%s/src" module)]
                    :class-dir class-dir})))

I don't get namespace sharing idea. you can look at https://polylith.gitbook.io/polylith/ who set monorepo idea as a corner stone but from my point of view it is overcomplicated and contains some opinionated arhitecture principles that might require changes for you project organisation.

ozimos06:05:45

Thanks. I was missing the :local/root coordinate and how each module grabs its deps from source files and not jars. Your original example handles everything and is much cleaner than what I had in mind. With this pipeline reusing output from namespace compilation across different modules is not necessary

2022-05-25

Channels