This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2022-05-25
Channels
- # announcements (9)
- # asami (69)
- # babashka (151)
- # babashka-sci-dev (34)
- # beginners (90)
- # cider (21)
- # clj-on-windows (17)
- # clj-otel (4)
- # cljsrn (5)
- # clojure (27)
- # clojure-austin (3)
- # clojure-europe (87)
- # clojure-gamedev (1)
- # clojure-nl (3)
- # clojure-norway (8)
- # clojure-poland (2)
- # clojure-uk (3)
- # clojured (10)
- # clojurescript (50)
- # core-async (73)
- # cursive (28)
- # data-science (2)
- # datomic (17)
- # etaoin (1)
- # honeysql (6)
- # introduce-yourself (3)
- # jobs (1)
- # joyride (12)
- # malli (5)
- # nbb (14)
- # off-topic (18)
- # pathom (4)
- # podcasts-discuss (2)
- # polylith (30)
- # project-updates (3)
- # re-frame (33)
- # reitit (1)
- # remote-jobs (13)
- # shadow-cljs (59)
- # sql (12)
- # tools-build (7)
- # xtdb (36)
Are there any tools based on tools.build to support parallel builds for a monorepo similar to lein modules
https://github.com/jcrossley3/lein-modules
For a monorepo of lein projects one can run
lein modules -p 4 build
and up to 4 sub projects can be built in parallel
I'm thinking of modifying b/compile-clj to run in parallel. Are there any existing tools that already do this
with tools build it is easy to make such tool: build.clj
(ns build
(:require [clojure.tools.build.api :as b]))
(def basis (b/create-basis {:project "deps.edn"}))
(defn clean [_]
(b/delete {:path "target"}))
(defn compile-module [module]
(let [class-dir (format "target/%s/classes" module)
src-dir (format "%s/src" module)]
(b/copy-dir {:src-dirs [src-dir]
:target-dir class-dir})
(b/compile-clj {:basis basis
:src-dirs [(format "%s/src" module)]
:class-dir class-dir})))
(defn compile-clj [_]
(clean nil)
(doall (pmap compile-module ["module1" "module2"])))
deps.edn
{:paths []
:deps {}
:aliases
{;; Run with clj -T:build function-in-build
:build {:deps {io.github.clojure/tools.build {:git/tag "v0.8.2" :git/sha "ba1a2bf"}}
:ns-default build}}}
files before:
❯ tree
.
├── build.clj
├── deps.edn
├── module1
│ └── src
│ └── foo
│ └── core.clj
└── module2
└── src
└── bar
└── core.clj
6 directories, 4 files
run compile-clj command:
clj -T:build compile-clj
files after:
❯ tree
.
├── build.clj
├── deps.edn
├── module1
│ └── src
│ └── foo
│ └── core.clj
├── module2
│ └── src
│ └── bar
│ └── core.clj
└── target
├── module1
│ └── classes
│ └── foo
│ ├── core$fn__142.class
│ ├── core$loading__6789__auto____140.class
│ ├── core.clj
│ └── core__init.class
└── module2
└── classes
└── bar
├── core$fn__142.class
├── core$loading__6789__auto____140.class
├── core.clj
└── core__init.class
13 directories, 12 files
by doing that you have more control over how to parallelise compilationso instead of changing internal b/compile-clj - build you own compile process on top
this is the whole benefit of builds being programs in a capable programming language
Thanks @U04V4KLKC This is a nice starting point. But there are more requirements • The modules in my monorepo depend on each other. Automatic ordering of module builds based on dependencies. For a parallel build with 4 threads for the example below, only 3 threads will be occupied for the first round. So a simple pmap will not suffice
[;; module dependencies
1 [2 3]
2 [4 5]
3 []
4 [5 8]
5 [10]
6 [8 10]
7 [8]
8 [9]
9 []
10 []]
• I also want separate jars for each module, so I'll build module 1 and all the others as well. (Perhaps separate jars not needed with tools.deps. Will need to investigate further)
• maybe share namespace compilation output across all modules to remove duplication of compilation but each modules jars contains only its own classes
So not so straightforward to put together in a build.clj but I think common enough use case that one can expect to find existing toolingIt definitely not straightforward, my example is just to illustrate the idea 🙂 dependencies have nothing to do with the way you build modules. You might have for each module its own deps.edn listing all module dependencies where you can refer to another module using :local/root coordinate. here is how I change compile-module function from my example
(defn compile-module [module]
(let [class-dir (format "target/%s/classes" module)
src-dir (format "%s/src" module)
basis (b/create-basis {:project (format "%s/deps.edn" module)})]
(b/copy-dir {:src-dirs [src-dir]
:target-dir class-dir})
(b/compile-clj {:basis basis
:src-dirs [(format "%s/src" module)]
:class-dir class-dir})))
I don't get namespace sharing idea.
you can look at https://polylith.gitbook.io/polylith/ who set monorepo idea as a corner stone
but from my point of view it is overcomplicated and contains some opinionated arhitecture principles that might require changes for you project organisation.Thanks. I was missing the :local/root coordinate and how each module grabs its deps from source files and not jars. Your original example handles everything and is much cleaner than what I had in mind. With this pipeline reusing output from namespace compilation across different modules is not necessary