Fork me on GitHub
#tools-deps
<
2022-01-15
>
Reily Siegel01:01:51

Is it possible to build tools.deps.alpha without a transitive dependency on clojurescript? I recently pacakged clojure-tools for Guix, which requires that all pacakges be built from source. However, building ClojureScript (and it's dependencies) from source is quite the undertaking. To avoid this, I simply strip out the S3 transporter, which is the only part of tools deps that depends transitively on clojurescript. However, this is for obvious reasons not really ideal in the long run. Is there a way to break this dependency chain somewhere? Currently we have tools.deps -> cognitect.aws -> core.async -> clojurescript.

Reily Siegel01:01:51

For context, here is how I strip s3-transporter (in scheme, for version 0.12.1104):

(lambda _
  (for-each delete-file
            (list
             (string-append
              "src/main/clojure/clojure/"
              "tools/deps/alpha/util/s3_aws_client.clj")
             (string-append
              "src/main/clojure/clojure/"
              "tools/deps/alpha/util/s3_transporter.clj")
             (string-append
              "src/test/clojure/clojure/"
              "tools/deps/alpha/util/test_s3_transporter.clj")))
  (substitute*
   "src/main/clojure/clojure/tools/deps/alpha/util/maven.clj"
   (("clojure.tools.deps.alpha.util.s3-transporter")
    "")))

seancorfield01:01:47

core.async does not depend on ClojureScript:

org.clojure/core.async 1.5.648
  . org.clojure/tools.analyzer.jvm 1.2.2
    . org.clojure/tools.analyzer 1.1.0
    . org.clojure/core.memoize 1.0.253
      . org.clojure/core.cache 1.0.225
        . org.clojure/data.priority-map 1.1.0
    . org.ow2.asm/asm 9.2
    . org.clojure/tools.reader 1.3.6

seancorfield01:01:59

tools.deps.alpha does not depend on ClojureScript either:

clojure -Sdeps '{:deps {org.clojure/tools.deps.alpha {:mvn/version "RELEASE"}}}' -Stree|fgrep script
<empty result>

seancorfield01:01:49

Here's the list of org.clojure deps that get pulled in for tools.deps.alpha:

(! 755)-> clojure -Sdeps '{:deps {org.clojure/tools.deps.alpha {:mvn/version "RELEASE"}}}' -Stree|fgrep org.clojure
org.clojure/clojure 1.10.3
  . org.clojure/spec.alpha 0.2.194
  . org.clojure/core.specs.alpha 0.2.56
org.clojure/tools.deps.alpha 0.12.1109
  . org.clojure/data.xml 0.2.0-alpha6
    . org.clojure/data.codec 0.1.0
  . org.clojure/tools.gitlibs 2.4.172
  . org.clojure/tools.cli 1.0.206
    . org.clojure/data.json 2.4.0
    . org.clojure/tools.logging 1.2.1
      . org.clojure/core.async 1.5.644
    . org.clojure/data.xml 0.2.0-alpha6
    . org.clojure/core.async 1.5.644
      . org.clojure/tools.analyzer.jvm 1.2.1
        . org.clojure/tools.analyzer 1.1.0
        . org.clojure/core.memoize 1.0.253
          . org.clojure/core.cache 1.0.225
            . org.clojure/data.priority-map 1.1.0
        . org.clojure/tools.reader 1.3.6

seancorfield01:01:18

What makes you think you have a dependency on ClojureScript @U6NJBB596?

seancorfield01:01:17

(if you built clojure-tools from source, that depends on t.d.a so you would have already run into this issue?)

Alex Miller (Clojure team)01:01:44

I second Sean’s confusion. core.async has a project.clj that depends on clojurescript for testing but that's not the project file we use to build core.async (built with Maven from the pom.xml)

Reily Siegel01:01:02

Ah, I found the problem. It was attempting to compile macros for clojurescript, which depend on cljs.analyzer. removing macro namespaces from the list of aot compiled namespaces should fix this.

Alex Miller (Clojure team)01:01:29

Again, there should be no clojurescript anywhere in this deps tree

Reily Siegel01:01:02

The problem here stemming from the fact that guix's clojure-build-system tries to aot compile every clj file.

Reily Siegel01:01:35

Guix doesn't use maven for dependency resolution, it uses its own packages, and trying to aot ClojureScript macros (in clj files) without ClojureScript failed.

Reily Siegel01:01:12

Thank you for helping me realize something obvious after staring at what was ultimately a stupid problem for way too long.

seancorfield01:01:13

Which namespaces are you talking about here?

Reily Siegel01:01:20

cljs.core.async.impl.ioc-macros

Alex Miller (Clojure team)01:01:30

Well, guix’s system is wrong to do that

seancorfield01:01:08

Yeah, definitely. And packaging a library as AOT'd code is likely to cause horrible problems for consumers of that library.

Alex Miller (Clojure team)01:01:10

Lots of Clojure libs have clj files that are not part of the lib

Reily Siegel01:01:39

Do you think it would be worh it to get that default changed in guix? Currently (afaict) it calls clojure.core/compile on every clj file that isn't excluded by default, unless specifically disabled. I am not super familiar with guix clojure-build-system, just trying to get a few things packaged. Currently the number of Clojure libraries packaged is abysmally small.

seancorfield01:01:48

@U6NJBB596 You're aware that lots of Clojure libraries are packaged as JAR files with no compilation of source code?

seancorfield01:01:10

"Packaging" Clojure libraries seems kinda wrong on several levels. The normal way to get Clojure libraries is as dependencies in a project -- fetched by the Clojure tooling itself, when it is run.

Reily Siegel01:01:13

I am aware of this, yes. I am neither familiar with the details of maven distribution, nor why guix's clojure-build-system was written like that.

seancorfield02:01:04

I can sort of understand wanting the clojure/`clj` tool itself as a package that one of these package managers can fetch and install for you, but there's no reason to package Clojure libraries in general.

Alex Miller (Clojure team)02:01:04

I personally just don't get the “rebuild” everything from source obsession

Reily Siegel02:01:10

Yeah, that is the normal way, which is why my first target for packaging was clojure-tools, to allow that use case. However guix likes to have everything packaged in its own system to facilitate reproducible builds of everything. And because guix as a package manager follows GNU guidelines on only providing software built from source, we get into situations where (possibly pre-compiled) maven artifacts are an issue.

Alex Miller (Clojure team)02:01:38

What could be more reproducible than using the exact globally published bits everyone else uses?

seancorfield02:01:04

It's like these package managers are stuck in the 1990s before the Java ecosystem appeared...

Reily Siegel02:01:33

I don't necessarily disagree, it /is/ a GNU project...

Reily Siegel02:01:06

But packaging clojure-tools solves things for 99% of cases, and for when people want to integrate a Clojure codebase with guix tools, they can package libraries themselves.

seancorfield02:01:49

Seems like GNU's guidelines are at odds with how the Java ecosystem works. Does Guix has brew as a package (linuxbrew) that can be installed via the package manager?

seancorfield02:01:38

(b/c once you have brew you can use that to install the Clojure tools directly from the Clojure repos)

Reily Siegel02:01:15

It does not. My patch for clojure-tools (sans S3 transport) was already accepted, Ill just have to send a new patch lifting that restriction once I sort the AOT issues

seancorfield02:01:13

I'm curious about the process for upgrading stuff? It seems like packaging clojure-tools relies on community volunteers and that means they have to repackage every time there's a new release of the CLI?

seancorfield02:01:21

What happens if a community volunteer moves on, or even is off on vacation for a while? Doesn't that just block people from upgrading their clojure-tools in a timely manner?

seancorfield02:01:22

What I like about brew on Linux is that I can install any version of the Clojure CLI, as soon as it is released by the core team -- and I can install prerelease versions just as easily as stable versions, and I can switch back and forth between them easily.

Alex Miller (Clojure team)02:01:42

This is all basically same as Debian, that redoes all this same work with their own slightly different build system that makes slightly different versions of libs too. They chunk it up into periodic release trains

Alex Miller (Clojure team)02:01:00

But what seems weird to me is that once you start using the custom Clojure cli, you are back again to using maven libs etc so …

Jelle Licht18:01:12

Fwiw, reproducible builds are perhaps not the best name for the concept: verifiable builds would perhaps be a better description. But it's a short ddg search away if you want to read up on it https://reproducible-builds.org. If a person does not experience (or care about) a problem, naturally they won't appreciate the trade-offs made to solve the problem. Edit: which is not a dig at anyone or any point made here, but just an almost tautological observation.