Fork me on GitHub
#tools-deps
<
2021-07-09
>
Alex Miller (Clojure team)17:07:41

Happy to announce a pre-release of the Clojure CLI 1.10.3.905 and tools.build - read lots more here: https://clojure.org/news/2021/07/09/source-libs-builds This is being provided for feedback and tire-kicking and there may be changes before it becomes a release. Notably if you have been following the pre-releases, we've removed the multi-function execution support.

🎉 14
😲 2
sheepy 2
Alex Miller (Clojure team)17:07:14

that's likely coming back, but might differ a bit

dpsutton17:07:37

thank you for releasing with the git history rather than copied to a new repo and everything "Initial Commit"

2
Alex Miller (Clojure team)17:07:44

it's a tortured road ... :)

dpsutton17:07:06

but its nice to see the thought in it. love reading how well-thought-out programs evolve to do things rather than just being presented wholly formed

borkdude18:07:59

@alexmiller Congrats on the release. I think this implementation in process can be problematic when the buffer of either stdout or stderr is full and the process will wait for those streams to be consumed. Afaik this is why clojure.java.shell does that copying in futures while the process is running, rather than letting it complete beforehand. https://github.com/clojure/tools.build/blob/master/src/main/clojure/clojure/tools/build/tasks/process.clj#L68-L70

borkdude18:07:55

And btw, the defaults are ProcessBuilder$Redirect/PIPE anyway, so no need to set those explicitly

Alex Miller (Clojure team)18:07:45

thanks, need to set up a test for that

borkdude18:07:40

Just try to make the process print a couple of megabytes of output

borkdude18:07:50

a few 100kbs should already be enough

borkdude18:07:29

in combination with the :capture or :ignore option

seancorfield18:07:26

@alexmiller In the tools.build guide, it says:

Invocation that activates :dev environment will look like this:

clj -X:build jar :env :dev

The kv params are passed at the end and will be used to invoke the first function, clean, then the return of that will be passed to jar.
but clean is not being invoked here. I assume at some point it said clj -X:build clean jar :env :dev but the multi-function execution stuff got pulled?

Alex Miller (Clojure team)18:07:10

Yeah, probably missed a couple places

seancorfield18:07:51

(we're already using multi-function execution with -X at work so I'm not sure what we'll do regarding tools.build -- we will most likely hold off testing it until multi-function execution comes back in a later prerelease)

Alex Miller (Clojure team)18:07:46

you're bleeding on the bleeding edge :)

3
💯 3
seancorfield18:07:29

Well, I was so excited about the multi-function exec stuff that we adopted it immediately 🙂

Alex Miller (Clojure team)18:07:00

it'll be back probably, just need to work out some of the potential ambiguities in arg parsing

2
seancorfield18:07:21

When I'm back at work (next Wednesday), I'll take a look at how/if it will affect us. I think, since we switched our build shell script to a build.clj Clojure script, we may have switched back from -X to -M until we knew what tools.build etc was going to look like in the wild, and I was only using multi-function -X from the CLI directly, rather than in our build system...

dominicm20:07:35

@alexmiller If I'm reading the source correctly, the uber task clobbers duplicate files. That means data_readers.cljc won't work correctly, and also that the apache-2.0 licence isn't followed (they require preservation of the licence). I think @seancorfield solved this by having merger functions which can be registered in some way for different matches of files.

dominicm20:07:46

ah, i did miss that then 😛

dominicm20:07:25

The LICENSE thing is still a problem though.

seancorfield20:07:28

Yes, depstar does all sorts of stuff to ensure merging of various files properly (including the log4j2 plugin cache). I plan to offer near drop in replacements for jar/uber soon.

dominicm20:07:55

I should probably get pack up to snuff with the new tools.build too.

dominicm20:07:38

It's been rotting while I wait for tools.build, so I could clarify the interfaces. But there's a parallelized jar generation tool in pack too which is probably worth pulling out independently.

Alex Miller (Clojure team)20:07:57

yes, this is an area that I got to a point but would like to take further

dominicm18:07:33

You probably know this, but the maven shade plugin implements a lot of this already. Worth looking through for what's worth stealing at the least.

Alex Miller (Clojure team)18:07:25

everything in tools.build is a clean room impl and I have largely avoided looking at other existing tools

Alex Miller (Clojure team)18:07:52

I'd rather think more about what we should be doing

dominicm20:07:19

I'm a little unsure about this :ensure key - it seems at risk of proliferation. e.g. what if I want to separate out my prod/dev builds into target/prod and target/dev so that I can build cleanly for production without any dev files leftover in there (short of cleaning, because then that breaks my background clojurescript build!). Different libraries/companies are going to have different ideas of where built output should live. I understand the purpose is to ensure the classpath is pre-populated, but I'm not sure this is the best overall solution. Could the folder be a param somehow to the library being prepped?

Alex Miller (Clojure team)20:07:59

can you talk more about the differences between dev and prod builds?

seancorfield20:07:22

How would that be affected by a third-party library dependency that needs the CLI to run some task before the library could be used after checkout?

Alex Miller (Clojure team)20:07:41

is that a question for me or dominic?

seancorfield22:07:27

(for Dominic, sorry)

dominicm18:07:34

Reading the following messages around it being local to the dependency, my concerns are allayed. I was thinking of application-specific dev vs prod such as different dependency versions or similar.

seancorfield20:07:32

@dominicm My reading is that it is local to that dep: for the CLI tools stuff to "prep" a checked out source lib before use the first time. So the :ensure directory is going to be within the .gitlibs checkout (only) and is going to be in the :paths vector for that dependency.

Alex Miller (Clojure team)20:07:31

git libs are per-SHA source checkouts and the assumption is that there is one way to prepare it to be on the classpath and if you do it once, you are done. (nothing is going to change in that sha)

Alex Miller (Clojure team)20:07:21

moreover, all builds on the machine are going to share the sha output, so parameterization is problematic (without handling this more explicitly)

hiredman20:07:34

that seems like it will run afoul of projects doing aot compilation with different versions of clojure?

Alex Miller (Clojure team)20:07:05

aot compilation is still something best done when you compile your app

hiredman20:07:21

ah, I see, prep is more for compiling java

Alex Miller (Clojure team)20:07:29

prepping is intended for things like java compilation, resource copying/filtering, etc

Alex Miller (Clojure team)20:07:35

the advice in https://clojure.org/guides/dev_startup_time doesn't change if you want to do dev time aot

seancorfield22:07:35

@alexmiller Just to let you know: I switched depstar over to use t.d.a/`create-basis` and it does exactly what I need, so I can remove all my custom basis code now -- thanks!

💯 3
seancorfield23:07:02

depstar now has CLI tools support, using the latest prerelease:

clojure -Ttools install com.github.seancorfield/depstar '{:git/tag "v2.1.250"}' :as depstar
and then you can do
clojure -Tdepstar jar :jar MyLib.jar

💯 3
catjam 3
seancorfield23:07:46

(it has :tools/usage {:ns-default hf.depstar} so that command invokes hf.depstar/jar)

seancorfield23:07:33

clj-new also has CLI tools support now:

clojure -Ttools install com.github.seancorfield/clj-new '{:git/tag "v1.1.314"}' :as new
# create a new app:
clojure -Tnew app :name myname/myapp
# create a new library:
clojure -Tnew lib :name myname/mylib
# create a new template:
clojure -Tnew template :name myname/mytemplate
# create a new project from a public template:
clojure -Tnew create :template electron-app :name myname/myelectron-app

Derek16:07:28

Waiting to publish to clojars?

seancorfield18:07:26

@UPEKQ7589 It doesn't need to be on Clojars for this: CLI tools takes it from GitHub.

Derek18:07:28

Thanks Sean. Very cool

seancorfield22:07:58

(I put it up on Clojars anyway, just now, for folks who are using it via deps.edn)

👍 3
seancorfield23:07:07

@alexmiller One problem I ran into: if you use the new CLI tools stuff to install a tool that depends on an OLDER t.d.a, because the EDN for the tool has :git/sha in it but not :sha, when the tool is run, the older version of t.d.a trips over the lack of :sha. You can see this if you install clj-new 1.1.309 instead of 1.1.314 and then try that last line:

(! 1063)-> clojure -Ttools install com.github.seancorfield/clj-new '{:git/tag "v1.1.309"}' :as new
Installed new
(! 1064)-> clojure -Tnew create :template electron-app :name myname/myelectron-app
Unqualified function can't be resolved: create
# ^ expected because 1.1.309 doesn't include :tools/usage
(! 1065)-> clojure -Tnew clj-new/create :template electron-app :name myname/myelectron-app

Failed with: Library com.github.seancorfield/clj-new has missing :sha in coordinate.
# ^ this is "unexpected" but happens because clj-new 1.1.309 depends on t.d.a 0.11.910 which doesn't recognize :git/sha
Execution error (ExceptionInfo) at clj-new.helpers/resolve-remote-template (helpers.clj:186).
Could not load artifact for template: electron-app
	Tried coordinates:
		[electron-app/boot-template "RELEASE"]
		[electron-app/lein-template "RELEASE"]

For more detail, enable verbose logging with :verbose 1, 2, or 3

Full report at:
/var/folders/p1/30gnjddx6p193frh670pl8nh0000gn/T/clojure-7876746592438264246.edn