Fork me on GitHub
#depstar
<
2021-07-14
>
seancorfield05:07:02

https://github.com/seancorfield/depstar/releases/tag/v2.1.267 -- the README has been split up into proper documentation https://cljdoc.org/d/com.github.seancorfield/depstar/2.1.267/doc/getting-started and several new sections added (I'm particularly interested in feedback on the new documentation!).

Eddie05:07:42

This documentation is excellent! It sounds silly, but breaking it up into pages is somehow cognitively easier to digest. Also, the new comparison/integration with tools.build is very useful.

seancorfield05:07:34

Thanks, @U7XR2PZFW -- at some point I need to split clj-new's docs up in a similar way (esp. since I just added a small piece about clojure -Ttools to that readme!).

Karol Wójcik13:07:46

Would it be technically possible to native-image compile depstar? I'm tempted to do it

seancorfield16:07:10

I've no idea and I have no interest in it -- I'm more concerned with the CLI tools functionality -- but if you, or someone else, wants to try producing a native image and can produce a script that will run it all in Docker images (so my machines don't need anything extra installed), I'd be happy to incorporate that into depstar's assets on GitHub. I don't know how @U04V15CAJ handles this for his tools?

borkdude17:07:46

@UJ1339K2B in babashka there is an older version of depstar that I use to make uberjars for bb projects itself (you invoke it via bb uberjar). This was before depstar switched to including tools.deps. Now that it includes tools.deps it's a bit harder, but not impossible.

Karol Wójcik17:07:58

Which version of depstar is available in babashka? I really don't care much about new features 😄

borkdude17:07:43

well, all bb uberjar does is combine files into a .jar file.

borkdude17:07:44

it's not really a specific version, more like a snapshot of which I branched of at a certain point

Karol Wójcik17:07:50

Is depstar somehow exposed via public api in babashka?

borkdude17:07:02

I'm not maintaining it for outside usage, for clojure project usage, no.

Karol Wójcik17:07:35

I don't really understand the difference between deps.clj and tools-deps

borkdude17:07:46

Why do you want to have this native? If you want to compile Clojure files to .class files, you need a JVM anyway

Karol Wójcik17:07:29

I got a lot of integrations tests which require Clojure compilation. Startup of depstar is like 4-5 seconds which makes the pipelines quite slow in the end.

seancorfield17:07:53

Why are you running tests after building JAR files?

Karol Wójcik17:07:16

Because I don't test Clojure code 😄 I test graalvm compiled binary

seancorfield17:07:42

And, to answer your question about an API, you can use depstar as a library already (in fact the latest docs should examples of using depstar in a build.clj script).

borkdude17:07:58

I bet the startup of depstar isn't caused by depstar but by your transitive deps

borkdude17:07:56

at least, could be

Karol Wójcik18:07:14

Kill me please guys

Karol Wójcik18:07:17

I'm stupid man

Karol Wójcik18:07:25

I did some profiling and I was suprised that regular depstar takes 6s to compile while on wrapped depstar in my script takes like 13 seconds. Turns out that I should not remove .cpcache 😕 before running depstar.

Karol Wójcik18:07:42

I'm really sorry for taking your time @U04V70XH6 @U04V15CAJ. I really do!

2
hansbugge13:07:28

In hf.depstar.task/preprocess-options there is a call to t/create-basis that gets invoked even if you have a :basis in the options map. I cannot work out if that is intentional or not.(https://github.com/seancorfield/depstar/blob/4e9964d5e56c384dad11bf2bf5809891e490db00/src/hf/depstar/task.clj#L23)

hansbugge13:07:14

My problem is that I'm trying to pass a bespoke basis created with (t/create-basis {:extra {:mvn/local-repo "some-project-local-folder"}}) in order to do some dependency caching in a CI environment. But that call to t/create-basis means that all deps are downloaded into ~/.m2 every time anyway.

hansbugge13:07:10

> If :basis is provided in the initial options, that is returned as-is instead of calculating a new project basis. from docstring of options-and-basis indicates that it is a bug. > Given an options hash map, if any of the values are keywords, look them up as alias values from the full basis (including user deps.edn) from docstring of preprocess-options indicates that it is not.

hansbugge13:07:38

In case it is a bug, I've opened a PR which hopefully fixes it: https://github.com/seancorfield/depstar/pull/96

seancorfield16:07:57

Hmm, interesting. The reason for that specific call is to get all available aliases for resolving aliases-as-data so it takes your user deps.edn into account whereas the other two uses (overall basis and compile-specific basis) do not. I think what I'll do here is revert to just reading and merging the EDN files (and not actually compute the basis completely). BTW, you specify :compile-aliases it will calculate a new basis, just for compilation.

hansbugge18:07:35

But with :compile-aliases there's no way to add an option such as :mvn/local-repo to the basis, right?

seancorfield18:07:18

That would need to be in your deps.edn as things stand today. If you don't need compilation to use a different basis to the one used for building the JAR, it wouldn't matter.

hansbugge19:07:09

I think I will end up putting it in deps.edn as you say, thank you. It seems a bit odd to me that the path to the m2 cache is so relatively inflexible – it can't be changed with aliases for example. With lein it could be configured with profiles. With maven it can be controlled with -Dmaven.repo.local. But that's because of tools.deps.alpha, not depstar, of course.

seancorfield19:07:45

By default, whenever depstar builds a basis itself, it does it from (just) the project's deps.edn file. If you feel you will need a custom basis for compilation, separate to the project's basis, open an issue requesting a :compile-basis option and I'll add it.

hansbugge19:07:50

I don't think that's what I want – all I want is to avoid downloading dependencies over and over again in my CI pipeline, and use the cached dependencies instead. It seems to me the best way to ensure that right now is to add :mvn/local-repo to deps.edn, which means that everybody needs to store the dependency cache in the same place as the CI machine.

hansbugge19:07:42

I can call depstar with clojure -Sdeps '{:mvn/local-repo ".depscache/m2"}' -X:depstar/uberjar [...], but that'll only mean I pull depstar from the cache, not the deps used for the build.

seancorfield19:07:34

Why can't you just cache the CI's ~/.m2 folder directly? That's what we do in CI.

hansbugge19:07:25

Yes that would be the easiest, but as far as I know that's not possible. It's Gitlab CI btw, with the docker runner.

seancorfield19:07:37

Hmm, BitBucket Pipelines lets you do it -- and it's pretty primitive compared to most other CIs 🙂

hansbugge19:07:17

It probably has to do with the way it runs the build with docker – I guess the runner mounts the project dir in the container, and only has access to that dir, and the rest gets wiped when the ephemeral build container is removed

seancorfield20:07:32

Could be. ISTR GitHub CI is also a bit like that. I haven't investigated any caching for my OSS projects yet...

seancorfield01:07:13

OK, depstar develop HEAD has the fix for that if you want to try it via :git/url.

hansbugge15:07:09

Awesome, that works 🙂