Fork me on GitHub
#tools-deps
<
2020-11-06
>
rickmoynihan16:11:10

@seancorfield: would you take a PR to refactor the bulk of this function into a new function that does everything except call System/exit?

rickmoynihan16:11:15

oh actually looks like it’s not system-exit that is the problem… but shutdown-agents

rickmoynihan16:11:12

well the system/exit is also bad as it kills my repl when I make a mistake in the config

seancorfield16:11:21

It calls System/exit in multiple places.

rickmoynihan16:11:33

yeah so I’ve seen

Alex Miller (Clojure team)16:11:41

I often try to separate the shutdown-agents/System.exit type stuff into a wrapper

rickmoynihan16:11:57

that’s what I’m suggesting

seancorfield16:11:37

Yeah, I would normally but I haven't quite gotten there yet.

seancorfield16:11:59

There's also the call to help-and-die in there. I'll give it some thought this weekend.

rickmoynihan16:11:27

shall I make an issue for it?

rickmoynihan13:11:39

Thanks sean, sorry I didn’t get around to this yesterday.

seancorfield20:11:40

@rickmoynihan release 1.1.133 has this refactoring so you can use hf.depstar.uberjar/run* from the REPL now. Same args as hf.depstar.uberjar/run but it returns a hash map for success/failure. The old entry points all still call (shutdown-agents) or (System/exit 1).

seancorfield20:11:51

Follow-up in #depstar if you want.

seancorfield16:11:44

Sure. It'll be a good reminder for me.

Alex Miller (Clojure team)16:11:34

another option is to parameterize the "on-exit" function

borkdude16:11:22

I usually write a -main and main. Main only returns the exit code (or a map with exit code) and -main does System/exit + shutdown-agents

borkdude16:11:30

I then use main in my tests

rickmoynihan16:11:33

yeah that’s the same pattern I do… though this isn’t main but an :exec-fn not that it makes much difference.

rickmoynihan16:11:24

it lets you essentially test command line opts etc

borkdude16:11:04

yes, this is how I test all my CLIs using the exact same input as from the command line

rickmoynihan16:11:43

Whilst I’m hear, as you may have seen I’ve been tweaking deps-deploy to support the new -X stuff. If you’re going to use deps-deploy however; you essentially need to use depstar (or similar), so it’d be good if these tools could be aligned a bit to respect the same keyword vocabulary. How do people feel about that? Similarly in the deps-deploy changes I made I thought it would be nice to use :mvn/version as it means the same as in tools.deps. However I also wanted :mvn/artifact-id and :mvn/group-id etc. Would those terms in principle be the same as tools.deps / tools.build etc might use? Is it legitimate for me to coin keys in the mvn namespace?

Alex Miller (Clojure team)16:11:48

to use in what context?

Alex Miller (Clojure team)16:11:12

like in calling with -X or something?

rickmoynihan16:11:27

yeah precisely, my patches to deps-deploy allow you to override the :mvn/version via :exec-args and the cli, but also :mvn/artifact-id etc…

Alex Miller (Clojure team)16:11:46

probably won't line up to tools.build but I don't know if you should care

rickmoynihan16:11:35

well I don’t know what tools.build does… so hard to know if I should care or not 😀

rickmoynihan17:11:17

I guess I have a slightly deeper question here though… Which is essentially how deep does RDF’s influence over best practice around keywords in clojure go? Rich has mentioned RDF publicly multiple times over the years. And clojure’s namespaced keywords etc are essentially equivalent to predicates etc. The promoted strategy to handling them has been “if they have the same meaning” it’s fine to reuse them (open world, open for extension etc…). However the linked-data world also has a convention of only coining new terms in namespaces you own. However clojure best practice glosses over that here. Does tools.deps own :mvn, is it legitimate for me to coin keywords in that namespace?

rickmoynihan17:11:55

Also is the concept of keyword vocabularies perhaps a useful one?

dpsutton17:11:59

What is "RDF"?

rickmoynihan17:11:01

TLDR; A web-scale datomic-like data model described as a set of W3C standards and practices for how people can build databases to support a global graph of knowledge

rickmoynihan17:11:29

(Also acknowledged by Rich to be a big influence on the design of clojure and datomic too)

Alex Miller (Clojure team)17:11:09

I do not consider tools.deps to "own" :mvn

Alex Miller (Clojure team)17:11:13

qualifiers and their usage are contextual. spec argues for sufficient qualification to create a universe of unambiguous attribute definitions

👍 3
dominicm22:11:40

Fwiw, I think this would be a valuable article on http://clojure.org. Explaining the purposes of namespacing and when you can skip it, when to use #db/id instead of #com.datomic/id Or #datomic/id.

Alex Miller (Clojure team)17:11:01

in more limited contexts (inside your project or inside another appropriately scoped data context), less qualification may be sufficient

Alex Miller (Clojure team)17:11:51

deps.edn has a competing design constraint of wanting to be succinct as this is a format people are frequently writing

Alex Miller (Clojure team)17:11:38

we ended up leaning more on the latter constraint in that balance

rickmoynihan17:11:49

Thanks for that @alexmiller, that’s essentially my perception too. What then do you think of collaborating around common shared definitions like :mvn/version etc and agreeing in principle on other ones to avoid future conflict in areas such as this?

Alex Miller (Clojure team)17:11:17

for the purposes of invoking tools, I don't think there's any need to collaborate

Alex Miller (Clojure team)17:11:34

and I don't see it as conflict

rickmoynihan17:11:58

well it’s not a conflict if I chose :mvn/group-id and you choose :mvn/groupid, but it is irritating if we mean the same thing and our terms flow through the same channels. It would be a potential conflict if we both independently chose :mvn/repository but I defined it as a map, and you a string; and they happened to flow through the same context.

Alex Miller (Clojure team)17:11:17

I don't see them flow through the same channels or exist in the same context, which is why I say this is not a conflict

Alex Miller (Clojure team)17:11:05

really, I'd say neither of these tools "own" :mvn. use a qualifier you own if you want to own it.

Alex Miller (Clojure team)17:11:31

otherwise, don't complain when it conflicts :)

rickmoynihan17:11:54

Well that’s certainly pragmatic. I’ll take that as permission to follow your example and use :mvn/group-id here. I mean it seems sensible to follow similar conventions for human reasons.

Alex Miller (Clojure team)17:11:22

I have no opposition to using that name :)

👍 3
seancorfield17:11:11

depstar's arguments to -main are pretty fixed at this point, and I've just named the :exec-args after the long-form command-line arguments I believe. Changing them would break backward compatibility so I don't want to do that. Re: deps-deploy -- I think it's pretty important that pom.xml be the source of truth for the group/artifact/version -- at least insofar as the version that is included in the JAR and uploaded to Clojars with the artifact -- so I don't think it's a good idea to override the version in deps-deploy and end up with the internal POM not matching the external POM.

vlaaad17:11:01

And then there is me who thinks that pom.xml it's just a temporary file that should be deleted as soon as deployment is done

seancorfield17:11:25

At this point, I would prefer to defer all of this to tools.build but I don't know whether deployment is in scope for that (we have local mvn-install already in t.d.a but no remote deployment).

seancorfield17:11:58

I don't want to have to keep maintaining a community tool if there is an official tool from the core Clojure team that does the same job.

rickmoynihan17:11:31

@seancorfield: yeah that’s fair… I’d much rather overriding the :mvn/version etc happened in depstar. I said this the other day in fact.

Alex Miller (Clojure team)17:11:41

I think deployment is one of many things that could be invoked by tools.build

Alex Miller (Clojure team)17:11:08

whether that's something provided as part of tools.build or community is a separate question. I've looked at it in depth, have a lot of opinions, have not built it.

seancorfield17:11:22

@vlaaad Well, the reality is that there's a lot of stuff in pom.xml that other tooling out there relies on, far beyond the minimum that clojure -Spom produces, so you pretty much have to maintain that information somewhere under version control.

vlaaad17:11:32

Of course, I just think that pom.xml is a horrible storage for this information, since there is a lot of complexity with regards to updating it

seancorfield17:11:58

Oh totally agree! XML is a horrible source format -- but we're "not supposed to use" deps.edn for this stuff, which is where I would otherwise expect it to be... although with :exec-args etc, that's getting more reasonable and "sanctioned" as appropriate by the core team.

seancorfield17:11:45

@rickmoynihan Yeah, there are parts of pom.xml that it would be nice for depstar to be able to override but if you override <version>, you pretty much have to override the SCM <tag> to match, because of tooling that uses that.

rickmoynihan17:11:26

yeah @U09LZR36F mentioned this the other day… that’s trivial to do too though.

seancorfield17:11:24

A guiding principle for depstar is no additional dependencies so the more functionality it accretes, the more I end up rewriting other tooling into the depstar codebase.

rickmoynihan17:11:25

Why no additional dependencies? A stable xml lib would be handy, and tools are usually separate from an apps :deps anyway.

rickmoynihan17:11:50

even tools.deps has dependencies!

rickmoynihan17:11:10

and clojure 🙂

seancorfield17:11:12

Look at the way depstar works and you'll see: it packages up what it finds on the classpath you used to run it, simply excluding itself.

seancorfield17:11:56

That ensures you get exactly what you expect in the JAR based on the aliases you provide, and mean that you don't have to tell depstar itself how to read/combine aliases/`deps.edn` etc.

rickmoynihan17:11:02

It also allows you to set an explicit :classpath though right?

seancorfield17:11:23

Yes, that's an alternative option but I don't want it to be the default.

seancorfield17:11:54

Lots of people seem to struggle with the other JAR-makers out there and end up at depstar because it is simple and does the obvious, intuitive thing.

rickmoynihan17:11:07

sounds more like easy to me… simple would be having the :classpath supplied explicitly.

seancorfield17:11:27

The original version of depstar that I inherited had pretty much no options and just created a JAR from the classpath. It was about as simple as it could be. Adding AOT and main-class and other stuff has made it less simple as a tool, but still easy 🙂

rickmoynihan17:11:49

yeah I understand, and there’s nothing wrong with easy. It just sounds like that’s a requirement for the uberjar side of depstar; but the library side is I think a far simpler problem. Perhaps it should be split into two separate libraries, depstar.jar and depstar.uber?

seancorfield17:11:43

It could be refactored to keep those two things separated. Lots of things could be done with depstar but, right now, I'm waiting to see where tools.build ends up.

seancorfield17:11:17

Like I say, I don't want to keep maintaining a community tool when there's an official core Clojure team tool that does the "same" job.

seancorfield17:11:48

If it seems like tools.build won't handle the cases that depstar can handle, then I'll be happy to invest a lot more time into it.

rickmoynihan17:11:53

that’s fair enough 🙂 but I need solutions now, not in 2+ years 🙂 Being able to build and deploy a library into a private s3 bucket is a common requirement that’s far more awkward than it should be right now.

Alex Miller (Clojure team)17:11:49

the code under -Spom has been refactored to make it more amenable to having more options/features and I think longer term that's going to be something we push to primarily being done as part of a build, with probably ever-increasing scope to refine as part of a build

rickmoynihan17:11:54

Yeah… For my needs I think treating pom.xml as a template, and moving my override feature from deps-deploy into depstar would make most sense.

Alex Miller (Clojure team)17:11:50

there are all sorts of special cases you might want to handle in making a pom for deployment

Alex Miller (Clojure team)17:11:29

many require some kind of data manipulation you'll want to do during a build

Alex Miller (Clojure team)17:11:54

just fyi on tools.build - it exists, it has a lot of functionality but we are trying to finalize a lot of the details around how tasks are defined, stitched together, invoked, etc. b/c that kind of affects everything, we're delaying the public release of it but we'll get there.

seancorfield17:11:08

And I'm still going to bleat on and on about some sort of built-in solution for either the monorepo case or where folks want to combine/insert an extra "master" deps.edn file into the t.d.a mix that everything should then use 🙂

seancorfield17:11:33

(ideally, in t.d.a so that tooling could rely on it easily)

seancorfield17:11:35

Some sort of "include" or "reference" functionality in deps.edn would work for that -- but I think you've said the core team are opposed to going down that path @alexmiller?

Alex Miller (Clojure team)17:11:14

my initial lean would be towards an additional deps.edn source but haven't talked about it enough

seancorfield17:11:17

That would satisfy me too. I'd like to get away from my current CLJ_CONFIG hack, and I'd like individual developers to be able to keep using their own ~/.clojure/deps.edn file for dev/test stuff without us needing to repro a common core set into our "master" deps.edn file 🙂

👍 3
rickmoynihan17:11:40

FWIW I think this would be useful too.

rickmoynihan18:11:22

@alexmiller: Which problem does tools.build solve more of? Orchestrating arbitrary tasks together, or providing ways to build jars, deploy stuff etc? My feeling is that it’s more like make than a way to build jars? Is that right? Or is it both? I appreciate you don’t want to go into too much detail; but it’d be useful to know if depstar etc might be replaced wholesale by it; or whether something like depstar would just integrate with it?

Alex Miller (Clojure team)18:11:57

it is both a framework and a set of built-in tasks, so both, and could probably replace a number of existing tools and/or integrate them

❤️ 6
👍 3
rickmoynihan18:11:01

Ok sorry folks this is going to have continue another time… Many thanks to everyone involved in the discussion. I hope you all have fab weekends. 👋 👋

Alex Miller (Clojure team)19:11:51

I would love to see the built-ins become de facto defaults and make them great with community help

dominicm19:11:45

I'm very curious to see how/if caching is handled. Notoriously difficult task.

vlaaad20:11:43

I would guess caching of task results

👍 3
Alex Miller (Clojure team)20:11:01

if you mean like compiled classes, that's one thing we're still thinking through