Fork me on GitHub
#tools-deps
<
2018-10-24
>
seancorfield03:10:28

I gave in and put seancorfield/clj-new and seancorfield/depstar on Clojars. Releases 0.5.4 and 0.1.2 respectively. JARs built with (my fork of) depstar. Initial pom.xml built by clj -Spom. Pushed to Clojars with mvn deploy:deploy-file (ugh!).

dominicm17:10:59

Topic of today. I want multiple aliases which compose and start multiple things. The goal would be using able to start a program like normal, but perhaps side load nrepl in. Best idea I have so far is a custom main and jvm options. The custom main would evaluate code based on the provided options, and then finally run the -m it was provided. Any other suggestions for ways to interface with such a thing? I did consider taking a list of mains, but then you have to detect when they are done in an arbitrary way, but that might be fine by looking at the thread state? Also main-opts don't concatenate.

codonovan09:10:01

Yes. I think we had to use lein-figwheel "0.5.16" and figwheel-sidecar "0.5.14" to get it working. https://github.com/drapanjanas/re-natal/issues/190 might be helpful

xfyre15:10:11

Interesting, let me check this in Xcode. I was experiencing reload issues before I even tried to upgrade Figwheel

xfyre15:10:51

I’m seeing

2018-10-25 18:34:41.958455+0300 DcnetApp[90232:23128804] [] nw_socket_handle_socket_event [C224.1:1] Socket SO_ERROR [61: Connection refused]

seancorfield18:10:52

@dominicm I thought about this long and hard as we switched our dev/test/build pipeline over to clj from boot -- and ended up with the "custom main and jvm options" approach, with several small "microprojects" containing the various -main we want that are added via aliases... :main-opts don't concatenate for a (good) reason but that does sometimes make it hard to "compose" multiple tasks in a "build pipeline". Our compromise there was to create a build shell script that runs clojure for each set of command line arguments (loosely... it implements a sort of mini-DSL around aliases and subprojects to make it easy for us to kick off a sequence of tasks from a single command line).

seancorfield18:10:55

(as of yesterday, we no longer use boot for any of our dev/test/qa/prod stuff -- with two legacy exceptions, one of which I'm working on right now 🙂 )

dominicm19:10:02

@seancorfield Just so I am clear, you have a custom main which calls other -main? Or do you look at JVM opts in order to determine what to start in your one custom main?

seancorfield19:10:53

"It Depends". We have some -main functions that use the supplied command line arguments to decide what to run (namespace-qualified function names, usually). But also, now we have our little build wrapper shell script, we often just run clojure multiple times with different (main) options...

seancorfield19:10:24

But the former approach could easily use JVM options instead of command line arguments. Or as well as.

seancorfield19:10:20

(the main issue is determining what arguments -- command line or otherwise -- to call each of the functions with, based on just a single -main entry point and its own command line arguments)

arrdem19:10:03

it’d be interesting to have a “multi clojure.main” that took say an EDN array of mains and ran each one in its own future / send-off.

arrdem19:10:22

or let you have “mixins” that get sent off and one “main”.

dominicm19:10:12

@arrdem I'm toying with that idea. The hard limitation being that if you actually want the main to run once and then block, it's more difficult. Also, if you want ordering (e.g. producing some state picked up for the next main), then you have to "detect" when the main is done.

arrdem19:10:52

Concatenating programs into a single JVM seems like … a job for a real program not some crazy main. We have a thing which does that, it’s a REPL 😉

arrdem19:10:55

For Katamari I’ve been toying with the idea of how to let kat reuse a JVM between taks if your dependencies haven’t changed. Not sure you can actually blow away enough Clojure state (in an unmodified Clojure) to do that safely or if it buys you anything.

dominicm19:10:23

I'm trying to leverage existing standards (e.g. -main). But I agree. I'm feeling like I might have to bite the bullet and "force" something new onto people for daemons like nrepl.

arrdem19:10:07

Yeah. Remote daemonized nREPL servers with a client hooked to your terminal is the best I’ve got for stuff like this.

arrdem19:10:54

I need to take a poke at pack.alpha, looks like it may provide more of the machinery I want for Katamari’s jar, uberjar and maybe pom targets than depstar does.

dominicm19:10:02

@arrdem I'm trying to move towards having some kind of API for consumers. But I've prioritized -main support for now. I'm very welcome to improvements to the "api" 🙂

arrdem19:10:37

@dominicm cool - I’ll see what I find 😉

arrdem19:10:54

kat is already a pile of alpha APIs what’s one more.

arrdem19:10:52

@seancorfield curious what wound up in your build script.

seancorfield19:10:43

@arrdem Well, our basic command to run anything is essentially

cd <repo>/clojure/<subproject>
CLJ_CONFIG=../versions clojure -A:defaults:<alias> <some> <args>
so mostly the build script maps pairs of <task> <subproject> on the command line into runs of that command pair.

seancorfield19:10:58

Since arguments are inherently variadic, we treat <task> <subproject> as a command with no args and [ <task> <subproject> <some> <args> ] as a command with some args. So build parses the [ and ] delimiters around commands.

arrdem19:10:27

Interesting.

seancorfield19:10:45

Plus a couple of convenient conventions: <task>+test means "map <task> to its alias(es) and also add :test:test-deps; and <task>s means "run <task> for each of the (subproject) arguments provided" -- so you can easily build multiple uberjars or run multiple subprojects' tests.

seancorfield19:10:38

Oh, and <task>-all which expands a list of subprojects to include all the others (recursively) that are pulled in as :local/root deps (so we can easily "test a subproject and everything it depends on")

seancorfield19:10:20

Very minimal really. And of course we could do it all from the command line but... we're lazy programmers...

seancorfield19:10:31

Everything beyond that "DSL" is handled with aliases (and command line arguments passed to clojure).

arrdem19:10:03

Hum. Do you just have standardized eg /test paths for the -all alias or how did you get dependees tests recursively included.

seancorfield19:10:04

A monorepo with subprojects all side-by-side in the clojure directory, each with src and test folders beneath them. Is that what you meant?

arrdem19:10:26

Cool, thanks for the info!

arrdem19:10:36

Interesting to see what other folks are coming up with.

seancorfield19:10:09

We deleted most of our 2,000 line build.boot file yesterday 🙂

seancorfield19:10:28

I expect to delete almost all of the rest of it by week's end.

dottedmag20:10:33

@seancorfield How big is the replacement?

seancorfield20:10:39

A very well commented and spaced out bash script with about 70 lines of help output -- total just under 300 lines 🙂

dottedmag20:10:09

So, 2000 lines of build.boot was just a dispatcher?

seancorfield20:10:24

No... it's kinda hard to explain 🙂

seancorfield20:10:26

We used to dynamically determine cross-subproject dependencies using tools.namespace -- seeing which namespaces a subproject required and which subprojects those were actually defined in. All that is gone and replaced with explicit :local/root dependencies now.

seancorfield20:10:20

Plus we had accrued a lot of now-unused build tooling as we evolved the project over several years (we switched to Boot at the end of 2015). We had a lot of per-subproject-specific stuff in the Boot file which was never made generic (which the shell script does).

seancorfield20:10:59

And, lately, this Boot bug was killing us https://github.com/boot-clj/boot/issues/715

seancorfield20:10:28

Boot had encouraged us to build complex tooling because it was easy to do so. The reason we switched from Leiningen to Boot three years ago was because we wanted to do more programmatic stuff -- which was much, much easier in Boot -- but we sort of got carried away 🙂 And moving to clj and deps.edn and aliases kind of focuses the mind on what's actually required.

dottedmag20:10:53

It reminds me of one of build steps (implemented in Go, to boot) I'm guilty of writing: it extracts from a Go binary a list of versions of packages it was built from, parses list of packages current module is dependent upon, and then decides whether that binary needs to be rebuilt. And to add to the insult, the list of binaries to check is created by parsing a Go source file.

dottedmag20:10:00

Very nifty, cuts down maybe 10s from the full build, but it's more than 300 lines of code just for that.

seancorfield20:10:55

Ah, yes, an easy trap to fall into 🙂

arrdem20:10:38

Curious what makes you see that as a trap.

seancorfield20:10:51

When your build is "just code" it's easy to just start writing code -- without thinking about whether it is a) needed or b) an over-complicated way to achieve your goal.

dottedmag20:10:05

Also this code is usually easy and fun to write, so one can waste enormous amount of time on it.