Fork me on GitHub
#tools-deps
<
2020-08-21
>
steveb8n01:08:26

Q: Is this a silly idea? I want a combination of git deps and “-LATEST” behviour for my CI/test builds. For prod builds I want static git shas but for automated/test builds, it would be great to automatically use the most recent git sha. Is this even possible?

souenzzo02:08:07

Temporal-basead versions are always bad IMHO If your CI fail and you run it again, it may take a new commit and success again. So success and fail says nothing. Pass the commit ID to your CI or fetch and use a sed / template. Then in case of success/fail, you will know which code succeed/failed

souenzzo02:08:28

And you will be able to run again the same CI in a older commit/retry

steveb8n02:08:56

thx. fetch and sed is an interesting idea. I could even use bb for this script.

steveb8n02:08:36

how would the script apply the latest sha? by writing it into the deps.edn? or is there some other trick?

steveb8n02:08:41

fyi: the reason I want this is to avoid the endless commits I currently do for all my git deps during dev/test. but I really like using git deps in CI

seancorfield02:08:23

Running CI and production against different versions seems like a recipe for hard-to-debug disaster...

seancorfield02:08:35

I can understand wanting your local dev-only tooling to always pick up the RELEASE (or perhaps occasionally the LATEST) release but I really don't want any variance in versions between dev, CI, production for anything I depend on.

seancorfield02:08:01

(so I guess my answer to your actual Q would be "yes, it's a silly idea" IMO)

steveb8n03:08:50

that is useful perspective. I’ll apply a hammock to this a little more. thanks

steveb8n03:08:27

I should have mentioned that these deps are all projects that I control but have split apart for re-use and/or separation of concerns. so I can be sure I know when changes are being introduced. I would never do this for an external dependency

seancorfield03:08:05

Consider using :local/root maybe, if you have them all checked out locally?

seancorfield03:08:54

You can always use a :dev alias to use :override-deps to replace the regular deps with your local ones.

steveb8n03:08:43

agreed. I am using :local/root for these deps for localhost. the annoying commits are required when I push a lib which I depend on in N other projects and then have to update shas in all of them

steveb8n03:08:42

ideally CI/test would use latest for all these deps. then when I cut a prod release, I would update the shas (once) for all the deps

steveb8n03:08:13

but I can see that I could introduce human error later i.e. forget 1 sha update in the prod build. that would be bad

steveb8n03:08:32

that would qualify as “silly”

steveb8n03:08:42

this maintenance problem seems to be pushing me towards a mono-repo but I like keeping these things separate

seancorfield04:08:10

Our 100K+ line codebase at work is in a monorepo, with 30+ subprojects, and we build about a dozen deployment artifacts from that...

steveb8n04:08:44

it’s something I will think about. thanks

borkdude06:08:45

@U0510KXTU I’m using git submodules to prevent endless commits updating git deps. Easy to update them all to latest using one git command. Works with all build tools.

steveb8n07:08:13

thanks. I’ll look into that as well. I’ve never used them before but, if they add no complexity, I’ll give them a try

borkdude07:08:03

@U0510KXTU I'm using them in babashka, I have 4 of them now.

steveb8n07:08:03

cool. I’ll look at your repos when I dig deeper

borkdude07:08:06

Not everyone likes them, I'm not entirely sure why, they work well for me.

practicalli-john11:08:46

I would certainly keep specific library dependency versions the same for any CI pipeline that had any chance of pushing things into production. Are you able to set up separate CI pipeline that uses a different alias to pull 'latest' versions of libraries? I prefer using :local/root rather than mono-repo or sub-modules as they feel simpler for the way I work. Nothing wrong with those approaches though...

borkdude11:08:53

local/root works, but not if the repo you're including is not part of the codebase proper, unless you're writing a script to set up the filesystem in CI the exact same way as on your machine

practicalli-john11:08:05

Is that the case when using jars built from other projects, I thought that worked. Maybe I am mis-remembering.

borkdude11:08:05

I mean, with local/root to some other project, you'll also have to git clone that other project in CI right

steveb8n23:08:37

I feel the same @U05254DQM in that I like them all separate. it has the added benefit of allowing me to outsource work on projects without needing to grant access to all

steveb8n23:08:36

so for me it’s.. local deps using local/root, CI using git/sha and prod using the same git/sha

steveb8n23:08:35

and the maintenance annoyance is updating the git shas for the CI builds anytime upstream projects change.

steveb8n23:08:32

from this thread, I’ve realised that this is also a good thing because it means that my CI builds always match prod builds. thus it’s probably the lesser of all evils, despite the chattiness of commits

rickmoynihan08:08:33

@U04V15CAJ: I’ve done the git submodule thing many times in the past. I’ve found it works quite well if you’re careful and have infrequently changing dependencies, but it can become pretty painful if you change branches a lot and have fast moving dependencies. The detached heads in the submodules can also lead to people unwittingly making many kinds of mistakes.

borkdude08:08:49

git pull --recurse-submodules

borkdude08:08:08

I almost never have such mistakes, but maybe for contributors it's different?

borkdude08:08:49

Wondering, is tools.build also going to have a test runner like https://github.com/cognitect-labs/test-runner or is this a separate concern?

Alex Miller (Clojure team)12:08:36

Not building any new test runner but you can make tasks that do whatever

otfrom10:08:15

Hi, I'm trying to reference the head of the manual-connector-anchors branch on dali here: https://github.com/stathissideris/dali/tree/manual-connector-anchors

otfrom10:08:45

and I'm struggling a bit with the deps.edn. Does anyone have a good example of how to reference a branch on github at a :git/url ?

Alex Miller (Clojure team)12:08:13

Git deps always use shas so you don’t need to specify the branch

otfrom15:08:36

thx! I've got it working now. It was a problem with the deps.edn in the repo I was referencing.

otfrom10:08:36

I have it working locally in the mean time