Fork me on GitHub
#tools-deps
<
2020-04-24
>
lispyclouds05:04:02

Hello, what is a recommended/nice way of organizing a mono-repo like project of multiple tools.deps backed projects in terms of being able to run all tests and produce JARs/docker images/run tests for each from the root of the project? I have something like this, using Makefiles:

.
├── Makefile
├── docker-compose.yaml
├── consumer
│   ├── Dockerfile
│   ├── Makefile
│   ├── deps.edn
│   ├── src
│   │   └── queue
│   │       └── consumer.clj
│   └── test
│       └── queue
│           └── consumer_test.clj
└── producer
    ├── Dockerfile
    ├── Makefile
    ├── deps.edn
    ├── src
    │   └── queue
    │       └── producer.clj
    └── test
        └── queue
            └── producer_test.clj

dominicm06:04:56

@rahul080327 I'd recommend looking at juxt edge which is my implementation of a monorepo. If you are looking to share makefiles, you probably want the include instruction in make (not shown in edge)

lispyclouds06:04:20

This looks really neat @U09LZR36F! I was considering going the shell approach but was thinking the need for manual orchestration and error handling might be complex hence opted for Make. But will try out your organized approach! Thanks! 😄

dominicm06:04:48

In edge I solve the reuse problem with bin scripts at ../bin/eftest, etc

zane13:04:41

Is it possible to use clj and friends private repositories via a password-protected SSH key?

ghadi14:04:04

there's no interactive ssh with the tools @zane

ghadi14:04:43

ssh-add the key to your running ssh-agent

ghadi14:04:50

then you can run clj

zane14:04:30

@ghadi Yes, I did that but I’m getting an error (`USERAUTH fail`) when I try it with a password-protected SSH key.

zane14:04:20

Works when I use a SSH key that does not have a password.

ghadi14:04:38

if you run ssh-add -l you see the key you intended to add?

zane14:04:24

I believe I added it, yes. Let me run through it one more time to be sure.

zane14:04:47

But it sounds like what you’re telling me is that this ought to work. 🙂

Alex Miller (Clojure team)14:04:59

I don't think it should work

Alex Miller (Clojure team)14:04:23

I don't think password protected ssh keys are supported in the set of stuff we're running

zane14:04:23

Ah, okay.

ghadi14:04:26

totally should work

ghadi14:04:43

it prompts for pw when adding to the ssh-agent

ghadi14:04:50

it's non-interactive after that

ghadi14:04:18

if you see the key in your agent, make sure it doesn't appear in ~/.ssh/config

ghadi14:04:58

JSch, the underlying ssh library, is stupid w.r.t. understanding .ssh/config when there is also an ssh-agent present

ghadi14:04:23

(It prefers to read the key directly from the file, if it knows about it)

ghadi14:04:31

(Rather than consulting the agent)

ghadi14:04:12

So, what's in your ssh agent (`ssh-add -l`) and what's in your .ssh/config?

zane14:04:14

% ssh-add -l
4096 SHA256:… /Users/zane/.ssh/id_rsa (RSA)
% cat ~/.ssh/config         
Host *
  AddKeysToAgent yes
  UseKeychain yes
  IdentityFile ~/.ssh/id_rsa

ghadi14:04:29

ok comment out that block in your .ssh/config

ghadi14:04:39

temporarily

ghadi14:04:56

then run clj again

zane14:04:23

clj succeeds, but I think it’s pulling from a cache.

zane14:04:48

i.e. It’s using the copy of the repository I downloaded when I was using a passwordless SSH key.

ghadi14:04:08

remove that from ~/.gitlibs to verify

zane14:04:10

Trying to remember how to clear that cache.

zane14:04:15

Rad. Thanks.

ghadi14:04:23

should be in two places in there

ghadi14:04:42

~/.gitlibs/{_repos and libs}

zane14:04:54

Any harm in just blowing away ~/.gitlibs entirely?

ghadi14:04:16

dunno how much you have in there

ghadi14:04:27

but anything will be redownloaded if you have creds

ghadi14:04:42

no worse than blowing away ~/.m2/repository

zane14:04:02

Awesome.

ghadi14:04:11

so did it work?

zane14:04:44

Started up instantly again.

zane14:04:49

That’s odd.

zane14:04:00

i.e. I didn’t see it download the private repository.

ghadi14:04:02

clj -Sforce

zane14:04:18

So, what should I do with ~/.ssh/config now?

zane14:04:23

Leave that stuff commented?

zane14:04:44

I confess I don’t understand SSH well enough fully understand the implications of that.

ghadi14:04:12

I don't think in your case there is much harm in removing just the IdentityFile portion of that block

zane14:04:29

Got it. I had hoped that was the case.

ghadi14:04:30

because ~/.ssh/id_rsa is one of the default searched keys

zane14:04:03

But for people with multiple SSH keys that might be an issue, sounds like?

zane14:04:20

(I’m writing up documentation on how to do this for the rest of my team.)

ghadi14:04:59

it depends

ghadi14:04:19

do you have 1 SSH key <> per repo?

ghadi14:04:38

or 1 SSH key that grants you access to a bunch of repos, but you also have unrelated SSH keys present too?

ghadi14:04:54

The former is not supported, the latter is, AFAICT

zane14:04:11

I personally only have this one key, but I imagine others on the team might have more.

zane14:04:59

Also, I just want to confirm one more thing: https://ask.clojure.org/index.php/8725/deps-support-newer-private-file-formats-types-such-ed25519 that clj won’t work with OpenSSH-format keys and that you must use -m PEM when generating them. Is that correct?

ghadi14:04:31

My personal recommendation is to not use files at all, only put your ssh keys into the agent

zane14:04:57

I see. I’ll read up on how to do that.

ghadi14:04:19

clj can use fancy newer keys like ed25519 keys as long as JSch is asking the agent to deal with them

ghadi14:04:35

If JSch is trying to read them from the key files directly -> no bueno

ghadi14:04:07

all of this is a pain IMHO

ghadi14:04:39

github style https auth tokens are a simpler model, and offer finer grained authority

zane14:04:54

But that’s not supported at the moment?

ghadi14:04:55

easier rotation/revocation

ghadi14:04:04

right, not supported

zane14:04:27

This is extremely helpful, @ghadi. Thanks again for your time.

zane14:04:46

I may try to write this up so others can benefit.

ghadi14:04:55

a buddy at Github told me <35% of the auth they see is ssh based

zane14:04:31

Right. I certainly wasn’t using SSH with GitHub until I had to get clj to work with a private repository.

✔️ 4
4
richiardiandrea15:08:10

Sorry to barge in on this one but I was searching for answer on how to use the clj cli (and :git/url) against github private repos Did you end up having that working?

zane15:08:21

The process Ghadi describes worked for me.

richiardiandrea15:08:50

Nice, ok I'am going to read up the history then - thank you for confirming!

zane15:08:00

You’re welcome!

richiardiandrea15:08:05

last q 😄 are Oauth still unsupported?

zane15:08:18

That I don’t know.

✔️ 1
richiardiandrea16:08:18

> The supported git url protocols are https and ssh. https repos will be accessed anonymously and require no additional authentication information. This approach is recommended for public repos. ssh repos may be either public or private.

zane16:08:54

What are you quoting there?

zane16:08:33

I know that there were plans to switch clj over to stop using the JVM implementation of git, which was the root of all of this pain.

👍 1
zane16:08:54

If that migration happened then perhaps https:// works now.

zane16:08:03

You could ask Ghadi.

richiardiandrea16:08:39

I am quoting the official docs https://clojure.org/reference/deps_and_cli Maybe it did not happen yet? let's ask @ghadi what is the status there :D

dominicm14:04:39

Weird, I always considered https the less-secure approach.

ghadi14:04:24

tokens can be ephemeral, ssh identities are long-lived

ghadi14:04:14

good pros/cons ^