Fork me on GitHub
#tools-deps
<
2020-01-21
>
avi15:01:36

Has anyone here ever managed to get Snyk to scan a Clojure project using tools.deps for vulnerabilities? I’m giving it a try and Snyk is helping out but I’m running into Maven problems.

avi15:01:45

[ERROR] Failed to execute goal on project project: Could not resolve dependencies for project project:project:jar:0.1.0: The following artifacts could not be resolved: com.gfredericks:test.chuck:jar:0.2.10, clj-commons:clj-yaml:jar:0.7.0, hawk:hawk:jar:0.2.11: Could not find artifact com.gfredericks:test.chuck:jar:0.2.10 in central () -> [Help 1]

avi15:01:03

I thought maybe I was running into this: https://github.com/snyk/snyk/issues/207

avi15:01:31

But I checked and the pom.xml generated by clojure -Spom already includes both Central and Clojars.

avi15:01:55

So I’m stuck…

Alex Miller (Clojure team)15:01:18

could you post the pom?/

avi16:01:19

Just re-running the CI job now to regenerate it.

avi16:01:39

Doh! picard-facepalm repositories is empty

avi16:01:42

Never mind!

avi16:01:23

Or… is there a way to get clojure to populate repositories?

avi16:01:38

I’m using 1.10.1.483 but could switch to something newer

Alex Miller (Clojure team)16:01:06

I'm trying to remember if it's supposed to do that now or not

avi16:01:31

I’m going to try it with 1.10.1.502 and report back

Alex Miller (Clojure team)16:01:32

looks like it is, so might be broken

Alex Miller (Clojure team)16:01:04

I guess it will only right now add repos explicitly listed in the deps.edn

avi16:01:22

I’ve got both repos explicitly in deps.edn

avi16:01:01

I’m testing locally now with 1.10.1.502

avi16:01:24

repositories is empty 😅

avi16:01:55

my deps.edn has this at the root:

:mvn/repos {"central" {:url ""}
             "clojars" {:url ""}}

Alex Miller (Clojure team)16:01:08

ah, it's subtle but I see the problem

avi16:01:53

any possible workaround? Or maybe I should brush up on my sed skills and try to just patch the pom for now?

Alex Miller (Clojure team)16:01:54

for the moment you'll need to manually modify your pom.xml after gen

avi16:01:05

will do — thanks!

avi16:01:16

Would you like a bug report somewhere?

Alex Miller (Clojure team)17:01:27

fixed for next release, but not sure when that will be (and added a test for future)

avi17:01:08

Fantastic — thanks so much!

danielcompton18:01:11

Is there a way to get tools.deps to print out all of the :paths and :extra-paths that a given set of aliases resolve to?

danielcompton19:01:26

-Sdescribe prints out some information, but nothing about the composed config

Alex Miller (Clojure team)19:01:05

not currently, although I've been making my way towards making something like that possible

Alex Miller (Clojure team)19:01:38

I've wanted something like mvn help:effective-pom so that's been a back burner project of mine

Alex Miller (Clojure team)19:01:50

do you care about :paths vs :extra-paths?

Alex Miller (Clojure team)19:01:20

or just "all local paths"

👍 4
danielcompton21:01:51

All local paths for my usage (send them to clj-kondo)

weavejester19:01:50

Does anyone know if there’s an uberjar library for tools.deps that supports merging of files like data_readers.clj?

ghadi19:01:08

depstar does it

ghadi19:01:29

It can be taught to, easily

weavejester19:01:55

I looked into depstar, but it looks like it has hardcoded that part. I could submit a PR, but I was hoping there was something around already.

ghadi19:01:51

have you tested it @weavejester ?

weavejester19:01:42

Yeah, that’s what I mean by it being hard-coded, @ghadi.

weavejester19:01:56

Notice data_readers.clj is explicitly checked for.

weavejester19:01:16

And clash-strategy doesn’t look like it can be overridden.

ghadi19:01:28

oh you meant other files, not that one in particular

ghadi19:01:30

not extensible as written

dominicm19:01:27

@weavejester pack avoids merging altogether.

dominicm19:01:51

Capsule is the one that works with most of your libraries :)

seancorfield20:01:40

@weavejester Happy to entertain PRs on depstar to extend the clash handling logic. I inherited that as-is and haven't enhanced that part yet...

weavejester23:01:04

Are there any disadvantages to Capsule jars? I guess I’m inherently wary of anything that builds on known deployment mechanisms.

dominicm07:01:02

They use a little bit more memory because they start trampoline jvm, much like lein does

dominicm07:01:31

Onejar is a little more special, that uses a class loader to do its work

dominicm07:01:54

You might also want to consider the docker mode, which will directly create a docker image.

tianshu06:01:14

@U09LZR36F can I package a clojure project into a docker image?

tianshu08:01:32

@U09LZR36F can you share your workflow?

dominicm08:01:37

@U0NBGRGD6 My understanding is that you provide a docker registry and it will upload it there. People are using it with Google.

rickmoynihan11:01:38

@weavejester: Yes there are definite disadvantages to capsule jars… Also they extra the jars contents into a cache to build the classpath. You also get a jvm dance at startup; and you then need to care about proxying various JVM options to the real app vm; i.e. if you want to set the heap etc. I’ve been meaning to look at onejar as an alternative. FWIW in spite of the occaisional frustration we currently avoid the issue you’re on about with capsule jars. I think there’s an argument that merging the files has problems too; it would be better to resolve all of them at runtime; i.e. uberjars are really just a hack. I just wish the JVM platform itself supported this.

rickmoynihan11:01:27

It might just be better to zip up the classpath and write it in a shell script that starts the jvm tbh.

rickmoynihan11:01:14

IIRC I wrote this stuff up on a duct issue somewhere… 👀

rickmoynihan12:01:10

Also lein-tools-deps and using lein for just the packaging and deployment may also be a solution for you. (Full disclosure: I wrote it)

dominicm14:01:32

If I had control over how people deployed, I'd tell them to use the skinny mode, aka, dump my project into a folder

dominicm14:01:06

There's an idea on what you just said though

dominicm14:01:45

Zip will ignore the prefix leading up to finding the header. So you can have zips which are scripts.

dominicm14:01:46

Unzip itself to a cache directory, and pass all arguments to the jvm