Fork me on GitHub
#tools-deps
<
2019-05-24
>
mynomoto00:05:51

I generated a pom for my project using -Spom so I could use dependency-check-maven to find vulnerabilities on dependencies. But since the way maven and tools.deps solve dependencies I got discrepancies on the dependency tree calculated by maven and by clj. Could -Spom return all dependencies including indirect ones or have another command that could do that?

Alex Miller (Clojure team)00:05:16

the trees are really different because mvn and clj resolve dependencies differently. which one do you actually care about?

Alex Miller (Clojure team)00:05:46

you can grab the actual full set of deps clj is resolving (as data) by looking at the .libs file that's generated

Alex Miller (Clojure team)00:05:43

if you do clj -Sverbose you'll see a cp file like .cpcache/723570742.cp - if you replace .cp with .libs, that's the file

Alex Miller (Clojure team)00:05:51

that's an edn file, so you could suck it in with clojure and manipulate it however you need

mynomoto00:05:56

I care about the clojure one, since that will go to production, but the tool to check dependencies uses the pom information.

mynomoto00:05:41

So reading the .libs file and generating a pom.xml with all dependencies should work.

seancorfield01:05:08

@alexmiller Does that mean that deps-based projects that use clj -Spom to generate/maintain a pom.xml that is published to Clojars can cause projects that use them to get the "wrong" dependencies?

Alex Miller (Clojure team)01:05:30

the generated pom should have only the same top level deps

Alex Miller (Clojure team)01:05:48

my understanding of the request above is that he is scanning the full transitive dep tree, which may indeed be different between clj and (mvn or lein)

mynomoto01:05:19

This is accurate.

Alex Miller (Clojure team)01:05:51

these differences are presumably mostly same libs but different versions, but in cases where transitive deps change, could be different deps

Alex Miller (Clojure team)01:05:46

in theory, that may be an issue. in practice, it seems to rarely matter, perhaps because most libs are subsequently used as someone else's lib, and then lose all control over their dep versions anyways.

seancorfield01:05:45

Thanks for the clarification!

mynomoto02:05:47

I have mixed feelings about my experience with the classpath differences today. lein and boot do the same as Maven in resolving transitive dependencies so this is a tools.deps only problem. If -Spom is supposed to create a classpath for libraries it's doing fine with the caveat that if you want to analyze the tools.deps library classpath using the generated pom using Maven tooling you can get results that are not real, but as Alex pointed out, in the end is the application classpath that matters. In the end I think that having an easy way of generating a pom.xml that will create the same classpath as tools.deps is important because of tooling that already exists for Maven.

mynomoto02:05:35

dependency-check-maven plugin pointed several high/critical vulnerabilities on dependencies used by pedestal and buddy mentioning only libraries that I think are popular in the clojure world. I had no time to check if those could be exploited. This is on the latest release of both libraries. The dependencies have new releases not affected by the vulnerabilities.

Alex Miller (Clojure team)02:05:29

my limited experience with dependency-check-maven and the underlying stuff is that the matching is very bad and often highlights libraries that have nothing to do with the CVEs it associates with it

Alex Miller (Clojure team)02:05:31

an example is https://github.com/jeremylong/DependencyCheck/issues/1788 which flagged data.zip afaict because it had "zip" in the name and that matched a totally unrelated "zip" project. I dug into the code just enough that I have a really low confidence in the results of this tool. So, fyi.

mynomoto02:05:05

Having spent a fun afternoon reading cves, looked legit. Cves were very specific on coordinates and versions affected.

Alex Miller (Clojure team)02:05:17

if you have a list of issues, would be great if you could just drop it somewhere so we could do necessary stuff to improve things

Alex Miller (Clojure team)02:05:56

issues on specific tools would be better, but if you've at least got a list, that would make it faster for someone else to log if you don't have time

Alex Miller (Clojure team)02:05:58

on the suggestion re generating a pom, the goal was to create a pom that was similar to the deps.edn, which meant just the top level deps. Stating all lower level deps in the pom would be (conceptually) much different. That's more than I'm planning to take on for -Spom (which itself may get pulled out of tools.deps/clj and put into a standalone tool)

Alex Miller (Clojure team)02:05:57

making a tool that used tools.deps and a deps.edn and did the dependency checks directly without involving Maven would be even better

Richard Gould01:05:36

We've written something internally that we've been using for a few months to do this. Have just gotten clearance to open source it, hope it's useful. Available now at https://github.com/ROKT/check-deps @U9Y2R7YAV

mynomoto02:05:13

I understand the trade-offs of generating a pom similar to what other tools generate. But that comes with the inconvenience of having different classpaths on different tools because the algorithm used for transitive dependencies is different and the results are different than one would expect. While a tools.deps specific dependency check tool would be indeed better that would mean another tool that someone would have to maintain. Using one that already exists and is maintained has lots of advantages too. I will rerun the analysis without my dependencies overrides and report on each tool tomorrow.

Alex Miller (Clojure team)02:05:32

thanks, that would be great

Alex Miller (Clojure team)04:05:32

looks like good reports, thx!

wei09:05:27

is there a good way to share code between two tools.deps projects living in the same git repo?

dominicm09:05:01

@wei use a local dependency when developing them, I think that's what you're asking.