Fork me on GitHub
#tools-deps
<
2018-08-02
>
martinklepsch16:08:58

Are there any restrictions to where tools.deps will look for the :sha provided?

ghadi16:08:24

what do you mean?

Alex Miller (Clojure team)16:08:37

where else would it look other than the git repo?

martinklepsch16:08:56

clj -Sdeps '{:deps {org.clojure-grimoire/lib-grimoire {:git/url "" :sha "767d68f605e5ec2ab740ae42cfc7b90a2948713c"}}}'
This fails with Manifest type not detected when finding deps for org.clojure-grimoire/lib-grimoire in coordinate {:git/url "", :sha "767d68f605e5ec2ab740ae42cfc7b90a2948713c"} In a more involved setup it fails with the following error/stacktrace :

ghadi16:08:35

the stack trace is a bit revealing... did you make edits to the file in .gitlibs/libs/... ?

Alex Miller (Clojure team)16:08:49

767d68f605e5ec2ab740ae42cfc7b90a2948713c doesn’t look like a commit in https://github.com/martinklepsch/lib-grimoire ?

ghadi16:08:02

I cloned and found that commit

Alex Miller (Clojure team)16:08:24

oh yeah, weird - github didn’t find it

ghadi16:08:18

(We should consider making the checkout dirs readonly)

dominicm16:08:36

@ghadi how do you make dirs read only? 😄

dominicm16:08:45

I guess you can chmod on linux, to prevent self-harm

Alex Miller (Clojure team)16:08:09

I got your second error trying to run your first command

martinklepsch16:08:00

I didn’t do any manual edits in .gitlibs (as far as I’m aware at least 😛)

ghadi16:08:54

I found it super easy to accidentally jump to source in Emacs to a gitlibs SHA dir.

ghadi16:08:11

make edits, hit save, break Content-Addressing guarantees

dominicm16:08:05

I've done the same thing to real jars

dominicm16:08:17

I had to uninstall zip so vim wouldn't let me :w inside a jar.

🙃 4
dominicm16:08:24

I should fix it properly

Alex Miller (Clojure team)16:08:14

I think the second error is the real error (the MissingObjectEx)

Alex Miller (Clojure team)16:08:32

and then after that, it looks like you have a checkout directory, but it didn’t finish so it’s missing files (like the manifest files)

martinklepsch16:08:35

I don’t even know how to jump to source so guess I’m safe 😄

martinklepsch16:08:47

should I try nuking .gitlibs?

Alex Miller (Clojure team)16:08:58

I think you’ll see the same problem again

ghadi16:08:02

no, not if Alex repro'ed immediately.

martinklepsch16:08:22

right, fair enough 🙂

bronsa16:08:26

i can confirm reproing w/o any particular setup

martinklepsch16:08:47

Interestingly there are some files in that SHA’s gitlib dir (LICENSE and README)

Alex Miller (Clojure team)16:08:21

yeah, it got part way done

ghadi16:08:37

➜  lib-grimoire git:(develop) git cat-file -t c7cd4f50871b20a580a66e77a34bfa9d336fe430
fatal: git cat-file: could not get object info

Alex Miller (Clojure team)16:08:43

I guess a) there is either something weird in this repo or a bug in jgit

Alex Miller (Clojure team)16:08:05

and b) in this kind of a failure, we should nuke the checkout dir

Alex Miller (Clojure team)16:08:01

a) is worth a bit more investigation

Alex Miller (Clojure team)16:08:12

maybe there’s a force push or something weird there?

ghadi16:08:31

There is something weird in this repo.

martinklepsch16:08:33

Would love a tarball deps procurer btw 😛

ghadi16:08:35

➜  lib-grimoire git:(develop) git cat-file -p HEAD^{tree}  
100644 blob 3bc48030cba1b0f250f8b3d2cb0a2635f5195339	.gitmodules
100644 blob 7689f30efd6dcaadfb4033a444fa1c3f1848dba3	LICENSE
100644 blob c87d64a78fcaebbdfe02f188168a3270962869a1	README.md
040000 tree 4c51b4bd0c9980a782b29286cdeefdb49e7bd899	bin  #### THIS
100644 blob 54f9aeba51d4b4a61c9ad3de046e770859a217c1	deps.edn
100644 blob f60a2b75223a9c8739d525ea62ba48aeeab878b0	project.clj
040000 tree 97a002c7db97cd89dc74606b77de343d2dbc878d	resources
040000 tree 28ffc92968bebaf4930b08ff59d9ff5a3a5c297b	src
040000 tree f4581b6f130115e5f0035945fc4ef6036076ba88	test
➜  lib-grimoire git:(develop) git cat-file -p 4c51b4bd0c9980a782b29286cdeefdb49e7bd899            ## Drill in
160000 commit c7cd4f50871b20a580a66e77a34bfa9d336fe430	hooks
100644 blob f3acf21ed723011f2586416e842ce41683bf3f29	install-hooks.sh

ghadi16:08:19

The second command cats out the SHA for the tree representing the bin directory, and it's pointing to a commit blob mysteriously

ghadi16:08:36

I expect only blobs and trees to be children of trees

martinklepsch16:08:02

I did force push to that branch, and it might have even been after I tried to use another revision from that repo — but seems weird that this would still replicate on a system that hasn’t had a clone yet?

Alex Miller (Clojure team)16:08:42

I agree with Ghadi that that looks like a bad git structure

ghadi16:08:12

try to overwrite bin/hooks and recommit it

martinklepsch16:08:03

Deleted bin/ entirely & commited as 9810f7b20c1a676060f939cfd8f2f785f4dea3c7 but still same issue

ghadi16:08:12

fetching....

martinklepsch16:08:33

did you have something else in mind when you wrote “overwrite”?

ghadi16:08:19

repo looks good to me, you updated to 9810f7b20c1a676060f939cfd8f2f785f4dea3c7?

martinklepsch16:08:33

seems to work now!

ghadi16:08:55

I don't know how that commit object got into the repo!

martinklepsch16:08:03

just saw an exception and was assuming it’s the same but it was a different one 🙂

ghadi16:08:05

git fsck didn't complain at all...

ghadi16:08:19

but that might not be looking for this prob

Alex Miller (Clojure team)16:08:37

I was just doing that myself :)

martinklepsch16:08:27

gotta say, I’m impressed by your git-fu 😄

Alex Miller (Clojure team)16:08:49

you’ve impressively horked this repo, kudos :)

martinklepsch16:08:08

don’t look at me, I’m pretty sure it was @arrdem 🙂

Alex Miller (Clojure team)16:08:52

just change my “you” ref via force-push

martinklepsch16:08:41

an aside to this: are there any plans to add more procurers? A tarball procurer would have been cool in this situation 🙂

dominicm16:08:36

I've been hoping that a procurer extension mechanism would appear, off the back of the work to locate the owner of a namespace.

Alex Miller (Clojure team)16:08:54

seems pretty far down the priority list

Alex Miller (Clojure team)16:08:17

why not just the existing jar support?

dominicm16:08:35

I imagine because github provides a tar.gz/zip on the releases page

Alex Miller (Clojure team)16:08:04

well, feel free to file an enhancement, preferably with a use case

ghadi16:08:28

seems like a weird use-case

Alex Miller (Clojure team)16:08:17

I can envision how to get a repo into this state via the plumbing api but I find it harder to envision how it happened via the porcelain

Alex Miller (Clojure team)16:08:46

I assume some kind of bad rebase

Alex Miller (Clojure team)16:08:16

and ghadi, it does seem like Git handles this better than jgit so maybe you’d never even see it shelling out

lilactown17:08:22

I’m trying to patch juxt/pack to allow me to pass in an alias to use while generating a jar

lilactown17:08:34

this is the relevant bit of code that is catching fire atm:

(let [deps-map (tools.deps.reader/slurp-deps (io/file deps))
            resolve-args (tools.deps/combine-aliases
                          deps-map
                          (get-in parsed-opts [:options :alias]))]
        (tools.deps/resolve-deps deps-map resolve-args)

lilactown17:08:10

I’m currently getting:

org.apache.maven.model.resolution.UnresolvableModelException: Could not find artifact org.apache:apache:pom:1
3

dominicm17:08:31

@lilactown try adding :mvn/repos to your deps.edn

dominicm17:08:59

See pack issues, just eating dinner else I'd give the full snippet

lilactown17:08:49

that seemed to fix the error 🙂 thanks @dominicm

dominicm17:08:00

@alexmiller to solve this, I'm considering inlining the system deps.edn into pack, do you think that would be a problem?

lilactown17:08:29

(and it works!)

Alex Miller (Clojure team)17:08:05

Not as long as you’re willing to roll with future changes

dominicm18:08:27

Happy enough for now.

seancorfield18:08:39

I originally had the system deps baked into boot-tools-deps then took it out in favor of calling (clojure-env) to get the list of deps.edn files to include -- which takes care of the :mvn/repos issue.

seancorfield18:08:55

Does pack not use the system/user deps.edn files?

dominicm18:08:39

@U04V70XH6 I don't want to do that in pack, I explicitly want to ignore the user deps.edn file, and I can't do that without (clojure-env) returning a map for the deps.edn files it considers.

seancorfield18:08:37

You could at least assume the first entry in that vector was the system deps tho'...

dominicm18:08:38

But that isn't part of the api, it could break at any point. I'd feel like a bad citizen depending on a coincidence rather than a contract.

seancorfield18:08:37

I thought we were "guaranteed" that the vector of deps files was in order? Or do you mean that whole -Sdescribe thing isn't documented/guaranteed? /cc @alexmiller

dominicm18:08:05

I recall asking and being told the order wasn't guaranteed. I asked when lein-tools-deps did this.

seancorfield18:08:20

I must admit, I find that very surprising since order does matter ...

dominicm18:08:56

@U04V70XH6 but being the first isn't attributed to system in anyway. A least-important override could be added at any time.

Alex Miller (Clojure team)20:08:58

you can be guaranteed that they are in order: install, user, project, -Sdeps

Alex Miller (Clojure team)20:08:22

however, the last 3 could all independently be either missing or present

Alex Miller (Clojure team)20:08:44

and in future the first may possibly not exist

dominicm21:08:44

If this guarantee is given, then I think the open ticket to convert it to a map is not needed.

seancorfield21:08:44

Well, you still couldn't tell the difference between system + user and system + project (except, perhaps, by inspection of the actual paths?) and that was something @U0567Q30W was after I believe.

seancorfield21:08:51

And you also wouldn't necessarily be able to tell the difference between system + user and user + project at the point where the system deps moved into the library itself (depending on how that is handled).

Alex Miller (Clojure team)21:08:47

I think it’s useful to differentiate

seancorfield21:08:21

Adding a new key to the result of -Sdescribe with some sort of map explaining where the deps files came from seems useful regardless of other concerns. I think if the system deps file moves into the codebase, then indicating that somehow in the list of config files, with a way to get the raw data structure, will also become important for tooling.

cfleming02:08:44

The latest version of Cursive also bundles the system deps.edn BTW

dominicm18:08:42

@alexmiller I might be being silly, but I don't see a way in the public api to slurp a dep which is a java.net.URL, e.g. io/resource. slurp-edn depends on being a file, and canonicalize-all-syms is a private function, so I cannot utilize it.

Alex Miller (Clojure team)20:08:08

could probably break slurp-edn into something smaller that takes a reader (although the file-ness is being used to construct a good error message there)

Alex Miller (Clojure team)20:08:28

canonicalize-all-syms is not public as it may go away

Alex Miller (Clojure team)20:08:12

there’s now a canoncialization step built into the extension apis

Alex Miller (Clojure team)20:08:44

you can of course invoke it via the private var if you want to call it for now