Clojurians Log v2

Is there anything against me mirroring the tools.zip on github releases of deps.clj? Or would you be in for solution 2, then I could just avoid doing so

Why is the jar invalid?

How do you download the jar?

there is a checksum file in maven already

Are the clojure tools uploaded to maven?

oh nvm, you're talking uber jar here

yeah the zip file

https://download.clojure.org/install/clojure-tools-1.11.1.1347.zip

for example

for zip and tar.gz files for other projects I now upload a .sha256 file for validation, e.g. see here: https://github.com/clj-kondo/clj-kondo/releases/tag/v2023.05.26 I directly copied this idea from graalvm: https://github.com/graalvm/graalvm-ce-builds/releases/tag/vm-22.3.2

If you checked the jar for clojure.main (or something) after downloading, that would serve the same purpose, right?

I’m still wondering why the jar is invalid without you knowing, though.

yeah I think so, although there are more files in that .zip file

the .tar.gz sha is in https://download.clojure.org/install/stable.properties but we don't currently make or publish a zip sha

Wait, the zip unzips but the files that come out aren’t valid?

yeah that's weird. so it could also be an unzip problem perhaps

Depending on what you’re using, I would have thought the checksumming in the zip would have prevented this.

Something smells fishy here.

maybe it's unzipping a partially downloaded zip?

or the user aborts while unzipping? dunno, but it sometimes happens

Do you keep the zip file around?

like the first half of the zip. I don't know anything about zip but maybe it supports this so you can zip while still downloading

deps.clj first downloads, then unzips

I vaguely recall that zip stores the file listing at the end just to be annoying.

https://unix.stackexchange.com/a/125102

the .jar file does exist on disk, next time this happens for someone I should ask them to send this .jar file to me, but it's hard to know what exactly goes wrong. Getting rid of ~/.deps.clj solves the problem though. Perhaps I could also try/catch and check for clojure.main in the error message and offer a better suggestion

seems like it still needs a bit more examination (is there somewhere you can detect the problem and save off the badness when it happens) but happy to publish additional sha files if that's helpful

I guess I could add to the error message: please go to the #babashka channel and post your .jar for examination ;)

Re-downloading automatically seems risky as if I make a mistake somewhere you could get into a loop

If you kept the zip around, you should be able to re-validate the jars against the zip.

yeah, good idea, I'll keep the .zip around. I currently delete it

If you wanted to check, you could .getCrc on your ZipEntry and calculate the Crc for the file on disk & compare them.

Not sure how fast that would be, but maybe worth doing on failure to launch or something. Crc32 is pretty fast.

oh that might be a good check, I'll try it out

I'm getting -1 on all entries from the tools.jar zip

which means "unknown"

ah, the crc is known after you read it

Something like this ought to do it. https://github.com/borkdude/deps.clj/pull/103/commits/8627f15d2278e78bf949861c589fa40c8781fdc1

I guess if I could manually change the crc code of the tools jar in an existing .zip file, I could test if the current deps.clj would detect this (and give the better error message)

You could also create a new zip with a different tools jar in ☺️

and then?

The CRC of the tools jar in the zip would be different than the one on disk.

but not if you unzip that zip with the different tools jar

I just want a zip file that I can throw at this function and then the function complains

Ah, then you’re probably into some kind of malicious crc fiddling of the zip file, yeah.

Time to pull out your hex editor 😄

I wonder how I can purposely damage a .zip file to test this, because I'm still not 100% sure if what I did makes any sense. I'm getting the crc32 of the entry and then read it through a checkedinputstream and then compare the crc32, but who says that those aren't always the same, even in the case of a weirdly downloaded zip file

When I truncate some of the last bytes, I can't even unzip it:

dd if=clojure-tools-1.11.1.1347.zip of=tools-corrupted.zip bs=1 count=17999800

I would expect that the file either unzips falsely or is "damaged" after the fact.

If you keep the zip around you can revalidate the files against it.

how would that work?

the crc32 codes are -1 when I read them, which means they weren't in the zip file when the the file got zipped, right

I'm not sure I follow

they only become some number after I've actually processed the entry, which tells me it's lazily computed based on the data that was already there, which is kind of pointless

You need to consume the stream for the CRC to be calculated, yeah

So you could just read the whole thing and then check it against the CRC of what's on disk

I'm checking this:

(= (.getCrc entry) (-> cis (.getChecksum) (.getValue)))

File on disk compared with the one in the entry

Sorry, not both *

so not based on the checked input stream?

I think they're both the same value tbh.

yeah, I think so too, the above comparison is always true, I think, no matter how corrupted

You can use the file Vs the zip as a checksum

Although the CRC should be written in the zip somewhere, too

so comparing to on disk would detect some kind of data write failure?

yeah, I think you need to produce a zip file with explicit crc32 on

Or if the file had been modified (eg by antivirus)

Oh I thought zips had them by default.

It could be a limitation of the Java zip library, too

ah right it has crc32 by default

ok, I added the extra check: https://github.com/borkdude/deps.clj/commit/82adf0cb15249a753851579bc9fac0db52921778

will keep an eye out until the next failure and will ask for the zip file so I can double-check if this really helps