depstar 2020-10-28 | Slack Archive

Eric Ihli17:10:40

Built an uberjar with clj -A:depstar -m hf.depstar.uberjar prhyme.jar Library has the following line: (nippy/thaw-from-file (io/resource "dark-corpus-2.bin")) File confirmed in jar with jar ft prhyme.jar | grep dark-corpus-2.bin jar depended on by another project with {:local/root "/home/user/code/prhyme/prhyme.jar"} Error thrown when starting a repl:

Caused by: java.lang.IllegalArgumentException: Not a file: jar:file:/home/user/code/prhyme/prhyme.jar!/dark-corpus-2.bin
	at $fn__11362.invokeStatic(io.clj:61)

Any thoughts on what would cause this? Is that ! in jar:file:/home/.../prhyme.jar!/dark-corpus-2.edn relevant? Or is it some artifact of an internal of java loading resources? I see a sorta similar issue at https://github.com/seancorfield/depstar/issues/27 but that person was getting an error about a file being skipped. Not the case for me. resources is on the path in deps.edn.

Eric Ihli17:10:19

I may have just answered my own question. I see there is a difference between an io/resource and an io/file.

(= (io/file (io/resource "dark-corpus-2.bin"))
;; => #object[java.io.File 0x4f44ce9c "/home/user/code/prhyme/resources/dark-corpus-2.bin"]   
   (io/resource "dark-corpus-2.bin")
   ;; => #object[java.net.URL 0x31865f0d "file:/home/user/code/prhyme/resources/dark-corpus-2.bin"]
   )

Eric Ihli17:10:23

Nope. Same thing.

Eric Ihli17:10:36

Or I was close... https://groups.google.com/g/clojure/c/8scFidro4ow > Just delete that line. The io/resource function returns a URL which all the Clojure IO functions can handle just fine-as is. When running in development the URL happens to be a file:// URL, and thus something io/file can handle. Once the resource is in a JAR that is no longer the case, and hence exceptions. Just don't require a file when any URL will do and you'll be fine. But nippy's thaw-from-file sends it back to an io/file.

(defn thaw-from-file
  "Convenience util: like `thaw`, but reads from `( <file>)`.

  To thaw from a resource on classpath (e.g in Leiningen `resources` dir):
    (thaw-from-file ( \"my-resource-name.npy\"))

  See also `freeze-to-file`."
  ([file          ] (thaw-from-file file nil))
  ([file thaw-opts]
   (let [file (jio/file file),
         ba (byte-array (.length file))]
     (with-open [in (DataInputStream. (jio/input-stream file))]
       (.readFully in ba))

     (thaw ba thaw-opts))))

seancorfield17:10:25

@ericihli Have you confirmed that file is actually in the JAR? jar tvf prhyme.jar | fgrep dark-corpus

Eric Ihli18:10:12

Yeah, it's definitely in there. I'm pretty sure now this issue has nothing to do with depstar. https://groups.google.com/g/clojure/c/8scFidro4ow I'm gathering you can't use io/file to access a packaged resource. Nippy is using io/file to get the length of the file to create a byte-array.

(defn thaw-from-file
  "Convenience util: like `thaw`, but reads from `( <file>)`.

  To thaw from a resource on classpath (e.g in Leiningen `resources` dir):
    (thaw-from-file ( \"my-resource-name.npy\"))

  See also `freeze-to-file`."
  ([file          ] (thaw-from-file file nil))
  ([file thaw-opts]
   (let [file (jio/file file),
         ba (byte-array (.length file))]
     (with-open [in (DataInputStream. (jio/input-stream file))]
       (.readFully in ba))

     (thaw ba thaw-opts))))

I'm trying an alternative.

(defn thaw-from-file
  ([file          ] (thaw-from-file file nil))
  ([file thaw-opts]
   (let [xin (io/input-stream file)
         xout (ByteArrayOutputStream.)]
     (io/copy xin xout)
     (nippy/thaw (.toByteArray xout) thaw-opts))))

That does appear to fix it.

seancorfield18:10:36

Interesting. Which line exactly throws the exception? Is it the .length call?

Eric Ihli18:10:09

(jio/file file). Says it can't find it when file is (io/resource "some-resource")

Eric Ihli18:10:49

Works fine until you package the code up as a jar and another package depends on it.

seancorfield19:10:18

Ah, so the docstring is misleading then:

To thaw from a resource on classpath (e.g in Leiningen `resources` dir):
    (thaw-from-file ( \"my-resource-name.npy\"))

seancorfield19:10:49

Well, glad you have it working now!

seancorfield19:10:04

We used to use Nippy at work but it's a strange beast so we decided to switch to something else. In the end, I think we stopped using all the taoensso libraries.

Eric Ihli19:10:47

Oh man that's interesting to hear. I've found myself migrating towards all of them.

seancorfield17:10:53

Also, if you build the jar like this, you'll see it tell you every file it puts into the JAR: clj -A:depstar -m hf.depstar.uberjar prhyme.jar -v

2020-10-28

Channels