2023-12-01 tools-build | Clojure Slack Archive

tools-build

jumar 2023-12-01T13:36:49.061049Z

I have a pretty simple wrapper over clojure.tools.build.api/uber which serves to build an uberjar in my project. The trouble I'm facing is in the presence of file conflicts, it fails with java.io.IOException: Stream closed I've found (I think) a similar issue here: https://github.com/healthfinch/depstar/issues/4 However, not sure what to make out of it. Sean said: > In my fork, I wrapped the .getNextJarEntry call in (try ... (catch Exception _)) and everything seems to work just fine... That doesn't seem right since it I think that would interrupt the process in the middle of uberjar packaging and could skip a lot of files. This is my code

(let [{:keys [basis target-dir]} (compile/compile-clj options)
      uber-file (str project-prefix "target/" uberjar-name)]
      (printf "Creating uberjar '%s' with target dir '%s'.\n" uber-file target-dir)
      (b/uber {:basis basis
               :class-dir target-dir
               :conflict-handlers {"^duct_hierarchy\\.edn$"
                                   (fn [{:keys [path in existing] :as _params}]
                                     (printf "Found conflicting file path '%s' with existing file '%s'. Merging...\n" path existing)
                                     (let [existing-content (edn/read-string (slurp existing))
                                           new-content (edn/read-string (slurp in))]
                                       {:write {path {:string (with-out-str (pprint/pprint (merge existing-content
                                                                                                  new-content)))}}}))}

               :main main
               :uber-file uber-file}))

It seems that the merge is done but almost immediately after that I get the error

java.io.IOException: Stream closed
        at java.base/java.util.zip.ZipInputStream.ensureOpen(ZipInputStream.java:70)
        at java.base/java.util.zip.ZipInputStream.getNextEntry(ZipInputStream.java:119)
        at java.base/java.util.jar.JarInputStream.getNextEntry(JarInputStream.java:129)
        at java.base/java.util.jar.JarInputStream.getNextJarEntry(JarInputStream.java:166)
        at clojure.tools.build.tasks.uber$explode.invokeStatic(uber.clj:178)
        at clojure.tools.build.tasks.uber$explode.invoke(uber.clj:169)
        at clojure.tools.build.tasks.uber$uber$fn__32890$fn__32894.invoke(uber.clj:286)
        at clojure.lang.PersistentVector.reduce(PersistentVector.java:343)
        at clojure.core$reduce.invokeStatic(core.clj:6885)
        at clojure.core$reduce.invoke(core.clj:6868)
        at clojure.tools.build.tasks.uber$uber$fn__32890.invoke(uber.clj:285)
        at clojure.core.protocols$iter_reduce.invokeStatic(protocols.clj:49)
        at clojure.core.protocols$fn__8230.invokeStatic(protocols.clj:75)
        at clojure.core.protocols$fn__8230.invoke(protocols.clj:75)
        at clojure.core.protocols$fn__8178$G__8173__8191.invoke(protocols.clj:13)
        at clojure.core$reduce.invokeStatic(core.clj:6886)
        at clojure.core$reduce.invoke(core.clj:6868)
        at clojure.tools.build.tasks.uber$uber.invokeStatic(uber.clj:283)
        at clojure.tools.build.tasks.uber$uber.invoke(uber.clj:270)
        at clojure.lang.Var.invoke(Var.java:384)
        at clojure.tools.build.api$uber.invokeStatic(api.clj:502)
        at clojure.tools.build.api$uber.invoke(api.clj:430)
...
Found conflicting file path 'duct_hierarchy.edn' with existing file '/var/folders/hn/tgwyrdmj1tb5pmmbdkd1g_qc0000gn/T/uber5850277775875523123/duct_hierarchy.edn'. Merging...%

In fact, as you can see, my message is printed only after the stacktrace. Is there maybe some laziness in place?

jumar 2023-12-01T13:42:37.155329Z

Maybe it doesn't have to do anything with my conflict handler? I now try to build the uberjar again and getting just the exception and not my logging message "Found conflicting file..."

jumar 2023-12-01T14:18:22.459769Z

I found the problem. It's the usage of (slurp in) in my conflict handler. That closes the input stream. I ended up copying these two functions and using stream->string instead in my conflict handler:

(defn- copy-stream!
  "Copy input stream to output stream using buffer.
  Caller is responsible for passing buffered streams and closing streams."
  [^InputStream is ^OutputStream os ^bytes buffer]
  (loop []
    (let [size (.read is buffer)]
      (if (pos? size)
        (do
          (.write os buffer 0 size)
          (recur))
        (.close os)))))

(defn- stream->string
  [^InputStream is]
  (let [baos (ByteArrayOutputStream. 4096)
        _ (copy-stream! is baos (byte-array 4096))]
    (.toString baos "UTF-8")))

Alex Miller (Clojure team) 2023-12-01T14:18:32.459369Z

the exception means that the stream has been closed

Alex Miller (Clojure team) 2023-12-01T14:18:47.313649Z

yeah, you got it

jumar 2023-12-01T14:19:24.663969Z

Yes 🙂 I guess my problem was that I didn't really understood what's in. And I'm not sure I still get it. I thought it's just the conflicting file, not the JAR itself.

Alex Miller (Clojure team) 2023-12-01T14:20:29.060509Z

https://clojure.github.io/tools.build/clojure.tools.build.api.html#var-uber defines this

Alex Miller (Clojure team) 2023-12-01T14:20:40.929239Z

Conflict handler signature (fn [params]) => effect-map:
  params:
    :path     - String, path in uber jar, matched by regex
    :in       - InputStream to incoming file (see stream->string if needed)
    :existing - File, existing File at path
    :lib      - symbol, lib source for incoming conflict
    :state    - map, available for retaining state during uberjar process

Alex Miller (Clojure team) 2023-12-01T14:21:15.098149Z

it's an InputStream into the incoming file (here from a jar entry)

Alex Miller (Clojure team) 2023-12-01T14:22:35.685469Z

if it would be helpful, could probably make those helpers public

jumar 2023-12-01T14:23:22.719319Z

Yeah, making them public would be useful 🙂

jumar 2023-12-01T14:25:32.722139Z

And why this wasn't obvious to me is because it's "incoming file" but from a JAR entry (as you said). That it's a JAR is crucial, because then it explains why it cannot be closed Also I should have read the javadoc (https://docs.oracle.com/javase/8/docs/api/java/util/jar/JarInputStream.html#getNextJarEntry--) to understand why stream->string (or slurp) actually only returns the content of the file and not the whole JAR file

Alex Miller (Clojure team) 2023-12-01T14:27:26.726089Z

I've logged it for the next time I do a pass on tbuild https://clojure.atlassian.net/browse/CLJ-2814

jumar 2023-12-01T14:27:36.782679Z

🙇

Alex Miller (Clojure team) 2023-12-01T14:27:39.332779Z

oops that was in the wrong project

Alex Miller (Clojure team) 2023-12-01T14:28:25.046679Z

moved to https://clojure.atlassian.net/browse/TBUILD-39

Clojurians Log v2

tools-build