tools-deps

kwladyka 2023-10-04T07:54:44.014039Z

How do you deal with Bill of Materials? https://cloud.google.com/java/docs/bom


  
    
      com.google.cloud
      libraries-bom
      26.22.0
      pom
      import
    
  



  
    com.google.cloud
    google-cloud-pubsub
  
I know there is https://clojure.atlassian.net/browse/TDEPS-202 Because BOM is not supported I have deps manually, and I have Bad type on operand stack on (.build (.setData (PubsubMessage/newBuilder) (ByteString/copyFromUtf8 (json/write-value-as-string body)))), but only in JAR. I see ByteString/copyFromUtf8 is com.google.protobuf.LiteralByteString in jar, but com.google.protobuf.ByteString$LiteralByteString in REPL. Why it works in REPL, but doesn’t in jar. I tried clojure.tools.build.api/write-pom and versions look the same as in deps.edn. Any hints what I can do to fix this? How to debug this to find the issue? PS I can’t be sure if the issue is luck of BOM or something else.

Alex Miller (Clojure team) 2023-10-04T10:50:49.902609Z

if you're not using BOM, then that can't be the problem - just work this as a java interop issue - what is the full error when it occurs?

kwladyka 2023-10-04T10:52:57.099829Z

#error {
 :cause Bad type on operand stack
Exception Details:
  Location:
    com/google/api/AnnotationsProto.registerAllExtensions(Lcom/google/protobuf/ExtensionRegistryLite;)V @4: invokevirtual
  Reason:
    Type 'com/google/protobuf/GeneratedMessage$GeneratedExtension' (current frame, stack[1]) is not assignable to 'com/google/protobuf/ExtensionLite'
  Current Frame:
    bci: @4
    flags: { }
    locals: { 'com/google/protobuf/ExtensionRegistryLite' }
    stack: { 'com/google/protobuf/ExtensionRegistryLite', 'com/google/protobuf/GeneratedMessage$GeneratedExtension' }
  Bytecode:
    0000000: 2ab2 0002 b600 03b1

 :via
 [{:type java.lang.VerifyError
   :message Bad type on operand stack
Exception Details:
  Location:
    com/google/api/AnnotationsProto.registerAllExtensions(Lcom/google/protobuf/ExtensionRegistryLite;)V @4: invokevirtual
  Reason:
    Type 'com/google/protobuf/GeneratedMessage$GeneratedExtension' (current frame, stack[1]) is not assignable to 'com/google/protobuf/ExtensionLite'
  Current Frame:
    bci: @4
    flags: { }
    locals: { 'com/google/protobuf/ExtensionRegistryLite' }
    stack: { 'com/google/protobuf/ExtensionRegistryLite', 'com/google/protobuf/GeneratedMessage$GeneratedExtension' }
  Bytecode:
    0000000: 2ab2 0002 b600 03b1

   :at [com.google.pubsub.v1.PubsubProto <clinit> PubsubProto.java 570]}]
 :trace
 [[com.google.pubsub.v1.PubsubProto <clinit> PubsubProto.java 570]
  [com.google.pubsub.v1.PubsubMessage$AttributesDefaultEntryHolder <clinit> PubsubMessage.java 112]
  [com.google.pubsub.v1.PubsubMessage$Builder internalGetAttributes PubsubMessage.java 910]
  [com.google.pubsub.v1.PubsubMessage$Builder buildPartial PubsubMessage.java 694]
  [com.google.pubsub.v1.PubsubMessage$Builder build PubsubMessage.java 682]
...

kwladyka 2023-10-04T10:53:37.776229Z

Why it work in REPL and doesn’t in jar?

kwladyka 2023-10-04T10:55:05.765539Z

I don’t see any conflicts or double versioning etc. in pom.xml or -Stree

Alex Miller (Clojure team) 2023-10-04T12:09:22.907879Z

not sure, but "Type 'com/google/protobuf/GeneratedMessage$GeneratedExtension' (current frame, stack[1]) is not assignable to 'com/google/protobuf/ExtensionLite'" makes me think of either getting mismatched lib versions, or a classloader issue

kwladyka 2023-10-04T12:09:56.879689Z

Any actions which I can do to verify it?

Alex Miller (Clojure team) 2023-10-04T12:10:40.065339Z

a VerifyError generally means the class bytecode is invalid. I assume here it's a class generated by protobuf

Alex Miller (Clojure team) 2023-10-04T12:11:22.768249Z

when you say "in jar" - what does that mean?

Alex Miller (Clojure team) 2023-10-04T12:11:34.864189Z

are you building an uberjar out of this?

kwladyka 2023-10-04T12:11:36.022459Z

I assume it is about: > I see ByteString/copyFromUtf8 is com.google.protobuf.LiteralByteString in jar, but com.google.protobuf.ByteString$LiteralByteString in REPL. Why it works in REPL, but doesn’t in jar.

Alex Miller (Clojure team) 2023-10-04T12:12:18.768299Z

I don't understand what you mean by that sentence

kwladyka 2023-10-04T12:12:35.659299Z

(b/uber
   {:class-dir class-dir
    :uber-file jar-file
    :basis     basis
    :main      'system.main})
To be precise, this is not my project and I am still learning this project.

kwladyka 2023-10-04T12:13:44.972959Z

I mean (ByteString/copyFromUtf8 (json/write-value-as-string body)) in REPL is com.google.protobuf.ByteString$LiteralByteString, but in jar is com.google.protobuf.LiteralByteString which suggest me there is an issue about type.

kwladyka 2023-10-04T12:14:03.249829Z

and this is the input for later function which fail

Alex Miller (Clojure team) 2023-10-04T12:15:28.608819Z

don't know why that would be unless the libs are different. could be you have two jars on your classpath with the same classes and on the repl, the ordering puts the "correct" one first, but after building the uber jar, the "wrong" one ends up in the uberjar

kwladyka 2023-10-04T12:16:08.971119Z

this was my first idea, but I don’t see any duplicate of libs in -Stree or pom.xml

kwladyka 2023-10-04T12:16:22.598349Z

Do you suggest any actions, code, commands, check which I can do?

Alex Miller (Clojure team) 2023-10-04T12:18:12.726259Z

it would not be duplicate lib, it would be two different jars that include the same classes. since you have two classes above, I would look for those in the jars you have (com.google.protobuf.LiteralByteString and com.google.protobuf.ByteString).

Alex Miller (Clojure team) 2023-10-04T12:19:46.597189Z

maybe something in the google-cloud-pubsub includes protobuf instead of depending on it

Alex Miller (Clojure team) 2023-10-04T12:20:12.071409Z

that kind of problem could definitely cause the error you're seeing

kwladyka 2023-10-04T12:51:46.059469Z

> com/google/protobuf/ByteString$LiteralByteString.class > com/google/protobuf/LiteralByteString.class I see both in Jar, but still I don’t know what actions I can do to fix it.

kwladyka 2023-10-04T12:55:26.089919Z

> maybe something in the google-cloud-pubsub includes protobuf instead of depending on it Do you mean the pubsub (or dependency of pubsub) is builded as uberjar instead of jar and include other lib jar with protobuf while at the same time have protobuf as dependency?

Alex Miller (Clojure team) 2023-10-04T13:48:48.733799Z

Yes

kwladyka 2023-10-04T13:54:39.551169Z

I don’t think there is any way to fix it in deps.edn, is it? Perhaps BOM fix it.

Alex Miller (Clojure team) 2023-10-04T14:00:39.541359Z

Bom has nothing to do with this - it’s just a way to specify multiple deps together

kwladyka 2023-10-04T14:01:32.023679Z

I mean maybe BOM set the same version in dependency as it is in jar

Alex Miller (Clojure team) 2023-10-04T14:06:33.544019Z

Ultimately you’re getting some set of jars that is on your classpath in repl case. In uberjar, that set of jars is being collapsed into a single jar. If the jars do not overlap, this identical from classpath perspective. If jars overlap, then ordering matters, in both cases.

kwladyka 2023-10-04T14:36:24.938699Z

interesting, I have tried to isolate problem (libraries and small code) to separate in small project: 1) with deps.edn clj -T:build ci I have > Cannot write META-INF/license/LICENSE.aix-netbsd.txt from io.grpc/grpc-netty-shaded as parent dir is a file from another lib. One of them must be excluded. 2) with lein uberjar it compiles and work in jar as expected

kwladyka 2023-10-04T14:38:15.071649Z

both have the same versions of the same libraries in the same order

kwladyka 2023-10-04T14:44:24.577719Z

also -Spath show io.grpc/grpc-netty-shaded only once

kwladyka 2023-10-04T14:44:43.805719Z

everything is only once

kwladyka 2023-10-04T14:47:36.796629Z

Can it be clj build bug somehow?

Alex Miller (Clojure team) 2023-10-04T14:49:53.187309Z

that error means probably means you have both a META-INF/license/ dir and META-INF/license file from different jars during uber jar'ing - in uber task you can add :exclude ["META-INF/license"] to exclude the file

kwladyka 2023-10-04T14:51:53.966709Z

oh sorry, I misinterpret it as a part of the libraries conlict

kwladyka 2023-10-04T14:55:18.636709Z

ok, it works both from lein and clj when isolated from the project.

kwladyka 2023-10-04T14:55:25.280419Z

finally, something

kwladyka 2023-10-04T14:55:29.566529Z

thank you for your help

kwladyka 2023-10-04T15:15:35.855509Z

you were right. This is one of third party library which has to include protobuf in hidden way. Just the presence of the library in deps.edn make it fail after build to jar.

kwladyka 2023-10-04T15:16:38.689659Z

I did it by commenting half of the deps, later other half etc. in experimental project. Is there any other way to detect such conflicts in the real project and real code?

kwladyka 2023-10-04T15:26:24.613329Z

I mean in the real project I can’t comment any library, because system will not build, because code needs each of them.

kwladyka 2023-10-04T15:29:46.188169Z

So I am thinking if there is any reasonable way to detect such conflicts and display them.

Alex Miller (Clojure team) 2023-10-04T15:37:40.398879Z

jars have files, jars with the same .class file are in conflict - there are some java tools out there that can help identify stuff like this

kwladyka 2023-10-04T15:38:23.681679Z

Can you give any name of such tool to start reading in google?

Alex Miller (Clojure team) 2023-10-04T15:38:46.933479Z

don't know one offhand

kwladyka 2023-10-04T15:40:11.150959Z

thank you, I will find something