Fork me on GitHub
#tools-deps
<
2023-10-04
>
kwladyka07:10:44

How do you deal with Bill of Materials? https://cloud.google.com/java/docs/bom


  
    
      com.google.cloud
      libraries-bom
      26.22.0
      pom
      import
    
  



  
    com.google.cloud
    google-cloud-pubsub
  
I know there is https://clojure.atlassian.net/browse/TDEPS-202 Because BOM is not supported I have deps manually, and I have Bad type on operand stack on (.build (.setData (PubsubMessage/newBuilder) (ByteString/copyFromUtf8 (json/write-value-as-string body)))), but only in JAR. I see ByteString/copyFromUtf8 is com.google.protobuf.LiteralByteString in jar, but com.google.protobuf.ByteString$LiteralByteString in REPL. Why it works in REPL, but doesn’t in jar. I tried clojure.tools.build.api/write-pom and versions look the same as in deps.edn. Any hints what I can do to fix this? How to debug this to find the issue? PS I can’t be sure if the issue is luck of BOM or something else.

Alex Miller (Clojure team)10:10:49

if you're not using BOM, then that can't be the problem - just work this as a java interop issue - what is the full error when it occurs?

kwladyka10:10:57

#error {
 :cause Bad type on operand stack
Exception Details:
  Location:
    com/google/api/AnnotationsProto.registerAllExtensions(Lcom/google/protobuf/ExtensionRegistryLite;)V @4: invokevirtual
  Reason:
    Type 'com/google/protobuf/GeneratedMessage$GeneratedExtension' (current frame, stack[1]) is not assignable to 'com/google/protobuf/ExtensionLite'
  Current Frame:
    bci: @4
    flags: { }
    locals: { 'com/google/protobuf/ExtensionRegistryLite' }
    stack: { 'com/google/protobuf/ExtensionRegistryLite', 'com/google/protobuf/GeneratedMessage$GeneratedExtension' }
  Bytecode:
    0000000: 2ab2 0002 b600 03b1

 :via
 [{:type java.lang.VerifyError
   :message Bad type on operand stack
Exception Details:
  Location:
    com/google/api/AnnotationsProto.registerAllExtensions(Lcom/google/protobuf/ExtensionRegistryLite;)V @4: invokevirtual
  Reason:
    Type 'com/google/protobuf/GeneratedMessage$GeneratedExtension' (current frame, stack[1]) is not assignable to 'com/google/protobuf/ExtensionLite'
  Current Frame:
    bci: @4
    flags: { }
    locals: { 'com/google/protobuf/ExtensionRegistryLite' }
    stack: { 'com/google/protobuf/ExtensionRegistryLite', 'com/google/protobuf/GeneratedMessage$GeneratedExtension' }
  Bytecode:
    0000000: 2ab2 0002 b600 03b1

   :at [com.google.pubsub.v1.PubsubProto <clinit> PubsubProto.java 570]}]
 :trace
 [[com.google.pubsub.v1.PubsubProto <clinit> PubsubProto.java 570]
  [com.google.pubsub.v1.PubsubMessage$AttributesDefaultEntryHolder <clinit> PubsubMessage.java 112]
  [com.google.pubsub.v1.PubsubMessage$Builder internalGetAttributes PubsubMessage.java 910]
  [com.google.pubsub.v1.PubsubMessage$Builder buildPartial PubsubMessage.java 694]
  [com.google.pubsub.v1.PubsubMessage$Builder build PubsubMessage.java 682]
...

kwladyka10:10:37

Why it work in REPL and doesn’t in jar?

kwladyka10:10:05

I don’t see any conflicts or double versioning etc. in pom.xml or -Stree

Alex Miller (Clojure team)12:10:22

not sure, but "Type 'com/google/protobuf/GeneratedMessage$GeneratedExtension' (current frame, stack[1]) is not assignable to 'com/google/protobuf/ExtensionLite'" makes me think of either getting mismatched lib versions, or a classloader issue

kwladyka12:10:56

Any actions which I can do to verify it?

Alex Miller (Clojure team)12:10:40

a VerifyError generally means the class bytecode is invalid. I assume here it's a class generated by protobuf

Alex Miller (Clojure team)12:10:22

when you say "in jar" - what does that mean?

Alex Miller (Clojure team)12:10:34

are you building an uberjar out of this?

kwladyka12:10:36

I assume it is about: > I see ByteString/copyFromUtf8 is com.google.protobuf.LiteralByteString in jar, but com.google.protobuf.ByteString$LiteralByteString in REPL. Why it works in REPL, but doesn’t in jar.

Alex Miller (Clojure team)12:10:18

I don't understand what you mean by that sentence

kwladyka12:10:35

(b/uber
   {:class-dir class-dir
    :uber-file jar-file
    :basis     basis
    :main      'system.main})
To be precise, this is not my project and I am still learning this project.

kwladyka12:10:44

I mean (ByteString/copyFromUtf8 (json/write-value-as-string body)) in REPL is com.google.protobuf.ByteString$LiteralByteString, but in jar is com.google.protobuf.LiteralByteString which suggest me there is an issue about type.

kwladyka12:10:03

and this is the input for later function which fail

Alex Miller (Clojure team)12:10:28

don't know why that would be unless the libs are different. could be you have two jars on your classpath with the same classes and on the repl, the ordering puts the "correct" one first, but after building the uber jar, the "wrong" one ends up in the uberjar

kwladyka12:10:08

this was my first idea, but I don’t see any duplicate of libs in -Stree or pom.xml

kwladyka12:10:22

Do you suggest any actions, code, commands, check which I can do?

Alex Miller (Clojure team)12:10:12

it would not be duplicate lib, it would be two different jars that include the same classes. since you have two classes above, I would look for those in the jars you have (com.google.protobuf.LiteralByteString and com.google.protobuf.ByteString).

Alex Miller (Clojure team)12:10:46

maybe something in the google-cloud-pubsub includes protobuf instead of depending on it

Alex Miller (Clojure team)12:10:12

that kind of problem could definitely cause the error you're seeing

kwladyka12:10:46

> com/google/protobuf/ByteString$LiteralByteString.class > com/google/protobuf/LiteralByteString.class I see both in Jar, but still I don’t know what actions I can do to fix it.

kwladyka12:10:26

> maybe something in the google-cloud-pubsub includes protobuf instead of depending on it Do you mean the pubsub (or dependency of pubsub) is builded as uberjar instead of jar and include other lib jar with protobuf while at the same time have protobuf as dependency?

kwladyka13:10:39

I don’t think there is any way to fix it in deps.edn, is it? Perhaps BOM fix it.

Alex Miller (Clojure team)14:10:39

Bom has nothing to do with this - it’s just a way to specify multiple deps together

kwladyka14:10:32

I mean maybe BOM set the same version in dependency as it is in jar

Alex Miller (Clojure team)14:10:33

Ultimately you’re getting some set of jars that is on your classpath in repl case. In uberjar, that set of jars is being collapsed into a single jar. If the jars do not overlap, this identical from classpath perspective. If jars overlap, then ordering matters, in both cases.

kwladyka14:10:24

interesting, I have tried to isolate problem (libraries and small code) to separate in small project: 1) with deps.edn clj -T:build ci I have > Cannot write META-INF/license/LICENSE.aix-netbsd.txt from io.grpc/grpc-netty-shaded as parent dir is a file from another lib. One of them must be excluded. 2) with lein uberjar it compiles and work in jar as expected

kwladyka14:10:15

both have the same versions of the same libraries in the same order

kwladyka14:10:24

also -Spath show io.grpc/grpc-netty-shaded only once

kwladyka14:10:43

everything is only once

kwladyka14:10:36

Can it be clj build bug somehow?

Alex Miller (Clojure team)14:10:53

that error means probably means you have both a META-INF/license/ dir and META-INF/license file from different jars during uber jar'ing - in uber task you can add :exclude ["META-INF/license"] to exclude the file

kwladyka14:10:53

oh sorry, I misinterpret it as a part of the libraries conlict

kwladyka14:10:18

ok, it works both from lein and clj when isolated from the project.

kwladyka14:10:25

finally, something

kwladyka14:10:29

thank you for your help

kwladyka15:10:35

you were right. This is one of third party library which has to include protobuf in hidden way. Just the presence of the library in deps.edn make it fail after build to jar.

kwladyka15:10:38

I did it by commenting half of the deps, later other half etc. in experimental project. Is there any other way to detect such conflicts in the real project and real code?

kwladyka15:10:24

I mean in the real project I can’t comment any library, because system will not build, because code needs each of them.

kwladyka15:10:46

So I am thinking if there is any reasonable way to detect such conflicts and display them.

Alex Miller (Clojure team)15:10:40

jars have files, jars with the same .class file are in conflict - there are some java tools out there that can help identify stuff like this

kwladyka15:10:23

Can you give any name of such tool to start reading in google?

kwladyka15:10:11

thank you, I will find something