Fork me on GitHub
#beginners
<
2023-04-23
>
Chris Lester00:04:05

I have a question around conversion to Java types .. I have an embedding vector:

[[-0.007481536
  0.0025613538
  0.021117128
  -0.010558564
  -0.024343925
  0.008863476
  -0.040273674
  -0.003334015
  -0.022710104
  -0.019551385
  0.024588997
  0.007284116
  -0.0025494406
  0.034282997
  0.0035875973
  -0.0033101882
  0.027611565
  0.003625039
  0.0056673144
  -0.023881009
  -0.035807896
  -0.013717284
  -0.0068756607
  -0.007651726
  -0.018394096
  -0.012219615
  -0.0021835328
  -0.017577184
  -0.024425615
  0.009401275
  0.038040787
  0.009435313
  -0.014595462
  -2.2550125E-4
  -0.022206342
  -0.016038671
  -0.010647063
  -0.014241468
  -0.015412373
  0.020654213
  0.007501959
  -7.416013E-4
  0.008053374
  0.008788592
  -0.003972226
  0.0068280078
  -0.016324589
  0.038830467
  -0.023976315
  0.012566802
  ...]]
That I need to pass to Milvus as a search parameter ... however the Java SDK wants a List<Float> so when I do I get a type conversion error. How do I convert this over to a Java generic list type?

hiredman00:04:04

List<Float> is a figment of the java compiler, it doesn't exist, generics are implemented via erasure

hiredman00:04:13

There is only List

hiredman00:04:33

Clojure vectors implement List, but the double [[ at the start of what you pasted indicates a vector that contains a vector that contains those numbers

Chris Lester00:04:27

Ah, yes .. forgot about that. So maybe the error is incorrect. I'm trying both the 1D version of the higher dimensional vector and the 1536dim vector stored in Milvus. (into-array Double [...]) .. worked, although it isn't a list of floats lol.

Chris Lester00:04:51

... ah, never-mind .. it didn't. Spoke too soon.

hiredman00:04:34

What is the exact error you get?

Chris Lester00:04:59

Float vector field's value must be Lst<Float> ... and the values in the vector are Doubles. I'm guessing that is the source of the problem. https://github.com/milvus-io/milvus-sdk-java/blob/c9f9750b908652234b09133bede6820b7ca9f60d/src/main/java/io/milvus/param/dml/SearchParam.java#L329

hiredman00:04:52

Yeah, clojure floating defaults to doubles, usually the jvm is fine about quietly casting, but the code explicitly checks for only floats

Chris Lester01:04:29

Mapping them to floats gets me past that error and into another one where ByteBuffer::putFloat is complaining about its inability to convert longs -> float. I'm going to try shoving it into a bytebuffer on my end next and seeing if that bypasses these problems. https://github.com/milvus-io/milvus-sdk-java/blob/c9f9750b908652234b09133bede6820b7ca9f60d/src/main/java/io/milvus/param/ParamUtils.java#L243

hiredman01:04:42

Complaining how

hiredman01:04:31

I would recommend getting it to the habit of sharing the actual exceptions you get, makes it much easier to determine what is happening

pppaul02:04:20

it may be the case that you have non-floats in your list, like 0 may be there as an int. you can map over your list coercing everything to a float.

Chris Lester00:04:14

Thanks @U0NCTKEV8 and @U0LAJQLQ1 ... that was essentially the actual exception ("cannot convert java.lang.long to java.lang.float") but yes, not quoting it leaves room for people to wonder. I am unsure why that was the error, but the real problem was that milvus wanted turtles all the way down. When I reshaped it to 1x1536 I'd missed converting the partitions to lists of floats and wound up with a List<Vector<Float>>>. The milvus source wants lists as the container (all the way down).

pppaul00:04:24

you can do a recursive conversion via clojure.walk, if that's still your error