This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2023-04-23
Channels
- # ai (10)
- # beginners (15)
- # biff (8)
- # calva (11)
- # clerk (11)
- # clojure-conj (6)
- # clojure-europe (28)
- # clojure-norway (2)
- # clojure-spec (55)
- # clr (9)
- # emacs (10)
- # etaoin (8)
- # helix (1)
- # hyperfiddle (3)
- # jobs (3)
- # off-topic (52)
- # re-frame (3)
- # reagent (7)
- # remote-jobs (3)
- # shadow-cljs (18)
- # specter (4)
- # xtdb (4)
I have a question around conversion to Java types .. I have an embedding vector:
[[-0.007481536
0.0025613538
0.021117128
-0.010558564
-0.024343925
0.008863476
-0.040273674
-0.003334015
-0.022710104
-0.019551385
0.024588997
0.007284116
-0.0025494406
0.034282997
0.0035875973
-0.0033101882
0.027611565
0.003625039
0.0056673144
-0.023881009
-0.035807896
-0.013717284
-0.0068756607
-0.007651726
-0.018394096
-0.012219615
-0.0021835328
-0.017577184
-0.024425615
0.009401275
0.038040787
0.009435313
-0.014595462
-2.2550125E-4
-0.022206342
-0.016038671
-0.010647063
-0.014241468
-0.015412373
0.020654213
0.007501959
-7.416013E-4
0.008053374
0.008788592
-0.003972226
0.0068280078
-0.016324589
0.038830467
-0.023976315
0.012566802
...]]
That I need to pass to Milvus as a search parameter ... however the Java SDK wants a List<Float> so when I do I get a type conversion error. How do I convert this over to a Java generic list type?List<Float> is a figment of the java compiler, it doesn't exist, generics are implemented via erasure
Clojure vectors implement List, but the double [[ at the start of what you pasted indicates a vector that contains a vector that contains those numbers
Ah, yes .. forgot about that. So maybe the error is incorrect. I'm trying both the 1D version of the higher dimensional vector and the 1536dim vector stored in Milvus. (into-array Double [...]) .. worked, although it isn't a list of floats lol.
... ah, never-mind .. it didn't. Spoke too soon.
Float vector field's value must be Lst<Float> ... and the values in the vector are Doubles. I'm guessing that is the source of the problem. https://github.com/milvus-io/milvus-sdk-java/blob/c9f9750b908652234b09133bede6820b7ca9f60d/src/main/java/io/milvus/param/dml/SearchParam.java#L329
Yeah, clojure floating defaults to doubles, usually the jvm is fine about quietly casting, but the code explicitly checks for only floats
Mapping them to floats gets me past that error and into another one where ByteBuffer::putFloat is complaining about its inability to convert longs -> float. I'm going to try shoving it into a bytebuffer on my end next and seeing if that bypasses these problems. https://github.com/milvus-io/milvus-sdk-java/blob/c9f9750b908652234b09133bede6820b7ca9f60d/src/main/java/io/milvus/param/ParamUtils.java#L243
I would recommend getting it to the habit of sharing the actual exceptions you get, makes it much easier to determine what is happening
it may be the case that you have non-floats in your list, like 0
may be there as an int
. you can map over your list coercing everything to a float.
Thanks @U0NCTKEV8 and @U0LAJQLQ1 ... that was essentially the actual exception ("cannot convert java.lang.long to java.lang.float") but yes, not quoting it leaves room for people to wonder. I am unsure why that was the error, but the real problem was that milvus wanted turtles all the way down. When I reshaped it to 1x1536 I'd missed converting the partitions to lists of floats and wound up with a List<Vector<Float>>>. The milvus source wants lists as the container (all the way down).