data-science 2023-09-30 | Slack Archive

Any tips on getting UMAP model working in http://scicloj.ml? I see a test in the project: Roughly the below:

(let [pinguins (-> (tc/dataset
                      ""
                      {:key-fn csk/->kebab-case-keyword}))
        pipe (mm/pipeline
              (tc-mm/drop-missing)
              (tc-mm/select-columns [:culmen-length-mm :culmen-depth-mm :flipper-length-mm :body-mass-g])
              (scicloj.metamorph.ml.preprocessing/std-scale :type/numerical {})
              {:metamorph/id :model}
              (ml/model {:model-type :smile.manifold/isomap
                         :args [4 5 false]}))
        fit-ctx (mm/fit-pipe pinguins pipe)]

    fit-ctx)

I can't seem to get the deps working. I get this error after trying to manually pull in deps from searching other git repos.

ava.lang.NoClassDefFoundError: Could not initialize class org.bytedeco.arpackng.global.arpack

deps:

com.github.haifengl/smile-interpolation {:mvn/version "2.6.0"}
com.github.haifengl/smile-core {:mvn/version "2.6.0"}
com.github.haifengl/smile-mkl {:mvn/version "2.6.0"}

org.bytedeco.javacpp-presets/openblas {:mvn/version "0.3.5-1.4.4"}
org.bytedeco.javacpp-presets/arpack-ng {:mvn/version "3.6.3-1.4.4"}

org.bytedeco/javacpp-platform {:mvn/version "1.5.9"}
org.bytedeco/javacpp {:mvn/version "1.5.9"}

org.bytedeco/arpack-ng-platform {:mvn/version "3.9.0-1.5.9"}
org.bytedeco/arpack-ng {:mvn/version "3.9.0-1.5.9"}

org.bytedeco/openblas-platform {:mvn/version "0.3.23-1.5.9"}
org.bytedeco/openblas {:mvn/version "0.3.23-1.5.9"}

scicloj/scicloj.ml {:mvn/version "0.3"}

Previously I was getting the same error but missing something from openblas. That was fixed after adding more deps, but I can't seem to get this one squashed. Lastly, I'm trying to use the umap model, but couldn't find an example of that. Thought I would get this working first then try to figure that one out next. Thank you

genmeblog07:10:30

I use different versions of bytedeco deps to work with Smile 2.6.0.

[com.github.haifengl/smile-interpolation "2.6.0"]
                 [com.github.haifengl/smile-core "2.6.0"]
                 [com.github.haifengl/smile-mkl "2.6.0"]
                 
                 [org.bytedeco/arpack-ng "3.7.0-1.5.4"]
                 [org.bytedeco/arpack-ng-platform "3.7.0-1.5.4"]
                 [org.bytedeco/openblas "0.3.10-1.5.4"]
                 [org.bytedeco/openblas-platform "0.3.10-1.5.4"]
                 [org.bytedeco/javacpp "1.5.4"]

az17:10:35

Thanks @U1EP3BZ3Q, unfortunately that didn't fix it.

genmeblog19:10:10

Maybe @U7CAHM72M can help here.

Carsten Behring20:10:55

I tried it using http://scicloj.ml 0.3, and your code (just adding the :require)

(ns scicloj.ml.umap
  (:require [tablecloth.api :as tc]
            [scicloj.metamorph.core :as mm]
            [tablecloth.pipeline :as tc-mm]
            [camel-snake-kebab.core :as csk]
            [ :as ml]
            [scicloj.metamorph.ml.preprocessing]
            [scicloj.ml.smile.manifold]))
  

(let [pinguins (-> (tc/dataset
                      ""
                      {:key-fn csk/->kebab-case-keyword}))
        pipe (mm/pipeline
              (tc-mm/drop-missing)
              (tc-mm/select-columns [:culmen-length-mm :culmen-depth-mm :flipper-length-mm :body-mass-g])
              (scicloj.metamorph.ml.preprocessing/std-scale :type/numerical {})
              {:metamorph/id :model}
              (ml/model {:model-type :smile.manifold/isomap
                         :args [4 5 false]}))
        fit-ctx (mm/fit-pipe pinguins pipe)]

    fit-ctx)

Carsten Behring20:10:22

Have you tried using only scicloj/scicloj.ml {:mvn/version "0.3"} in deps.edn ?

Carsten Behring20:10:18

The org.bytedeco.* deps should get pulled automatically transitively. I get for example :

clojure -X:deps list | grep arpack
org.bytedeco/arpack-ng 3.7.0-1.5.4  (Apache-2.0)
org.bytedeco/arpack-ng$linux-arm64 3.7.0-1.5.4  (Apache-2.0)
org.bytedeco/arpack-ng$linux-armhf 3.7.0-1.5.4  (Apache-2.0)
org.bytedeco/arpack-ng$linux-ppc64le 3.7.0-1.5.4  (Apache-2.0)
org.bytedeco/arpack-ng$linux-x86 3.7.0-1.5.4  (Apache-2.0)
org.bytedeco/arpack-ng$linux-x86_64 3.7.0-1.5.4  (Apache-2.0)
org.bytedeco/arpack-ng$macosx-x86_64 3.7.0-1.5.4  (Apache-2.0)
org.bytedeco/arpack-ng$windows-x86 3.7.0-1.5.4  (Apache-2.0)
org.bytedeco/arpack-ng$windows-x86_64 3.7.0-1.5.4  (Apache-2.0)
org.bytedeco/arpack-ng-platform 3.7.0-1.5.4  (Apache-2.0

az20:10:30

Hi @U7CAHM72M, thank you, yes, I tried that alone and remove all other deps, but will try again now

Carsten Behring20:10:03

which OS do you use ?

az20:10:10

I'm on an osx m2 chip by the way, not sure if that has anything to do with it.

Carsten Behring20:10:48

could be. I never tested it on OSX

az20:10:06

Yeah same error.

; ERROR: Unhandled REPL handler exception processing message {:op stacktrace, :id 20, :session b386c172-3029-4685-b7f5-55ef7e85f1cf}
; java.util.concurrent.ExecutionException: java.lang.NoClassDefFoundError: Could not initialize class org.bytedeco.openblas.global.openblas_nolapack
; 	at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
; 	at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
; 	at clojure.core$deref_future.invokeStatic(core.clj:2317)
; 	at clojure.core$future_call$reify__8744.deref(core.clj:7055)
; 	at clojure.core$deref.invokeStatic(core.clj:2337)
; 	at clojure.core$pmap$step__8757$fn__8761.invoke(core.clj:7106)
; 	at clojure.lang.LazySeq.sval(LazySeq.java:44)
; 	at clojure.lang.LazySeq.seq(LazySeq.java:53)
; 	at clojure.lang.RT.seq(RT.java:535)
; 	at clojure.core$seq__5579.invokeStatic(core.clj:139)
; 	at clojure.core$map$fn__6053.invoke(core.clj:2774)
; 	at clojure.lang.LazySeq.sval(LazySeq.java:44)
; 	at clojure.lang.LazySeq.seq(LazySeq.java:53)
; 	at clojure.lang.Cons.next(Cons.java:41)
; 	at clojure.lang.RT.next(RT.java:713)
; 	at clojure.core$next__5563.invokeStatic(core.clj:64)
; 	at clojure.core.protocols$fn__8447

az20:10:43

when I eval the ns form I get this.

..instrumented #'
..instrumented #'
..instrumented #'
..instrumented #'!
..instrumented #'
..instrumented #'
..instrumented #'
..instrumented #'
Register model:  :smile.manifold/isomap
Register model:  :smile.manifold/laplacian
Register model:  :smile.manifold/lle
Register model:  :smile.manifold/tsne
Register model:  :smile.manifold/umap

Then when I try to run the code I that that error above

az20:10:10

I was able to squash the org.bytedeco.openblas.global.openblas_nolapack error, when I manually import, but couldn't figure out how to get the other dep working, arpack

Carsten Behring20:10:42

I would say it is a "Smile on MAC" issue. Looking in smile issues revelaed something similar: https://github.com/haifengl/smile/issues/619

az20:10:44

Also, I can run parts of this example, just not the ml/model portion.

(let [pinguins (-> (tc/dataset
                      ""
                      {:key-fn csk/->kebab-case-keyword}))
        pipe (mm/pipeline
              (tc-mm/drop-missing)
              (tc-mm/select-columns [:culmen-length-mm :culmen-depth-mm :flipper-length-mm :body-mass-g])
              (scicloj.metamorph.ml.preprocessing/std-scale :type/numerical {})
              {:metamorph/id :model}
              (ml/model {:model-type :smile.manifold/isomap
                         :args [4 5 false]}))
        fit-ctx (mm/fit-pipe pinguins pipe)]

    fit-ctx)

az20:10:53

I'm going to try those deps listed

Carsten Behring20:10:35

These are for Smile 2.5.3, while 0.3 use smile 2.6.0

az20:10:44

I remember now I came across this thread and brought the deps in but bumped the versions to latest, maybe that's the issue

Carsten Behring20:10:56

maybe here is a better deps.edn: https://github.com/haifengl/smile/tree/v2.6.0/clojure

Carsten Behring20:10:29

Or maybe passing -Dorg.bytedeco.openblas.load=mkl_rt to JVM is needed.

Carsten Behring20:10:43

On linux none of this is required.

az20:10:15

trying this now

az20:10:08

@U7CAHM72M, yeah no luck, added the additional startup arg for mlk as well

genmeblog20:10:08

Can you show full dependency tree from the project?

az20:10:35

yes, one moment

az20:10:14

I'm going to setup a new project and strip all the other deps, it's a polylith project I haven't tried to strip everything else out yet

genmeblog20:10:09

Maybe there is something what injects different version of arpack-ng or so...

az20:10:01

ok, yeah so still getting the same issue with just these deps: https://github.com/brancusi/test-smile/blob/main/deps.edn

az20:10:24

that's the sample project

genmeblog21:10:48

I'm getting different error...

1. Unhandled java.lang.NoSuchMethodError
   'void org.bytedeco.arpackng.global.arpack.dseupd_c(boolean, byte[],
   int[], double[], double[], int, double, byte[], int, byte[], int,
   double, double[], int, double[], int, int[], int[], double[],
   double[], int, int[])'

               ARPACK.java:  180  smile.math.matrix.ARPACK/syev
               ARPACK.java:  113  smile.math.matrix.ARPACK/syev
               IsoMap.java:  188  smile.manifold.IsoMap/of
               IsoMap.java:  120  smile.manifold.IsoMap/of
              manifold.clj:   61  smile.manifold/isomap
              manifold.clj:   21  smile.manifold/isomap
                  AFn.java:  165  clojure.lang.AFn/applyToHelper
                  AFn.java:  144  clojure.lang.AFn/applyTo
                  Var.java:  705  clojure.lang.Var/applyTo
                  core.clj:  669  clojure.core/apply
                  core.clj:  662  clojure.core/apply
              manifold.clj:   26  scicloj.ml.smile.manifold/manifold
              manifold.clj:   23  scicloj.ml.smile.manifold/manifold
              manifold.clj:   31  scicloj.ml.smile.manifold/train/fn
                    ml.clj:  542  
                    ml.clj:  517  
                  AFn.java:  156  clojure.lang.AFn/applyToHelper
                  AFn.java:  144  clojure.lang.AFn/applyTo
                  core.clj:  667  clojure.core/apply
                  core.clj:  662  clojure.core/apply
                 core.cljc: 2582  malli.core$_instrument$fn__40000/doInvoke
               RestFn.java:  421  clojure.lang.RestFn/invoke
                    ml.clj:  709  
                  AFn.java:  154  clojure.lang.AFn/applyToHelper
                  AFn.java:  144  clojure.lang.AFn/applyTo
                  core.clj:  667  clojure.core/apply
                  core.clj:  662  clojure.core/apply
                 core.cljc: 2582  malli.core$_instrument$fn__40000/doInvoke
               RestFn.java:  408  clojure.lang.RestFn/invoke
                  core.clj:   39  scicloj.metamorph.core/pipeline/local-pipeline/fn
     PersistentVector.java:  343  clojure.lang.PersistentVector/reduce
                  core.clj: 6885  clojure.core/reduce
                  core.clj: 6868  clojure.core/reduce
                  core.clj:   33  scicloj.metamorph.core/pipeline/local-pipeline
                  core.clj:  181  scicloj.metamorph.core/fit-pipe
                  core.clj:  176  scicloj.metamorph.core/fit-pipe
                      REPL:   26  test-smile.test-smile/eval43697
                      REPL:   16  test-smile.test-smile/eval43697

Carsten Behring21:10:16

same for me. The deps.edn are wrong in my view.

genmeblog21:10:17

looks like some versions mismatch

Carsten Behring21:10:49

have a deps.edn like this:

{:paths ["src" "resources"]
 :deps {org.clojure/clojure {:mvn/version "1.11.1"}

        scicloj/scicloj.ml {:mvn/version "0.3"}}
}

does work on Linux.

genmeblog21:10:32

this works for me:

{:paths ["src" "resources"]
 :deps {org.clojure/clojure {:mvn/version "1.11.1"}

        com.github.haifengl/smile-interpolation {:mvn/version "2.6.0"}
        com.github.haifengl/smile-core {:mvn/version "2.6.0"}
        com.github.haifengl/smile-mkl {:mvn/version "2.6.0"}

        org.bytedeco/javacpp {:mvn/version "1.5.4"}

        org.bytedeco/arpack-ng-platform {:mvn/version "3.7.0-1.5.4"}
        org.bytedeco/arpack-ng {:mvn/version "3.7.0-1.5.4"}

        org.bytedeco/openblas-platform {:mvn/version "0.3.10-1.5.4"}
        org.bytedeco/openblas {:mvn/version "0.3.10-1.5.4"}

        scicloj/scicloj.ml {:mvn/version "0.3"}}
 :aliases
 {:dev {:jvm-opts ["-Dorg.bytedeco.openblas.load=mkl_rt"]}

  :build {:deps {io.github.clojure/tools.build
                 {:git/tag "v0.9.2" :git/sha "fe6b140"}}
          :ns-default build}
  :test {:extra-paths ["test"]
         :extra-deps {org.clojure/test.check {:mvn/version "1.1.1"}
                      io.github.cognitect-labs/test-runner
                      {:git/tag "v0.5.1" :git/sha "dfb30dd"}}} :neil {:project {:name test-smile/test-smile}}}}

az21:10:00

@U1EP3BZ3Q are you on mac? With those deps I get:

; ERROR: Unhandled REPL handler exception processing message {:op stacktrace, :id 19, :session cc1a7e9c-6910-4880-bf61-6ac018f69927}
; java.util.concurrent.ExecutionException: java.lang.NoClassDefFoundError: Could not initialize class org.bytedeco.openblas.global.openblas_nolapack
; 	at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
; 	at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
; 	at clojure.core$deref_future.invokeStatic(core.clj:2317)
; 	at clojure.core$future_call$reify__8544.deref(core.clj:7041)
; 	at clojure.core$deref.invokeStatic(core.clj:2337)
; 	at clojure.core$pmap$step__8557$fn__8561.invoke(core.clj:7092)
; 	at clojure.lang.LazySeq.sval(LazySeq.java:42)
; 	at clojure.lang.LazySeq.seq(LazySeq.java:51)
; 	at clojure.lang.RT.seq(RT.java:535)
; 	at clojure.core$seq__5467.invokeStatic(core.clj:139)
; 	at clojure.core$map$fn__5939.invoke(core.clj:2774)
; 	at clojure.lang.LazySeq.sval(LazySeq.java:42)
; 	at clojure.lang.LazySeq.seq(LazySeq.java:51)
; 	at clojure.lang.Cons.next(Cons.java:39)
; 	at clojure.lang.RT.next(RT.java:713)
; 	at clojure.core$next__5451.invokeStatic(core.clj:64)
; 	at clojure.core.protocols$fn__8249

genmeblog21:10:13

nope, Linux on WSL2

genmeblog21:10:44

probably this is mac issue as Carsten suspects

az21:10:50

yeah

Carsten Behring21:10:48

I suggest to reproduce it in Clojure without using at all. is calling this: https://github.com/haifengl/smile/blob/1485bdce329d5502305026173257436fe47caba1/clojure/src/smile/manifold.clj#L21 So if you have an error with calling it using

[org.clojars.haifengl/smile "2.6.0"]

and this setup: https://github.com/haifengl/smile/blob/1485bdce329d5502305026173257436fe47caba1/clojure/README.md?plain=1#L21 you can report an issue against Smile (v 2.6.0)

az21:10:26

Thank you

az21:10:44

I'm going to try it

az21:10:00

I really appreciate the help on all this

Carsten Behring21:10:48

This https://github.com/haifengl/smile/issues/743

Please set the "org.bytedeco.javacpp.logger.debug" system property to "true" to get more information on the console.

genmeblog21:10:02

Ok, I probably found the problem. Not every bytedeco stuff is compiled for M2 chips. When you look at the pom file of arpack-ng there is no javacpp.platform.macosx-arm64 platform (which exists in other projects).

genmeblog21:10:28

So the first step is to ask bytedeco people to include M2 platform for builds. The second step is to make it work with SMILE...

genmeblog21:10:42

Here are some issues regarding OpenBlas/Lapack on M1/M2, looks like solved already... https://github.com/bytedeco/javacpp-presets/issues/1171

az23:10:23

I'm going to keep going through this and see if I find anything new.

2023-09-30

Channels