Fork me on GitHub
#clojure
<
2021-09-08
>
Empperi05:09:09

Asked this at #numerical-computing but got no answers maybe due to low amount of people there. So, I'll ask again in here: Trying out Neanderthal but immediately facing issues with MKL. I have included `org.bytedeco/mkl-platform-redist` as dependency and I also tried installing MKL manually but I'm still facing issues. Basically I'm getting a macroexpansion level error at `mkl.clj:90:35` of `UnsatisfiedLinkError`. Basically it is complaining of `neanderthal-mkl-0.33.06349245150535543233.dll: Can't find dependent libraries`. I'm a bit stuck here. Running win 10 x64 with Ryzen CPU. Would appreciate any ideas on this matter.

Empperi13:09:08

Forever alone 😭

Empperi05:09:10

Interesting. Thanks, I'll look into this

Empperi05:09:45

It's at least something to hopefully get me forward

blueberry22:09:16

@U7MRZK43B All this generic JNI answers are probably not relevant to you, as Neanderthal and/or javacpp should handle all stuff automatically. Please check the getting started guide at the neanderthal homepage. As a first time windows user, you should probably try the javacpp route. Why that is not working for you now, I can't be sure withoud more details about the stacktraces you're getting and the details of your setup.

Empperi05:09:56

Cheers @blueberry. I understand JNI can be tricky. Just FYI, I also did try the examples in Neanderthal repo with same end result. As this is a hobby thing my progress and time is limited with small kids. Is it ok for me to ping you if I have something more concrete to ask and I do get forward with this? But what I do have right now is easy to copy-paste here:

Empperi05:09:00

(ns digit.core
  (:require [uncomplicate.commons.core :as cmn]
            [uncomplicate.neanderthal.native :as native]
            [uncomplicate.neanderthal.core :as core]))

(cmn/with-release [x  (native/dv 0.3 0.9)
                   w1 (native/dge 4 2 [0.3 0.6
                                       0.1 2.0
                                       0.9 3.7
                                       0.0 1.0]
                                  {:layout :row})
                   h1 (native/dv 4)]
  (println (core/mv! w1 x h1)))

Empperi05:09:19

(defproject digit-recognizer "1.0.0"
  :dependencies [[org.clojure/clojure "1.10.3"]
                 [uncomplicate/neanderthal "0.43.2"]
                 ; MKL accelerated math libraries
                 [org.bytedeco/mkl-platform-redist "2021.3-1.5.6"]]

  :exclusions [[org.jcuda/jcuda-natives :classifier "apple-x86_64"]
               [org.jcuda/jcublas-natives :classifier "apple-x86_64"]]

  :profiles {:dev {:global-vars {*warn-on-reflection* true
                                 *assert*             false
                                 *unchecked-math*     :warn-on-boxed
                                 *print-length*       128}}}

  :main digit.core

  :jvm-opts ^:replace ["-Dclojure.compiler.direct-linking=true"
                       "-XX:MaxDirectMemorySize=16g" "-XX:+UseLargePages"
                       "--add-opens=java.base/jdk.internal.ref=ALL-UNNAMED"
                       "--add-opens=java.base/sun.nio.ch=ALL-UNNAMED"])

Empperi05:09:49

And the stacktrace I get at macro expansion time:

Empperi05:09:51

#error {
 :cause C:\Users\Niklas\AppData\Local\Temp\neanderthal-mkl-0.33.08800107790577691303.dll: Can't find dependent libraries
 :via
 [{:type clojure.lang.Compiler$CompilerException
   :message Syntax error macroexpanding at (mkl.clj:90:35).
   :data #:clojure.error{:phase :execution, :line 90, :column 35, :source mkl.clj}
   :at [clojure.lang.Compiler$InvokeExpr eval Compiler.java 3711]}
  {:type java.lang.UnsatisfiedLinkError
   :message C:\Users\Niklas\AppData\Local\Temp\neanderthal-mkl-0.33.08800107790577691303.dll: Can't find dependent libraries
   :at [jdk.internal.loader.NativeLibraries load NativeLibraries.java -2]}]
 :trace
 [[jdk.internal.loader.NativeLibraries load NativeLibraries.java -2]
  [jdk.internal.loader.NativeLibraries$NativeLibraryImpl open NativeLibraries.java 383]
  [jdk.internal.loader.NativeLibraries loadLibrary NativeLibraries.java 227]
  [jdk.internal.loader.NativeLibraries loadLibrary NativeLibraries.java 169]
  [java.lang.ClassLoader loadLibrary ClassLoader.java 2407]
  [java.lang.Runtime load0 Runtime.java 747]
  [java.lang.System load System.java 1857]
  [uncomplicate.neanderthal.internal.host.NarSystem loadLibrary NarSystem.java 48]
  [uncomplicate.neanderthal.internal.host.MKL <clinit> MKL.java 16]
  [uncomplicate.neanderthal.internal.host.mkl$create_stream_ars5 invokeStatic mkl.clj 87]
  [uncomplicate.neanderthal.internal.host.mkl$create_stream_ars5 invoke mkl.clj 85]
  [clojure.lang.AFn applyToHelper AFn.java 154]
  [clojure.lang.AFn applyTo AFn.java 144]
  [clojure.lang.Compiler$InvokeExpr eval Compiler.java 3706]
  [clojure.lang.Compiler$DefExpr eval Compiler.java 457]
  [clojure.lang.Compiler eval Compiler.java 7186]
  [clojure.lang.Compiler load Compiler.java 7640]
  [clojure.lang.RT loadResourceScript RT.java 381]
  [clojure.lang.RT loadResourceScript RT.java 372]
  [clojure.lang.RT load RT.java 459]
  [clojure.lang.RT load RT.java 424]
  [clojure.core$load$fn__6856 invoke core.clj 6115]
  [clojure.core$load invokeStatic core.clj 6114]
  [clojure.core$load doInvoke core.clj 6098]
  [clojure.lang.RestFn invoke RestFn.java 408]
  [clojure.core$load_one invokeStatic core.clj 5897]
  [clojure.core$load_one invoke core.clj 5892]
  [clojure.core$load_lib$fn__6796 invoke core.clj 5937]
  [clojure.core$load_lib invokeStatic core.clj 5936]
  [clojure.core$load_lib doInvoke core.clj 5917]
  [clojure.lang.RestFn applyTo RestFn.java 142]
  [clojure.core$apply invokeStatic core.clj 669]
  [clojure.core$load_libs invokeStatic core.clj 5978]
  [clojure.core$load_libs doInvoke core.clj 5958]
  [clojure.lang.RestFn applyTo RestFn.java 137]
  [clojure.core$apply invokeStatic core.clj 669]
  [clojure.core$require invokeStatic core.clj 5996]
  [uncomplicate.neanderthal.native$eval463$loading__6737__auto____464 invoke native.clj 9]
  [uncomplicate.neanderthal.native$eval463 invokeStatic native.clj 9]
  [uncomplicate.neanderthal.native$eval463 invoke native.clj 9]
  [clojure.lang.Compiler eval Compiler.java 7181]
  [clojure.lang.Compiler eval Compiler.java 7170]
  [clojure.lang.Compiler load Compiler.java 7640]
  [clojure.lang.RT loadResourceScript RT.java 381]
  [clojure.lang.RT loadResourceScript RT.java 372]
  [clojure.lang.RT load RT.java 459]
  [clojure.lang.RT load RT.java 424]
  [clojure.core$load$fn__6856 invoke core.clj 6115]
  [clojure.core$load invokeStatic core.clj 6114]
  [clojure.core$load doInvoke core.clj 6098]
  [clojure.lang.RestFn invoke RestFn.java 408]
  [clojure.core$load_one invokeStatic core.clj 5897]
  [clojure.core$load_one invoke core.clj 5892]
  [clojure.core$load_lib$fn__6796 invoke core.clj 5937]
  [clojure.core$load_lib invokeStatic core.clj 5936]
  [clojure.core$load_lib doInvoke core.clj 5917]
  [clojure.lang.RestFn applyTo RestFn.java 142]
  [clojure.core$apply invokeStatic core.clj 669]
  [clojure.core$load_libs invokeStatic core.clj 5974]
  [clojure.core$load_libs doInvoke core.clj 5958]
  [clojure.lang.RestFn applyTo RestFn.java 137]
  [clojure.core$apply invokeStatic core.clj 669]
  [clojure.core$require invokeStatic core.clj 5996]
  [digit.core$eval157$loading__6737__auto____158 invoke core.clj 1]
  [digit.core$eval157 invokeStatic core.clj 1]
  [digit.core$eval157 invoke core.clj 1]
  [clojure.lang.Compiler eval Compiler.java 7181]
  [clojure.lang.Compiler eval Compiler.java 7170]
  [clojure.lang.Compiler load Compiler.java 7640]
  [clojure.lang.RT loadResourceScript RT.java 381]
  [clojure.lang.RT loadResourceScript RT.java 372]
  [clojure.lang.RT load RT.java 459]
  [clojure.lang.RT load RT.java 424]
  [clojure.core$load$fn__6856 invoke core.clj 6115]
  [clojure.core$load invokeStatic core.clj 6114]
  [clojure.core$load doInvoke core.clj 6098]
  [clojure.lang.RestFn invoke RestFn.java 408]
  [clojure.core$load_one invokeStatic core.clj 5897]
  [clojure.core$load_one invoke core.clj 5892]
  [clojure.core$load_lib$fn__6796 invoke core.clj 5937]
  [clojure.core$load_lib invokeStatic core.clj 5936]
  [clojure.core$load_lib doInvoke core.clj 5917]
  [clojure.lang.RestFn applyTo RestFn.java 142]
  [clojure.core$apply invokeStatic core.clj 669]
  [clojure.core$load_libs invokeStatic core.clj 5974]
  [clojure.core$load_libs doInvoke core.clj 5958]
  [clojure.lang.RestFn applyTo RestFn.java 137]
  [clojure.core$apply invokeStatic core.clj 669]
  [clojure.core$require invokeStatic core.clj 5996]
  [user$eval13 invokeStatic form-init7205182847745397661.clj 1]
  [user$eval13 invoke form-init7205182847745397661.clj 1]
  [clojure.lang.Compiler eval Compiler.java 7181]
  [clojure.lang.Compiler eval Compiler.java 7170]
  [clojure.lang.Compiler eval Compiler.java 7170]
  [clojure.lang.Compiler load Compiler.java 7640]
  [clojure.lang.Compiler loadFile Compiler.java 7578]
  [clojure.main$load_script invokeStatic main.clj 475]
  [clojure.main$init_opt invokeStatic main.clj 477]
  [clojure.main$init_opt invoke main.clj 477]
  [clojure.main$initialize invokeStatic main.clj 508]
  [clojure.main$null_opt invokeStatic main.clj 542]
  [clojure.main$null_opt invoke main.clj 539]
  [clojure.main$main invokeStatic main.clj 664]
  [clojure.main$main doInvoke main.clj 616]
  [clojure.lang.RestFn applyTo RestFn.java 137]
  [clojure.lang.Var applyTo Var.java 705]
  [clojure.main main main.java 40]]}

Empperi05:09:51

This happens with org.bytedeco/mkl-platform-redist and without it. I've pointed java.library.path to separately installed MKL dlls. No dice.

Empperi05:09:07

And I guess that stacktrace doesn't help you at all in understanding what the issue is since it does not provide any important information beyond what I've already told you

Empperi05:09:07

Oh, and I have checked getting started guides and whatnot. 🙂 I spent quite some time on this issue before going to Slack. I'm a Neanderthal beginner but definitely not Clojure beginner and I've been following Neanderthal for years

Empperi05:09:27

But my take on this is that "it should work" but it doesn't. Guess I have to clone Neanderthal repo and start debugging 🤷

blueberry19:09:13

@U7MRZK43B It seems to me that it's worth checking your javacpp-platform copy in the .m2 folder. NOTE: it's a huge (500+ MB or something like that) jar. Some people had the following gotcha with such libraries: 1) they run lein repl 2) lein starts huge download in the background 3) user thinks nothing is happening and kills the process 4) javacpp jar is broken. If you suspect this might be your case, just delete bytedeco from your maven repository, go to https://github.com/uncomplicate/neanderthal/tree/master/examples/hello-world and run lein.deps. In any case I'd recommend trying this hello world before running your own project. Anyway, I don't have a copy of windows, so I'm not sure what else to try, but some people gave helpful pointers here and in older threads (please search google or github issues).

Empperi05:09:55

Thanks @blueberry! This sounds like a possible culprit. I'll definitely try it out

jjttjj13:09:24

Any interesting libraries related to applying reducers/transducers to tabular data? For instance, I have a vector of row maps all with the same keys, each key representing a column. I want to derive other columns from that data and add them to each row, like adding a column that is a moving average of another column, or adding a column that is a Boolean comparison of two other columns. I really like the composable nature of transducers for this and have some hacky things that sort of let you work with transducers for this type of thing. But transducers are not quite a perfect fit out of the box without adding on more assumptions (like making sure there's a 1:1 output for every input). Curious if there's anything out there that explored this

jjttjj16:09:37

Yup I've been leaning on that heavily so far. It's definitely a key piece to the puzzle but I haven't fully been able to come up with a coherent layer on top.

;;; this works if each transducer yields exactly one input per output
(defn assoc-xf [& kvs]
  (assert (even? (count kvs)))
  (let [k->xf (apply hash-map kvs)
        k->xf (assoc k->xf ::this (map identity))
        ct    (count k->xf)]
    (comp (x/multiplex k->xf)
          (x/partition ct (x/into {}))
          (map (fn [m] (merge (::this m) (dissoc m ::this)))))))

(defn windows
  "like partition-all with step of one but outputs window on each item"
  ([n]
   (x/window n conj (fn [acc _]  (subvec acc 1 n)))))

(def data [{:a 1 :b 2} {:a 3 :b 5} {:a 9 :b 20} {:a 2 :b 5} {:a 3 :b 5}])

(->> data
     (into []
       (assoc-xf
         :sumA         (comp (map :a) (x/reductions + 0) (drop 1))
         :rolling-sumA (comp (map :a) (windows 2) (map (fn [xs] (reduce + 0 xs)))))))

;;=>
[{:a 1, :b 2, :sumA 1, :rolling-sumA 1}
 {:a 3, :b 5, :sumA 4, :rolling-sumA 4}
 {:a 9, :b 20, :sumA 13, :rolling-sumA 12}
 {:a 2, :b 5, :sumA 15, :rolling-sumA 11}
 {:a 3, :b 5, :sumA 18, :rolling-sumA 5}]

Sidestep15:09:06

Just learned that blocking take and put on a channel (<!! and >!!) has no timeout. I get that channels have to be bounded by size, to add backpreasure and design withing queue limits in mind. Ok, but how do you avoid cascading failure if there's no time-out on blocking take and put then? What are the patterns/practices in cloujure?

ghadi15:09:56

look at alts!!

👆 4
raspasov19:09:18

There’s also offer!

Sidestep15:09:06

aaa ok add timeout channel

Chase15:09:22

I've been exploring deployment options like using a Digital Ocean droplet or something like Heroku (AWS seems super intimidating but should I look at that too?) for a full stack Clojure web app. I'm curious about the RAM requirements you folks are seeing. The lower level production options only provide 512mb ram or 1gb. Is that going to be enough for a fledgling saas project or is the JVM going to want more RAM? Am I even asking the right questions here?

p-himik15:09:42

For a low traffic server, 512 MB could be enough. But, of course, the real answer is "it depends". In my own experience with Heroku, despite its JVM settings my Clojure apps still take a bit more, resulting in a deluge of warnings like "RAM consumption is at 103%, the app might become slower" or something like that. At this point, I'm just ignoring them since neither raising the memory nor adjusting the JVM settings helped. Perhaps I've done something wrong, so I should try again at some point. I also wanted to play with G1 GC - apparently, it gives the memory back to the OS when it's freed.

p-himik15:09:02

It shouldn't be too hard to estimate the lower boundary for your SaaS and how it depends on the amount of users. You might be able to calculate whether Heroku would be worth it given the amount of resources you'll need for the amount of users you'll have.

emccue16:09:53

my advice - as long as you can avoid total lock in, go for heroku and overprovision

emccue16:09:11

figure out the amount of time it would take you set up aws and determine the right size for your app, multiply it by a reasonable hourly wage - if that number is less than the extra cost for heroku and a big instance just do it and fine tune later

Chase16:09:46

That's what I was thinking. The 1gb dyno is $50 plus $50 for the postgres db. Sadly that is still tight for me but if the app has any kind of success that cost is covered almost immediately. I just wasn't sure if 1gb would be enough.

javahippie17:09:28

Switched my stuff from Heroku to DigitalOcean in June, and DB pricing was one of the big reasons. The apps running there are on the smaller side, so I don’t need to spend 50$ on a DB instance.

javahippie17:09:35

Now I’m at ~60$ a month for a K8S cluster with two nodes (preference, Droplets/Apps do fine, too), a Postgres Instance and a private Docker registry

Chase17:09:57

And how would you compare the effort and dev ops know how needed for that switch? Is it pretty simple?

javahippie17:09:08

With Kubernetes I don’t think so, that was a learning curve for me and I had prior experience. But the Digital Ocean Apps work similar to buildpacks on Heroku, although Clojure Support is not as “out of the box” as on Heroku.

p-himik17:09:13

It's not that OOTB with Heroku either if you use deps.edn. The default buildpack supports only project.clj.

javahippie17:09:39

One thing I just remembered: When adding shadow-cljs + npm, the default clojure buildpacks on heroku stopped working for me, so I pushed a Docker Image to Heroku. That also got rid of the memory warnings @U2FRKM4TW described

mikeb03:09:55

What about cloud run set to minimum of 1 instance? It looks like cost would be $13/month for 1GB and only only $9 if you can get by on 512mb.

mikeb03:09:35

I wonder if OpenJ9 could help keep the memory footprint low too.

jumar15:09:31

We have been running our staging enc on aws t2.micro for years and it was good enough. It really depends on tge workload but in our case it is a nontrivial web app

Chase18:09:18

Nice to see some other options. I have heroku up and running fine now so I'm going to roll with that. They do have some cool JVM metrics running so this will be fun to explore and see how far the lower tiers take me.

Lucas Jordan16:09:22

Greetings, some time ago I decided to attempt to implement d/entity (datomic) so I could use it (carefully) with diatomic client api. For the most part it works. I used deftype to implemented a bunch of map interfaces, which I call DatomicReferenceMap. This works great. I can create it with an eid from datomic and it looks and acts like a map, and when you access a key that is a sub entity, it goes and fetches it. I also implemented clojure.lang.IFn, so I can do (entity :key) like a real map. However, when I try and create a new datomic entity that references one of these things, I get :db.error/not-an-entity Unable to resolve entity. If I replace with my entity with a ‘normal’ map, it works right. I am hoping I just need to implement another interface or two to make it work. Does anyone know what interfaces datomic expects for referenced maps? Thanks in advance!

Lucas Jordan16:09:46

Here is basically what I am talking about

Lucas Jordan16:09:42

(defn add-and-get [conn entity](let [temp-id1 "temp-id1"
                                     tx (d/transact conn {:tx-data [(assoc entity :db/id temp-id1)]})
                                     eid ((:tempids tx) temp-id1)
                                     after  (:db-after tx)
                                     pulled (d/pull after '[*] eid)] ;pulled (de/entity after eid) ;pulled (d/pull after '[*] eid)]
                                 pulled))

zane18:09:46

I’ve got an application that utilizes some polymorphic functions (currently defined as protocols, but I’m likely going to switch them over to being multimethods). I’d like third-parties to be able to define their own implementations for those polymorphic functions for types that they control. Ideally all that would be required for the new implementations to be utilized by the application is for the library in which they’re defined to be on the classpath. I’ve seen this sometimes described as a “https://yogthos.net/posts/2015-01-15-A-Plugin-System-in-Clojure.html”. Are there best practices for going about this that I should be aware of? The approach in the linked blog post would certainly work, but I’m trying to get a sense of what possible alternatives might be / learn more about the design space.

Russell Mull18:09:02

I wouldn't recommend load-plugin as written there. You should be able to do something with require instead, which should work more cleanly with AOT scenarios. (Edit: I see they're doing that, after reading the source... still, it's strange. You should be able to put together something that works without needing to inspect the source)

hiredman18:09:43

it isn't inspecting the source

hiredman18:09:13

it is using a "well known" resource name to find possibly multiple instances of plugins

Russell Mull18:09:27

OH I see, it's a manifest. n/m.

hiredman18:09:37

the issue with that is uberjaring will often collide those things

Russell Mull18:09:20

"Ideally all that would be required for the new implementations to be utilized by the application is for the library in which they’re defined to be on the classpath" is the source of a fair amount of complexity. If it's just there for convenience, it would probably be better to just require all your plugins from the top level of your application, in some way. You'll still get the structural benefits of the abstraction you set up with multimethods or protocols.

hiredman18:09:46

like, if you are built on polymorphic constructs (protocols or defmethods) and I extend those, and call you with something that takes part in my extension, then you just work

hiredman18:09:00

why have a plugin system?

Russell Mull18:09:40

If you really want users to be able to supply these things at runtime, then the options that come to mind are: • Java-style: use protocols, and require that all the plugin jars are AOTed and the impls are gen-classed. Scan the classpath for implementations of your plugin interface. • Manifests-in-jars, like the article you linked. As @U0NCTKEV8 says, uberjar-ing may bite you in the future. • Say that plugins have to be .clj files, or that your jars can never be AOTed. • There may be some other way to identify things on the classpath that are actually Clojure namespaces, but I don't know what it is.

zane18:09:31

This is all very helpful. Thanks!

noisesmith19:09:58

an uberjar can contain multiple files with the same path, but it's up to the consumer of those files to iterate and load them all, as opposed to the first found

zane19:09:18

The current usage pattern is that the application is launched via the Clojure CLI. Additional implementations for the polymorphic functions come via libraries that are included via -Sdeps. Those additional implementations may sometimes be in .cljc files, though I can see why such a plugin system would not work in the ClojureScript context.

noisesmith19:09:58

it's probably worth looking at the way clojure currently does this exact thing with *data-readers*

(cmd)user=> (doc *data-readers*)
-------------------------
clojure.core/*data-readers*
  Map from reader tag symbols to data reader Vars.

  When Clojure starts, it searches for files named 'data_readers.clj'
  and 'data_readers.cljc' at the root of the classpath. Each such file
  must contain a literal map of symbols, like this:

noisesmith19:09:39

except instead of the root of the classpath you'd probably want /my/project/construct/plugins.clj or whatever

zane19:09:47

I’ll have a look!

hiredman18:09:27

in general, as a library, I would not want to load code, and would say that approach general leads to things being globally defined and less flexible, e.g. if your library loads and registers plugins, then how do I opt out of using the plugins while still making the plugins available for use to other libraries I am using