This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2021-09-08
Channels
- # announcements (32)
- # aws (2)
- # babashka (21)
- # beginners (143)
- # cider (3)
- # cljsrn (13)
- # clojure (65)
- # clojure-dev (7)
- # clojure-europe (20)
- # clojure-losangeles (8)
- # clojure-nl (13)
- # clojure-norway (39)
- # clojure-uk (9)
- # clojurescript (39)
- # code-reviews (10)
- # conjure (2)
- # cursive (3)
- # datascript (6)
- # datomic (40)
- # events (5)
- # exercism (23)
- # fulcro (9)
- # funcool (2)
- # girouette (2)
- # graphql (4)
- # helix (8)
- # improve-getting-started (4)
- # integrant (7)
- # introduce-yourself (5)
- # jobs (3)
- # luminus (32)
- # malli (3)
- # off-topic (10)
- # pathom (9)
- # pedestal (4)
- # polylith (25)
- # practicalli (1)
- # re-frame (4)
- # sci (3)
- # shadow-cljs (5)
- # tools-deps (25)
- # vim (31)
- # xtdb (32)
Asked this at #numerical-computing but got no answers maybe due to low amount of people there. So, I'll ask again in here: Trying out Neanderthal but immediately facing issues with MKL. I have included `org.bytedeco/mkl-platform-redist` as dependency and I also tried installing MKL manually but I'm still facing issues. Basically I'm getting a macroexpansion level error at `mkl.clj:90:35` of `UnsatisfiedLinkError`. Basically it is complaining of `neanderthal-mkl-0.33.06349245150535543233.dll: Can't find dependent libraries`. I'm a bit stuck here. Running win 10 x64 with Ryzen CPU. Would appreciate any ideas on this matter.
What is the java.library.path
set to?
https://stackoverflow.com/questions/6092200/how-to-fix-an-unsatisfiedlinkerror-cant-find-dependent-libraries-in-a-jni-pro
@U7MRZK43B All this generic JNI answers are probably not relevant to you, as Neanderthal and/or javacpp should handle all stuff automatically. Please check the getting started guide at the neanderthal homepage. As a first time windows user, you should probably try the javacpp route. Why that is not working for you now, I can't be sure withoud more details about the stacktraces you're getting and the details of your setup.
Cheers @blueberry. I understand JNI can be tricky. Just FYI, I also did try the examples in Neanderthal repo with same end result. As this is a hobby thing my progress and time is limited with small kids. Is it ok for me to ping you if I have something more concrete to ask and I do get forward with this? But what I do have right now is easy to copy-paste here:
(ns digit.core
(:require [uncomplicate.commons.core :as cmn]
[uncomplicate.neanderthal.native :as native]
[uncomplicate.neanderthal.core :as core]))
(cmn/with-release [x (native/dv 0.3 0.9)
w1 (native/dge 4 2 [0.3 0.6
0.1 2.0
0.9 3.7
0.0 1.0]
{:layout :row})
h1 (native/dv 4)]
(println (core/mv! w1 x h1)))
(defproject digit-recognizer "1.0.0"
:dependencies [[org.clojure/clojure "1.10.3"]
[uncomplicate/neanderthal "0.43.2"]
; MKL accelerated math libraries
[org.bytedeco/mkl-platform-redist "2021.3-1.5.6"]]
:exclusions [[org.jcuda/jcuda-natives :classifier "apple-x86_64"]
[org.jcuda/jcublas-natives :classifier "apple-x86_64"]]
:profiles {:dev {:global-vars {*warn-on-reflection* true
*assert* false
*unchecked-math* :warn-on-boxed
*print-length* 128}}}
:main digit.core
:jvm-opts ^:replace ["-Dclojure.compiler.direct-linking=true"
"-XX:MaxDirectMemorySize=16g" "-XX:+UseLargePages"
"--add-opens=java.base/jdk.internal.ref=ALL-UNNAMED"
"--add-opens=java.base/sun.nio.ch=ALL-UNNAMED"])
#error {
:cause C:\Users\Niklas\AppData\Local\Temp\neanderthal-mkl-0.33.08800107790577691303.dll: Can't find dependent libraries
:via
[{:type clojure.lang.Compiler$CompilerException
:message Syntax error macroexpanding at (mkl.clj:90:35).
:data #:clojure.error{:phase :execution, :line 90, :column 35, :source mkl.clj}
:at [clojure.lang.Compiler$InvokeExpr eval Compiler.java 3711]}
{:type java.lang.UnsatisfiedLinkError
:message C:\Users\Niklas\AppData\Local\Temp\neanderthal-mkl-0.33.08800107790577691303.dll: Can't find dependent libraries
:at [jdk.internal.loader.NativeLibraries load NativeLibraries.java -2]}]
:trace
[[jdk.internal.loader.NativeLibraries load NativeLibraries.java -2]
[jdk.internal.loader.NativeLibraries$NativeLibraryImpl open NativeLibraries.java 383]
[jdk.internal.loader.NativeLibraries loadLibrary NativeLibraries.java 227]
[jdk.internal.loader.NativeLibraries loadLibrary NativeLibraries.java 169]
[java.lang.ClassLoader loadLibrary ClassLoader.java 2407]
[java.lang.Runtime load0 Runtime.java 747]
[java.lang.System load System.java 1857]
[uncomplicate.neanderthal.internal.host.NarSystem loadLibrary NarSystem.java 48]
[uncomplicate.neanderthal.internal.host.MKL <clinit> MKL.java 16]
[uncomplicate.neanderthal.internal.host.mkl$create_stream_ars5 invokeStatic mkl.clj 87]
[uncomplicate.neanderthal.internal.host.mkl$create_stream_ars5 invoke mkl.clj 85]
[clojure.lang.AFn applyToHelper AFn.java 154]
[clojure.lang.AFn applyTo AFn.java 144]
[clojure.lang.Compiler$InvokeExpr eval Compiler.java 3706]
[clojure.lang.Compiler$DefExpr eval Compiler.java 457]
[clojure.lang.Compiler eval Compiler.java 7186]
[clojure.lang.Compiler load Compiler.java 7640]
[clojure.lang.RT loadResourceScript RT.java 381]
[clojure.lang.RT loadResourceScript RT.java 372]
[clojure.lang.RT load RT.java 459]
[clojure.lang.RT load RT.java 424]
[clojure.core$load$fn__6856 invoke core.clj 6115]
[clojure.core$load invokeStatic core.clj 6114]
[clojure.core$load doInvoke core.clj 6098]
[clojure.lang.RestFn invoke RestFn.java 408]
[clojure.core$load_one invokeStatic core.clj 5897]
[clojure.core$load_one invoke core.clj 5892]
[clojure.core$load_lib$fn__6796 invoke core.clj 5937]
[clojure.core$load_lib invokeStatic core.clj 5936]
[clojure.core$load_lib doInvoke core.clj 5917]
[clojure.lang.RestFn applyTo RestFn.java 142]
[clojure.core$apply invokeStatic core.clj 669]
[clojure.core$load_libs invokeStatic core.clj 5978]
[clojure.core$load_libs doInvoke core.clj 5958]
[clojure.lang.RestFn applyTo RestFn.java 137]
[clojure.core$apply invokeStatic core.clj 669]
[clojure.core$require invokeStatic core.clj 5996]
[uncomplicate.neanderthal.native$eval463$loading__6737__auto____464 invoke native.clj 9]
[uncomplicate.neanderthal.native$eval463 invokeStatic native.clj 9]
[uncomplicate.neanderthal.native$eval463 invoke native.clj 9]
[clojure.lang.Compiler eval Compiler.java 7181]
[clojure.lang.Compiler eval Compiler.java 7170]
[clojure.lang.Compiler load Compiler.java 7640]
[clojure.lang.RT loadResourceScript RT.java 381]
[clojure.lang.RT loadResourceScript RT.java 372]
[clojure.lang.RT load RT.java 459]
[clojure.lang.RT load RT.java 424]
[clojure.core$load$fn__6856 invoke core.clj 6115]
[clojure.core$load invokeStatic core.clj 6114]
[clojure.core$load doInvoke core.clj 6098]
[clojure.lang.RestFn invoke RestFn.java 408]
[clojure.core$load_one invokeStatic core.clj 5897]
[clojure.core$load_one invoke core.clj 5892]
[clojure.core$load_lib$fn__6796 invoke core.clj 5937]
[clojure.core$load_lib invokeStatic core.clj 5936]
[clojure.core$load_lib doInvoke core.clj 5917]
[clojure.lang.RestFn applyTo RestFn.java 142]
[clojure.core$apply invokeStatic core.clj 669]
[clojure.core$load_libs invokeStatic core.clj 5974]
[clojure.core$load_libs doInvoke core.clj 5958]
[clojure.lang.RestFn applyTo RestFn.java 137]
[clojure.core$apply invokeStatic core.clj 669]
[clojure.core$require invokeStatic core.clj 5996]
[digit.core$eval157$loading__6737__auto____158 invoke core.clj 1]
[digit.core$eval157 invokeStatic core.clj 1]
[digit.core$eval157 invoke core.clj 1]
[clojure.lang.Compiler eval Compiler.java 7181]
[clojure.lang.Compiler eval Compiler.java 7170]
[clojure.lang.Compiler load Compiler.java 7640]
[clojure.lang.RT loadResourceScript RT.java 381]
[clojure.lang.RT loadResourceScript RT.java 372]
[clojure.lang.RT load RT.java 459]
[clojure.lang.RT load RT.java 424]
[clojure.core$load$fn__6856 invoke core.clj 6115]
[clojure.core$load invokeStatic core.clj 6114]
[clojure.core$load doInvoke core.clj 6098]
[clojure.lang.RestFn invoke RestFn.java 408]
[clojure.core$load_one invokeStatic core.clj 5897]
[clojure.core$load_one invoke core.clj 5892]
[clojure.core$load_lib$fn__6796 invoke core.clj 5937]
[clojure.core$load_lib invokeStatic core.clj 5936]
[clojure.core$load_lib doInvoke core.clj 5917]
[clojure.lang.RestFn applyTo RestFn.java 142]
[clojure.core$apply invokeStatic core.clj 669]
[clojure.core$load_libs invokeStatic core.clj 5974]
[clojure.core$load_libs doInvoke core.clj 5958]
[clojure.lang.RestFn applyTo RestFn.java 137]
[clojure.core$apply invokeStatic core.clj 669]
[clojure.core$require invokeStatic core.clj 5996]
[user$eval13 invokeStatic form-init7205182847745397661.clj 1]
[user$eval13 invoke form-init7205182847745397661.clj 1]
[clojure.lang.Compiler eval Compiler.java 7181]
[clojure.lang.Compiler eval Compiler.java 7170]
[clojure.lang.Compiler eval Compiler.java 7170]
[clojure.lang.Compiler load Compiler.java 7640]
[clojure.lang.Compiler loadFile Compiler.java 7578]
[clojure.main$load_script invokeStatic main.clj 475]
[clojure.main$init_opt invokeStatic main.clj 477]
[clojure.main$init_opt invoke main.clj 477]
[clojure.main$initialize invokeStatic main.clj 508]
[clojure.main$null_opt invokeStatic main.clj 542]
[clojure.main$null_opt invoke main.clj 539]
[clojure.main$main invokeStatic main.clj 664]
[clojure.main$main doInvoke main.clj 616]
[clojure.lang.RestFn applyTo RestFn.java 137]
[clojure.lang.Var applyTo Var.java 705]
[clojure.main main main.java 40]]}
This happens with org.bytedeco/mkl-platform-redist
and without it. I've pointed java.library.path
to separately installed MKL dlls. No dice.
And I guess that stacktrace doesn't help you at all in understanding what the issue is since it does not provide any important information beyond what I've already told you
Oh, and I have checked getting started guides and whatnot. 🙂 I spent quite some time on this issue before going to Slack. I'm a Neanderthal beginner but definitely not Clojure beginner and I've been following Neanderthal for years
But my take on this is that "it should work" but it doesn't. Guess I have to clone Neanderthal repo and start debugging 🤷
@U7MRZK43B It seems to me that it's worth checking your javacpp-platform copy in the .m2 folder. NOTE: it's a huge (500+ MB or something like that) jar. Some people had the following gotcha with such libraries: 1) they run lein repl 2) lein starts huge download in the background 3) user thinks nothing is happening and kills the process 4) javacpp jar is broken. If you suspect this might be your case, just delete bytedeco from your maven repository, go to https://github.com/uncomplicate/neanderthal/tree/master/examples/hello-world and run lein.deps. In any case I'd recommend trying this hello world before running your own project. Anyway, I don't have a copy of windows, so I'm not sure what else to try, but some people gave helpful pointers here and in older threads (please search google or github issues).
@U7MRZK43B Just in case, please try vanilla javacpp mkl example https://github.com/bytedeco/javacpp-presets/tree/master/mkl
Thanks @blueberry! This sounds like a possible culprit. I'll definitely try it out
Any interesting libraries related to applying reducers/transducers to tabular data? For instance, I have a vector of row maps all with the same keys, each key representing a column. I want to derive other columns from that data and add them to each row, like adding a column that is a moving average of another column, or adding a column that is a Boolean comparison of two other columns. I really like the composable nature of transducers for this and have some hacky things that sort of let you work with transducers for this type of thing. But transducers are not quite a perfect fit out of the box without adding on more assumptions (like making sure there's a 1:1 output for every input). Curious if there's anything out there that explored this
Have you looked at Tesser? https://github.com/aphyr/tesser
Yup I've been leaning on that heavily so far. It's definitely a key piece to the puzzle but I haven't fully been able to come up with a coherent layer on top.
;;; this works if each transducer yields exactly one input per output
(defn assoc-xf [& kvs]
(assert (even? (count kvs)))
(let [k->xf (apply hash-map kvs)
k->xf (assoc k->xf ::this (map identity))
ct (count k->xf)]
(comp (x/multiplex k->xf)
(x/partition ct (x/into {}))
(map (fn [m] (merge (::this m) (dissoc m ::this)))))))
(defn windows
"like partition-all with step of one but outputs window on each item"
([n]
(x/window n conj (fn [acc _] (subvec acc 1 n)))))
(def data [{:a 1 :b 2} {:a 3 :b 5} {:a 9 :b 20} {:a 2 :b 5} {:a 3 :b 5}])
(->> data
(into []
(assoc-xf
:sumA (comp (map :a) (x/reductions + 0) (drop 1))
:rolling-sumA (comp (map :a) (windows 2) (map (fn [xs] (reduce + 0 xs)))))))
;;=>
[{:a 1, :b 2, :sumA 1, :rolling-sumA 1}
{:a 3, :b 5, :sumA 4, :rolling-sumA 4}
{:a 9, :b 20, :sumA 13, :rolling-sumA 12}
{:a 2, :b 5, :sumA 15, :rolling-sumA 11}
{:a 3, :b 5, :sumA 18, :rolling-sumA 5}]
Just learned that blocking take and put on a channel (<!! and >!!) has no timeout. I get that channels have to be bounded by size, to add backpreasure and design withing queue limits in mind. Ok, but how do you avoid cascading failure if there's no time-out on blocking take and put then? What are the patterns/practices in cloujure?
I've been exploring deployment options like using a Digital Ocean droplet or something like Heroku (AWS seems super intimidating but should I look at that too?) for a full stack Clojure web app. I'm curious about the RAM requirements you folks are seeing. The lower level production options only provide 512mb ram or 1gb. Is that going to be enough for a fledgling saas project or is the JVM going to want more RAM? Am I even asking the right questions here?
For a low traffic server, 512 MB could be enough. But, of course, the real answer is "it depends". In my own experience with Heroku, despite its JVM settings my Clojure apps still take a bit more, resulting in a deluge of warnings like "RAM consumption is at 103%, the app might become slower" or something like that. At this point, I'm just ignoring them since neither raising the memory nor adjusting the JVM settings helped. Perhaps I've done something wrong, so I should try again at some point. I also wanted to play with G1 GC - apparently, it gives the memory back to the OS when it's freed.
It shouldn't be too hard to estimate the lower boundary for your SaaS and how it depends on the amount of users. You might be able to calculate whether Heroku would be worth it given the amount of resources you'll need for the amount of users you'll have.
figure out the amount of time it would take you set up aws and determine the right size for your app, multiply it by a reasonable hourly wage - if that number is less than the extra cost for heroku and a big instance just do it and fine tune later
That's what I was thinking. The 1gb dyno is $50 plus $50 for the postgres db. Sadly that is still tight for me but if the app has any kind of success that cost is covered almost immediately. I just wasn't sure if 1gb would be enough.
Switched my stuff from Heroku to DigitalOcean in June, and DB pricing was one of the big reasons. The apps running there are on the smaller side, so I don’t need to spend 50$ on a DB instance.
Now I’m at ~60$ a month for a K8S cluster with two nodes (preference, Droplets/Apps do fine, too), a Postgres Instance and a private Docker registry
And how would you compare the effort and dev ops know how needed for that switch? Is it pretty simple?
With Kubernetes I don’t think so, that was a learning curve for me and I had prior experience. But the Digital Ocean Apps work similar to buildpacks on Heroku, although Clojure Support is not as “out of the box” as on Heroku.
It's not that OOTB with Heroku either if you use deps.edn
. The default buildpack supports only project.clj
.
One thing I just remembered: When adding shadow-cljs + npm, the default clojure buildpacks on heroku stopped working for me, so I pushed a Docker Image to Heroku. That also got rid of the memory warnings @U2FRKM4TW described
What about cloud run set to minimum of 1 instance? It looks like cost would be $13/month for 1GB and only only $9 if you can get by on 512mb.
We have been running our staging enc on aws t2.micro for years and it was good enough. It really depends on tge workload but in our case it is a nontrivial web app
Nice to see some other options. I have heroku up and running fine now so I'm going to roll with that. They do have some cool JVM metrics running so this will be fun to explore and see how far the lower tiers take me.
Greetings, some time ago I decided to attempt to implement d/entity (datomic) so I could use it (carefully) with diatomic client api. For the most part it works. I used deftype to implemented a bunch of map interfaces, which I call DatomicReferenceMap. This works great. I can create it with an eid from datomic and it looks and acts like a map, and when you access a key that is a sub entity, it goes and fetches it. I also implemented clojure.lang.IFn, so I can do (entity :key) like a real map. However, when I try and create a new datomic entity that references one of these things, I get :db.error/not-an-entity Unable to resolve entity. If I replace with my entity with a ‘normal’ map, it works right. I am hoping I just need to implement another interface or two to make it work. Does anyone know what interfaces datomic expects for referenced maps? Thanks in advance!
Here is basically what I am talking about
(defn add-and-get [conn entity](let [temp-id1 "temp-id1"
tx (d/transact conn {:tx-data [(assoc entity :db/id temp-id1)]})
eid ((:tempids tx) temp-id1)
after (:db-after tx)
pulled (d/pull after '[*] eid)] ;pulled (de/entity after eid) ;pulled (d/pull after '[*] eid)]
pulled))
I’ve got an application that utilizes some polymorphic functions (currently defined as protocols, but I’m likely going to switch them over to being multimethods). I’d like third-parties to be able to define their own implementations for those polymorphic functions for types that they control. Ideally all that would be required for the new implementations to be utilized by the application is for the library in which they’re defined to be on the classpath. I’ve seen this sometimes described as a “https://yogthos.net/posts/2015-01-15-A-Plugin-System-in-Clojure.html”. Are there best practices for going about this that I should be aware of? The approach in the linked blog post would certainly work, but I’m trying to get a sense of what possible alternatives might be / learn more about the design space.
I wouldn't recommend
load-plugin
as written there. You should be able to do something with require
instead, which should work more cleanly with AOT scenarios.(Edit: I see they're doing that, after reading the source... still, it's strange. You should be able to put together something that works without needing to inspect the source)
it is using a "well known" resource name to find possibly multiple instances of plugins
OH I see, it's a manifest. n/m.
"Ideally all that would be required for the new implementations to be utilized by the application is for the library in which they’re defined to be on the classpath" is the source of a fair amount of complexity. If it's just there for convenience, it would probably be better to just require
all your plugins from the top level of your application, in some way. You'll still get the structural benefits of the abstraction you set up with multimethods or protocols.
like, if you are built on polymorphic constructs (protocols or defmethods) and I extend those, and call you with something that takes part in my extension, then you just work
If you really want users to be able to supply these things at runtime, then the options that come to mind are:
• Java-style: use protocols, and require that all the plugin jars are AOTed and the impls are gen-classed. Scan the classpath for implementations of your plugin interface.
• Manifests-in-jars, like the article you linked. As @U0NCTKEV8 says, uberjar-ing may bite you in the future.
• Say that plugins have to be .clj
files, or that your jars can never be AOTed.
• There may be some other way to identify things on the classpath that are actually Clojure namespaces, but I don't know what it is.
an uberjar can contain multiple files with the same path, but it's up to the consumer of those files to iterate and load them all, as opposed to the first found
The current usage pattern is that the application is launched via the Clojure CLI. Additional implementations for the polymorphic functions come via libraries that are included via -Sdeps
. Those additional implementations may sometimes be in .cljc
files, though I can see why such a plugin system would not work in the ClojureScript context.
it's probably worth looking at the way clojure currently does this exact thing with *data-readers*
(cmd)user=> (doc *data-readers*)
-------------------------
clojure.core/*data-readers*
Map from reader tag symbols to data reader Vars.
When Clojure starts, it searches for files named 'data_readers.clj'
and 'data_readers.cljc' at the root of the classpath. Each such file
must contain a literal map of symbols, like this:
except instead of the root of the classpath you'd probably want /my/project/construct/plugins.clj
or whatever
in general, as a library, I would not want to load code, and would say that approach general leads to things being globally defined and less flexible, e.g. if your library loads and registers plugins, then how do I opt out of using the plugins while still making the plugins available for use to other libraries I am using