Clojurians Log v2
data-science 2024-07-21
Hi. It is not too late to join the Scicloj real-world-data group. https://clojureverse.org/t/real-world-data-meeting-10/
I'm prototyping a wrapper for https://github.com/ggerganov/ggml, a tensor library for machine learning that powers llama.cpp, whisper, and about a dozen other ML libraries. ggml has multiple backends including cpu, nvidia gpus, and apple silicon. It seems like it's possible to get the boilerplate down to a pretty reasonable API:
(def cpu-sched (cpu-scheduler))
(def gpu-sched (gpu-scheduler))
(def my-graph
(fn my-graph [ctx a b]
(let [out (raw/ggml_scale ctx (raw/ggml_add ctx a b) -1)]
;; multiple outputs
[out
(raw/ggml_sum_rows ctx out)])))
(def n 10000)
(def a (float-array (repeatedly n rand)))
(def b (float-array (repeatedly n rand)))
(def results-cpu (time (compute cpu-sched my-graph a b)))
(def results-gpu (time (compute gpu-sched my-graph a b)))
(prn (-> results-cpu second seq)
(-> results-gpu second seq))
Interesting!
(def my-graph
(fn my-graph [ctx a b]
(let [out (raw/ggml_scale ctx (raw/ggml_add ctx a b) -1)]
;; multiple outputs
[out
(raw/ggml_sum_rows ctx out)])))I presume you’re creating a graph, then sending that graph off to ggml to “do things with” for performance?All the
raw calls are calling into the ggml native code. However, nothing happens when the graph is created. The full graph is sent to the gpu all at once to minimize copies between the cpu and gpu.
> Have you put any effort into timing how pure ggml performs relative to your wrapper?
Generally speaking, there is a small amount of overhead for each call to native code, but once the native code is running, the performance will be identical. I haven't spent much time comparing ggml vs wrapped ggml since the use cases where the gpu wins, it tends to win big (10-20x) and the overhead will be negligible (it's the same idea behind using python for ML and data science).
Currently, I'm using JNA for native calls which isn't the fastest ffi (foreign function interface) option. At some point, I'll add support to clong for other ffi options that have very little overhead like JNI or panama.Understood, thanks for explaining! I'm interested in approaches for fast native code with clojure. I've done finite element analysis with commercial software before, in that software a Python layer was built upon an effective Fortran core. Certain pieces of code simply need to be fast in order to be useful. One can certainly write a lower level in C, Fortran or something else - but personally, I'd like to do as much in clojure as possible. A REPL is super helpful when working on numerical software, in my opinion.
If you are already familiar with native code and manual memory management, I think there are some great options for leveraging native code from clojure. In some cases, I think it's easier to use a c library than the equivalent pure java or java wrapper. C libraries are often less imperative and more data oriented than java!
I don't consider myself super comfortable with manual memory management yet - but it seems to be a good thing to learn. It's one of those skills I feel complement the "clojure ethos" nicely - it let's you go places pure data oriented clojure can't go.
To clarify: by "done finite element analysis with commercial software" i meant use the software, not write it. I don't have Fortran experience except for a university course.
Do you find it better to go direct to ggml vs through llama.cpp?
Depends on the use case. For generating tokens from an LLM, the llama.cpp api is much higher level. It would probably be a lot of work reimplementing the models llama.cpp supports just with ggml.
I see, thanks!
Theoretically, it doesn't seem like it would be that bad to implement models directly, but I'm not sure there's a better way to learn the model architecture than reading a bunch of papers, c++, or python.