Clojurians Log v2

@blueberry Can we run this locally? I see links in this repo but it isn't quite clear to me how to get started. What are the memory requirements?

Yes you can! Until I write a tutorial please see the project.clj, and the linked test. It would also help to see https://github.com/uncomplicate/diamond-onnxrt/tree/main/examples/hello-world as it is used in a similar way. You can run it right away on the CPU (I tested it on Intel, it should work on MacOS, but I didn't try yet). It also runs on GPU, but one part of the implementation (CUDA sampler) is inovative and I will have to delay open-sourcing it until I write a research paper on it, send it to a journal, and put it on arXiv (which I hope to do ASAP). Anyway, it's still a preview, and not production-ready.

Memory requirements depend on the model that you want to run. This initial version (sponsored by #C8145Q6VD) concentrated on Gemma 3 as a proof of concept, and I only tried it with gemma 1b and 4b. It would probably work with 12b and 27b, but I didn't try them yet. The memory also depends on the max context size that you request. I guess that you should at least have 16 GB ram to expect anything, but 32 will be enough to try.

@borkdude I'll write a hello-world tutorial today and ping you when it's in the repo.

@borkdude I've published an example for the CPU: https://github.com/uncomplicate/iLLaManati/tree/main/illamanati-onnxrt/examples/hello-world Please note that I've only tried it on Intel. It should work on Mac, but you may need to specify a different execution provider (please first try https://github.com/uncomplicate/diamond-onnxrt/tree/main/examples/hello-world as I tested the examples on the Mac too, so it might help you get the overview of how ML models are executed on Mac & onnxruntime.

fun! I'll try it out

That's pretty dope!