I started to work on a library related to running LLMs localy- using Diamond ONNX RT. It needs to me three components: • downloading such large models (from huggingface at least) • tokenize text ◦ most models require text input in a certain "structure" ('harmony' for example: https://github.com/openai/openai-cookbook/blob/main/articles/openai-harmony.md) • execute the model using Diamond ONXX RT It started here: https://github.com/scicloj/llm very, very draft , but has working code to download from huggingface: https://github.com/scicloj/llm/blob/main/src/huggingface.clj First short term goal is to get gpt-oss working: https://huggingface.co/onnxruntime/gpt-oss-20b-onnx
11-13GB download. Not bad. 😎
just added nicer progress bars... working best under Linux and when using 'clojure' (not 'clj')
Java Mascot Generator (with a Clojure mode!) https://duke.mccue.dev/