Clojurians Log v2

> I'm curious if we could just spec gen billions of functions and record the output This sounds like a good idea. It's also plays well to clojure that you can generate code as data (harder to generate valid java from just a schema). > When playing with chatgpt, it's clear that it's transferring knowledge about programming solutions in other languages sometimes when giving a clojure solution. You need a high quality underlying model - it's not enough to just be good at Clojure the LLM also needs a good model of the world to be able to interpret the user's request. > where having a language coding assistant is as common as having a code linter or an LSP server, so we may want to think about how to make sure we have a clojure assistant story Yes, I think an LLM to assist with Clojure coding either in Editor (like GitHub co-pilot) or as a chat bot (like chatGPT) - could be very important for Clojure communities continued growth. It may be that Open AI/Microsoft/etc will do this for us - or it may be that we can do better with our own fine tuned ClojureLLM. > A full dump of clojurians chat logs for clojure instruction tuning could be useful for that too. We could use an LLM to analyse each message "Is this a Question or answer?" "Does this answer the question above?" "Do people seem satisfied with this answer based on the subsequent replies?" - then we can build a nice training set from it. > We could also host a chat-based "clojure coaching" service, That's a good idea, alternaively we could just use Slack as we are today and use the above classification approach mentioned above. > They've got over 150 thousand conversation trees. But we might be able to get away with just a few tens of thousands of high quality clojure coding instruction trees, plus existing chatlogs and gen'd repl interactions. Agreed, Github Clojure Code + Clojurians Slack + Clojure StackOverflow + some extra examples could provide enough examples for fine tuning.

> We could use an LLM to analyse each message "Is this a Question or answer?" "Does this answer the question above?" "Do people seem satisfied with this answer based on the subsequent replies?" For the highest quality data there, in the shortest time, we could use gpt3/4 to gen all that. But the licensing gets tricky. But I'm not sure it matters if it's an open source project. I believe the licensing is only tricky if you want to commercialize the output.

We could also get the LLM to tidy up the message "Make this message more readable and understable for programmers" "Remove any references to usernames" etc

Classification of question/answer could probably work on a slightly less capable model like Pythia.

Might want to replace each username with a stable anonymized name. I heard a recent story that stabilityAI's recent model problems came from blanking out usernames and the model didn't know who was talking to who lol

Yeah - anonymization needs to happen at the right step in the fining tune data generation pipeline.

Aye. Well some new clojure llm mutl-gpt hotness just dropped: https://clojurians.slack.com/archives/C06MAR553/p1682349923753599

So I'm down to help out with a ClojureLLM project 🙂 Looks like that project is just using OpenAI for now. I wouldn't mind just starting with that, with the option to bolt on private LLMs when they're ready

Agreed - I'm down to help too. A few high level tasks that come to mind: • Collect fine tuning data (raw data -> data cleanup pipeline) • Run fine tuning on top of open source models • Provide APIs, Command lines interfaces, User interfaces, Websites and IDE integrations for interacting with the new model.

Those are pretty good, broad three categories

I'm probably the grayest around the second point. I've been doing a lot of research but I haven't yet gotten my hands dirty with doing a fine tuning run

I can probably do that and cover compute cost for it too.

And fine tuning one of the smaller open source models to be a reliable coding assistant might take a whole lot of experimentation... I haven't yet assessed how good they all are yet

If it doesn't work great - then we still have the data and wait for a better model to come to fine tune on top of that.

No guarantees with this project- could be that we do this work and end up with a model that is too low quality in the end.

Nice! We should probably do an assessment. yeah, if there are any other llama code assistant models that are really good, we could probably run with that. If we're sure this bot won't be commercial, and will always be a clojure community asset, then I think we can use existing high quality llama

But we can have multiple models in there, it's whatever

Yeah - no reason to be stuck to just one model. I guess another task is assessing the output of the model - perhaps running it's suggestions side by side with another AI and asking an GPT-4 to decide which is best.

The same data can be used on any of the models as they come out

right

Yeah, that's what folks are doing

Or might even get picked up and used by other groups and used in their models by defaut.

Having gpt4 score the output against itself

yeah, there might be a lot of model sexing going on in the near future lol

People folding weights back in from branch LLMs that were trained on other datasets

https://github.com/sourcegraph/awesome-code-ai

This is interesting https://replit.com/site/ghostwriter Aren't there clojure folks over at replit?

Codium is free for individuals and claims to support clojure here: https://codeium.com/blog/code-assistant-comparison-copilot-tabnine-ghostwriter-codeium

Open source version of copilot: https://github.com/fauxpilot/fauxpilot

https://github.com/Hannibal046/Awesome-LLM

Might be worth looking first into dolly or pythia as they allow commercial use

Open Assistent will likely have some decent pythia models coming on soon too

https://www.anyscale.com/blog/how-to-fine-tune-and-serve-llms-simply-quickly-and-cost-effectively-using

> use the combination of HuggingFace, DeepSpeed, and Ray to build a system for fine-tuning and serving LLMs, in 40 minutes for less than $7 for a 6 billion parameter model

If I can be of help here, I'm all for it! In my experience trying to get llms to spit reasonable clojure, chatgpt is really the only one that does an acceptable job. But then, maybe we just need to gather or generate a training dataset and finetune codealpaca?

The issue with fine tuning openai stuff is that you can't finetune the chat models; you can finetune the davinci llm, which has a quality comparable to chatgpt while costing 10x more per token. On coding tasks the tokens pile up fast, on top of the privacy issues

I'm not sure how good a 6billion parameter model can get at generating code, I have huge doubts but I'm curious 🙂

Using logs as a starting point for a high quality dataset seems a great idea!

> If I can be of help here, I'm all for it! Great, the more the merrier. If we have a good fine tuning data then we can use it to fine tune multiple models, both now, but also future models as they come out. If we have the end to end workflow we can optimise it later or at least learned the limitations.