Clojurians Log v2

This is really the original sin of LLMs, the fundamental problem that most if not all other problems stem from. OpenAI became religously convinced that if they could only scale language models big enough, they would achieve AGI. Brute force doesn't even begin to describe how wasteful this approach is. But ChatGPT took the world by storm, and a frantic race of FOMO began. Google/OpenAI/Microsoft/Anthropic/xAI all need to one up each other, and collectively they need to win from the Chinese, and so that means having the biggest meanest model. And then it turns out you can't actually make chips and memory fast enough, build data centers fast enough, ramp up electricity generation fast enough. Forget any notion of data provenance, these things now need so much training you have to hover up not just the internet but every book ever printed, every video ever made. And since now you're also training on all the deplorable aspects of humanity you try to patch over that with another model deciding if things get inappropriate. There is so much interesting stuff you can do with smaller models, mixture-of-experts, purpose-specific models, still so much space for research... if 1% of the funding going to LLMs would have gone to other fundamental research in this space who knows what we would have gotten in the next decade. It is all so so dumb. https://archive.is/J3VLi