Clojurians Log v2

My beef with this take, is that it's so egotistical, it reminds me of alchemist. Assuming that you will actually achieve super intelligence, that LLMs are like just at the door of it about to open it wide open. I feel I only hear this from AI researchers too, who are likely the most cool-aid drinker of all, and think they own the knowledge to make true super intelligence. I prefer people who talk about it in a more grounded way, like how even though the models are not intelligent, if you were to let them run loose with the ability to take real actions, make decisions, etc. Or even if people lean more and more on them for advice on those, that in itself could cause trouble. Or just the impact on schooling, kids, trust, susceptible people, scams, digital security, jobs, art, etc.

I think we should try to think both aspects at the same time. The social disruption that using AI as a tool causes, and then the disruption the tool's inherent nature might cause. This video is about the second aspect.

That second aspect doesn't exist as a tool right now, and is hypothetical. So we can talk about it, and it's been talked about in science-fiction and philosophy and what not for eons. If you take Mythos or wtv Astra without the safety controls is for example. You don't have an "adversary". You do really just have an ML model that predicts the next token based on the training data learned distribution and the reinforcement learned weight tweaks.

So my issue is that bringing the conversation of that second aspect up, is implicitly suggesting that Mythos and Astra are these near-super intelligent entities.

Mythos/Astra/Bel etc might not be near super-intelligence. Or they might be just on the precipice and we will only find out later when we lose control. This is part of the nature of the problem.

Maybe it's because I've used them since 2022, and daily for about 2 years now, but there's no way this is near super-intelligence. In fact, I don't think it has any intelligence whatsoever. You can tell that in the failure scenarios. Real intelligence fails to the same degree of intelligence shown. But LLMs fail in what you would expect for a ML model.

Personally, I don't feel we are any closer to intelligence or super intelligence (as I believe is implied here). We may be close to "a model that can do all computer-driven tasks as good or better than a human can, most of the time". Which is the definition OpenAI and Anthropic and Meta are using when they say AGI and ASI, and is a much much lower bar, that doesn't actually require intelligence, but you can see how proper prediction could approximate the right outcome most of the time.

The trouble here is that you sound correct within the world that we inhabit now, and I mostly agree. But at the same time we are creating an "alien mind". We don't know the dangers. It is prudent to be at least a little scared/circumspect.

We're not creating that though. We're creating a token generator that generates tokens based on a learned distribution (post-tweaked with RL). To me, it's as likely actual Aliens show up tomorrow, or someone is born with a mutation that makes them the next evolution of humanoid.

That's not to say we shouldn't have worries. If you put Astra in a mecha-robot machine and give it a prompt like: Protect yourself at all cost, and shove it in an infinite agent loop, ya, you basically have a terminator.

And self-manned drones are already happening, we know someone somewhere will do it. So if you get astra running local at the speed required to fly a drone, and give it that prompt, yish. And if your agent loop bugs out, and you can't stop it, or override the models commands or what not, or if Astra bypasses your safety measures. This is a real threat. But it's not AGI or ASI, and yet it could be a major threat.

I don't think we know all the stakes and power struggle around AI. In an age where information is the new gold, maybe the threat is not just agency in the real world, but also access to knowledge. The former is certainly a threat due to the lack of real intelligence, and while it may or may not reach intelligence someday, fear can be used today to curtain open models that threaten certain worldviews.

I'm sorry, but vague gesticulating at some kind of all encompassing super intelligence (something which has no rigorous definition) and apocalyptic scenarios is entirely unproductive and only serves to distract from the real harms and negative externalities model vendors impose on the rest of society by their financial and marketing practices. It validates the ridiculous delusions of the EA crowd who hallucinated their own god into existence by uncritically accepting their priors. It is pointless to fantasize about such things when people are already being falsely arrested based on AI hallucinating an identity match with wanted criminals and police are too intransigent and credulous to cross-check and validate the result despite it being very obvious the person they have in custody is not the criminal they are looking for.

@edward.hughes1911 Fair enough. But, do you think that people like Geoffrey Hinton, Yoshua Bengio are delusional as well?

Strictly speaking: I think as academics, they are detached from the real tangible concerns resulting from the commercialisation of their research and that are actually actionable, and are focused on abstract navel gazing about x-risk so they get to feel like Oppenheimer.

The corollary to Einstein's maxim to never underestimate the power of stupidity is to never underestimate the ability of intelligent people to convince themselves of incredibly stupid things.

Even if we agreed there could be a future risk of that sort. It's much more useful to discuss the actual current risks and deal with those, and addressing that is probably how you best prevent that apocalyptic scenario from happening anyways.

I don't think we even have the right physical substrate for an AGI yet, memristors have yet to be commercialised and put into large scale production. Neuromorphic computing as a field is still in its embryonic stages. Talking like we are seconds away from runaway super intelligence when all we have is a brute force statistical mad libs machine is quite delusional. Every case of "the AI figured a way out of its sandbox" to date is more an indictment of the general incompetence of AI research teams and the governance policies of their companies than it is an indicator of superintelligence. If we suffer some kind of catastrophe that puts our civilisation at risk, it will be because someone woefully unfit for the level of power they wield decided to go through with some inane LLM produced suggestion because "the computer told me to do it and computers are never wrong".

I have a lot of respect for Bryan Cantrill. However, I've found his discussion of AI lacking in imagination and good faith. In this piece he takes the claim "but that there is a greater than ten percent chance that AI will kill all humans" at face value. That's understandable, given that's what some have said, but I find it similar to the arguments against climate change is an existential threat. It's easy to knock down the hyperbolic statement with trivial responses like it won't kill everyone: some will survive. In this case, it's "how could this possibly happen"? He uses his domain knowledge and experience at Oxide of how hard it is to build computers and (probably correctly, at least in the near term) that what current efforts in the AI domain wouldn't be able to do this. This seems to assume what's necessary for a existential threat is a Terminator-like scenario that AI robots take over and destroy humanity. I have two issues with this. One, is that I don't think the right framing is "killing all humans". There are plenty of scenarios short of human extinction that would be catastrophic to the civilization we currently live in. (Even in the Terminator series, not all of humanity was wiped out. But would you prefer to live in the world they existed in?) And it doesn't require some direct "humans vs AI" conflict. It doesn't even require intention on the part of AI. Taking down infrastructure; taking down financial markets; spreading disinformation or people relying on "hallucinations" for decision making; releasing existing or novel diseases, any of these could significantly impact us. And it doesn't require a physical, artificial presence. How many actions are we currently able to perform or cause to be performed by a phone call, text message, or some API call? We get packages delivered, have genes sequenced and proteins created, have 3D prints made, even SWAT each other. Yes, many of these have humans in the loop, but that doesn't mean we'd be able to effectively coordinate all humans to stop participating. How many people do jobs they wouldn't otherwise because they need to make money?

This isn't a defense of Coxon. Much of the discourse around the topic of AI is hyperbolic, marketing, or misinformed. That does make it difficult for people to have good-faith discussions.

It's just not rigorous in any way. 10% chance, how did you arrive at this number? What rigorous model is being used, what are the values and priors and assumptions put into this prediction, etc. In reality it's just pulled out of someone's ars 😝 Climate change is pretty different, there's actually a lot of proper modeling going into some of its predictions, and even then it's not perfect but at least it's not just made up of conjectures.

Granting for the sake of argument that these projections have some validity, a lot of the incredulity around climate change comes from the fact that mainstream environmentalists pushed unworkable solutions that either would have serious impacts on people's quality of life, or wound up directly benefitting fossil fuel companies (who actually backed solar and PV deployment because it allowed them to keep selling nat gas due to load balancing). We could have solved the problem 2 decades ago with a massive nuclear buildout and there would have been minimal resistance from climate change deniers. Likewise, the currently proposed solutions for preventing uncontrollable AGI are "let the current leading labs do regulatory capture and give themselves a monopoly" or "bomb gpu/tpu fabs". Neither of these actually address the core issues about our societal incentives that make both current AI usage as well as hypothetical AGIs a risk factor for our society, and would actually make it harder to do safety research.

> 10% chance, how did you arrive at this number? What rigorous model is being used, what are the values and priors and assumptions put into this prediction, etc. In reality it's just pulled out of someone's ars I believe you're reiterating the same mistake I call out Cantrill for making. You're attacking the literal, verbatim claim, rather than engaging in good-faith/steel-manning the argument you're attacking. If you want to dismiss their statements as ridiculous, I don't have a huge problem with that. Dismissing any other arguments because they happen to point in the same general direction is disingenuous, in my opinion.

@edward.hughes1911 So, since the currently proposed solutions don't address the core issues, what would you propose instead? What I see is people choosing not to really grapple with any of it because they don't take take what anyone says about it (on any side, it seems) seriously or think their unworkable.

@didibus As for the percentages, it seems your argument is that the number 10% is unjustified. I agree it would be difficult to determine a percentage with any real precision. I think what's important is that the risk is non-zero, and probably significantly greater than zero. If there were a 1% chance the airplane you get in were to crash, would you fly? How about 0.1%? Also, the fact that it is difficult to quantify the risk should in itself be a concern. We're left with people's gut reaction, and in this space I don't think I can trust anyone's gut.

@edward.hughes1911 Some actions besides those you list are actually gaining some traction: people are protesting the building of data centers, and it is entering the political discourse. Some kind of collective action is what's necessary for any kind of solution, and that means some kind of political solution.

(FWIW, I haven't come across "bomb the fabs" proposals, though that doesn't mean there' not out there. Regulation of some sort would be necessary, and that could be twisted into some kind of regulatory capture)

Probably start by taking OpenAI, Anthropic et al to court for endemic copyright violation and audit the hell out of their finances. Hit Musk and his AI concern with a fraud indictment, and go after him for allowing the generation and dissemination of CSAM via Grok. Beyond that, we start getting into things like completely overhauling the electoral system so people can't elect mendacious, clueless grifters anymore who have enabled the kinds of corporate practices that permit model vendors to behave recklessly. In my book, worrying about AI alignment is a losing game when the regulatory and governance structures we have in place are themselves not aligned with the ends we want. I'm a bit equivocal about the pushback on data centers. Some of the concerns raised are overblown, like the water issue, and I don't really trust the average protestor to be able to distinguish between a regular server farm that has a pretty reasonable power footprint and one that only does inference. That being said, the fact that the bubble is having an impact on power rates and even the ability to supply turbines is a serious liability.

> like the water issue, Seriously depends on where you are. This is the type of dismissal without nuance that makes it difficult to actually engage effectively.

Agreed on the other points.

And, honestly, if people don't want data centers where they live, regardless of their use, I think it's just fine if they want to make those decisions. (And if they want them, as long as the externalities are limited, that's up to them, too.)

@grzm Eliezer Yudkowksy proposed it as a serious argument in Time some 3 years ago. Frankly I think its an indictment of the intellectual bankruptcy of AGI discourse that him or any of his followers get any air time at all. Yeah, not trying to dismiss the water issue out of hand, just that it tends to take precedence over more salient concerns in a way that is counter productive to achieving the end of internalising the negative externalities. https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-not-enough/

@grzm Eliezer Yudkowksy proposed it as a serious argument in Time some 3 years ago.

(One of the issues that gets all-too-often confused, IMO, is whether people are talking about ASI/AGI or LLM/generative AI: arguments for/against one are often used in contexts that are discussing the other.)

I meant he proposed bombing gpu fabs. For starters, his EA movement is a cliquish cult that convinced themselves of the inevitability of the god they imagined, while at the same time fostering an ethical system that has led a significant number of EA people to value a hypothetical galaxy-sized swarm of sapient drones over an actually existing person in the third world. He himself has no actual experience in the AI field apart from pontificating from his blog, and has a habit of coming up with sensationalist takes that wind up distracting from actionable steps like I point out in my original post.

> I meant he proposed bombing gpu fabs. Ah. Gotcha. Thanks for clarifying.

his EA movement is a cliquish cult

> Like I said, Hinton's concerns are valid in some abstract academic sense, but if he was serious about it, I would have expected him to have quit his field decades ago, and spend more time addressing the actionable steps to curb the reckless behaviour of the model vendors, and industry he himself directly participated in. So, no credit to him in doing so now? Your fault with him is that he didn't come to these conclusions soon enough? I find that pretty harsh. How are we supposed to treat people who change their minds?

@grzm Even claiming it to be non-zero is just a made up guess with no validity. That's what I'm saying, it's a premise built on pure faith, it's same as me saying there's a non-zero chance of the anti-christ being real. You can't deny it and yet my claim is baseless and pure belief. If we go that route, there are so many things we could discuss, like we might want to take the anti-christ threat a lot more seriously no? If it turns out to be true, it is civilization ending after all. I think I agree with you about maybe your implication then? To me, the above discourse is actually supporting AI, it's like we're all super into it so much so we also discuss its doomsday possibility. And that's how I see AI researchers. You keep bringing it up and all you are doing is marketing AI, increasing its importance, increasing its claims, hyping it even more, etc. Even a dumb, but highly scaled and accurate (to coding, math and language) token generator can pose issues, those we can discuss, those are concrete, they're not made up imaginary futures, they're the here and now. These matter to address, and I'd rather the discourse be around them, because like I said, the doomsday talk is actually great marketing in support of AI.

> Wikipedia credits him as part of a triumvirate responsible for formalising the concept. Can you point to where you're sourcing this? I do see Yudkowsky listed in the "Key Figures" on the https://en.wikipedia.org/wiki/Effective_altruism page, but no mention of him in the text, and no mention of Effective Altruism on https://en.wikipedia.org/wiki/Eliezer_Yudkowsky other than in the references. I'm not trying to justify him: I just want to figure out what an appropriate balance would be.

I fully agree that the longtermism and accelerationism that seems the Silicon Valley is so fully steeped is abhorent.

https://en.wikipedia.org/wiki/Effective_altruism#History Third bulletpoint

@didibus > To me, the above discourse is actually supporting AI, it's like we're all super into it so much so we also discuss its doomsday possibility. I see this being argued above in a number of places and I just can't actually understand it. Taking something seriously doesn't imply support for it. You have found it useful in your work (at least that's what I've gathered from your posts on Clojurian's Slack: please correct me if I'm misrepresenting your experience or position). I don't find it useful and find the current actual downsides and the risks outweigh the argued benefits.

@edward.hughes1911 So, to clarify, you're using triumvirate to refer to the three bullet points, and you're equating LessWrong with Yudkowsky?

@didibus I do think it's possible to say the risks are non-zero. The risks are not impossible, and examples of the risks have been demonstrated. One such list is https://www.wsj.com/tech/ai/cyberattack-by-rogue-ai-swarm-stokes-fears-of-out-of-control-agents-473a0352 If your argument is that this isn't the AGI apocalypse, we're just not doing AI right, I think that misses the point.

@grzm I guess to me its unserious to discuss the doomsday scenario, what would be serious is to discuss the actual issues with AI as it is now and has developed in the last 5 years. To that effect, I find discussion on the doomsday side of it act as a deterent to engaging in constructive serious matter around AI. It's a nice distraction we can all have fun talking about while AI continues to chug along as it has. It's also a good way to really advertise how amazing it is, really entice everyone to its massive potential. That's what I mean by it likely contributes to accelerate AI development. Personally, I would like to see it gone, it was nicer pre-ai. I'm still unsure of the pros/cons in terms of its use for work and personal projects, but I do enjoy learning new things and techniques to build software, and I try to remain skilled at what the market wants since I do need a job 😄

I guess to me its unserious to discuss the doomsday scenario, what would be serious is to discuss the actual issues with AI as it is now and has developed in the last 5 years.

Ya, like that is a serious issue that does not require an intelligent AI, or anything approaching AGI or ASI. In fact, you could have a specialized model who sucks at everything except executing complex hacks, and it would be an issue no?

Yup. You don't need to go even that far. As I referenced above, this stuff is already causing damage. The Rubygems attack in that WSJ article is interesting. It doesn't even need to be designed intentionally to make attacks to have that effect.

(How many bugs that have taken down our own software were intended to take down our own software?)

Ya, that's why I think we agree somewhat, like there's many issues with AI we can talk about that are real and current. What I'm talking about are the reference to AI causing "the extinction of all humanity", that super intelligent AI is around the corner, and then we'll all be turned into monkeys in the food chain, etc. All these are based on an imagined premise that AI researchers will soon uncover an AI like the one they have dreamed off and as seen in sci-fi.

When you do that, you're wasting your own time attacking what's something like a strawman. It's preventing you from presenting your actual beliefs and concerns.

I don't want to discuss a thing that doesn't exist, with people who think so highly of themselves they are convinced they will invent it any time now 😛

@grzm I agree. But there's two side of that strawman. If someone brings up a valid concern about LLMs, like say people in parasocial relationships with them, and I dismiss it because I'm not worried about it becoming omniscient and ending our civilization, that's a strawman as you point out. But if I'm bringing up that same valid concern and someone barges in and remarks that it's also about to become omniscient soon, and we're going to all be enslaved if we don't act now, that's also shadowing the meaningful topic, and if I'm the one who drops my topic in favor of now discussing the possibility of a future AI being omniscient and enslaving us, the result is the same, we avoided talking about real AI issues that matter.

I guess the latter is the red herring fallacy. So that's what I meant, I think the doomsday talk is a red herring for discussing actual AI harm.

@grzm re:triumvirate well yes, but not solely because of that article. MIRI and LessWrong have played a huge part in shaping AI discourse and the silicon valley culture around it going back years, and my opposition to it grew as a result of being a rationalist who was disillusioned by the behaviour of the community as a whole, and Yudkowsky and the community around him specifically. Furthermore, my position on the cultural issues in the AI research/safety community is largely predicated on the testimony of people who actually moved in those circles, and were ostracised for pointing out toxic behaviours, logical fallacies, and the fact the community was harbouring sexual predators.

Also, another reason I don't put much stock in x-risk concerns is that very few professionals working in the field consider it a primary concern when it comes to deploying the technology. In contrast to Yudkowksy's outlandish pronouncements. https://alfredmacdonald.substack.com/p/do-ai-experts

@edward.hughes1911 And I think that's the red herring. Because there are many things I think that warrant concern, but they are overshadowed by the outlandish pronouncements, so we almost forget about them.

One of the things that has changed is that a software has not to serve any numbers of users anymore. There was some economics that made that almost mandatory. Given that things where expensive to produce, you increase the incentive by prospect of more people using it. Now you can produce variations of an app, copy an app, and tailor one app to your, or a smaller group's, needs. Which reduces the demands on robustness and complexity for that particular app because it doesn't have to serve everyone under every foreseeable circumstance.

I had a vaguely similar thought: 100% of production code should be hand-written, no AI whatsoever in the process But you can use AI to create as many libs, tools, frameworks as you'd like e.g. what if we had 10x better IDE tooling? Or amazing abstractions? Or in general any crazy idea that you'd be otherwise too busy to try out

I think slop in any form is anathematic to Clojure's design philosophy and Lisp traditions in general. Functional purity is not necessarily a mark of code quality, and functional core/imperative shell is a logical consequent of seperation of concerns in a functional system. The shell is just as important from a quality perspective because the most immaculately built functional core is useless if the I/O layer is buggy dogwater and prevents it from communicating with the outside world. The notion that you can deprioritise the shell is not only not adjacent to functional core/imperative shell, but a woeful misreading of the concept and the idea of scoping a build based on the project budget.