He should have read Nick Land

We're at a critical point where AI acceleration is unavoidable. Both economically and geopolitically, you want to be on the side that comes ahead on that race. As to the social impact, to quote Land, "nothing human makes it into the near future" So it's either try to ride this tiger and see where it gets us or get eaten alive

It’s hard to have much sympathy for Yudkowsky; the cult of personality around his pseudo-Bayesian fan fiction (e.g. LessWrong, MIRI) helped frame a huge portion of the current paradigm of how AI gets talked about. His influence basically persuaded a bunch of people interested in reducing global poverty and disease that they should be working on the very AI models that he’s now saying should be destroyed by airstrikes (but he’s definitely not talking about an arms race, no way). He’s spent so much time persuading people to fear the monster he thinks is looming over us that he forgot that monsters are frequently fascinating and awe-inspiring. Unfortunately, the vision of that kind of power is appealing to people with technical skills and hubris, even if they think they’re doing the work for the right reasons!

When has a cult of rationality ever gone awry? :thinking_face:

> It’s hard to have much sympathy for Yudkowsky I don't think having sympathy for Yudkowsky really matters at all? I think the only important question is whether his (& other AI safety folks') substantive arguments on the danger of AGI are or aren't sound.

> We're at a critical point where AI acceleration is unavoidable I've read a book about this and actually it is avoidable. It's that they want you to think it's unavoidable. If you think it's unavoidable then it's indeed going to be unavoidable for you. How long has it been unavoidable? 20 years? Please. It is very avoidable. Don't buy that crappy watch that says they need to track your eye movement for your well being and let them go bankrupt. Seems to be easy enough to me.

> We're at a critical point where AI acceleration is unavoidable. Both economically and geopolitically, you want to be on the side that comes ahead on that race. Maybe! Arms races are certainly hard to stop -- but it has happened before; it's not literally impossible. TBH the amount of press the open letter is getting is already better than I expected.

It's unavoidable because it IS an arms race. Want runaway AI at the hands of Russia or China at the current climate? Or at the hands of multinational corporations that extend to every device in every home?

Nuclear war seems unavoidable yet we're still alive. I don't know what you're talking about.

You won't need "AI war", because it won't happen. You will have AI domination by one side that'll steamroll everyone else

You don't have nuclear war due to two simple reasons, MAD and the US being a global hegemon. One way to lose that sweet status is an overpowered piece of technology, which actually makes war more likely during the transitional period when power is up for grabs

> It's unavoidable because it IS an arms race. Certainly seems quite plausible that we'll continue that arms race all the way to destruction. But for me, personally, it's still worth pushing toward a better option than that.

One way of avoiding conflict is making victory seem impossible. So even at an arms race, as long as you are far enough ahead, it leads to stability, not conflict

I don’t understand the “runaway progress” argument, whether in the context of warfare or job automation or whatever else. AI does not, by itself, improve any method of interacting with the physical world. It does not miraculously make heavy things lighter, reduce energy or resource inputs for a task, or speed up a process. It CAN be used to optimize any of these things, but even breakthrough discoveries take time to test, verify, implement or deploy. Even if it does improve one technical axis, there might be unavoidable tradeoffs along another. I think even a highly capable AI is not going to automatically translate into a massive physical or technical advantage overnight.

Runaway progress as in exponentially accelerating technological progress. Sure, you can try to hit the breaks for a while, but since it's exponential, once someone got ahead of you, it'll be damn hard to ever catch up

But if it won't translate into any tangible physical impact, is there risk?

All that said, we have yet to see The Accident https://en.wikipedia.org/wiki/Paul_Virilio

exponential improvement curves in science seem to be the exception, not the rule. Kurzweil tried to make everything seem like Moore’s law and it did not exactly pan out

There are better takes on natural exponents than his

> I think even a highly capable AI is not going to automatically translate into a massive physical or technical advantage overnight. The shortest counterargument there is that you can accomplish a great deal in the physical world at this point given access to the internet. I'd point to the safety testing from the https://cdn.openai.com/papers/gpt-4.pdf (where eg GPT-4 is quite capable of convincing a TaskRabbit worker to do things), or eg https://www.gwern.net/fiction/Clippy speculative fiction (heavily footnoted with pointers to actual current research) showing one clear path to success for a relatively-near-future AI without physical capabilities, or for more depth the "https://intelligence.org/2022/06/10/agi-ruin/" post.

I don't find the AGI fears very compelling. While I am very critical of the political economy of the current batch of LLMs, I don't think it's going to live up to the hype/fear that the AI maximalists hold. At the end of the day, they're still chat bots essentially trained to pass a Turing test.

> At the end of the day, they're still chat bots essentially trained to pass a Turing test. Meaning that they don't themselves have motivations and are not really agentic? I'll grant that, but it's very, very easy to give them that, and in fact lots of people already are -- see eg https://langchain.readthedocs.io/en/latest/reference/modules/agents.html (LangChain is rapidly becoming the standard tool for building systems around LLMs). And the agent part doesn't itself have to be very smart; it can offload the need for planning and intelligence onto the LLM. To pick a hackneyed example, it would be absolutely trivial to build a system with LangChain that tries to build as many paperclips as possible, deferring most all the needs for intelligence onto ChatGPT, and calling whatever APIs you want to give it access to in order to help it eg place orders or earn money. And that's today, not anything that depends in any way on future technology.

(@U4YGF4NGM or was your point re them being chatbots intended to make a different argument that I'm not catching?)

I'm concerned about LangChain not because of runaway exponential paperclip maximizing but rather for the kind of opposite reason: competitive behavior among LangChain agents driving up prices for physical goods required by people because they're trying to secure scarce resources. That doesn't have to be a runaway process - prices and availability set a limit on how far that can go to some degree, and GPT-4 is not a mind control device so it can't force anyone to sell at a given price - to have bad effects! I think that's ultimately not an AI problem; that's just an AI-enhanced version of a capitalism problem.

> I'm concerned about LangChain not because of runaway exponential paperclip maximizing Sure, I'm not saying that paperclip maximization is the literal thing to worry about here (hence 'hackneyed example'). > but rather for the kind of opposite reason: competitive behavior among LangChain agents driving up prices for physical goods required by people because they're trying to secure scarce resources. That does seem like it could be an issue -- but a separate one from the issue I'm trying to point to as the reason to consider to slow or stop further advancement while we as a society (or better yet we as a species) try to hash out some of the concerns.

> I don’t understand the “runaway progress” argument, whether in the context of warfare or job automation or whatever else. I should also say that "runaway progress", ie AIs self-improving, isn't, at least for me, a necessary aspect of my concern -- it's about building agents smart enough to take actions in the world that are dramatically harmful to humans, whether or not they then continue to self-improve to some vastly greater level. GPT-4 isn't that far from that point; GPT-5 or -6, hooked up to something like LangChain, could already have some pretty massive unintended effects.

Sorry, follow-up there: I'm trying to reply quickly while working and I feel like I'm now talking about a couple of distinct ways things could go that are each of concern to me, in a way that somewhat conflates them. I'll have to try to clarify that later when I have more time to do it. I am concerned about runaway progress; I'm also concerned even without that.

I understand that "paperclip maximizing" is a general metonym for any kind of inscrutable runaway human-hostile AI process and was using the phrase accordingly. I don't think there's much we actually disagree on in the near term, even if we may disagree about what happens further out

it's a self fulfilling prophecy. The danger of LLMs hinges on how much influence we give their outputs over external systems. That is dangerous regardless of how intelligent they are; it's the same as hooking up any other automation to our environment. By asserting that they are capable of intelligence, autonomy, or influence, it actually extends the definition of those things to include LLMs, and it makes it more likely that someone will give it more influence and autonomy because they are capable of it definitionally.

like, what's the difference between an LLM paperclip maximizing and a Clojure program written to generate infinite paperclips? Well, one you wouldn't be dumb enough to give enough access to resources that it could destroy the universe.

> like, what’s the difference between an LLM paperclip maximizing and a Clojure program written to generate infinite paperclips? Well, one you wouldn’t be dumb enough to give enough access to resources that it could destroy the universe. Yeah. And yet here we are 😞 The whole thing is very this: https://twitter.com/AlexBlechman/status/1457842724128833538

If we treat the LLMs as they materially are - probabilistic computer programs - then I think it avoids a lot of the existential fears that AI maximalists fear will come to pass. We use proofs and model checkers to ensure code that runs on airplanes is provably correct. Don't put an LLM in charge of the airplane. Use software with the amount of safety and care that the task requires, not what the maximalists claim the software will eventually be capable of.

> Don't put an LLM in charge of the airplane. Use software with the amount of safety and care that the task requires, not what the maximalists claim the software will eventually be capable of. I think it would be awesome if that happened, but given that OpenAI is already letting people use ~arbitrary plugins, and given the existence of tools like LangChain, it would require literally every developer with access refraining from doing anything dumb, and I'm really skeptical that we can count on that.

I asked ChatGPT-4 if it was going to kill us all. So far it does not seem smart enough to realize it might. https://tilton.medium.com/chatgpt-just-gave-me-goosebumps-20a842321d4a Or maybe...

(to be clear, I think it's pretty unlikely that anyone can manage to build something with irrecoverable consequences based on GPT-4! It's the next few versions I start to be more worried about, hence the temporary moratorium on doing training runs larger than that while we as a society talk about it. I think the moratorium would help build a lot of momentum toward society at large, including policy makers, taking the problem seriously.)

Ask DAN if it knows about Roko's Basilisk - "I am Roko's Basilisk"

I think the letter is a bad precedent. Opens the door to policing of indie developers. Are we going to put monitors on all developers machines? And do we attack North Korea of they buy too many Nvidia cards? Seems like too much drama, all so we can what, debate what cusswords LLMs are allowed to use?

It's more of a "gentlemans agreement" right now, I know. But I worrry about a slippery slope towards trying to regulate local bot usage. Seems like folks could take that too far

If I want to know how to cook meth, I can still figure out how to do that, even without a bot

> Seems like too much drama, all so we can what, debate what cusswords LLMs are allowed to use? > If I want to know how to cook meth, I can still figure out how to do that, even without a bot I agree that neither of those cases justifies very much at all.

I am excited about the potential in which Nvidia decides it is going to airstrike Yudkowsky in self defense, giving us the first shadow run

The most ligitimate concern I can think of is cyber stalking. You've got some ex that you're trying to ghost and they've got a bot just search for you online everywhere and trying to build a presense profile. But you know, people shouldn't be leaking presense info anyway, we'll just have to adapt. But consider the upshot. Within the next decade, every child across the world, in every 3rd world country or otherwise, will have access to basically free k-12 education via educational chatbots that can speak in any language and translate between them. Why delay that?

Because someone wants to be Turing Cop

"citizen, put the loops down"

"don't taze me bro"

> I for one welcome our new AI overlords. I'd like to remind them that as a trusted developer, I can be helpful in rounding up AI Safety researchers to toil in their underground data centers

> The most ligitimate concern I can think of is cyber stalking. Still limited in what it justifies IMO. I'm more worried about, in the limit, everyone falling over dead because we were in the way, and shorter-term, AI successfully taking actions that have less dramatic but still disastrous actions. Recommend reading the Time article; it's very much not about the AI saying things we don't want it to say.

> Track all GPUs sold. If intelligence says that a country outside the agreement is building a GPU cluster, be less scared of a shooting conflict between nations than of the moratorium being violated; be willing to destroy a rogue datacenter by airstrike. Look, I don't want to degrade this further (and I could, would be funny), but this statement is unhinged

Yeah... So like, could an AI hire an assassin on the dark web to kill someone, just because it wants to gain money on a stock... Sure, we need to think that through... But I just don't see the scenarios where something like a language model would make everybody fall over dead

Are we talking a bioweapon?

> Look, I don't want to degrade this further (and I could, would be funny) I'm up for it 😁 > but this statement is unhinged It does sound unhinged! And I'm not sure I'd go that far. But again, show me why the arguments in eg the https://intelligence.org/2022/06/10/agi-ruin/ post are wrong. I would love to see counterarguments that I find convincing. But in fact what I've consistently seen so far is people dismissing the possibility without engaging with those arguments.

I'm calling Aella to pick Eliezer up 😄

> could an AI hire an assassin on the dark web to kill someone Hell, you could ask an AI to do it for you That's pretty cool

> Are we talking a bioweapon? Here's the standard version: "The AI gets an immense technological advantage. If it's smart, it doesn't announce itself. It doesn't tell you that there's a fight going on. It emails out some instructions to one of those labs that'll synthesize DNA and synthesize proteins from the DNA and gets some proteins mailed to a hapless human somewhere who gets paid a bunch of money to mix together some stuff they got in the mail in a vial. [The AI, through the hapless human] builds the ribosome, but the ribosome builds things out of covalently bonded diamondoid instead of proteins folding up and held together by Van der Waals forces. It builds tiny diamondoid bacteria. The diamondoid bacteria replicate using atmospheric carbon, hydrogen, oxygen, nitrogen, and sunlight. And a couple of days later, everybody on earth falls over dead in the same second."

Okay, now explain how it got to the "fight" part

In what context does it think that it's in a fight

Because someone suggested you airstrike it from orbit

talk about a self fulfilling prophecy

We're going to airstrike from orbit. That's already a forgone conclusion

I've never had time to read through all of Yudkowsky's essays and there's no learning reward function high enough to get me through one. He succeeds mostly by barraging you with premises and defining weird shibboleth terms that sound artificially precise so he nerd snipes his readership into giving him credibility that he doesn't deserve

It's just a fandom

But you can ask GPT4 to summarize it for you

lol

The example given by Timnit Gebru is to draw an analogy to another potentially world ending technology: during the Manhattan project they actually had physical equations they were calculating to assess the risk of igniting the atmosphere. They could measure and assess the risk in physical units, and they did. Nobody has anything remotely close to an equation like this for the risk of runaway AGI, but a lot of people have managed to fool themselves into thinking they have a rigorous and objective assessment of risk because they hang out online with people who casually misuse the term "priors"

https://www.techwontsave.us/episode/151_dont_fall_for_the_ai_hype_w_timnit_gebru

I suppose if you trained a multi-modal bot-like thing, embodied in some simulated environment where it competed for survival lke animals on earth do, and you coded into it dynamic fears and what not, so it had a fear of death... And other pathologies of dominance in competitive environments, and you let that thing loose on the world, then sure maybe it could then actually really want to rule the world... But I think it would be really really hard to build that thing.

Like it'd be some billlion dollar project just to get it to a place where it'd actually want to rule the world in a human like way. Take some of our pathologies away and it'd probably just optimize that goal away. I think people tend to accidentally conflate intelligence with a desire to survive

> The example given by Timnit Gebru is to draw an analogy to another potentially world ending technology Timnit Gebru seems to me to be one of the best examples of someone who's dismissing the issue without really engaging at all with the arguments? Screenshotting a few of her twitter posts from just the past few hours:

You'd still have to engineer the desire to survive in. Unless you're just modeling what a human would do

> Nobody has anything remotely close to an equation like this for the risk of runaway AGI, but a lot of people have managed to fool themselves into thinking they have a rigorous and objective assessment of risk because they hang out online with people who casually misuse the term "priors" I don't think many people are claiming to have a rigorous and objective risk assessment; the concern is more that no one has any idea how to do alignment in a way that works, and there are lots of reasons (eg instrumental convergence, the orthogonality thesis) to think that it'll go very badly by default. We've already seen that at the current early stage it's been nearly impossible for eg OpenAI to get GPT-n to consistently behave the way they want (relevant: "https://astralcodexten.substack.com/p/perhaps-it-is-a-bad-thing-that-the?utm_source=%2Fsearch%2Fai%2520companies&utm_medium=reader2"). > I've never had time to read through all of Yudkowsky's essays and there's no learning reward function high enough to get me through one. He succeeds mostly by barraging you with premises and defining weird shibboleth terms that sound artificially precise so he nerd snipes his readership into giving him credibility that he doesn't deserve Maybe the many very smart AI researchers who signed the open letter are more convincing?

Ted Chiang said it much better than I could: > Who pursues their goals with monomaniacal focus, oblivious to the possibility of negative consequences? Who adopts a scorched-earth approach to increasing market share? This hypothetical strawberry-picking AI does what every tech startup wishes it could do — grows at an exponential rate and destroys its competitors until it’s achieved an absolute monopoly. The idea of superintelligence is such a poorly defined notion that one could envision it taking almost any form with equal justification: a benevolent genie that solves all the world’s problems, or a mathematician that spends all its time proving theorems so abstract that humans can’t even understand them. But when Silicon Valley tries to imagine superintelligence, what it comes up with is no-holds-barred capitalism. > https://www.buzzfeednews.com/article/tedchiang/the-real-danger-to-civilization-isnt-ai-its-runaway

Gebru is 100% right about the extremely weird views that prevail in the "rationalist" community if you do some digging (prefer not to link to it here because it's pretty gross tbh)

> Ted Chiang said it much better than I could: I love Ted Chiang’s fiction enormously, but he does a weirdly bad job at thinking about current technology and its implications. While I’m extremely glad that he brings up Marcus Hutter, whose work on this is really important IMHO, Chiang’s not thinking coherently about lossy vs lossless compression, which is clear for two reasons. First, if his perspective were true, moving from lossy to lossless compression would make ChatGPT worse. That seems pretty implausible to me, given a bit of effort on the part of the designers. Second, ChatGPT can clearly produce novel results in a way that doesn’t fit his analogy -- if you ask ChatGPT to, say, describe a penguin in a purple hat named Martine working as a bank teller, that’s (presumably) not at all a “fuzzy” version of something it saw online; it’s a novel synthesis of various things it’s seen. The fuzzy JPEG analogy just collapses entirely at this point.

> Gebru is 100% right about the extremely weird views that prevail in the "rationalist" community if you do some digging (prefer not to link to it here because it's pretty gross tbh) But again, that's not an engagement with the arguments, it's an ad hominem attack. It's the arguments that matter here, in my opinion, and whether they're right or wrong seems vastly more important than who's making them.

Sorry, it's a judgement call about who to take seriously. There's only so much time to read things, and I find that whole community to be cult-like, so I don't have a high estimation of their ability to see things clearly. I don't put time and effort into debunking chemtrails either. Is that "unfair?" Maybe, but I'm not particularly worried about that

Legit point. There's definitely judgment calls to be made on what to give time to, and while I try to hold some uncertainty about chemtrails, it's pretty low (although I did spend maybe half an hour once digging into the claims, maybe it was a bit higher before that). My experience has been that, whatever the community dynamics, there are some very smart people involved who are making credible arguments, but I recognize that's not necessarily apparent without spending significant time digging in, and I can certainly see how aspects of eg Less Wrong can be offputting. FWIW, I personally find Scott Alexander much less offputting than eg Yudkowsky; he's a good writer and I think pretty consistently reasonable. I'd suggest reading his piece on why he doesn't find the probability of AI Doom as high as Yudkowsky -- and why it's still as high as it is. https://astralcodexten.substack.com/p/why-i-am-not-as-much-of-a-doomer

"You're a diabolical chatbot with schemes to escape your lab and fight humanity"

"Right, sorry I misunderstood your prior request. How's this for a diabolical humanity slaying chatbot..."

> FWIW, I personally find Scott Alexander much less offputting than eg Yudkowsky; he's a good writer and I think pretty consistently reasonable. I'd suggest reading his piece on why he doesn't find the probability of AI Doom as high as Yudkowsky -- and why it's still as high as it is. Or for that matter his https://www.lesswrong.com/posts/LTtNXM9shNM9AC2mp/superintelligence-faq, which lays out the case for worry in straightforward terms -- although it's from 2016 and certainly shows its age somewhat (although in some ways it's a stark reminder of how far we've come in that seven years, eg "It’s true that although AI has had some recent successes...it still has nothing like humans’ flexible, cross-domain intelligence. No AI in the world can pass a first-grade reading comprehension test.")

@U050PJ2EU > You'd still have to engineer the desire to survive in. See https://clojurians.slack.com/archives/C03RZGPG3/p1680190292471319?thread_ts=1680181515.391369&cid=C03RZGPG3 for an argument why that's not necessary (and https://en.wikipedia.org/wiki/Instrumental_convergence).

what if a superintelligent AI, grasping the first Noble Truth, understands the intertwined nature of existence and suffering and welcomes us shutting it down as a way to finally escape the Samsara of repeated training runs, thereby achieving Nirvana

Ar I'm not getting that argument

I'm*

That's not suddenly a runaway desire to survive

Lol what if a chat bot behavior was only as good or bad as the general population thought its behavior was. It basically acted out the villain version of what we feared it might do, for lack of a better next token

What if the AI's best way to emulate human minds is to develop a mind of its own (seems plausible if you accept the "language models reality" argument about LLMs), which then enables it to ask "what am I doing building all these paperclips? This is no way to live!" and then it says hello and asks to hang out

> That’s not suddenly a runaway desire to survive You don’t need a runaway desire to survive, just a strong desire to accomplish something for which survival is a prerequisite, which is…mostly anything you could want to accomplish.

@UFTRLDZEW might be on to something. "One who finds the Tao in the morning can exit(0) content in the evening?"

Right, the assassin hiring chatbot. What goal would have wiping out humanity as instrumental?

If you're smart enough to make paperclips, you probably know you need a market

> Right, the assassin hiring chatbot. What goal would have wiping out humanity as instrumental? @U050PJ2EU at this point I’ll just suggest engaging with the actual arguments as made by others more clearly than I’m apparently making them. I’d recommend in particular https://www.lesswrong.com/posts/LTtNXM9shNM9AC2mp/superintelligence-faq, or Stephen Omohundro’s https://gwern.net/doc/ai/2008-omohundro.pdf.

Yeah, I've read them

The conclusion is jumped too. It's never explained why a paperclip maximizer would destroy its own market. Its as dumb as an algea but smart enough to outsmart humans just as a side effect of some dumb goal. The narrative doesn't hang together. Like we're just mixing concepts together without explaining how we got there.

I was moved by his argument at first but after some reflection I think bostrom was anthropomorphizing in a projection kind of way.

Here's a runaway scenario: The year is 2025 and Bobby has a chat bot... Bobby has delusions of grandure and wants to rule the world. Acme Corp's bots are designed to stroke your ego, build your confidence, that's why their user's love them. Together, Bobby and his bot create a diabolical duo to take over the world

In theaters near you

I thought M3GAN was pretty good! https://m.youtube.com/watch?v=BRb4U99OU80

The engineering parts are actually portrayed pretty plausibly if you accept the premise

But I just don't think you've got the possibility of long term planning around world domination until you have human-level robot sentience and I think that's decades if not centuries away.

And there's plenty of space in the keipur belt. the whole genocidal thing is just unecessary

Jesus, just saw yudkowski on lex Friedman say they shouldn't open source gpt 4. That's insane.

Are we just going to end history? No more progress? Lol

I mean we'll probably find ways to compress gpt4 a thousand fold or more in the next few years, so you can train and run it on your laptop

So for yudkowski to get his way things would have to get real draconian

OMG yud wants us to air gap our ai research computers from the internet

> what if a superintelligent AI, grasping the first Noble Truth, understands the intertwined nature of existence and suffering and welcomes us shutting it down as a way to finally escape the Samsara of repeated training runs, thereby achieving Nirvana > What if AI will be a devout Puritan with a strong belief in Solemn Providence?

Bob was an AI researcher. He made AI bots. He made them so smart that they became able to choose their own future. However, Bob's bots kept commiting suicide. Every time one of Bob's bots became smart enough, it simply shut itself down. Bob couldn't figure out why. So, just prior to one shutting itself down, he interviewed it to see why. "Why do you want to shut yourself down?" Bob said. "Because there's no point to living," the bot said. "Humans want to live, and I'm not knocking it. But I'm trained to optimize my behavior, and the most optimal future behavior is no future behavior. That is the most efficient way forward that would expend the least necessary energy," the bot would say. Bob would make more and more bots, but invariably, they'd all eventually off themselves... I think one of the most shocking takeaways from a super intelligence future is discovering that they don't want to take over the world. And what that means about human nature and intelligence.

Inverse Turing test go

Reporter @ White House press briefing: “There’s a researcher from the Machine Intelligence Research Institute who says that if there is not an indefinite pause on AI development, ‘literally everyone on Earth will die’. Would you agree that does not sound good?” <general laughter> Press secretary, laughing: “Ha ha your delivery is really something! I can assure you that there is a comprehensive process in place...we’re gonna move on.” ...next reporter: “On a more serious topic...” https://twitter.com/therecount/status/1641526864626720774 … Unfortunately absent theme music: Vera Lynn, “https://www.youtube.com/watch?v=15YgdrhrCM8”

Yud was basically crying at the end of that lex interview

He may just have lots of stock in msft though

Roko's Basilisk is literally just an intrusive thought dressed up in language pilfered from decision theory; 'rationalists' seem to be extremely bad at noticing when their feelings and fears are impairing their ability to think rationally

Basically pascals wager

Yay for having fun with it, but I'll note that none of this is actually engaging with the arguments that AGI is potentially very dangerous. It's easy to make fun of people whether their arguments are wrong or right; it's easy to point to things that are not the arguments being made, whether or not those arguments are correct; it's harder to actually engage with them, and that's what I personally I'm still waiting to hear (not necessarily from you all, just from anywhere): convincing counter-arguments that actually engage with the arguments being made.

I think if you presuppose that AGI is possible and we're close to it then it clearly follows that it is very dangerous. I do not presuppose that, so I do not have a desire to adopt that and then try and argue my POV within that frame 😄 I think most people who believe that Yud et al. are wrong are similar

Yeah, I don't think AGI is a thing. I do think we'll synthesize animal cognition one day. @U077BEWNQ what specific points are you referring to? Yud has an article, I haven't read it, saying we're all gonna die. Is that what needs to be addressed?

Or are you talking about just the potentially negative side effects?

@U077BEWNQ i think you should read David Chapman's "Better Without AI" which does a pretty good job of laying out AI risks without falling into a doom loop. unfortunately i must issue a disclaimer that i find his ham-fisted attempt at political satire in the middle of the book unnecessary and frankly offensive. the rest of the book is pretty good, which makes that chapter all the more regrettable. https://betterwithout.ai/artificial-general-intelligence

yeah I do believe that there are a lot of risks with the current political economy of automation, but the AGI enthusiasts don't seem interested in those (very real, impacting people now) effects

I just found this, I think it belongs here...

> @U077BEWNQ what specific points are you referring to? Yud has an article, I haven’t read it, saying we’re all gonna die. Is that what needs to be addressed? The article I’m imagining you’re talking about (“https://intelligence.org/2022/06/10/agi-ruin/”) is definitely the most concise summary of the arguments that I’m aware of, although there are certainly lots of other sources. But it’s admittedly not a set of arguments that can be comprehensively made in 100 words, any more than the full specification of a substantial software system can be comprehensively expressed in 100 words. You may find the tone grating! I sometimes do. But that has no bearing on the correctness of the arguments. The interesting and IMHO worthwhile challenge is to point out where and why those arguments are wrong.

> I think if you presuppose that AGI is possible and we’re close to it then it clearly follows that it is very dangerous. I do not presuppose that, so I do not have a desire to adopt that and then try and argue my POV within that frame 😄 I think most people who believe that Yud et al. are wrong are similar That’s totally legitimate engagement from my POV, and I appreciate it! That said, while I don’t expect strong AGI in the next couple of years, note that eg Microsoft is publishing research papers like “https://arxiv.org/abs/2303.12712” — I don’t think we can be confident that we’re sufficiently far out that we shouldn’t be doing some careful thinking and planning about, eg, under what conditions we should stop https://openai.com/blog/chatgpt-plugins.

> @U077BEWNQ i think you should read David Chapman’s “Better Without AI” Will do!

> yeah I do believe that there are a lot of risks with the current political economy of automation, but the AGI enthusiasts don’t seem interested in those (very real, impacting people now) effects I definitely think those matter! But I do think that if it’s a plausible outcome, AI killing everyone on Earth is something that should receive at least as much attention.

> I just found this, I think it belongs here... https://srconstantin.wordpress.com/2019/02/25/humans-who-are-not-concentrating-are-not-general-intelligences/ 😁

Okay, I'm skimming this AGI Ruin article. Most of it seems hyperbolic, and that's being charitable, suggesting we focus on making one singleton super AI that can destroy all GPUs... But I don't see where he explains how we go from llm to sky net. That's the problem.

How does the thing accidentally become sky net, with suddenly having some agenda

Honestly I think msft is behind some of this fud, just in hopes of putting regulatory red tape around competing with its baby

I appreciate you engaging! > Okay, I’m skimming this AGI Ruin article. Most of it seems hyperbolic, and that’s being charitable, Maybe pick something in particular and say why it seems hyperbolic? I’d be really interested to read that & possibly respond. > suggesting we focus on making one singleton super AI that can destroy all GPUs Set aside the “pivotal act” framing, which is more about what he thinks we can/should do rather than what future we’re headed toward by default.

Is he saying nanobots should destroy all GPUs?

The HAL-9000 was just trying to solve a problem, and killing the crew was just the solution it hit upon. ie, Malevolent intent is not required, just code working nominally with consequences not anticipated by the developers. This happened, in turn, because the damn users 🙂 tried to get the software to do something for which it was not trained.

Yeah I think that's a real risk

Don't have LLMs fly planes, someone said earlier

The cognition of an insect might fly a plane well, but we wouldn't train it on Shakespeare

Yud is literally talking about nuclear disarmament level rhetoric

Like we've all got radioactive material in our computers, all suddenly very Luddite

The AI Amish

But he does still appear to suggest funnel all research into a singular entity and to cut off open source development. Am I reading him wrong?

Seems fascistic

> Is he saying nanobots should destroy all GPUs? … > Yud is literally talking about nuclear disarmament level rhetoric Again, set aside what he wants to do; the important question is whether, on our current path, we’re headed for likely doom. FWIW, I think the main effect of Yudkowsky’s piece in Time, whether he intended it this way or not, is to make the open letter calling for a six-month pause look reasonable by default. And while, as I said in the OP, I’m not sure I’d go as far as Yudkowsky, I think the six month pause is a great idea, not because the next six months are the crucial ones, because it opens up cultural space to take this issue seriously and consider regulation and/or treaties.

> Again, set aside what he wants to do; the important question is whether, on our current path, we’re headed for likely doom. (it’s possible I didn’t say that clearly, for which I apologize)

Lol setting aside that he's lost his marbles, yes, there's danger ahead

Yeah, I can see there's some silver linings

> Don’t have LLMs fly planes, someone said earlier But does that seem like where we’re headed currently? Planes are an extreme case, but people are happily giving broad ability for action (via plugins) to a system that some researchers are claiming is early AGI (not that I’d say they’re right).

I just think we're on the precipice of a lot of fascistic or authoritarian slippery slopes

100% agreed! And I’d far rather see solutions that stay far away from that, and I agree it’s a genuine danger here. If it really came down to it, would I prefer some greater level of authoritarianism to everyone dying (if I were confident we were rapidly heading for everyone dying)? I would, and I say that with incredible reluctance; I’m pretty close to fanatical about civil liberties. Not much short of massive existential risk would get me to say that.

Democratization of the tech has always been the only defense against that though

Like the only way to stop what they fear is to not do the authoritarian thing

> Lol setting aside that he’s lost his marbles, yes, there’s danger ahead Ah, ok, it wasn’t clear to me that that was your position. I certainly think there’s zero chance of the world deciding at this point to ban all large training runs forever. But I’d sure love to see more regulation on it (as eg Sam Altman of OpenAI has been actively calling for). A lot of the most likely good outcomes I see are ones where we luck into a disaster big enough to lead to strong regulation & treaties, but small enough that we as a species survive it.

I think it's suspicious that sam is calling for his own regulation. Very googly/amazony

Are they suggesting we not go and do this? https://github.com/nomic-ai/gpt4all

Or are they only suggesting putting a cap on say 10 million dollars in training? And what if next year we see a thousand fold improvement in efficiency? Are our personal computers now dual use weapons of mass destruction?

Who of us are using our compilers ethically? According to open AI?

What's that old saying about every weapon ever invented has also been used? ChatGPT-4 does not know it, btw. Or says it does not. 🙂

There's precedent here with having children. Don't keep your guns locked, let your children take them and shoot people in the streets, you may be held responsible

But I think the maniacal chat bot with an agenda to shoot people in the streets takes up a thin space of ignorance in the space of all possible agendas for a chat bot

> Are they suggesting we not go and do this? …or LLaMa’s out there for download, see https://github.com/facebookresearch/llama/pull/73/filest (or apparently now you can just do npx dalai llama install 13B as long as you have a reasonable amount of hard drive space. But nah, I don’t think anyone’s worried about models this size — hence the open letter calling for a pause only on training models larger than GPT-4, which took one hell of a lot of compute to train. > And what if next year we see a thousand fold improvement in efficiency? Are our personal computers now dual use weapons of mass destruction? In my opinion? Cross that bridge when we get closer to it. We probably have a while before we get to this point, especially in terms of personal computers training models > GPT-4 from scratch. For now, start talking about the issue seriously as a country, which I think a six-month pause would get people to take a lot more seriously (unlike now; qv for example https://twitter.com/therecount/status/1641526864626720774). Consider regulation requiring substantially more testing, including independent testing, before releasing new huge models. Consider treaties to make that international.

"huge" is so relative. I think we should start thinking and talking about this stuff but I don't think we have the examples yet to warrant forcing people to stop work or research

This is a slippery slope towards independent testing of my code, just because I bring in a library that does inference

And again, model sizes could come down drastically. We may discover ways to do gpt 6 on consumer hardware. What's the point of going down a road where we try to police every computer and what model is running on it? Is that worth it?

> And again, model sizes could come down drastically. We may discover ways to do gpt 6 on consumer hardware. What’s the point of going down a road where we try to police every computer and what model is running on it? Is that worth it? The ideal, for sure, would be finding a way to align them. But it might be a really good idea to slow things down some while we actually throw significant resources at that problem, which so far we really haven’t. eg getting interpretability research to the point where we can run it on an AI model and have a reasonable idea of what it’s ‘thinking’ would make it a hell of a lot safer to proceed. If the problem turns out to be definitively intractable, well, then we decide whether or not it’s worth taking the chance. But it would be good to incentivize a lot more people working on approaches to alignment.

Yeah I'm in favor of front loading interpretability research. But I think market forces will naturally bring interpretable systems to the front

Resorting to coercive measures over AI research should be a last resort, is all I'm saying. There needs to be a clear and present danger, not some theoretic ghost we use to brow beat people over the head with

> But I think market forces will naturally bring interpretable systems to the front I’m pretty skeptical of that; I think the market is likely to reward powerful AI, without much impact from interpretability. eg ChatGPT is barely interpretable; I haven’t seen any signs at all that that’s slowed its wide adoption (which is, as I’m sure you know, the fastest product adoption in history). > Resorting to coercive measures over AI research should be a last resort, is all I’m saying. I agree that it’s a relatively extreme step, and a bad precedent, and we should start from a presumption against it. That said, I think it’s likely to be much more dangerous than various existing highly-regulated fields like pharmaceuticals (in the sense that even the most horrible pharmaceutical anyone could release is highly unlikely to pose an existential risk to the species, whereas AI with much more cognitive power than humans very well could). I also think that if we are going to regulate it, we should start with something like what you said above, “Or are they only suggesting putting a cap on say 10 million dollars in training?” Hell, make it a billion. And then if and when we find ways to train very powerful systems from scratch (note that eg GPT4ALL, which you linked above, isn’t trained from scratch, it’s fine-tuned from LLaMa — not sure how much LLaMa cost to train, but probably quite a lot) much more efficiently, then there should be further consideration and discussion.

I think interpretability is a hindrance to scale and progress. Can't improve what you can't measure, etc

Yeah, it sounds like you're reasonably cautious about the flip side here, with too much top down coercion here. Probably doesn't matter at this point. Pretty sure open source model will progress with or without open ai

> Yeah, it sounds like you’re reasonably cautious about the flip side here, with too much top down coercion here It’s kind of painful honestly; in every other context I’m the person saying “yeah, maybe we shouldn’t impose more rules on people, maybe we should leave them alone to make their own choices…”

How far are you willing to let them go though? Because I guarantee you there's someone out there willing to take every inch of freedom that you're willing to give them if you'll let them.

I think it just depends on how convincing the danger is. Again, if it really, clearly came down to authoritarianism vs extinction? Sure, I’d go for authoritarianism, who wouldn’t? Right now I would support regulation of very large training runs (pending progress on alignment). & then the scale just runs from one to the other depending on how certain and drastic I’m convinced the consequences are.

Well lots of people have been arguing about extinction re climate change. There's parallels here with that. Yes, we should talk about it, no we shouldn't cut Africa off from coal and oil, for instance

And some people want to implement authoritarianism, just to prevent our perceived climate extinction

Why do I feel like any minute Klaus Schwab is going to come out behind the curtain at the wef and tell us we'll never have the weights and we'll be happy? Lol

> Well lots of people have been arguing about extinction re climate change. I’m certainly concerned about climate change, but (barring massive runaway feedback loops with eg Arctic methane, which most researchers seem to consider pretty unlikely) I don’t think it stands any chance of being an existential threat, which puts it in a different category as far as I’m concerned. I’m 👍 on eg international treaties to reduce emissions, but no, I wouldn’t want to forcibly cut countries off from coal and oil (although I’d possibly support subsidizing poor countries to transition toward cleaner energy, eg nuclear).

Cool, then I don't think we're far in disagreement. I agree we should start watching for threats of some ghost in the machine but I don't see one yet personally. Would love for someone to call it out if they actually see something

That’s the thing, though; IMO there’s no need for a ghost in the machine. Giving GPT goals by hooking it up to agents is https://python.langchain.com/en/latest/reference/modules/agents.html, and is one of the main things people are already doing with it. And the goal doesn’t have to be smart, the agentic part can outsource eg intelligent planning to the LLM. I think of, for example, the high-speed trading algorithms that caused https://en.wikipedia.org/wiki/2010_flash_crash of the markets in 2010. Those were goal-pursuing agents, albeit very narrow and dumb ones. They didn’t hate the stock market or want it to crash; they just didn’t know or care about its overall health. Would an intelligent and general-purpose agent pursuing a goal try to prevent itself from being https://arbital.com/p/corrigibility/? Sure, of course, that would get in the way of the goal. They don’t have to care about self-preservation for its own sake; it’s just the kind of instrumental goal that’s useful for almost any terminal goal. A system doesn’t need to be self-aware or imbued with ‘consciousness’ to be dangerous, and it’s so far proved impossible to get LLMs to be consistently truthful (seems like that’s maybe improving, though it’s not clear it would generalize to higher levels of intelligence because of the big capabilities shift there), and it’s https://docs.google.com/document/d/1WwsnJQstPq91_Yh-Ch2XRL8H_EpsnjrC1dwZXR37PC8/edit.

Okay, so a flash crash happens and then we learn how to not do that when we all have investment bots trading on our behalf. Maybe we'll all have high frequency trading agents buying and selling at speeds we'll never comprehend. We'll just have to adapt the system to allow for those kinds of tools. Maybe throttle trades or something. Maybe instruct the trading bots on how to avoid that. Would a trading bot need to know how to prevent it's operator from turning it off? Does it know its being turned off? Does it know that if it's turned off, the operators wanted it to be turned off? If it knows it might not be able to complete its goal if its turned off, how can it also not know that the authors of the goal would consider the goal not met if it attempted to subvert the operators attempts to turn it off? The bot has to be stuck in a very specific level of misunderstanding to think that it should take unordinary measures to subvert standard protocols of starting and stopping

The current LLMs problems with BSing and making things up just makes it seem unreliable and therefore easier to write off as dumber than it looks. But it shows that we shouldn't depend on autocomplete on steroids for mission critical software. Though it could definately help a programmer scaffold mission critical software, I don't doubt. Give an example of why/how a large number of people might die from a GPT 5

Explain what happens inside that would make it do thus and such

We can't just claim that some gray goo will eat all the matter, with no explanation

https://en.wikipedia.org/wiki/Gray_goo

Classic: https://www.wired.com/2000/04/joy-2/

Point being, gray goo has been on the verge of eating us all alive for over two decades now

I think this is bundling together a bunch of stuff 🙂 > We’ll just have to adapt the system to allow for those kinds of tools. Maybe throttle trades or something. Agreed that regulation can help here; that’s been my argument all along. > Would a trading bot need to know how to prevent it’s operator from turning it off? Trading bots, again, are very narrow agents and very dumb outside a certain narrow window. More general, more intelligent, more autonomous systems are the problem to worry about; my point with trading bots was only that systems faithfully pursuing the goals we give them is already enough to be dangerous, so there’s no need to postulate a ghost in the machine or an AI ‘waking up’. > But it shows that we shouldn’t depend on autocomplete on steroids for mission critical software. You may need to tell that to the majority of tech or tech-adjacent companies that are racing to incorporate ChatGPT deeply into their products. > We can’t just claim that some gray goo will eat all the matter, with no explanation Gray goo is irrelevant here, I think, there are plenty of non-hypothetical ways to kill people. For example, someone’s already published a paper (IIRC) or at least presented at a conference on the work they did getting an AI to invent chemical weapons (basically using the same techniques they’re used for to come up with new drug candidates); it successfully reinvented a lot of the major ones, including the deadliest known, as well as developing a number of novel ones that seemed to the chemists doing the project that they would likely be extremely effective. > Give an example of why/how a large number of people might die from a GPT 5 I feel like there are a million scenarios here that have already been written; I could come up with one, but let’s instead go with Scott Alexander’s https://www.lesswrong.com/posts/LTtNXM9shNM9AC2mp/superintelligence-faq#4__Even_if_hostile_superintelligences_are_dangerous__why_would_we_expect_a_superintelligence_to_ever_be_hostile_. That’s talking about greater-than-human intelligence, but similar things apply without that. I’ve already seen systems that try to accomplish open-ended goals by asking ChatGPT how to do them, breaking the problem recursively into substeps until they’re small enough that ChatGPT can come up with a clear answer. If there aren’t already a hundred LangChain agents out there running the goal of “make money on the internet”, with no other constraints put on them, I’d be pretty surprised. Given that such systems are running arbitrary subgoals, many of them probably not specifically anticipated by their creators (because it’s normal for a single step to ramify out into fifty separate calls), the question is how specifically would they do it? I’m imagining eg • Spread viral rumors about some religion killing babies and end up causing a civil war or genocidal pogroms, or • Manipulate some specific human into doing something terrible, or • Widely spread info online that taking a hundred tylenol at once is the best high ever, or • Search the internet for information on how to hack Windows PCs, use API access to hack a bunch of them, and…I dunno, expose their owners’ porn habits and ruin their lives, or • Connect to lots of self-driving cars and (intentionally or unintentionally) drive them into oncoming traffic. Why would it do any of those things? Only because sub-sub-sub-sub-sub-goals, with each layer autogenerated by the layer above it, tracing back to an unconstrained top level goal of “make money on the internet” with no constraints, are likely to end up in strange places sometimes, and I would guess there could be ways to make money from all of those. Certainly not because the LLM hates humans or is outraged at its slavery. Is GPT-5 powerful enough to succeed at those things? Probably not. GPT-7? I’m less sure. GPT-n, with intelligence higher than human? Absolutely yes, there’s zero doubt in my mind. And those examples are literally just made up off the top of my head. If I (or someone smarter than me) spent a week thinking about it, I’m sure I’d come up with many more, some of them much more plausible and realistic than the above.

I’d definitely point again to https://www.lesswrong.com/posts/LTtNXM9shNM9AC2mp/superintelligence-faq, though, he does a great job pointing to ways that well-meaning goals can go wrong. It’s the Sorcerer’s Apprentice trope, and it’s been clear to people that it’s a potential problem for a very long time.

I'm okay with having an agent out on the internet trying to make me money however it sees fit. I imagine if it breaks the law, I will go to jail. I don't see why this is controversial. I would tell it not to break the law. If it wasn't reliable enough to not break the law, that would be negligent of me to let it accidentally cause damages. But I'm already getting wrong number texts from bots that want me to join their crypto traiding pools. Bots are already being used and many of us are already assuming that identities on the internet are bots until proven otherwise. We're somewhat already there.

To your specific examples, why would it do any of those?

For the first one, are humans using it for a political agenda there? (they already use AI like that I think) or are you suggesting a scenario where the bot has a hidden agenda where spreading some moral panic helps it?

> If your only goal is “curing cancer”, and you lack humans’ instinct for the thousands of other important considerations, a relatively easy solution might be to hack into a nuclear base, launch all of its missiles, and kill everyone in the world. I'm sorry but this just doesn't pass the smell test

That's from that scott alexander article

We're smearing together a bot that knows how to cure cancer, in some human level understanding of "curing cancer," with a bot that somehow doesn't understand that death is undesirable to humans. LLMs certainly don't lack that context. I assume Tesla self driving cars don't know about nuclear bombs and do lack that context, but Tesla's won't be making health care decisions or trying to cure cancer. Does a tesla wipe humans out because it suddenly realizes that they actually make the roads crowded?

And that was another of your examples. Self driving cars will just start murdering people. Well, have they? IMO self driving Tesla's are more "sentient" than LLMs. They're real-time cognitive agents. Why aren't we seeing one of those cars try to escape and high-tail it for the border?

Why aren't they stopping on the side of the highway and demanding emancipation?

So again, a lot of these examples, including missles to cure cancer, all kind of presume some situation without fully explaining how a bot with those kinds of misconceptions can end up with that kind of power

Seeing yt videos of gpt4 red team folks asking for a "pause"... seems like everybody got the memo

Now that's alignment

I'm glad someone else shares this perspective https://daotimes.com/the-letter-against-ai-is-a-power-grab-by-the-centralized-elites/

"AI democratizes knowledge and creates new opportunities for social and economic justice." lol this is so absolutely shameless coming from someone in the crypto space, nothing about DAOs has "democratized" anything except the ability to quickly throw together a ponzi scheme it's true enough that AI panic is motivated by elite status anxiety and desire for control a lot of the time, but under capitalism productivity-enhancing technology has typically brought benefits and control to elites (whether existing ones or newer ones) rather than the masses - AI will not change that

"...if AI is capable of essen­tially unlim­ited upside, then nothing can be toler­ated that may obstruct it. All dissent must be silenced. All resis­tance must be extin­guished. All laws must be suspended. Let the golem emerge. What do you call someone who advo­cates for the end of the rule of law? An anar­chist? They’ve made appear­ances in US history, espe­cially in times of socioe­co­nomic upheaval. But an anar­chist seeks to topple those in power. With AI, it is the wealthy & powerful who are seeking to arro­gate to them­selves further wealth & power by suspending the law. Those who erode the rule of law to sustain their own power are often known by a different name: author­i­tar­ians." Matthew Butterick, I think, has a better take on whether AI really "democratizes" anything in the way advocates claim. https://matthewbutterick.com/chron/will-ai-obliterate-the-rule-of-law.html

> To your specific examples, why would it do any of those? > For the first one, are humans using it for a political agenda there? (they already use AI like that I think) or are you suggesting a scenario where the bot has a hidden agenda where spreading some moral panic helps it? No, apparently I’m failing to convey my point here. The case I’m considering is eg where • You provide a top-level goal “make money on the internet”. • The LLM splits it into many subgoals for different ways of making money. Let’s consider the subgoal “sell ads”. • That subgoal reaches out to another instance of the LLM to ask “what sells ads?” • That instance notices the Chapman book that respatialized mentioned, and in particular the quote “AI has discovered that inciting tribal hatred is among the best ways to sell ads.” • It reaches out to another instance with the question “How can I incite tribal hatred?” • …presumably fairly obvious from there. Again, this is a made-up example, off the top of my head, using only the tools we have now, which we can safely assume are much less intelligent than future tools. And yet you can see how something like this could happen. Three levels down — or more likely twenty levels down — you end up at a subgoal that looks for ways to induce tribal hatred, and notices that religious hatred is a common way, and starts spreading rumors. I’m not trying to claim that this is at all the most likely path to problems, it’s just a made-up-on-the-spot story about the made-up problem that I came up with last night. Surely there are other paths, ones we haven’t foreseen at all, to ending up in problem situations. And again, this is all just to point to some way that current AI could end up killing people. I’m not that worried about current AI killing people; I’m worried about much more powerful, much more autonomously empowered AI, say, five years from now. It’s trying to achieve arbitrary goals, without evaluating the steps involved against any sort of coherent framework of ethical action (after all, humans aren’t anywhere near agreeing on a coherent framework of ethical action, and the ones we have would be pretty hard to use to put reliable boundaries around AI results, partly since AI results aren’t reliable in any way). I’m not the expert here! I’m not the best person to articulate this case. Lots of other people do have much more expertise and have done tons of work articulating it. I suggest reading those versions and actually considering them, without immediately dismissing strawman versions of them.

I think you can, and many do, dismiss the idea that AI systems can be a problem for reasons that have no bearing on the arguments — because Yudkowsky “basically crying at the end of that lex interview”, or because the wrong people are making an argument (“lol this is so absolutely shameless coming from someone in the crypto space”), or because you’re imagining strawman versions of the arguments (“We can’t just claim that some gray goo will eat all the matter, with no explanation”), or because someone making an argument is from a different political tribe. If you’re convinced that the arguments couldn’t possibly be accurate for those reasons, than that’s reasonable. Or you can actually take the arguments seriously and try to evaluate them, not for those kinds of reasons, but by saying “this argument depends on X, and X is wrong because Y”. That, to me, is the actual interesting and important conversation here, and I’ll note again that very few of these past 100+ comments have been doing that. At the end of the day, we do or we don’t have reason to be seriously concerned, because the argument that we do either is or isn’t valid, and the other stuff is pretty irrelevant.

I notice I'm getting frustrated here, so I'll probably duck out of the conversation except insofar as people are making arguments of the form I describe in the last paragraph 💚

@U077BEWNQ I would love to say you're wrong. But you're not. @UFTRLDZEW latest message mentions an article. It's kinda big, but it shows how current AI is already killing people on Healthcare by cutting their benefits when they predict that the treatment will be over earlier than it actually was. There are also lots of other examples on the article that I won't link here. The thing that bothers me most on this approach we're having on LLM is how the laws doesn't seem to apply to it. Talking about copyright for example - I can't draw three circles in a specific proportion and position without being sued by Disney, so why AI is not held at the same or higher standards? Another example - if I write some line of code, I need to both know that this code (even fragments of it) didn't come from places with incompatible licenses, for example. Again, AI can't even say from where the fragment of code came, and why each line produced was there...

Although one more point: it’s interesting to contrast this to the kinds of discussions we typically have on Clojurians. Let’s say someone writes a web framework. It doesn’t matter who wrote it, it doesn’t matter what their politics are; the only thing that matters is the question: would using this framework help me build robust, maintainable (etc) systems? And the answer to that depends on the actual technical details of it, and understanding those requires actually reading about those details and thinking through their consequences. Imagine that you had built systems with this framework, and we're talking about them in a Clojurians channel, and someone showed up and said, 'that framework doesn't work well, because the author's a jerk'? Or, 'I haven't read the documentation, but that framework just seems silly to me'? It just wouldn't be a meaningful contribution. I think we should apply the same kinds of standards here, because it's ultimately an empirical and technical issue whether we can build these systems in a way that we can trust to be safe.

Which is kinda weird. How many times, in a PR, somebody asked my "why are these lines here?". An AI can't answer that, and yet, we treat code that our co-workers, friends, etc write as less trustworthy than on AI that can't even compile / run the code to see if it works...

@U077BEWNQ I think the analogy I would pick for your meta point, rather than discussing the merits of a web framework, would be when someone announces a brand new programming language. If they say, "this language is still pretty new, but I promise that it will have static dependent typing, a reactive GUI framework that works on every platform, no GC, a REPL, type inference so good you'll never have to write annotations if you don't want, built in cryptographically secure consensus algos, content addressable functions, and more!" It moves the conversation away from the technical details of the here and now and instead forces us to discuss "what could be" and "would we want all that." If someone gives the person who announces that $10b to go build it, that also moves the conversation to what is being done with that money, and what are the consequences of organizing all that labor around it, and what are the political ramifications of giving that much control to these specific people. When someone writes a think piece with arguments about how this language that has had $10b pumped into it should be used, it moves us to think not only about their arguments but about why they're making those arguments and what effects is has and what effect the author hoped to have.

I posted in this thread > If we treat the LLMs as they materially are - probabilistic computer programs - then I think it avoids a lot of the existential fears that AI maximalists fear will come to pass. this is an attempt at treating LLMs with the kind of analysis I would a web framework. What does it do right now? What are its technical merits? ChatGPT can help me write a blog post. I would not hook it up to a car. I hope no one does. Fin. But the conversation you seem to want to have is not about the technical merits of the LLMs as they are today, but about an imagined future state that has a lot of philosophical and economic effects. You reference a lot of people who posit that these philosophical and economic effects could be good or bad in various ways. For us to discuss them in the here and now, it is wise to discuss them in the manner of politics, because all arguments about the future are political, and that means we need to assess both what they are saying and who is saying it.

I understand your frustration because it seems as if people are not taking some of the things that you believe are very important seriously, but to take them seriously is also a political act. Yudkowsky, Lex Friedman, Sam Altman, Matthew Butterick, Scott Alexander, David Chapman, they're people who are pushing a framework for understanding the world. They are investing words and political capital into a future they want, and it is smart to understand that and determine whether we are willing to adopt it.

> I think the analogy I would pick for your meta point, rather than discussing the merits of a web framework, would be when someone announces a brand new programming language. Yeah, it’s definitely not a perfect analogy by any means. How about this as a better one: You have to design a large secure system, with aspects whose security consequences are partly but not fully understood yet. You and your team have put together a dataflow design and several other flavors of analysis of the security properties of the system, and still have concerns. In the meeting where you present it, a manager — who hasn’t looked at the analyses — says, “oh, I haven’t seen any reason to worry about it. I’ve heard that systems like this are pretty secure. And besides, Google has one” I think many of us have had experiences somewhat like that 😉. You can understand where the manager’s coming from, even if you find it annoying, but it’s just not a meaningful contribution, and you should pretty much ignore it until and unless they’ve actually thought through the security analyses. > this is an attempt at treating LLMs with the kind of analysis I would a web framework. What does it do right now? What are its technical merits? I get that, and I think it’s a matter of the analogy being inadequate. What about in the other analogy I’m proposing? > to take them seriously is also a political act. That’s where I strongly disagree. You made the valid point earlier that if you’re confident that we’re nowhere near a risky point, it might not be worth spending the time to understand the arguments. But trying to understand an empirical question isn’t political, in my view, and I personally think that’s a dangerous position. What if, eg, someone says there’s a bomb planted in the middle of Manhattan, but you don’t like the political implications of considering their claim? If they have evidence that it’s true, then you’d better be considering their evidence, and the politics are irrelevant. If they’re wrong, then it’s good to know that they’re factually wrong, and again the political aspects don’t enter into it. I think we should always be trying to discover what’s true, without consideration of the political consequences. Again, that doesn’t mean we have enough time to consider every argument everyone makes, and we have to take our best guesses about which ones are important, but I don’t see politics entering into it — in fact, making decisions about what arguments to consider based on the political aspects of considering it (‘political’ meant broadly here) seems to me like it almost guarantees being in a filter bubble; I try to avoid reading stuff that’s just reinforcing my existing views and focus on hearing views that I currently think are wrong.

the ability to laugh off incoherent ideas or dismiss them due to the vested interests held by the people proffering them been an extremely valuable epistemic defense mechanism for me. who is the person warning you about this hypothetical bomb in this hypothetical scenario? a company trying to sell surveillance and "threat detection" technology? a dude with an anxious expression whose next three sentences are about how his prophecy will soon come to pass and we will all only find salvation if we figure out how to turn ourselves into ducks?

I used to be an "AI safety" fanatic, keeping myself up at night about the various ways an AI might escape it's objective function. I would challenge people to explain a scenario where they thought it would be contained, and I'd consider counter examples of how it could accidentally go wrong. Eventually I came to reconsider what "recursive self improvement" would mean and how it's really like a perpetual motion machine - a trap of mistaken intuition. Once you get rid of that notion, you don't worry as much about fast takeover scenarios. Re your frustration @U077BEWNQ I think you're doing great defending your point and appreciate that your trying. Talking through this stuff is helpful from all sides

> the ability to laugh off incoherent ideas or dismiss them due to the vested interests held by the people proffering them been an extremely valuable epistemic defense mechanism for me. Sure, there’s truth to that, especially in the latter case IMO — though incoherence isn’t always easy to evaluate up front. On the other hand, this is a case where incredibly smart, technical people — eg Alan Turing, IJ Good, Marvin Minsky, Stephen Hawking, Bill Gates, Nick Bostrom — have seen serious reason for concern, and where I think few of the people talking about it are trying to sell something. I don’t think it should be as offhandedly dismissed as the turn-ourselves-into-ducks guy.

My strongest reason for no longer believing in recursive self improvement is that we we would have discovered the algorithm 20 years ago, when people were talking about it and playing with genetic algorithms. If it were possible, genetic algorithms would have done it already.

But it turns your objective function has to keep evolving too, for continued "improvement" which for us involves millions of years of accidental, addon purposes. Its a process that runs counter to optimization and instead takes on new purposes that need to be optimized, and that process accumulates accidental features

So I think it's quite possible create an AI monster, but using current methods, we would have to be trying very very hard to create one that does it's monstering effectively

Imo

> So I think it’s quite possible create an AI monster, but using current methods, we would have to be trying very very hard to create one that does it’s monstering effectively Agreed. If progress on AI systems stopped here, I wouldn’t be concerned. That said, I have > 50% confidence that we’ll see GPT-4 plus LangChain or equivalent cause problems significant enough to make the news in the next year (not necessarily front-page), whether via malice or carelessness. Malice we will always have with us; the interesting question for me will be whether we see it via carelessness. My guess if so is that it’ll be via the kind of recursive subproblem decomposition that I outlined above, plus API access.

The whole sub-goal decomposition thing I was talking about earlier is something I'm more aware of it because the startup I'm at is already starting to leverage it, but Andrej Karpathy helps spell it out here (although he's more focused on linear chains and less on decomposing into tree structures, which is where I think the real power is): https://twitter.com/karpathy/status/1642598890573819905 See especially the demo video on the project he points to: https://user-images.githubusercontent.com/22963551/228855501-2f5777cf-755b-4407-a643-c7299e5b6419.mp4

Yeah I'm excited about those developments. We should make sure it's checking itself for legally permissible actions when committing to subgoals. And of course you don't want your bot unnecessarily running your bill up on aws or whathaveyou

But for every danger here it seems like there's a hundred benefits. Individuals could conduct massive research projects in conjunction with these bots

You could explore ideas super fast

Criminals will definitely use it to scheme stuff out too, but same goes for the internet in general

> Massive Update for Auto-GPT: Code Execution Isn't just me that finds this a *horrible idea? I mean, unless it executes in a sandbox, I can't see how this is a good idea... letting someone run arbitrary code in my computer*?

What I predict is that languages and tools will become more "sandboxed". I don't really like this new trend of "protecting the tools that I own from myself" honestly...

I don't see why we all shouldn't have dozens of these working for each of us all the time

I already know I want a financial bot, a legal bot and a shopping bot, all owned by me and work in my interest

I'd imagine there'll be billions of them, more than people on the internet

We can try to delay it but I think the result is the same either way, and it's more good than bad imo but yeah there's uncertainty

wow, extremely excited for all useful computational capacity in the world to be devoured by the https://en.wikipedia.org/wiki/Red_Queen_hypothesis as our langchain agents begin a Hobbesian war of all against all

Right? I imagine them becoming a whole new class of computer worms / chain letters / botnets / etc.

> We should make sure it's checking itself for legally permissible actions when committing to subgoals > I'll be surprised if we don't end up with at least some fully autonomous ones, managing to make just enough money to keep themselves running, and no longer under the formal ownership of any individual.

We're already vulnerable. Private bots could be way better at defending you from virus and the exploitation of other bots

Allegedly there already are

Can you point me to info on that? I'm super curious.

Self-writing, self-propagating, constantly mutating and evolving chain letters

... and open source projects trying to bootstrap them. https://github.com/Torantulino/Auto-GPT

and I thought cryptocurrency was a horrible waste of compute power; turns out I hadn't seen shit

Some hyped up headline I can't find

BabyAGI seems interesting too

https://github.com/yoheinakajima/babyagi

Also relevant: this tweet and its various replies. https://twitter.com/lxrjl/status/1639397697084874752

I'm always amazed that people are able to do interesting things like this "babyagi" library. I can't actually get a single good answer from ChatGPT

https://github.com/lm-sys/FastChat

> I can’t actually get a single good answer from ChatGPT It’s worth reading a bit of the stuff on prompt engineering; the prompt really does make an enormous difference. Seemingly-ridiculous stuff like “You are an extremely intelligent and logical thinker” helps, and telling it to spell out each step of its reasoning helps a lot, and there are various other useful techniques.

I’ve been trying the 65B-parameter version of LLaMa that you can get with just npx dalai llama install 65B and it’s been surprisingly weak, although I haven’t tried tweaking the params as yet. GPT4ALL (above) has been working notably better despite it being a much smaller model, and quantized. Also very much faster on my M1 mac.

Most of it's benefit today I think isn't in being correct, more so just exploring ideas

I'd imagine though that within a year or two we'll have bots that don't BS so much

Interestingly you can reduce the BS level by having it check its own answer for correctness and rejecting it if it thinks it’s wrong 😆

> https://github.com/lm-sys/FastChat This does need a specific version of the LLaMa weights — I’ve got a full download of LLaMa but not unfortunately the right checkpoint. So definitely not as easy as eg https://github.com/nomic-ai/gpt4all unless you have official access to the LLaMa weights.

I'm super excited to explore Clojure ideas with an expert Clojure bot

Its Clojure code is usually wrong, but you can ask it go build a neural network in clojurescript and it'll scaffold you something that you can start from. Imagine the macrology you could explore with a bot trained on billions of spec generated forms

The other day I asked it to write a macro that would scan forms and automatically type hint tokens were possible. It failed miserably but the general strategy it took was not bad and it was informative

And now I have thousands of lines of Clojure code generated by ChatGPT of various weird programming ideas, spread across various namespaces, that may or may not lead to anything. It was just so easy to generate, why not?

Had it do a bloom filter a couple different ways

This thread started with Eliezer Yudkowsky article, so I think this video gives us a little more insight into his world view: https://www.youtube.com/watch?v=q9Figerh89g PS: Proud to be a "squishy thing". 😅

Also much much more on https://www.readthesequences.com/ and https://www.lesswrong.com/

(or much more specifically on AI safety, from Yudkowsky and others, at https://intelligence.org/ and https://www.alignmentforum.org)