Clojurians Log v2

In this sense I think AI can't outsmart the user in general, because it primarily 'reacts' to what the user says. The context in the chat is like a limit that caps the performance.

It is narrative autocomplete

And it is atrophying the brains of a generation

More than a generation

IMO the thing that's made LLMs useful for me is the ability to run it in a automatic feedback loop (E.g. make a button that does X, then click the button and make sure it does X, repeat), with the corrolary being that for any task for which the LLM can't run in an automated feedback loop, it's a lot less likely to be useful. so I like it for spitting out a rough draft of probably working code, but not for making design decisions or writing docs.

My experience is that, it can make things more convenient. For example, I can fire off a prompt and go make coffee. Or I can code on my phone while I feed my baby a bottle by just talking a prompt to it. Those are nice features. And it seems I need less brain power to get going with it. But loops like that always make me question if I'm really getting things done faster. It seems I can only go faster by lowering my bar on the code, if I start doing LGTM basically 😛. "Oh well, it's not the cleanest, not the greatest abstractions, overly verbose, factored weirdly, and a few things confuse me, but whatever it seems to work push it." Else, as soon as I try to not do that, the above loop enters. Basically, giving the AI "taste" takes many iteration of that loop, and likely eats away at a lot of the initial gains. Sometimes the loop is infinite, and I have to manually intervene as well, as all attempts of steering it to a tasteful place fails.

> the mental model is that asking it to write X isn't really the tool writing X for you. It's the tool producing a likely approximation of what X might look like if someone had written it. A very good "fake it until you make it" engine, if you will @jason.bullers I like that, I think that's a good mental model. The fake it until you make it engine is so true, it's literally faking being an intelligent developer. I've also seen it as an optimized version of "randomly generate ASCII chars and eventually you'll get a generation that's exactly the code you wanted". It's like if you could steer the random generator towards a tiny subset near what you want but still within that it's a matter of luck of the draw if you get the right thing.

And it is atrophying the brains of a generation

Fear of brain atrophy is one of the main reasons I don’t want these things too close. I know what smartphones and social media did to my attention span, I have zero interest in repeating the exercise with my reasoning skills.

@christian767 apart from simply not using brain powers like before, what symptoms of brain atrophy do you actually experience? how do you sense it?

I don’t experience it, because I haven’t tossed myself at these tools. (Edited previous message in an attempt to clarify).

I too have a vague fear of that 'brain atrophy'; but there's no way to quantify or measure it. perhaps it's overrated?

What additionally really bothers me about AI is that this feedback loop you talk about can be really slow. Sometimes I ask the AI something simple (“how to read JSON”), and it goes on tangents trying to figure out what JSON libraries are in my ~/.gradle/cache or whatever. Thinking about my query for minutes and still producing the wrong result.

@tlonist.sang I used to be pretty good at mental arithmetics when this was taught at school; I haven’t practiced it in 15 years and I am really bad at it now — thats what I’m afraid of, but for critical thinking instead.

> Fear of brain atrophy is one of the main reasons I don’t want these things too close. I know what smartphones and social media did to my attention span, I have zero interest in repeating the exercise with my reasoning skills @christian767 love this point - I’m slowly clawing back my attention span by reading novels. It’s taken a long time to actually start finishing books again. So easy to lose, so hard to get back. God knows what I’d need to do to regain reasoning…

> asking it to write X isn't really the tool writing X for you. It's the tool producing a likely approximation of what X might look like if someone had written it I like this

@tlonist.sang as someone that peeked too close to the abyss - in my case, it happened when I had to write something that the LLM could not do, in a project I didn't know. I felt the urge to do something else instead of looking at the code; from time to time, I could feel my concentration fading and a huge urge to ask the LLM - kind of like "one more prompt, and that will be solved". It felt like I was on drugs: the sense of fulfillment that I delivered a task without any of the effort, then the regret of me not absorbing anything from my solution, and needing to prompt again to solve something that should be very simple.

It literally felt as if my brain was unwilling to think alone, and that it was refusing to even try; as if it found a way to cut the effort and decided, by itself, that the effort was not worth it.

The dopamine hits are a real concern. I've found the same on some hobby stuff. It feels good to have a cheerleader at your fingertips that validates you (and shortcuts the need to think critically or seek human input)

It also bothers me. Sometimes I want the LLM to tell me that I’m doing something dumb and I should really hit a hammock :’).

I too have a vague fear of that 'brain atrophy'; but there's no way to quantify or measure it. perhaps it's overrated?

You remember the big debate around calculators? Maybe I'm showing my age, but there was a thing (pre/early internet meme?) back in the day that you need to know how to so math in your head because you won't always have a calculator in your pocket. And haha, joke's on them right? We've got that and so much more now. But even at the time, how common was it to reach for the calculator even for simple sums? Anecdotal, but I'd say very common. You could say that it doesn't matter much: doing math mentally isn't worth the brain cells when you can outsource it to the calculator. No big deal, good savings. Maybe that's true. But LLMs promise to let you offload so much more, and they make it so easy, it's practically frictionless. I think there's an important line we need to draw for what's good outsourcing and what's so much that it's detrimental to ourselves. I don't know if that's a very fuzzy, personal line, or if there's a broader societal line that's super important to stay on the right side of

Oh for sure, I mean kids can't even read or anything anymore, and that's just from brain rot attention media. Throw LLMs on top, and I don't know...

Strong parents give birth to weak kids. Extremely literate parents give birth to extremely illiterate kids... oops 😄 Back to dark ages we go 😄

Man can't wait for the Sea Peoples to show up. Then we'll know it's bad.

So I think our teachers were right in insisting we know some maths in our heads, but the reasoning “because you won’t always have a calculator” was an easy scapegoat that backfired.

@christian767 I agree with you. I'm really just trying to explore thresholds, because, like most problems, I think there's a lot of grey here. I have genuinely felt dumb in the past picking up a calculator on autopilot to check something trivial that I could have calculated in my head in less time than it took to punch in the numbers. But I hope you would agree that if you had to sit down and do large sums, it's more efficient to just punch all the numbers in than to sit there and write out or try and keep in your head a tally of a hundred numbers. You haven't lost the ability to do the math, but you're eliminating cognitive load from low value work. Another example in this space to maybe illustrate what I'm driving at: auto complete in editors. I could just type everything out character by character, but if I know what I'm trying to say and the calls get filled in for me, that's a net savings. Same with automated refactorings or paredit key strokes. I haven't figured out yet where the line is for AI that gets us the above without stealing useful skills away from us. Many of the tools we've used before have a natural maximum of what they can take away; even if you pick up a calculator for literally everything, the thing you've lost is mental math. If you auto complete everything, you may not remember APIs anymore, but you know how to pull up a list to choose from and use your judgement from there. AI feels like it can bleed into everything, and we need to put our own boundaries on it. And I think that's hard because the whole thing is designed to be as low friction as possible, and, intentionally or not, is basically addictive

For me its “learning from the ground up” to some extend. I am happy to use a calculator because I know how calculation works. I am also happy to use autocomplete because I know how to write code. What LLMs threaten is the need to understand. If someone shoved me a calculator in my hands when I was 7, and explained me that the quiz question 7 + 5 can be answered by pressing, 7, +, 5 , = and faithfully copying the output to my paper then I would lack understanding what arithmetic actually is. I am struggling with where this boundary lies. I also didn’t learn to program in assembly, nor did I make my own CPU from logic gates. So how far down the stack do I need to understand? But, having this fundamental understanding of programming makes me able to unravel the stack. If it really bothers me, I can inspect how my program gets converted into assembly and going down the stack. So if anything, we need people to continue to learn to learn. Have enough of a grip on subjects to be able to unravel complex problems. I think this is what education is actually about.

(My group theory teacher suggested learning group theory before arithmetic. His argument was that it was the foundation of arithmetic. I think the 7-year-old-me would have stuggled with that)

I like how they share insights on the practical downsides of LLM usage for a non-profit code hosting organisation. Apart from the normal objections against LLMs this I can fully understand in isolation. It also makes you wonder why for-profit organisations are still happily running all your slow CI jobs and hosting your unlimited private git repos for free 🙈

> If you personally work on developing LLMs et al, know this: I will never work with you again, and I will remember which side you picked when the bubble bursts. ✊

Wonderful article. Fully convinced me to finally move all my projects there.