Clojurians Log v2

Ugh! People keep quoting that study and it is horribly flawed...

Oh? Say more!

It was discussed a while back https://clojurians.slack.com/archives/C068E9L5M2Q/p1752205978324349?thread_ts=1752188521.201919&cid=C068E9L5M2Q

The outlier -- the dev who was much faster with AI, he was the (only) dev really familiar with the AI tooling they made everyone use.

I feel like I've debunked this single study multiple times in multiple places at this point 😞

(sorry, had to step away) So the devs were randomly assigned issues from their projects and randomly made to use AI or not, so they had no choice over whether to use AI appropriately (see my quoted comment) and, as far as I could tell, were all made to use the same AI setup (Cursor) regardless of whether they were familiar with it or not. I'm new-ish to AI so I'm nowhere near as effective with it as some of the folks here (in #ai-assisted-coding at least). I can look at an issue in my OSS projects and get a good sense of whether I can get an AI to do it quickly or whether it'll be faster to do it myself. Even with that sense, sometimes an AI just gets in a mess and it takes me longer to either get it back on track or to start over and just do it myself. I'm sure most of those devs with in a similar place. As for the incorrect estimations of faster/slower -- we're all pretty terrible at estimating, and I'm sure the "fun" of using an AI made time feel like it was going faster, even when it wasn't, so I can imagine a dev getting focused on driving the AI and completely misjudging how long it really took compared to how long it would have taken without using AI.

I have to disagree with Sean and wtv criticism people have here. If you think that study was poorly done, than stop trusting all studies 😛 , they seldom ever get better than this.

Plus this person isn't just quoting the study, they are their own mini study, and it replicated the METR study's finding. In their case, it sounds like they are quite familiar with AI, used the AI tooling they were already familiar with.

I also, just, it takes like 2 days max to get proficient with AI coding tools. It's not super hard. This isn't like, you need to work with AI coding agents for 200+ hours before you see increased productivity. After like 8 hour of using them you should have figured how to use them.

Also, there's some things a bit innacurate. The study claims that, the engineers were as familiar with AI as any other average professional engineer at the time of the study. So like pick engineers at random, some might be using it all the time already, some just starting. Then what really matters, is that they ALL said they were faster with AI. No matter how "familiar" with it they were. And they were ALL wrong, except for 1. That 1 person might have a magic AI usage sauce, or they might just be a slow developer that is a bit faster with AI. I think the only criticism worth mentioning of the study, is it was Cursive early this year, and I don't know how good that was as a full on coding agent that does edits itself, versus say Claude Code or Cursor now. (and the sample size)

I have another anecdata, the 3 teams I oversee, there's ample coding agent usage, a lot of people quite verse in how to best use them, but not everyone, kind of 50/50. But the team's velocity is the same as it was before. Maybe they code faster, but we still don't get more task points done. It's strange. If this was an experiment, I'd hypothesize as teams start leveraging coding agents, velocity would go up, and that's not happening.

And I say all that as someone who uses coding agents daily. But the boost in productivity is simply non-obvious to me, and I think very much up for debate. The big difference is, I'm just lazy, and using AI is lazier, I can turn off more of my brain, and I get props for using AI and being at the "forefront" 😛. Also, ya, it often feels faster/rewarding, like how the windows install had a bar that animated as if it kept getting bigger but it didn't really.

A question: what does it take to become "familiar" with a tool that wraps a popular text editor with autocomplete suggestions? as didibus suggests, I have some doubts that the learning curve is steep, or maybe even that it exists at all.

As for the choice whether or not to use an LLM based on the nature of the task, I also wonder if people are making such considered judgments or are instead nearshoring their entire mental process to an LLM (that seems to be the case for many non-technical users, and I don't have any anecdotal reason to assume it would be different among the vast majority of software developers.)

> Sure, it may take only one hour to describe what a system should do. But the reasons it takes a year to build are fundamental and unavoidable. Anyone who says they can bridge that gap is selling fairy dust. > Unless it's AI slop generated project templates. In that case, so true!!

But the truth is, I don’t want it to stop. If people actually started building coherent systems, then what would I even do with myself? I’d have nothing left to yell about.True, we need job security. Today's bug is tomorrow's bug fixing paycheck 😄

This is why I'm not pessimistic about LLMs