Clojurians Log v2

More specifically, I'm interested in where people get slowed down by LLMs. It's rare for me to see people write out complaints from experience.

i could write unsafe code in any language. and agent to agent comms are kind of a niche

@danlentz wrong thread again

man why does anyone trust my programming

lol

I thought the conclusions from https://clojurians.slack.com/archives/C8NUSGWG6/p1790116305800989 were a nice honest take of their experience, which I appreciate. It's not 100% good or 100% bad, so maybe no one will be happy, but I think these honest experience reports are worth sharing.

what kind of testimony can i offer? i've had good experiences

i'm also an ai skeptic, but not in the sense of abject moral opposition. i'm using it in practice and working on ways to constrain it to produce the result i want

“skeptic” in the traditional sense — where one can be convinced by evidence

https://blog.jsbarretto.com/post/i-remain-a-skeptic https://joshcollinsworth.com/blog/productivity

More code that does not give more sales means that the code is not productive in an economic sense.

I'll say that I'm significantly more productive in terms of being available for my coworkers and less annoyed/slowed by interruptions but that's mostly anecdotal and tracking would involve some privacy issues

It kind of depends. I vibe coded this: https://github.com/didibus/go-to-sleep and it took me about 10 hours. I don't know how much longer it would have taken me the old-school way, likely more than 10h. But the biggest difference is possibly that those 10h, a lot of it was spent waiting around for models to do things. Where-as the old-school way, the time would have been spent where I'm focused and engaged mentally throughout each minute of it. At work it's hard to see actual gains in project releasing sooner. You get to "code complete" faster, but QA takes longer. Getting things exactly as you want takes a while, as you have to go back and make changes to how the AI did things. If you kind of just accept wtv it did as-is, then you avoid that churn. It's also hard to tame it, to do less, be more KISS, not just in how it implements, but less tests, less security audits, less docs, less features, etc. What happens is you often end up with almost too much, and you can't audit it, or focus on what matters, and you get distracted.

Basically, at work, I think we're realizing that devs often coded in parallel to other things, and those other things are not faster. The decision making, alignment, how to solve for problems that show up, the reporting to leadership overhead, setting up infra, tooling, managing databases, clusters, ops concerns, test plans, working through all scenarios, etc. All that stuff still takes the same amount of time. In the past, devs would be coding in-parallel to that, and the coding effort often helped to figure those things out as well, as devs got really deep mentally into it.

We’re pretty close to requiring an indepth review from an LLM for merging. I wouldn’t want to ship without it. I agree i see lots of people using LLMs to create software factories that in theory can make stuff with LLMs. But the world has fundamentally changed from what I can see and coding seems to be pretty solved. Architecture is still wide open, and api design. But once those are decided the LLM can exceed human capability as far as i can tell

QA takes longer that's for sure

Because PRs can be far larger?

i you need to learn to take smaller steps, and enforce that discipline on yourself. or, yeah

Not related to large PRs in my experience. It's all about getting the AI to do what you want. There's a lot of things in QA that pops up that you realize, ok ya that is not what we wanted, or this should not work like that, this isn't the correct behavior in this case, etc.

Sometimes there's whole feature you didn't even asked for but AI added 😛

it is exactly about getting the AI to do what you want. just like a junior engineer. i’ve described it as a new puppy with a PhD in computer science that still poops in the floor

I don't know that I'd say AI coding exceeds human capabilities. It's still often not to my tastes at least, but it can definitely produce good enough code, that works, and much faster than it taken me. Though I admit, it's better at leetcode than I am 😛

You have to use Opus 5.5 class models though. Pre Opus 5.5, and Astra, I'd say it wasn't true yet that "AI can code well enough"

Or maybe Fable-level and up to be more precise.

@didibus excluding overall architecture, what do you think you could code better than a frontier LLM right now?

i started getting reasonable results at 4.5 if i gave it examples and patterns to use. but the progress since then is kind of breathtaking

i think i’d catch a few silly mistakes, and push back on api design and work with it to design architecture. but the code it would make once that is settled is equivalent to or better than me and way, way faster than me

i feel like that. it's like, nomatter what, even if i need to go back and clean up a few things it's still doing 5x my productivity

Most things that don't lean too heavily into complex algos or data-structures. My code would often be cleaner, but it is possibly to my preference, but in general I feel most good human coder would produce nicer code. You can see that if you ignore code, in how AI generates english text. It works, you understand the points, it makes good arguments, it has good knowledge, but its just not the clearest nicest way of expressing the ideas compared to a good professional human writer. Same for code, except at some point it matters less for code if no one but other AIs will read or work with it.

fair point on some stuff being cleaner

i agree that it can produce slightly to much more tasteful code. Im still figuring out if this matters

It's also inherent to the model. They shifts the weights away from what humans would like, to what rewards the output accuracy. This is especially true for code. So it's normal that it's not the nicest looking code for humans, they train it to produce working code, more so than nice looking code.

me too

It also explains why some of the new models talk even more annoyingly than before 😛

Since they are doing less RFHL than before, in favor of more automated RL.

it's difficult because the name of the game is constraining behavior, yet it's so difficult to establish metrics

my approach is with a graph based meta model. but i'm also struggling with usable metrics

@dpsutton I think it doesn't matter anymore. When they would make mistakes, introduce bugs, and so on, it did matter, cause you had to review/work with it still. Now that they can make the code work almost always. And debugging an issue is just telling them the bug and they fix it. You really don't need to work with the code much. So you just need to make sure the code structure will remain healthy and that the AI themselves won't get themselves into a hard corner because they structured things in a non evolvable way, or where changing one thing can break many others.

The challenge I found now is almost more about how you remain in the loop. How do you still know how it works, what are the things it does/does not do, how it modeled things, etc.

yeah. if the apis/contracts are correct, changing the impl is immaterial. and then LLM code can be better and it doesn’t matter if its idiomatic or what a human would have written

my approach is with RDFS / memory model. i'm interested in others’

asking it for a callgraph diff has been really nice for me

check out clojure-xref for ai digestible call graphs 😎 https://github.com/danlentz/clj-xref

I'm struggling now to figure out what grain I need to work at. Like do I even care about namespace design? Or just app-level APIs ? Or should I also care about per-function interface design (like their input/output and side-effect contract). And so on.

10000% care about design of some kind of module or namespace. Give it pseudocode > I expect it to be something like (-> options compile transform diff update-stuff) push back if i’m missing anything staying in the loop is where it’s most valuable when you are designing contracts. For per-function i let my taste take over. If IO gets spread out a lot, or if there’s something that offends sensibilities. But often this is just fine, even if not exactly how i would have done it

@dpsutton yeah. architecture. it doesn't see that on its own

i just continually re-read code. it’s just like working with junior engineers though

Ya, that's what I used to do, but since Opus 5.5, you can tell it just an API and guidance on how to change data-models, and it does a pretty damn good job 😅

Heck, if you are doing a full-stack app, you can even let it figure out what APIs to add to the backend to support the frontend changes, and just describe the UX changes and the overall data model.

I just don't know yet if doing too much of that will hit a wall down the road or not.

it’s hard to get it right for sure. and the human is still quite necessary

@didibus what general domain of applications are you working on?

i guess, that's interesting to open up to the floor. in order to have context for discussion

i'm “in” commodities — as in the software i work on for work gets used by traders — not me. i have a couple of open source libraries but that's it

Personal projects are like libraries for Clojure, those I have to remain focused on the function-level design, since that's all a lib is really, good namespace/function/macro interfaces. Other personal projects, are desktop apps, websites, etc. Those I can vibe code pretty well, work at UX/scenario level, API-level, etc. But their scope remains small. At work, it's multi-team, multi-service, enterprise grade distributed systems, mix of backend, frontend, service to service, background jobs, queues, async processes, automation workflows, etc.

i hate the term vibe-coding

it's so unfair

Oh, I love it. Nothing pleases me more than dropping it in some leadership meeting

so, do you have any positive experiences sharing AI over multiple teams? my approach is my graph memory mcp server — but it's invented in-house. there's so little prior art

Ya, but it's likely veering more appropriate for ai-assisted-coding, I discussed some of what I'm trying out here: https://clojurians.slack.com/archives/C068E9L5M2Q/p1789103420212109

to me, vibe coding means i should be out riding my motorcycle all day and check in 8am next morning. not spending 14 hours laboriously building out an architecture

Had a convo about that too here: https://clojurians.slack.com/archives/C068E9L5M2Q/p1790197777487029 If AI makes you work more or less

i do much more than i used to. and it's not like anyone has asked me to. i'm just having a blast

in my opinion i've never had this freedom of creativity

i just wish there were a distinction between “vibe coding” and developing methods and systems to use “AI” in intentional ways. but i think that will work itself out of the vernacular eventually

there definitely is that distinction. I have several things where i don’t read the code. And then many where i use an LLM augmented, wrestle with the apis and contracts, do a few different POCs, read the code, offer suggestions and others. There is, and should be, a meaningful difference between these two

developing methods and systems to use “AI”

> But once those are decided the LLM can exceed human capability as far as i can tell

> But once those are decided the LLM can exceed human capability as far as i can tell I'll push back on that a bit. Abstractions still matter. I think a human using well defined understandable abstractions can still create a system that is more reliable, sustainable, cost effective, and fit for use than one mass generated from ambiguous human language. The apparent "slowness" of humans here is in part the time needed to understand what they are building - intrinsic - and part challenges of the environment - accidental. I would see us as an industry use agents to accelerate experiments with building better abstractions over the accidental complexity rather than paper over them with reams of more generated accidental complexity. The danger is we allow them to take over the intrinsic complexity of our domain because the line between the two is so blurry. I don't want to build things I don't understand. I use LLMs, but every time I use them, I'm painfully aware I'm giving up on better abstractions. There are better ways to work. Immediately reaching for LLMs means you absolutely are choosing a tradeoff. I'm thinking of Alan Kay's research group and little languages. Everyone pulling the LLM lever ensures we stay at a local maxima. This may not be true for all languages, but with lisp, I think we do have a chance to build the right abstractions. Weavejester's thoughts here particularly resonate. https://mastodon.social/@weavejester/117343057782006627

i think you are putting my point more eloquently than i did. Because the things the human decides are the abstractions. that’s what i’m saying the LLMs aren’t nearly as good at. And if you wrestle with that and make sure things are good before letting hte LLM rip you can have far better experience

Why do people say that programming is solved? Is the halting problem solved?

or maybe expressing the sentiment far better. I don’t want to say you are just repeating me. I think that’s a great summary of where coding with LLMs stands. Reaching for them too early can remove you from the loop for sure. And putting it in terms of local maxima is a very interesting point to ponder

abstractions still matter. yes. and i feel like that's the fun part

We probably are close ... ish. I may have misunderstood your position. Coding has always been about building abstractions. I can see how LLMs might help when the abstractions are sub-optimal. But if that isn't the case, we are better off without LLMs IMO.

no. we invent the abstractions. llms just do the typing

but i've been asked twice now not to participate in the channel so it's been fun and i apologize if i caused any offense. from my end i thought it's been worthwhile and i learned a lot about where you guys are coming from. so i appreciate the discussion

I have no idea where this "LLM can code better than us" came from, and at this point, I'm really afraid to ask. My experience with Opus 5.5, lots, *lots* of dumb mistakes. Like, A LOT - for example, I asked "port Promesa to Jank". Took five times, and I'm still not 100% sure that it works as I expect. It's fast, and it allowed me to iterate faster on some stuff, but the level of "wrong things" that it does is atrocious

Performance improvements with LLM are a complete joke. They suggest usually the wrong things, start to go haywire to rewrite parts of the app that have nothing to do with the hot spots, and for me at least, it never worked - I got 5% performance improvements at best and it's normal to get a worse experience; and to make things even worse, the LLM will write the worse code, benchmark it somehow, and say it's faster - but a simple profiler will show the opposite.

i have a very different experience. I’m currently having an LLM port sqlglot from python to java and having a nice experience

The "RFC approach", I tried. Lots of times. Never worked either for me: it usually says "I'll implement this and this and that" and it simply "forgets" to implement one or two things.

@dpsutton porting systems it's where I use a lot. Still get something wrong, but because I have the "oracle" as they say, I can double-check if something works as expected or not. But even then, still works 80% of the time - for example, on my Promesa port, the LLM wanted to write a thread pool "by hand" by synchronizing atom things. I pushed back and pointed to a library. The LLM added the library... and didn't use it. Then I pushed back again, it implemented the thread pool API... and left it there, none of the constructors like future used it facepalm

ha yeah. gotta stay in the loop. it can go off the rails. having guardrails is necessary. they are incredible, and incredibly frustrating, technology

There's no real incentive for engineers to come out against ai, whereas the upside of billing yourself as some 10x guru is self evident.

I think the main problem for me is how "randomly unreliable" it is. For example - I asked a LLM to write a VSCode API -> Atom (Pulsar) API translation layer. It........ works..... but there are still a lot of thing that's missing. The issue is: if a human wrote that layer, they would know exactly what it was implemented (and know some of the things that are stubs, "won't work", etc). The LLM? No idea. I asked to fully implement the webview API, and it did, it told me it did, then I tried another extension... and it failed. Without any exception, because LLMs are champions of "on error resume next"-based programming. Which bring me to the point: is that translation layer done? No. Is it close to be done? No. How much was implemented? Absolutely no idea - nobody knows. When I ask the LLM to implement the APIs for a plug-in to work, it usually works after some tries... but then, did it break something? Did it fixed more things? Did it implemented correctly? My experience is yes, no, no - in fact, I put on my harness to not change the autocomplete API ever, because the LLM (Claude, Codex, Kimi, all three that I tested) keep implementing it by guesswork instead of following up the documentation...

I'm glad this thread prompted some introspective discussion. I wasn't trying to find ways to dunk on LLMs or people who use them, more of what I wanted was to be strictly a skeptic and find counterexamples to the prevailing opinions/experiences of LLM usage.

In think @dpsutton sums up some of my experience as well: > it can go off the rails. having guardrails is necessary. they are incredible, and incredibly frustrating, technology. Having more ways for the LLMs to measure and validate leads to better results. I have been burned getting insane amount of slop when I’m stressed and starting to lean too heavily on it. One coworker told me: > Previously when you had shitty and messy code you could have humans pick up the slack by having understanding of that code. Now, using best practices becomes a requirement.

I think I'd be more inclined to use LLMs more often if it was simpler to squeeze them into clojure.spec or similar shaped boxes.

@mauricio.szabo I'd say "port Promesa to Jank" isn't really a coding task. That's a design task at first, you have to understand what is the essence of Promesa, what does it mean to port it to Jank, does the Promesa abstraction need adapting to jank? And then how do you go about doing the port? Validating? Etc. All that you have to still do, and steer the AI. But none of that involves coding, so once that's established, the coding I'd assume Opus 5.5 should be able to do. That said, Jank is niche, likely the model operates with limited knowledge, so you also need to give it a good rundown of Jank so it doesn't mess up I'd assume.

@didibus my prompt was about 20 lines long, explaining what Promesa was. I asked LLM to make the API fully compatible and use the same namespaces. I also gave it the C++ thread pool lib that it was supposed to replace the Fork/Join in Java. Did a plan, max effort, best model I had access. Reviewed the plan, asked for modifications, reviewed again. Asked to reuse the tests, just adapting things that didn't work. It was literally chewed up for the LLM, it only needed to implement the code. This... unfortunately also is part of my experience with LLM. It feels kind of frustrating that it's always "you are doing it wrong", never "the tool didn't work". I know it wasn't your intention, really, but I really feel that LLMs never take the blame.

@mauricio.szabo Oh, I agree, they need to take the blame sometimes. I just wanted to distinguish if it was a difference in our definition of "coding". So even though it had a working plan with details of what to do and how to do it, it couldn't achieve working code? With Opus 5.5 max ?

It achieved working code, but not the right way. It implemented the thread pool, but didn't use; all entrypoints that needed to dispatch a task for the pool were using unbounded threads from Jank's core API (not even the C++ lib I asked to use), and the few functions that did use a bounded thread pool, also didn't use the C++ lib - it used a weird atom-based sync that I explicitly rejected and asked Claude to not use.

Yes, Opus 5.5 Max

Cool, ya I can believe it. Even though I'm also a little surprised as I've been getting pretty good results with Opus 5.5 following the plan.

@mauricio.szabo I guess the next question is, do you still feel that even though you had to be more in the loop at verifying what it's doing, and steering it back in the right direction, do you still feel like it saved you a lot of time and effort?

In this case, yes. I think there's a "sweet spot" for LLMs - for example, I want to implement an API and I already have my own API, so it's just a "facade"; or I want to port library X to language Y, and both languages are very similar, things like that. It kind of bothers me that even on these cases, you still get this weird "oh, I said I was going to do that, I confirmed with you, but didn't do it" So I don't actually use for new code, nor for parallel code (my own experience, it ends up with horrible, horrible code that's slow as hell and full of problems) nor for things that I need a high level of trust.

Nor performance improvements or calculations - these are insanely bad in my experience too

The former.

this is also a framing that won’t allow for any useful discussion

my only interest is in useful discussion. it's apparent that i have a different perspective. but ive been approaching it to learn

i thought this would be a good place to discuss.

it is. i’m just saying “is this a safe space where only group think is allowed” kinda walks in brashly. Just try your question instead

i mean it’s fair. but i didn't introduce the concept of safe space or getting blocked for asking questions. i want a friendly discussion.

then go for it. i’m interested in your question

i appreciate it. i'm not looking to upset anyone. i'm just someone on the “other side” looking to understand the opposition arguments better. i love clojure.

sometimes it’s devolved into “well, americans just have hero worship” i don't think the channel maintains the standards of discourse that i do

The channel doesn't maintain anything, it's a passive entity. Admins and occasionally other members do, and not everyone has time and will to read through each and every message.

In other words, if you find that something doesn't adhere to our Code of Conduct, let an admin know. I'd use a DM for that.

i am passionately for free speech even if i don't agree. don't get me wrong. i'm just saying the community standards in the group seem different than normal conversations. i don't want speech regulated. i would just like to be allowed into the discussion

this just all feels like meta about a question

liek just ask a question

In a new thread though. :)

> i would just like to be allowed into the discussion What made you think you aren't allowed in the first place? It's not like anyone is banning anyone here for normal words.

haha. yeah. my question here was if i join the discussion politely am i going to upset people and have them block me. that's what made me feel bad

yes that’s exactly what happened

just now

People can't block you. They can simply ignore your thread and move on. > yes that’s exactly what happened > just now What do you mean? Can you elaborate?

https://clojurians.slack.com/archives/C0B7Q4XRM1U/p1791240584979099

Ah, right. Anyway, nobody has blocked you. A user can hide some other user so that they don't see their messages' contents and don't receive any notifications - that's slightly different. In any case, I gotta say that you did appear like a troll there by asking whether Stephen writes insecure APIs. No clue what's up with that. Maybe you meant something else, but the message doesn't feel ambiguous in any way.

lit was a rhetorical question— obviously no one says yes i write unsafe APIs — mcp or otherwise

I can't really add anything here except repeating my perception that the question did seem like utter trolling. :)

contribute to that thread then?

yeah. this isn't the right group to discuss ai for me i’m afraid. i did learn a lot of your concerns and i appreciate the friendships ive made.

You're judging the whole group based on a reaction of a single individual under his own post.

> contribute to that thread then? That would require reading that article and engaging with the content. :D Maybe I'll do it, dunno.

i want to stay. i felt bad i upset that guy

i have a lot of things i want to talk about

then discuss them

> i felt bad i upset that guy That's no different from real life though, so dunno why you contrasted the two. Real life is even worse in many regards. Better in some though - text is inherently lossy. Anyway, yeah, just make another thread with whatever you wanted to ask.

yes. the most boring room to be in is the one where everyone agrees with you.

#C068E9L5M2Q tends to be slightly less bitter (they're more at peace with the global status quo as it were). But they also have some rose tinted glasses about some things. Might be another place of interest for you

Ambiguously aggressive stuff will probably be treated similarly in either channel

ambiguously aggressive. ok. can you explain how i did that. i'm nearing my limit of walking on eggshells to have a discussion

I didn't watch the other thread

i’m just confused because this is a discussion about discussions. there is no real topic

ok. well, not my intention. and i definitely apologize no offense was intended. my question was about the insecurity of MCP servers. and can't you write stuff in a regular API that’s insecure. and that agent to agent comms is quite a niche thing to criticize

if live it if this group talked about technical things

because of the post about it

Much of ai discourse in general is as vibes-based as the ai coding itself 🫠

And no worries on the meta-discussion, we all have to learn each other's tolerances somehow or another (based on what I've been told of X and other public spaces I'm not in, this is much gentler)

You should stay, the friction is good

& i don't really have a horse in this race but it does feel a bit mean to subtly announce after a debate with someone that you've put them on ignore https://clojurians.slack.com/archives/C0B7Q4XRM1U/p1791212810834349?thread_ts=1791150229.259749&cid=C0B7Q4XRM1U

fwiw, @danlentz’s messages seem indistinguishable from trolling to me. Many of them are off-topic in a way that derail the main discussion. Some of the messages presume bad faith on the part of other people in the conversation. Some of the messages restate what other people have said in a way that (to me) intentionally misrepresents their message. I'm happy to engage with anyone here participating in good faith. I'd be happy to provide examples and explanations here or via DM.

i think we finally got the other discussion going. so hopefully we’ve corrected any rough edges

this is a rough crowd. once again i appreciate your patience and im only trying to learn your positions and ask questions and occasionally give my personal experience. we’re all learning here. and we all love clojure.

Yes, why not? Did anyone disrespect you?

After this Hollywood-level teaser, I really would love to see the actual question/topic! 😃