Clojurians Log v2

One counterargument is that your QA process is so potent that how code passes it doesn't matter. We didn't believe this before LLMs, but now it has become a fallacy of convenience.

Too many variables to track, tbh. Different coding tasks are not easily comparable. The model matters. The harness matters. The prompt, the session history, the skill files. It's all way too many variables to get anything systematically useful out of it.

I agree that there is a boundary to be fleshed out, but it is well north of "track nothing".

Personally I rely on LLM benchmarks to get a general sense of which model is better. It's not perfect but tracking which model was used for a PR seems like it would provide zero signal, at least for my workflows. Often it's not even just one model.

Personally I am imposing my own opinions into the process often enough that I wouldn't want an unattended software factory. The signal from recording the model used would also be swamped by the far larger signal of my design opinions and prompts.

it's a "your project, your rules" scenario. For example, Clojure would not pass many company standards with its Java source code because it's not formatted and probably lacks any static analysis (linting).

QA process

I don't agree it's free. I do enough hand editing of commit messages that it would add to my workload to ensure that model attribution is preserved. There's also a fundamental problem with indiscriminate data collection. If you look at data that is easy to collect but mostly irrelevant/noisy that data may lead you in a bad direction. Good data may be hard to come by and mostly invisible. Collecting data just because you can will flood the zone with irrelevant, noisy information. I won't stand in anyone's way if they want to gather that metadata but I'll wait to see someone produce useful results from it before I choose to do it myself.

There's also something I don't like about including a model attribution in the commit message. I feel it undermines the idea that I'm responsible (as in "on the hook for") for the content of the commit. I don't want to abdicate responsibility saying "oh Claude wrote that, not me".

Other ways of tracking model use could avoid that connotation though.

I don't think it is bad to track what model was used to generate some code, but I don’t think it is especially important. What matters is the end state result: what was produced. In this context, the generated code is the artifact, and the model used to produce it is a construct tool. For hand-written code, no one signs their commit message with the text editor they used to write it. I think this is similar.

I do think there is a categorical difference between using AI and using emacs but even so I don't think the model used captures nearly enough to be useful to track on its own.

Personally I would love to have a system that automatically records the full input/output of an llm and how it was used in the development process, if for no other reason than I'd like to see what my coworkers are doing and learn from them.

I'm most interested in the prompts actually.

One “bad” instruction can have a much bigger impact than the model itself. And vice versa. At the same time, when implementing a single feature, I typically use multiple models. One is better at reasoning, another writes better code, a third does better reviews and abstractions, and so on, so it would be really difficult to break that down within a PR. We definitely need the ability to report the entire stack (at least to give benchmarks and various examples more credibility), but so far I don't see any solution other than sharing the whole log, which in turn poses different issues…

I would like to do more pair programming with teammates where we both get to see and discuss the approaches we're using and the results we're getting. Human inference from directly experienced anecdotes is probably better at figuring out good patterns than trying to do some systematic data analysis.

@U0BGC50RM26 That is one aspect; another is ego. Certain prompts can be embarrassing, exposing a lack of knowledge or ignorance about resource consumption which can lead to judgment on a completely new level in SWE, etc.

I’ve not seen the video, I agree with the sentiment

Stuart takes a much broader view to discuss this topic. Its quite good.

that very much resonates with my experience

Interesting piece. I have a slightly more granular take on the camps software developers are divided into: https://politechs.dev/episodes/ai-06-programmers/

I didn't do any research on the different groups so I didn't want to assume too much. I fully agree with you @jmglov that it is a lot more granular and nuanced. The two mentioned are probably just a bit louder about their believes.

@U0BGC50RM26 I agree, rhetoric isn't helping anyone. I can understand frustration, but projecting that frustration onto others usually tends to lead to situations where nuance is lost and discussion becomes difficult. The threat that AI can pose on a career is real and should not be dismissed.

There are reports coming out that AI isn't delivering the efficiency that CEOs seem to think it will. It seems quite clear to me that it is just used as another excuse to cut costs everywhere.

Well, AI promises to deliver efficiency gains. I’d say that because they are good salespeople, IT industry need to conform to the expectation by applying existing solutions to improve efficiency and say “AI helped us, we can help you do the same “.

There are known solutions. Solutions that are not silver bullets, but do add up when you apply them.

I thought this was a nice explanation from a developer (instead of a consultancy company that explains impact on the market): https://oioannou.com/blog/ai-promised-less-work/

@U0BGC50RM26 That's a valid call-out, though I think the https://politechs.dev/episodes/ai-06-programmers/#timestamp-203 from which you extracted it is actually important: > [03:23] There's some people who—and maybe this is a general point, but let's see—there are some people we know in our own community that use these tools very enthusiastically, and they don't give a fuck. They have no cares in the world about what's the externalities of these tools. They assume—either they're just monsters and they don't care about all of this stuff, which is one option, I guess. But without judgement, I would say that they're more enthusiastic and more excited—like children I would say to some extent—about the shiny new thing then they are about all these externalities. I mean, and again, let's just be clear: these people know there are problems. They know there are issues, but they don't give a fuck. The point being made here is that we don't think the people in group 3a are monsters. I agree with the point you're making that calling people monsters is dehumanising, a vital step in removing the normal human reluctance to do violence unto other humans—a page ripped right out of the fascist playbook—and a better phrasing here would have been "either they're just acting monstrously...". Have you listened to the whole episode or skimmed the transcript past the first five minutes? If not, I think you're not actually engaging with the substance of argument we're making, but rather finding local faults with the words we use to make the argument. As I said, your point is valid and I appreciate people pointing to where we could improve our phrasing to be more consistent with our values. Those values are stated in our https://politechs.dev/rules.html, from which I'll draw one relevant excerpt: > 2. Humans are made and must be treated equally > No human can be reduced to a category such as "terrorist", "murderer", "immigrant", etc. We can debate their actions and disagree on whether what they are doing is right or wrong, good or evil, but we must insist on their humanity.

@alexander.vanelsas I disagree with this point: > rhetoric isn't helping anyone. Resistance to fascism has to be total, in the arena of rhetoric as well as action. I believe that the project of Artificial Intelligence is inherently fascist, which I think is worth saying. > I can understand frustration, but projecting that frustration onto others usually tends to lead to situations where nuance is lost and discussion becomes difficult Speaking for myself, I'm not acting out of frustration (most of the time), I'm acting out of hatred. Hatred for what this socialtechnical project to doing to humanity and the ecosystems that sustain human and other-than-human life itself. In Politechs, we are attempting to engage in good faith with the contradictions and pull out and address the nuances. People can decide for themselves if we do that successfully. Anyone is welcome to explore our long-form argument against "AI" and see the evolution of our thinking, which is why we put in on the public record to begin with.

@jmglov It isn't up to me to dictate how anyone should act or feel. You feel this way and that is your choice. I would point out that there is a very fine line between hating the tech and its consequences and hating the people that (often have to) use the tech. Its all fine if you can afford to chose, but the majority of developers cannot live without a salary, without a job so refusing has serious consequences.

Personally I think it is important to realise that even though you feel you are right about something, it will be impossible to convert everyone to your position. Using strong language will certainly make clear what your position is, but it rarely helps others to move to your side. You only have to look at the completely messed up politics in the USA to see what extremism does to 2 sides of the argument. Not suggesting anything about either side, just that listening to the other is at least as important as being able to articulate your own views. Perhaps that could be a legitimate goal (not discussing), but in general it feels a bit counterproductive.

I find it's difficult to convince people of things that conflict with their existing world view. The ideas we have are tend to tie into many different concepts that together constitute our world view. So, changing any single idea means having to rebuild a whole conceptual graph of everything related to the idea. And unless we have a good reason to do that, it's just easier to discard the new idea that doesn't fit. Hence you really have to understand the general world view of a person you're having a discussion with, and present the ideas in a way that creates the least friction with it.

I've had several good friends who had strong views about what happened in the world during Covid. That too was a moment where it seemed there were only two options (for or against in general terms). I also decided that trying to convince each other of one side or the other was a useless exercise. Instead, listening to each other was a lot better and at the very least ensured that friendships remained (which is important imo).

@U0BGC50RM26 thanks for the link, this is such a saturating and refreshing read

Just skimmed through and immediately found two projects allowing AI to identify bugs and issues, but blocking AI code over license concerns. The statement in the image goes too far…

Oh fascinating. Can you point me to one of the projects? I can message the author

Sure: https://schismtracker.org/ https://fna-xna.github.io/

@U0BHKM127SQ I’m not bothered about those projects. It’s just that the description on http://noAiList.org says that these projects decided not to use AI to move forward. And finding bugs with the help of AI is the complete opposite, right?

Thanks I asked the author.

I'm slightly embarrassed lol

The owner of the list replied and asked me to let them know which projects. Good news!