Human in the Loop

It May Just Be the Human IS the Loop

• By Howard M. Cohen
• 08/21/2026

The popular phrase we use for the title of this column refers to the cycle through which an AI system performs a task, produces an output, receives feedback or correction, and continues. The human is inserted into that cycle at a point where human judgment is needed. The phrase implies that the "loop" is the province of the large language model and the software harness that directs and manages it, with the human needed mainly to confirm the results.

It’s becoming increasingly obvious that the entire loop is really about the human, what we need from the technology, and what we must do to keep the LLM on track and honest. We've been conceptualizing the AI as the center of the loop when perhaps the human should have been at the center all along.

Engineering, Engineering, and More Engineering
In the 18 months since Andrej Karpathy first coined the term “vibe coding,” we've been through prompt engineering, context engineering, harness engineering, agentic engineering, and now everybody's talking about loop engineering with graph engineering on the horizon.

Loop engineering, the most recently fully defined activity, means you give the model a goal, let it iterate, walk away, then come back to a finished result.

When Boris Cherny and Addy Osmani talk about “loop engineering,” they're not talking about a loop the human performs. Humans design the loop, but the AI runs it. Cherny has said that instead of continually prompting Claude himself, he creates loops that prompt the model and determine what it should do next. The AI then works through that cycle, performing a task, checking the result, and trying again until it reaches the goal. The idea is to let the AI do the circling while the human focuses on defining the goal and judging the outcome.

That's exactly the problem, because you're going to find yourself stuck in a loop anyway. Not a designed one, an accidental one. You and an AI, going back and forth, back and forth, about a process that isn't converging on anything, and the AI sounding completely sure of itself the entire time.

Don't Misread Your Model's Confidence
Here’s a telling response from a recent conversation I had with Claude: “You're right, and I got that wrong — it's currently morning on the 18th, not evening. Today's 12:05 PM run hasn't happened yet, and now that its .bat is fixed too, it should run clean this afternoon and cache the morning's readings. Then tonight's 11:58 PM run will find that cache, merge it with its own trailing fetch, and overwrite today's row with a full-day summary instead of the partial one sitting there now — no extra action needed from you, it'll just correct itself tonight. Apologies for the mix-up.”

My fondest wish is that this kind of response is unusual or infrequent. On the contrary, I find myself spending copious amounts of time discussing dev work with my favorite AI, correcting it or recommending alternatives it hadn’t considered. So much time that I find myself resenting it when it tells me I’ve used 90% of my credits. No consideration for my correction of its errors.

At the end of the day, however, I must admit a certain amount of satisfaction knowing that the AI is not some towering intellect vastly superior to me. Actually, it often needs me to guide it to the right solutions. I really have no fear of being replaced anytime soon.

Confidence As Strategy
As confident as your AI assistant may sound, that confidence has little to do with whether it’s right. LLMs are designed to produce plausible, authoritative-sounding responses, even when the information they have is incomplete or uncertain. And when they don’t have enough information to answer accurately, they may fill in the gaps rather than simply say, “I don’t know.” As I've said, they make stuff up, something most of us have experienced by now. What is a little shocking is that this behavior isn’t an occasional glitch. It’s a consequence of how these systems generate answers.

Carnegie Mellon researchers tested the phenomenon last year and found that chatbots stayed just as confident when they were wrong as when they were right. Not slightly less sure, not hedging, just as certain either way. Separate research on calibration backs this up. In return, users consistently overestimate how accurate these answers are, especially when the model states something with authority and doesn't hedge. Feeling gullible yet?

There's actually a term for the specific thing that happens next. Researchers call it "choice-supportive bias." The model inflates its confidence in whatever answer it lands on first, and it keeps defending that answer at rates well past what good decision-making would justify, often even when you hand it evidence that it's wrong.

That's not a bug you can patch. That's just how the tone works. And how it works on you.

Pattern Matching Isn't Judgment
So, what actually makes you think your model knows what it's talking about? What makes you think it knows exactly what to do?

It's in the dark as much as you are. Perhaps even more!

It's not reasoning its way to the best possible process; it's reaching for the process that resembles solutions it's seen before. That resemblance produces something that looks like good judgment, because it echoes judgment you've encountered elsewhere. But looking like good judgment and being good judgment are not the same thing, and the model has no way to tell you which one you're getting.

A Half Hour in Notion
I ran into this myself a couple of weeks ago, and it's a small example, but it made the whole pattern visible.

I was building a task-management workflow in the digital workspace application Notion. I wanted each call task to include a button that would create a new record in a separate, linked Interactions database, where I could take notes during the call. Keeping those records in their own database would also give me a running history of my interactions with each contact, rather than forcing me to scroll through a long list of notes on a single page. When I encountered a call task in the workflow, I made the call and wanted to create just such a record.

Every task is part of a project, so I wanted that interaction to be related to and linked to not just the task it came from, but also to the project it belonged to.

Claude and I went around and around for quite some time trying to get the interface to do that, but couldn’t. An hour in, still stuck, still confident, still circling. So, I stopped conversing with Claude and turned instead to Notion's own built-in AI and explained the problem.

Much more immediately, the Notion AI hit the same wall Claude did.

The next thing that happened was instructive and truly inspired greater confidence from me in the ability of LLMs, some more than others. Instead of continuing to push on the same broken approach, Notion came back with a completely different solution: relate the interaction to the task, then let a rollup incorporated in the actual project pull that task in automatically. The desired result Claude couldn’t achieve, just taking an entirely different route to it. When I brought that solution back to Claude, it agreed immediately that it was the better answer.

Two AIs: essentially the same underlying model family, the same problem, and wildly different quality of thinking depending on which one I happened to be talking to. That’s not consistency. That's a room full of helpers, some sharp, some not, and it shifts from problem to problem which is which. And the only one who could break the cycle and seek a viable alternative was me.

Know When You're Stuck
I'm now getting better at spotting this in the moment, though I couldn't have told you exactly what tipped me off until I sat down to write this. Perhaps it was instinct, a capability unknown to any LLM. Perhaps it was insight gained from experience, another mystery to AI. Perhaps it was experience with previous problems that tickled my memory and inspired me to act. I’m not sure which it was, but I do know I want to get better at identifying when I’m stuck in a psychic spiral with a synthetic being.

A few signals I’ve found worth watching for:

• You find yourself rephrasing the same request three or four different ways, hoping a different wording produces a different answer.
• The proposed fix keeps getting more complicated and more circuitous instead of simpler and more elegant.
• You catch yourself debugging the AI's approach instead of questioning whether the approach is the right one at all.
• You find yourself thinking more and more about what else you could be spending all this time on.

When any of those show up, that's your cue. You're not in a designed loop anymore. You're in an accidental one, and nobody's driving forward. Just around and around.

The Loop Was Never Meant to Be You
Loop engineering is a genuinely productive and valuable idea, when someone uses it by design. The model iterates, checks its own output, converges, and you get a finished result without babysitting every step. Your time remains yours.

But if you're the one stuck going around and around in an AI conversation, rephrasing the same request, watching the fix get more tangled instead of cleaner, that's not loop engineering. That's just getting lost in a loop. And that loop is you. The human is the loop.

The moment you notice you're in one, don’t grind even harder. Instead, step outside it, trust yourself and what you already know, and if the AI in front of you isn't getting you anywhere, go ask a different one. Or tell the one you’re working with how you want it done.