Prompt to Loop: How Fast AI Engineering Grew Up

In roughly three years, the hard part moved from wording prompts to designing context, tools, harnesses and feedback loops.

For a short stretch, AI engineering looked embarrassingly simple.

Write a better prompt, add a few examples, tweak the system message, then ship the chatbot.

That phase did happen. It just did not last.

What changed so fast is not only model quality. The shape of the work changed. In 2023, a lot of teams still believed the main problem was phrasing. By 2026, the harder problems live elsewhere: what context the model sees, which tools it can call, how state survives across turns, how you evaluate behavior that changes from run to run, and what loop you use when the system gets something wrong in production.

That is the real story of AI’s speed. The center of gravity moved from prompt -> context -> tools and workflows -> agents -> harness -> loops in about three years.

If you still think the job is “prompt engineering,” you are solving the 2023 problem with 2026 expectations.

The Prompt Era Was Real, and It Was Small

The prompt era deserves more respect than people give it now. It taught teams two things quickly:

• LLM behavior is steerable.
• Tiny wording changes can produce very different outputs.

That mattered because it was the first time many application engineers could influence behavior without retraining a model. OpenAI’s 2023 function-calling release made this moment even more visible: suddenly the conversation was not only “what text should the model write?” but also “what shape should the model return so software can act on it?”

Still, the core mental model was narrow. You had one request, one context window, and one chance to phrase the job well enough that the model would behave.

That worked for:

• summarization
• rewriting
• extraction
• single-turn classification

It broke down as soon as the model needed fresh information, external actions, or memory of what happened before.

The reason is simple: a prompt can steer behavior, but it cannot magically create missing state.

Context Is Where the Real Engineering Started

Once teams moved beyond toy use cases, the next problem showed up immediately: the model usually did not fail because the wording was bad. It failed because the model did not have the right information at the right time.

That is why context engineering became the more useful term.

Anthropic describes this shift directly: after a few years of prompt engineering getting most of the attention, the work became less about finding the right words and more about deciding what configuration of context is most likely to produce the behavior you want. That is a much better description of modern AI systems.

This is also where the speed of the field becomes obvious. Context windows grew from something you treated like a scarce chat buffer into something large enough to hold books, repositories, transcripts, tool definitions, and long-running work state. Google now documents Gemini long-context use cases at 1M tokens and above. Anthropic documents 200K-token and 1M-token Claude contexts, plus explicit context-awareness features for long-running sessions.

The raw window size matters. But the more important change is what engineers learned to do with it:

• separate static instructions from dynamic input
• retrieve only the evidence that matters
• summarize stale turns before they pollute the next decision
• keep the model aware of what it still knows, what it forgot, and what it should re-read

That is a different craft from prompt tweaking.

Prompting tries to persuade the model.

Context engineering curates the world the model sees.

That second job is much closer to systems design.

Bigger Context Did Not Remove Structure. It Made Structure More Valuable

The naive interpretation of long context was predictable: if models can read more, just stuff more into the request.

That turned out to be lazy engineering in nicer clothes.

Long context increased capacity. It did not remove the need to choose. In practice, more context introduced three new concerns:

• cost: long prompts are expensive unless you organize them well
• latency: large prefixes slow down every turn if you keep recomputing them
• attention quality: a model can technically see a million tokens and still focus on the wrong ones

That is why prompt caching became part of the stack. OpenAI now documents automatic prompt caching that can reduce latency and input-token cost sharply when requests share stable prefixes. Anthropic documents automatic and explicit cache breakpoints so multi-turn conversations can reuse repeated context instead of paying to reprocess it every time.

This is an important inflection point. Once teams started caring about cacheable prefixes, context blocks, and prefix stability, AI engineering stopped looking like copywriting and started looking like performance engineering.

You can tell a field matured when people stop arguing about wording and start arguing about cache boundaries.

Tools Changed the Job Again

Context solved one class of failure: “the model answered without enough information.”

Tools solved the next one: “the model answered without doing the work.”

Function calling was the early bridge. It let models return structured intent that an application could turn into real actions. From there the stack widened quickly. OpenAI’s current tools guide now treats web search, file search, function calling, tool search, and remote MCP servers as normal building blocks for agentic systems. The Responses API goes further and frames one request as an agentic loop where the model can call multiple tools before returning a final answer.

That is a large shift in what an application means.

Before tools:

• the model mostly transformed text

After tools:

• the model can inspect the web
• retrieve from files
• call internal functions
• route into third-party systems
• decide what action sequence is needed before it answers

This is where workflows appeared.

Not every system needed a full agent. Many teams got more value from deterministic workflows around a model than from autonomous orchestration inside the model. Anthropic’s guidance on building effective agents makes this distinction clearly: the strongest systems are often simple, composable patterns rather than oversized frameworks pretending to think for themselves.

That distinction matters because it saves teams from fake sophistication.

A tool-using workflow is already a major step beyond a one-shot prompt. You do not need to call it an agent just because the market likes the word.

The ReAct Pattern Made the Loop Legible

If there is one conceptual bridge between old prompt systems and modern agents, it is the ReAct idea.

The ReAct paper described a simple but powerful pattern: interleave reasoning traces with actions so a model can think, act, observe, then decide what to do next. That sounds obvious now because the field absorbed it so quickly. At the time, it was a major shift. It turned the model from a one-turn predictor into a participant in a loop.

That loop is the ancestor of a lot of what teams now call agent behavior:

1. inspect the current state
2. decide the next action
3. call a tool or request more information
4. observe the result
5. continue until a stop condition is met

The key point is not the paper’s exact formatting. The key point is that modern AI systems are rarely one inference anymore. They are a controlled sequence of inferences plus environment updates.

Once that clicked, “agent” stopped meaning “fancy chatbot” and started meaning “a system that can plan, call tools, keep enough state, and continue across multiple turns.”

That is also how OpenAI’s current Agents SDK defines it: applications that plan, call tools, collaborate across specialists, and keep enough state to complete multi-step work.

That is a very different object from a prompt template.

Agents Were the Visible Hype Layer, but Orchestration Was the Real Change

People noticed agents because they are easy to demo.

Give the model a browser, a code tool, a terminal, or a document store and the result looks dramatic. But the dramatic part is not the most important part. The important part is orchestration.

The practical questions became:

• when should the model call a tool instead of answering directly?
• who owns approvals?
• how much state lives inside the model context versus the application?
• when do you hand off to a specialist model, sub-agent, or deterministic function?
• how do you resume work after interruption?

This is why the best agent stacks do not begin with autonomy. They begin with boundaries.

Anthropic’s effective-agent guidance and OpenAI’s Responses-vs-Agents split both point in the same direction:

• use one model call plus tools when that is enough
• add orchestration only when the task actually needs planning, state, approvals, or delegation

That is a much more sober engineering posture than the public “just give it more autonomy” narrative.

The field evolved fast, yes. It also got less romantic once real systems hit production.

Harnesses Are What Separate Demos From Products

This is the stage many teams still underestimate.

Once a system can use tools and run across multiple turns, you are no longer evaluating a prompt. You are evaluating a process. That means your test surface explodes:

• final answer quality
• tool choice quality
• tool parameter quality
• cost
• latency
• number of turns
• failure recovery
• behavior under partial context

At that point, an article about “prompt engineering” is barely talking about the real job.

You need a harness.

By harness, I mean the surrounding machinery that lets you run scenarios repeatedly, inspect transcripts, verify outcomes, grade traces, and stop regressions before they hit users. OpenAI’s evaluation guide is blunt on why this exists: generative AI is variable, traditional software testing is insufficient, and evals are the mechanism for measuring performance in spite of that variability. Anthropic’s eval guidance for agents pushes the same idea further, recommending combinations of code-based, model-based, and human graders because no single grading method sees the whole system.

This is the part most executives do not see in demos.

The visible feature is the agent.

The expensive feature is the harness around the agent.

That is not overhead. That is the product.

The Save-Worthy Section: Where Your Team Actually Is on the Curve

Most teams are not “doing AI” in a single way. They are sitting at different layers of this stack and calling all of it one thing.

Use this decision framework instead.

You are still in the prompt phase when:

1. one call usually solves the task
2. the input is self-contained
3. no external action is required

Your main risk:- wasting time on wording while the real problem is missing information

You are in the context phase when:

1. outputs depend on large private data
2. answer quality changes with retrieval quality
3. conversation history and token budget affect behavior

Your main risk:- stuffing too much into the window and calling it architecture

You are in the tools and workflows phase when:

1. the model must fetch, verify, calculate, or execute
2. some steps are deterministic and should stay that way
3. one-turn responses are no longer enough

Your main risk:- handing too much control to the model too early

You are in the agent phase when:

1. the system must plan over multiple steps
2. state has to persist across turns
3. approvals, handoffs, or specialist roles matter

Your main risk:- calling a workflow an agent and then paying agent complexity anyway

You are in the harness and loop phase when:

1. failures are costly
2. you need reproducible evaluation
3. production errors must become future regression tests

Your main risk:- shipping a clever system you cannot measure, debug, or trust

That is the curve. Most teams are not late because they lack a better prompt. They are late because they are solving the wrong layer.

Loops Are the Current Endgame

The latest shift is subtle, and it matters more than the agent headline.

Strong teams do not just run agents. They run loops around agents.

A production loop usually looks like this:

1. capture a real task
2. run the workflow or agent
3. grade the outcome and the trace
4. inspect failures
5. change context, tool design, routing, or prompt structure
6. rerun the eval set
7. promote only when the system improves on the failures that mattered

That is why Anthropic’s tools article talks about building and testing tool prototypes, creating comprehensive evaluations, then repeating the cycle of evaluation and improvement. The loop is the product discipline. The model is only one moving part inside it.

This is also why the most interesting AI engineering work now feels familiar to good backend teams:

• instrumentation
• replay
• regression control
• approval gates
• versioned prompts and tools
• observable traces
• failure budgets

In other words, the field did not escape software engineering. It circled back into it.

What Changed Fastest Was the Bottleneck

That is the real answer to “how fast did AI evolve?”

The models improved, yes. But the bigger story is that the bottleneck moved almost every year.

In 2023, the bottleneck looked like prompting.

Then it looked like context.

Then tool design.

Then orchestration.

Then evals.

Now it looks like loops: the system around the system, the mechanism that turns failure into the next better run.

That is why so many engineers feel like the field keeps moving under their feet. It does, but not randomly. The stack is getting more complete.

The old question was: “What prompt should I write?”

The better 2026 question is: “What loop will keep this system honest after the first demo works?”

That is a much harder question. It is also the one that matters now.