When the right answer is for the agent to do nothing

My Fujifilm body comes out of the camera with the photo I wanted. Most of the time. The film simulation, the white balance, the exposure compensation - those are decisions I made when I clicked the shutter. The sensor and color science do the rest. When I started building an AI agent to edit my photos automatically, I assumed the agent's job was to grade every frame. After enough iterations to know better, I rebuilt the system around a different premise: the camera's processing is the product, and the AI's job is to leave the good photos alone and lift the small number that need it.

That premise sounds obvious written down. It was not obvious when I started. The first version of the agent graded every photo I gave it, because grading was the response shape the conversation rewarded. Given a normally-exposed shot of my Yukon parked in the garage, the agent pulled saturation negative, lifted the shadows, and pushed exposure up half a stop. The output looked like a vintage Instagram filter sitting on top of a clean image. I had told the agent, in its system prompt, in a hard rules block in capital letters, never to pull saturation negative. The model had read the rule. The model had agreed with the rule. The model had violated the rule on the first photo it ever saw.

Any designer or product manager who has tried to keep an LLM inside a brand voice guideline, a copy deck, or a style guide has had this happen. The model knows the rule. The output ignores it. The fix you reach for first - making the prompt louder, the rules more emphatic, the caps block longer - is the same intervention with more confidence. It does not put a floor under the worst behavior. The fix that actually held was sixty lines of Python that ran after the agent and clamped its output regardless of what it had decided. I called the function _sanitize_params and it has become the shape I reach for whenever an LLM is doing creative-judgment work with strong domain priors. Prompts shape reasoning. A deterministic post-processing step enforces intent. Both belong, in that order, in any agent that is allowed to be wrong.

What the pipeline is trying to do

The agent sits inside a personal photo pipeline that takes a Fujifilm shoot from the camera to my NAS with as little manual editing as I can get away with. Most of my photos do not need editing. The Fuji film simulation already encodes a look I trust. The job of the AI agent in the editing stage is to do nothing on the photos that are fine, and to fix the small number that are not. The agent's bias toward action - the bias that produced the filter on the garage shot - was the design problem. Getting it to do nothing was harder than getting it to do something.

The bigger taste call came first

Before the sanitizer story is the JPEG-first story. I was originally rendering RAFs through rawpy and grading the resulting TIFFs. The output was technically defensible and visibly worse than the out-of-camera JPEG the camera had already produced. The reason is specific to Fujifilm: the X-Trans sensor and the film simulation engine encode a look in the camera's MakerNote that rawpy does not honor. The TIFFs were rendering as generic Bayer output with no film simulation applied. Every grade the AI agent produced was lipstick on a downgrade.

The fix was not better grading. The fix was abandoning RAW rendering for batch delivery. The pipeline now ships out-of-camera JPEGs directly. The same code that was supposed to grade every photo got repurposed as a triage classifier - it flags the small fraction of photos that have an actual exposure or white balance problem, and only those photos get graded. Everything else passes through untouched, because the camera already did the job.

This is the move I was slowest to make and the one that mattered most. The architectural mistake was assuming the AI belonged in the path of every photo. The correction was recognizing that the camera's processing is the product, the AI's job is to stay out of its way, and the system should only invoke the AI on the photos where the camera missed. The sanitizer pattern that follows only makes sense once that decision is made. The sanitizer is what keeps the AI inside the boundary the architecture defines.

v1 - grading by sensation

The first version of the agent graded every photo I gave it. It read the scene, matched a lighting scenario from a rulebook, and emitted parameter changes. None of those parameters were ever zero. If the agent was asked to grade a photo, it produced grading. The concept of this photo does not need grading was not in the prompt's vocabulary, even when the rule was written explicitly. Brian's verdict on the first batch was two words: muddy - stop, different route.

v2 - tightening the words

I rewrote the system prompt. I added a CORE PRINCIPLE block: the Fuji out-of-camera JPEG is the reference, never the starting point. I added five HARD RULES in caps. Saturation never negative. Contrast never negative. EV at zero unless the histogram has a real problem. Shadows lift as the primary positive lever. Sharpness off by default. I added render-level floors so even if the agent emitted negative numbers, the pixels could not follow them past a threshold.

Most of the photos came back close to the camera reference. A few still leaked. One came out with EV pushed half a stop and contrast pulled below zero, both in direct violation of rules the agent had just been told, in caps, never to violate. The lesson took longer to land than I want to admit. More emphatic English is not a different intervention. If the agent slipped on the quiet version of a rule, it would slip on the louder version too. The next move had to live somewhere other than the prompt.

v3 - the sanitizer makes intent code-enforceable

The sanitizer is sixty lines of Python in agentic_loop.py. It runs after the agent emits its parameters and before the renderer reads them. Sharpness, saturation, and contrast are forced to zero. Always. If the scene was tagged well_balanced_already, every parameter is zeroed. EV is gated on whether the scene brief explicitly flagged an exposure problem. Shadows lift is clamped. Highlight protect is clamped. The function is shorter than the system prompt it backstops.

Every clamp logs at INFO level. The raw agent output and the sanitized output are both visible in the pipeline log, side by side. The sanitizer is not a black box that hides the agent's mistakes from the operator. It is the operator's view of which decisions got overruled and why - a running ledger of where the prompt is leaking and which rules are doing the catching. The first time I read a calibration log I could see, in two columns, exactly how often the agent was still pulling for negative saturation even though I had told it not to four different ways.

The hard floor finally landed. The filter-look outputs stopped. The EV gate at v3 was still too permissive though, and one more iteration of design was needed before the system stabilized.

v4 - the gate gets the allowlist it needed

v4 narrowed the EV gate. The sanitizer would unlock non-zero EV only on two signals: an explicit underexposed / overexposed / clipped flag in the scene brief, or a match to a small allowlist of canonical EV-adjustment scenarios. The allowlist is four names. Backlit subject. High-contrast midday. Snow or beach bright. Dark subject filling the frame. Anything else, EV gets zeroed.

high_iso_low_light_handheld was deliberately not in the allowlist. The code comment explains why. High ISO alone does not mean a photo needs lifting. If the scene agent matched that scenario erroneously on a normally-exposed shot, the EV correction got zeroed and the photo passed through unedited. What I left off the list mattered as much as what I put on it. The bias-toward-action that produced the original filter look was bounded, not by what the agent was allowed to do, but by which scenarios were allowed to unlock the action.

What the pattern is

The shape that fell out: prompts shape reasoning, a deterministic post-processing step enforces intent, and both belong in any agent doing creative-judgment work with strong domain priors. Defense in depth, with the depth in the right order - the prompt teaches the agent what to consider; the sanitizer locks in what it is allowed to emit.

The sanitizer is what the series intro calls a harness. The agent runs inside it: every parameter it emits passes through the clamp before the renderer ever sees it. A negative saturation can be reasoned into; it cannot be emitted into the renderer. The clamp happens at the per-emit level, which is what makes the sanitizer a runtime harness specifically. The prompt can fail and the floor still holds.

The sanitizer is not the agent's safety net. The sanitizer is the agent's vocabulary. An agent whose emit-vocabulary is bounded can be more interesting in its reasoning, because the cost of being wrong is bounded by the floor. The agent can experiment with how to describe the scene because no description it produces will let a saturation: -0.5 reach the renderer.

This generalizes to anything an agent does under hard constraints. Write the prompt. Write the sanitizer. Run both on every emit. Treat the sanitizer's log of overrules as the truth about where the prompt is leaking. The override log is more honest than the prompt itself - the prompt is what I told the agent to do, the override log is what the agent actually tried.

What ships and what doesn't

The system does what I built it to do. The pipeline pulls photos off the card, culls the blurry ones, identifies the small fraction that need grading, and ships the rest untouched. The sanitizer is live in agentic_loop.py and runs on every photo the Grade Station grades. The agent has not produced a filter-look output on a normally-exposed photo since v4 calibration locked. The agent still tries for negative saturation about one shot in twenty. The sanitizer still zeroes it. The override log gets reviewed periodically and tells me where the prompt is still leaking.

The same instinct shows up in the other instruments I have been building. The UX audit had to survive its own audit. The grade agent had to lawyer its way around its own prompt. The memory layer has to push back on the operator who built it. Every instrument I build with strong domain priors needs a constraint that does not depend on the instrument deciding to honor it.

The deeper lesson is the one that came before the sanitizer. The first version of the system assumed the AI belonged in the path of every photo. The version that works assumes the opposite - the camera is the product, the AI sits on the side, and the architecture's job is to keep it there. The sanitizer is how the architecture holds. Building an instrument to bound an agent's judgment is not a feature you ship. It is a practice you grow into.