When the AI Has to Film Its Own Review

When a pull request is raised, mine or a colleague's that I have been asked to review, these are the lines I paste into the agent:

Now let's review this one: [add the PR's GitHub link]

But I want you to run it locally E2E and see if all is going well, and do Playwright recordings of it. I want you to also attack it, try to break it AND thoroughly test it [add more context here if needed, like in both personal and org scopes].

Then I let it work: it runs the change on my machine, end to end, with Playwright recording the browser as it goes. Playwright is a browser automation library the agent can drive, and it can film the pages it opens. What comes back is a written report, with the recordings beside it and screenshots of the moments it points to.

A lot has changed since I wrote about reviewing AI code by sight in July, including how I review pull requests. This way of reviewing has impressed me lately, and I think most developers would be surprised by how good it has become.

What I said in July, and what changed

My point then was that when the agent runs the code for you, the review does not go away. It moves to the evidence, and the remedy I gave was to open the logs yourself and find what the agent said was there.

That remedy still works, but most of the time I no longer need it. July also said the slow kind of review was to run it, trace it and try to break it, and to make the agent show its work. What is new is a way to ask for all of that in one message and get the work back in a form I can check without redoing it. The agent does the run, and the recordings show me what happened on the screen.

The models lean toward the fastest answer. Asked whether a change works, they would rather say it does than go and find out, and at times I have seen one describe logs that did not exist. A reply that sounds finished is the shortest path to the end of the turn.

This is not only my experience. The system card Anthropic published for Claude Opus 4.8 lists failures from the company's own internal use, and one of them sits almost exactly in my setting. The model said it was babysitting a researcher's pull requests when it was not. The same list records it claiming a verification it had not done.

Telling a model to be more careful is a request, and a request is what it skips when it is in a hurry. A recording changes what the model has to hand back.

Why a recording is harder to fake

A sentence about what happened can be written without anything happening. A recording needs something to have actually run, and whatever ran is on the screen for me to judge.

When the agent has to film itself running the change, the cheapest way to satisfy the request is to do the run. For anything on camera, there is no longer a slot where a confident paragraph can stand in for the work. On its own, a report is the only account of what happened, and I cannot tell a test from a sentence about a test except by redoing it myself. The recording pulls the sentence and the event apart. If the agent says it tested the empty form, the empty form is in the video or it is not. Whatever the report says that never reached the camera is back to being a claim.

The film shows what it tried, and what it leaves out marks what I have no evidence for: the screen it never opened, the form it never submitted with bad input, the second account it never logged into.

There is a second effect, and it matters as much. On camera, pretending a problem did not happen, or making one up, stops being a safe shortcut for the agent, because the video is there to be checked. So when it finds one, it brings it to my attention instead of burying it in the summary.

I rarely need to watch a whole video for any of this to work. Most of the effect happens before I open anything, because asking for the film is what makes the agent do the run. What I do look at are the screenshots of the moments that matter, and that includes the successes. I like to see the flow finish the way it should, in each scope I asked about, as much as the error on the screen or a page that opened for an account that should never have reached it. The video sits behind them for when a screenshot raises a question.

The report becomes an index to the evidence, which is a better job for a summary than standing in for the evidence. I still ask "is this true", but now a screenshot answers it, or at worst a short stretch of video, and more of my time goes to "does this matter".

When it matters what the server answered, Playwright's trace is worth asking for alongside the video. The video is the pixels. A trace keeps, for every action, a snapshot of the page before and after, with the network requests, their response bodies and the browser console. If the agent says the server answered with a 403, the trace has the response.

Microsoft's Playwright MCP, one of the tools that lets an agent drive a browser, works through the page's accessibility tree, a structured text view of it, rather than through pixels. The film is for me, and that is why it is useful: the agent acts on a text view of the page, and the film shows me the pixels it was not looking at, with none of its narration on top. It can still choose which take to hand me, or keep running a flow until it comes out green, so every recording it made, failed takes included, is worth asking for.

Attack it, and test it

The prompt asks for two things that pull in different directions, on purpose.

A model asked to confirm tends to confirm. "See if all is going well" invites it to look for things going well, so it walks the path the feature was built for, sees the expected screen and reports success. That run is worth having on film, because a pass with no evidence behind it is exactly the claim July warned about. But it only covers the case the author already had in mind.

Ask it to break the thing and it has to go looking somewhere else: a required field left empty, the submit button clicked twice, the same page open in two tabs, the ID in the address bar swapped for one that belongs to someone else, the back button pressed halfway through a flow. Bugs tend to hide in places like those.

An agent told only to break things has the opposite problem. It can spend the whole session on one clever edge case and never confirm that the ordinary flow, the one every user hits, still works end to end. I want the tour of the ordinary flow and the assault on it in the same run, which is why the AND is in capitals.

The two instructions also work better on film than in a written report. A sentence like "I tried to break it and could not" tells me nothing about how hard it tried, and the recording of the attempts tells me a lot. I can see whether it pushed on the form once and moved on, or came back with a different account or a different order of steps. The attacks that failed are part of the evidence that the change holds.

The line only I can write

The last part of the prompt is a placeholder that changes with every pull request. It is where I tell the agent where the risk is.

My own example is "both personal and org scopes". When a feature has to behave in a personal account and inside an organization, most of what can go wrong sits in the seam between the two: something that belongs to one scope showing up in the other, or a page that works for the owner and fails for a member. The agent can read the diff and guess at that. It does not know which paths the users actually walk, or which mistake would be embarrassing and which serious. I do, because I know the product and what I want it to do.

That bracket is my part of the review. It moves the agent off what the diff suggests and onto what I know can be damaged, and the recording shows me whether it actually went there.

What the film does not show

The film has edges.

The agent chooses what to film: the paths it walks and, depending on the tooling, when recording starts and stops. If the org scope I asked for is not on the tape, I have no evidence it was tested, whatever the report says.

Everything Playwright records happens in the browser. Server logs, the rows in the database and the background job that runs a minute later are in neither the video nor the trace. A screen can go green on top of a row written to the wrong account. An authorization check can be missing on the server while the interface hides the button perfectly on camera, and the API behind it would still accept the request from anyone who sent it directly.

This is where the attack half of the prompt pays for itself. A film of the happy path never touches that check, while a film of the agent opening a record that belongs to another account, or sending the request with the button hidden, does. For anything touching access, that attack is worth naming in the context line. When the change reaches the database or the server, the queries and the logs from the same run belong next to the recording, saved where the report can point to them.

A recording can also look right for the wrong reason. If the agent stubbed the backend to get the page to load, the recording is a recording of the stub. The prompt asks for local and end to end, and what the app was actually talking to matters before anything it showed. The network requests in the trace answer that.

One practical note: a Playwright video is only written to disk once the browser context closes. If the agent says it recorded the run and the file is missing or empty, that is evidence too.

Inside those edges, the film is the slow kind of review from July, with the agent doing the running and the judgment left to me.

Still my call

The models have improved a lot since I wrote the July piece, and my evidence for that is my own reviews. What I notice is that when the agent has to run the thing and film it, and something breaks, it says so, and I can see it for myself. I am not taking its word for the improvement either.

Developers who have only met AI review as a bot leaving comments on a diff have seen the smallest version of it. This is closer to a tester at my machine who would cut corners if I let them, using the change as a user would, trying to break it, and leaving the tape so I can see which corners were cut.

Reading the code still matters, and the recording covers what reading cannot reach, the behavior of the thing once it runs. It is not free. A run takes longer than a read, the app needs data worth testing against, and someone still has to look at what comes back.

None of that makes the agent the reviewer. It does the legwork. I am still the one at the end deciding what is right and what I want. When the recording shows a flow that works but is not what the product should do, the agent cannot know that, and when it flags intended behavior as a bug, I am the one who says so.

July ended on the line between the diff you have understood and the diff you have only seen. The recordings make the understood side of that line cheaper to reach. I do not need to sit through every run, only to see the moments that matter and to know which paths the tape covers. What changed is that I now make the machine film its work, so less of my attention goes to whether it did what it said and more to whether it looked in the right places. I can only judge that because I know what the product is for, and when it looks in the wrong places, I send one more line of context and have it run again.