How to measure a coding-agent pilot without relying on activity counts

Task sample: cosmetic, test-repair, cross-file, and permission-sensitive changes

A pilot needs tasks that resemble the work a team would actually delegate. A cosmetic edit, a failing test repair, a small feature across several files, and a change that touches deployment or permissions reveal different operating risks. Each task should begin with a short specification, a clean starting commit, and an acceptance rule that a reviewer can apply without guessing what the author intended.

The repository also needs the commands a new engineer would use: install, build, test, lint, type-check, and start the application. GitHub's repository-instruction documentation treats these commands and repository-specific preferences as context that an agent needs before it can work consistently. A pilot should test whether that context is complete instead of assuming that the agent will infer it.

Receipt fields: runtime, tool use, blocked actions, checks, review, and repairs

A merged patch hides most of the cost. The useful record includes elapsed agent time, model and tool usage, commands attempted, blocked actions, test results, reviewer minutes, requested repairs, and the final disposition. Review time matters because a patch that takes ten minutes to generate and ninety minutes to verify may be a poor candidate for use under the repository controls even when the code is eventually accepted.

Reproduction is the strongest check. Another person or a clean runner should start from the recorded commit, apply the patch, execute the named checks, and obtain the same result. If the task only works in the first operator's environment, the pilot found an environment dependency rather than a repeatable repository workflow.

Task-class dispositions: supported, held, or excluded

The final report summarizes results by task class rather than as one overall productivity score. Task classes can behave differently in the same repository, so the decision belongs at that level. A team might approve test repairs with deterministic checks, hold cross-service changes for a second pilot, and exclude permission or deployment work. Google's DORA research describes AI-assisted development outcomes as dependent on existing workflows and internal platforms, which is why the report preserves a local task receipt instead of relying on a generic benchmark.

Coding Agent Repository Pilot prepares the replay and repository evidence through Reality Contact, LLC. The buyer decides which task classes the repository controls support and approves every production merge.

Where the service stops

Reality Contact, LLC configures and measures the supervised pilot, but does not approve production changes, replace the buyer's security or legal review, manage employees, or allow an agent to merge outside the written controls. The buyer reviews the evidence, decides which task classes the repository controls support, and retains responsibility for accounts, permissions, merges, and deployment. This is technical implementation and review support; it does not replace the customer's security, legal, employment, procurement, or production-change review. We do not promise productivity, defect reduction, autonomous operation, successful deployment, or approval of any task class.