Evaluating Agent Workflows: Traces, Cost, Recovery
Connects trajectory evaluation, goal-level cost, semantic recovery, and confidence calibration into a practical evaluation frame for agent workflows.
ai-engineering
20 public items tagged ai-engineering.
AI Agent Operational Governance
Defines AI agent operational governance: the layer of authority, evidence, tools, traces, costs, and human gates that makes agents usable in real work.
Commitment Tracking and Agent Drift
Explains residual drift in long-running agent sessions and why serious workflows need commitment tracking, not just memory or contradiction checks.
Evidence and Source Authority
Shows how evidence-carrying actions and source authority rules prevent fluent text from overriding structured data, tool output, or source evidence.
Runtime Authority and Autonomy Gates
Explains why agent actions need runtime authority checks and autonomy gates, instead of relying only on plan-time approval or a confident task plan.
Tool Registries Are Control Surfaces
Treats MCP servers, plugins, tool descriptions, and dependencies as part of the agent control surface that needs governance and supply-chain review.
Why Codex Should Be a Worker, Not the Scheduler
Explains why Codex and similar AI coding tools should remain bounded workers, not long-running schedulers or release controllers.
Give AI the Context It Should See, Not the Whole Repository
Many AI task failures do not happen because the model cannot modify code. They happen because the model reads the wrong context.
Evidence Contract: AI Delivery Must Come With Proof
Explains why AI delivery must include verifiable proof: tests, logs, screenshots, risk notes, and a review path, not only a claim that work is done.
Evidence Gate: Why AI Pull Requests Need Evidence Bundles
Explains why AI pull requests should include reviewable evidence bundles, not just code changes and a claim that tests pass.
From AI Failure to Project Memory
Shows how repeated AI mistakes should become project memory, updated rules, and regression checks so the delivery pipeline improves after each failure.
Use Labels, Branches, and Comments as an AI Engineering State Machine
Shows how GitHub labels, branches, and comments can become an AI engineering state machine that keeps issue repair work observable and controllable.
Put Humans at Risk Boundaries, Not Only at the Final Approval
Shows where humans should stand in an AI delivery pipeline: requirements, risk boundaries, release decisions, rollback choices, and final acceptance.
Stage-Gated AI Workers Need Isolated Workspaces
Explains why real projects should put AI work in isolated branches or workspaces, then move changes through explicit gates before they reach the main codebase.
An Issue Is Not a Todo: It Should Be an Executable Contract for AI
Argues that an issue should be an executable contract for AI work, with scope, context, gates, evidence, and release boundaries instead of a loose todo.
Open-Sourcing an Issue-Driven Agent Workbench
Explains how to safely extract public methodology, templates, and toy examples from private production AI workflow experience.
PR Merged Is Not Done: The Release Boundary in AI Engineering
Explains why a merged PR is not the end of AI engineering work; release verification, rollout checks, and post-merge evidence still define done.
What a Project-Specific AI Delivery Pipeline Means
Defines a project-specific AI delivery pipeline: AI acts as a worker while the project owns task intake, context, gates, evidence, and release boundaries.
From Chat Request to Task Contract: Route the Work Before AI Executes
The most common risk in AI-assisted development is not that the model cannot write code. It is that the model starts writing code too early.
From Triage to Implementation: Why AI Fixes Need Gates
Explains why AI repair work should move through triage, analysis, implementation, evidence, and review gates instead of jumping straight to code.