When a software update slows down an application, engineers need to go through multiple stages, including inspecting metrics, tracing requests across services, identifying the responsible code, and testing potential solutions. 

With AI agents taking on the main burden of writing and changing code, manual performance review of code changes becomes one of the key bottlenecks of the software development lifecycle.

To address this challenge, OpenAI developed the Performance Factory, a new, internal AI system powered by Codex that automates the investigation and mitigation of performance regressions. OpenAI is already using Performance Factory to keep ChatGPT fast.

According to an exclusive engineering report shared with VentureBeat, the system tracked more than 1,700 investigations and identified the root causes of over 600 cases in just over three months. That's an average of 17 automated investigations per day.

The Performance Factory provides a glimpse into how coding agents can evolve from solving individual tasks into building automated workflows that coordinate between AI agents and human engineers. This approach can help engineering teams leverage the speed of AI-assisted software development while reserving human attention for decisions that require judgment.

From manual debugging to automated workflows

OpenAI's performance engineering team initially relied on manual investigations. When a service slowed down, engineers examined logs and traces, looked at code changes, and debugged live systems. The team developed more than 30 specialized tools to streamline these investigations.

Codex changed how engineers worked with these tools. Instead of manually searching through codebases and telemetry, engineers could ask Codex why a service had become slower. The agent could investigate logs and use tools to provide the answer. But engineers were responsible for steering the process.

To reduce this dependence, the team developed a reusable Codex skill called perf-forensics, which captures the performance team's investigation procedures for the agent to reproduce.

After testing and refining the skill, OpenAI could give Codex a performance alert and receive a diagnosis in a single run.

At this point, the engineers became intermediaries between monitoring systems and Codex. They would copy alerts into Codex and then paste the findings into Slack. So, naturally, the team simply connected the alerts to cloud-based Codex sessions. 

But the investigations ended with a diagnosis that required review and implementation by an engineer. Meanwhile, the ChatGPT performance team was dealing with regressions almost daily and struggling to keep track of them.

In comments provided to VentureBeat, OpenAI's performance team explained that it initially built a regression tracking system to manage the growing workload. This led to the realization that each tracked regression could become a task that Codex would automatically investigate and advance through different stages, such as identifying the root cause, proposing a fix, and verifying recovery.

This became the foundation of the Performance Factory's task-based architecture.

How the Performance Factory works

The Performance Factory organizes engineering work into tasks that move through predefined workflows. A task might begin with a performance alert and progress through investigation, root cause identification, code changes, review, and verification.

Each task maintains its state and findings independently so that different agents can work on the same problem at different stages. According to the OpenAI performance team, the task record contains a free-form field where agents leave notes, along with structured metadata such as when a regression began and when its root cause was identified.

Instead of copying large amounts of profiling data into the task, agents record the relevant metrics, queries, and high-level findings. The next agent receives the task record and can rerun those queries if it needs to verify the results. This keeps the information passed between agents manageable while preserving access to the underlying evidence.

OpenAI Performance Factory

OpenAI Performance Factory (source: OpenAI)

The system has four main components.

  1. The task service stores tasks and their progress. Engineers define workflows through configuration files and provide markdown prompts that determine what the agents must do at each stage.

  2. A dispatch loop periodically checks on the tasks. When a task is ready to advance, it starts a Codex session and assigns the task to it. The agent records its findings and updates the task when it completes its work.

  3. The third component is an internal API for running Codex in the cloud. Agents operate in environments that have access to OpenAI's code repositories and observability infrastructure. They are designed in a way that lets OpenAI control permissions and track agent actions. 

  4. Finally, subscriptions connect tasks to external events. For example, if an agent opens a pull request and needs an engineer's review, the task can pause until it receives the engineer’s feedback on GitHub.

Performance Factory in action

OpenAI's first concrete implementation of Performance Factory is the Regression Workflow, which focuses on performance regressions. Regression Workflow kicks into action when a configured alert fires. It creates a task that agents investigate using the perf-forensics skill that the OpenAI team had previously developed.

The agent first determines what changed by examining the affected metrics and identifying when performance started deteriorating and which requests were affected. 

After homing in on the problem the agent tries to establish the cause by examining several artifacts, including code changes, logs, and traces.

However, OpenAI found that agents sometimes stopped at the first plausible explanation, even when the evidence did not fully support it.

To address this problem, the team introduced an “adversarial review” process that uses a separate Codex session to challenge the original diagnosis. The reviewer receives the entire task record but starts with a fresh context. Its instructions explicitly tell it to find gaps in the diagnosis, verify the supporting evidence, and test alternative explanations. This allows the reviewer to devote its entire session to challenging the original agent's conclusions.

For example, in one investigation, an agent attributed failing ChatGPT conversations to errors in image downloads. The reviewing agent noticed that the internal service responsible for locating files had already recovered while image requests continued failing. The reviewer therefore reopened the investigation because the original diagnosis did not explain the continuing failures.

According to the OpenAI performance team, the adversarial reviewer initially sent back a substantial number of diagnoses for further investigation. The team used this feedback to improve the original investigator's instructions, reducing the number of cases returned for additional work. Currently, the adversarial review sends back around 25% of diagnoses.

Once a diagnosis passes adversarial review, Codex proposes a fix and opens a pull request. An engineer reviews the code and decides whether to merge it.

The results show the gap between diagnosing problems and implementing reliable fixes. According to the OpenAI performance team, of the more than 600 investigations that reached a root cause, around 400 progressed to a proposed fix. Approximately 50 fixes that were eventually shipped were influenced by or matched agent suggestions, while around 15 shipped fixes were the exact pull requests authored by Codex.

The team acknowledges that these results fall short of its ambitions for automated remediation. However, much of the work of resolving regressions involves detecting them, identifying their causes, tracking their progress, and verifying recovery. The Performance Factory has automated much of this process, while improving the quality of proposed fixes remains a priority. OpenAI has also completed a workflow evaluation suite that it plans to use to improve agent-generated fixes.

The OpenAI performance team found that Codex performs best when the cause of a regression is clear and the fix is relatively contained. For example, a code change might cause an application to serialize the same conversation several times while processing a request. Codex can identify the redundant work and propose serializing the conversation once and reusing the result.

More complex changes remain challenging, particularly when a new feature adds latency and requires architectural modifications. Agents sometimes propose overly complicated fixes or introduce new problems. For example, adding a cache might improve response times but create stale data. OpenAI has therefore added adversarial review to proposed fixes as well.

After an engineer deploys a fix, another agent checks whether the performance issue has been addressed. If the problem persists, the workflow sends the task back for further investigation. 

Another challenge OpenAI had to address was managing the volume of agent-generated findings. Engineers were initially given a user interface full of investigation results, and they had to determine which findings mattered.

The system now makes it easier for engineers to navigate results by grouping related incidents, ranking their estimated impact, and generating concise reports with graphs and metrics. This helps engineers evaluate proposed changes without reconstructing each investigation from raw agent output.

Lessons for enterprise AI teams

Engineers of agentic workflows can take away a few key lessons from OpenAI's work on Performance Factory. The practices are applicable to agentic systems beyond performance engineering.

Avoid AI fatigue. Where humans must review an AI agent’s output, try to make sure that the most pertinent information is presented to them to avoid overwhelming them with too much content (aka AI slop).

Preserve evidence without overloading agent context. OpenAI's approach of storing high-level findings and reproducible queries in task records allows agents to hand off work without transferring large volumes of raw telemetry. This pattern can help teams build multi-stage workflows that remain manageable as investigations become more complex.

Evaluate agents and their interactions with humans. OpenAI backtests investigations against historical incidents, measuring how accurately agents identify when problems started, their impact, and potential causes. These evaluations require care because historical evidence can expire, while information collected after an incident might reveal the correct answer. The team also evaluates agent-generated summaries separately to determine whether engineers can understand the evidence and proposed actions.

Monitor the agents themselves. OpenAI runs a daily agent that examines recent agent logs for recurring tool failures and permission problems. When it finds an issue that repeatedly blocks investigations, the Performance Factory can create a task and propose a fix for an engineer to review. Evaluations then check whether the change resolves the problem.

From reactive maintenance to continuous optimization

The Performance Factory currently focuses on responding to configured alerts, but OpenAI plans to expand it into proactive optimization.

The team is exploring ways for agents to compare CPU profiles, memory data, and request traces across releases to discover slowdowns that monitoring alerts might miss. It is also developing workflows for tasks such as service rightsizing, where agents can identify inefficient autoscaling configurations.

OpenAI is investigating whether agents could eventually deploy some low-risk fixes without human approval.

According to the OpenAI performance team, automating regression investigations has allowed engineers to spend more time on long-term performance improvements instead of constantly responding to incidents. The team also hopes to turn what engineers learn from individual investigations into reusable workflows that can improve services across the organization.

This approach could enable engineering teams to preserve and apply performance knowledge across projects and services. Lessons from one investigation could inform how agents identify and resolve similar problems months later, reducing repetitive debugging work and allowing engineers to focus on more strategic optimizations.