VentureBeat Intelligence surveyed 117 people in October 2026, and 106 qualified: people who work on AI at organizations with 100 or more employees, and consultants serving such organizations. We asked whether AI agents may push changes to production with no human review and whether AI that passed pre-release evaluations went on to fail. We also asked how organizations would find out about bad output and which evaluation platforms they use.
Executive summary
Respondents whose organizations use automated checks or monitoring alerts are nearly twice as likely to report AI that passed pre-release tests and then failed: 63%, against 34% of respondents whose organizations rely on people to notice. The likeliest reading is that organizations without automated monitoring miss many of their failures. Overall, 42% of respondents who test AI before release report such a failure, and in 49% of those cases the most significant failure reached customers.
Among respondents whose organizations deploy autonomous AI agents, 24% say an agent may already push a change to production on automated evaluation results alone, for specific low-risk agents or changes. That is down from 47% in September. The share who say their organizations do not expect to permit fully automated deployment in the foreseeable future rose to 40% from 24%. Among October respondents who did not come through the research panel, neither change is statistically significant.
We compared respondents whose organizations deploy autonomous AI agents, run evaluations before release and could say whether a failure happened. Of those who let agents push some changes without human review, 76% say AI that passed their evaluations then failed in the past 12 months. Of respondents whose organizations require human review, 46% say so. In September the figures were 80% and 39%. Among in-house respondents alone, the October gap is not statistically significant.
Of respondents whose organizations run pre-deployment evaluations, 42% say AI that passed those evaluations went on to fail in the past 12 months, and 25% say it happened more than once. Another 19% do not track this or cannot say. Of those reporting a failure, 49% say the most significant case reached customers, through incorrect output or a customer escalation, churn or a damaged commercial relationship. Another 37% say it was caught internally with no external impact.
Asked how their organizations would normally first find out that AI was producing incorrect or low-quality output, 39% of respondents name neither automated checks nor a monitoring threshold. They name engineers sampling output, internal users, customers or rising support tickets. Another 57% name automated checks or a monitoring threshold, and 5% do not know.
Of respondents, 57% say their organizations use Anthropic's Claude Console to evaluate AI, and 43% use OpenAI's developer platform. LangSmith, the most used specialist platform, is used by 15%, and custom in-house tooling by 31%. Anthropic's console is the primary evaluation platform for 33%.
Finding 1. 24% of respondents let AI agents push changes to production without human review, down from 47% in September
40% say their organizations do not expect to allow it in the foreseeable future, up from 24%.
Fully automated deployment means an AI agent pushing a change to production on automated evaluation results alone, with no person validating it.
We asked whether the organization would currently allow an autonomous AI agent to deploy a code or system change to production based on automated evaluation results alone, with zero human-in-the-loop validation.
Finding 1 — Would allow an AI agent to deploy on automated evaluation results alone
Base: 106 qualified respondents. One answer each. Shares in the text and the table below leave out respondents whose organizations do not deploy autonomous agents (base: 102).
Finding 1 — Respondents whose organizations deploy autonomous AI agents
Bases: 89 September and 102 October respondents whose organization deploys autonomous agents. One answer each. The changes in the first, third and fourth rows are Findings. The change in the second row is Directional.
Of October respondents who did not come through the research panel, 34% say their organizations already allow fully automated deployment for some changes, against 47% of all September respondents, a difference that is not statistically significant. We cannot establish from these surveys how much of the change reflects who answered.
We asked respondents whose organizations are engineering toward fully automated deployment what has to be true before their organizations would allow it.
Finding 1 — What has to be true before allowing fully automated deployment
Base: 34 respondents whose organizations are engineering toward fully automated deployment within 12 months, the group the question was shown to. One answer each.
Finding 2. Respondents who let AI agents ship changes without human review report far more failures slipping past testing
76% report at least one in the past year, against 46% of respondents whose organizations require human review. September showed the same pattern, at 80% against 39%.
Human review here means a person approving an AI agent's change before it reaches production. Pre-deployment evaluations are tests an organization runs on an AI agent or LLM feature before release.
We compared the answers to two questions. The first was the question on fully automated deployment in Finding 1. The second was whether, in the past 12 months, the organization had deployed an AI agent or LLM feature that passed its internal evaluations but then caused a customer-facing failure. Examples given were an incorrect output, a broken workflow or a quality incident.
Respondents whose organizations let agents push changes without human review answered that they already allow it for specific low-risk agents or changes. Respondents whose organizations require human review answered No: Their organizations would not currently allow an agent to deploy with zero human validation, whether or not they are engineering toward it.
Finding 2 — Respondents whose organizations deploy autonomous AI agents
Base: 80 respondents whose organizations deploy autonomous agents, run pre-deployment evaluations, gave their policy and could say whether a failure happened. Of the 102 respondents whose organizations deploy autonomous agents, 3 did not know their policy and 4 run no pre-deployment evaluations. Another 15 do not track failures or could not say: 2 letting agents push changes without review and 13 requiring review. Counted as reporting no failure, those 15 make the figures 70% and 38%, still a statistically significant gap. September's figures use the same base.
Of respondents whose organizations let agents push changes without human review, 76% report such a failure, against 46% of respondents whose organizations require review, a statistically significant gap. The gap in failures reported more than once, 48% against 25%, is not statistically significant, and neither is the gap between the two groups requiring review.
Among respondents working on AI for their own organization, the figures are 71% against 41%, a gap that is not statistically significant. Respondents who let agents skip review are also more likely to use automated checks or a monitoring threshold, 86% against 58%, and Finding 4 shows that organizations monitoring automatically report more failures. So part of this gap may reflect which organizations can see their failures. Among respondents whose organizations monitor automatically, the gap holds, at 83% against 53%, a statistically significant difference.
On the same questions and base, September's figures were 80% against 39%. Neither group's share changed in a statistically significant way between the two months. We did not ask whether a failure came from a change an agent pushed without review, or how much AI each organization deploys.
Finding 3. 42% of respondents who test AI before release say failures slipped past testing, and about half of those reached customers
In 49% of those cases, the most significant failure reached customers. Another 19% of respondents who test before release do not track this or cannot say.
We asked whether, in the past 12 months, the organization had deployed an AI agent or LLM feature to production that passed its internal evaluations but then caused a customer-facing failure.
Finding 3 — Deployed AI that passed internal evaluations, then caused a customer-facing failure
Base: 106 qualified respondents. One answer each. Shares in the text leave out respondents whose organizations run no pre-deployment evaluations (base: 102).
Of respondents whose organizations run pre-deployment evaluations, 42% report at least one such failure in the past 12 months, and 25% report more than one. In September, 55% reported at least one, a Directional change.
We asked respondents who reported a failure to think of the most significant instance and say what happened as a result.
Finding 3 — What happened in the most significant case
Base: 43 respondents who reported at least one failure, the group the question was shown to. One answer each.
Of respondents reporting a failure, 49% say the most significant case reached customers: Customers received the incorrect output, or a customer escalated, churned or a commercial relationship was damaged. That is 21% of all respondents whose organizations run pre-deployment evaluations.
The question asked about a "customer-facing failure," yet 37% of respondents who reported one say their most significant case was caught internally, with no external impact. So this report describes these cases as failures after passing evaluations, and the 21% figure counts only cases that reached customers.
Finding 4. 39% of respondents rely on people to catch bad AI output, with no automated check or alert
They name engineers sampling output, internal users, customers or rising support tickets. Another 57% name automated checks or a monitoring threshold, and 5% do not know.
We asked how the organization would normally first find out that a deployed agent or LLM feature was producing incorrect or low-quality output for users.
Finding 4 — How the organization would normally first find out
Base: 106 qualified respondents. Several answers allowed, so shares sum past 100%. Of the 106, 57% named automated checks or a monitoring threshold, 39% named neither and 5% answered only "Don't know."
We compared these answers with the failure question in Finding 3, which the questionnaire set up as its check. Among respondents whose organizations run pre-deployment evaluations and could say whether a failure happened, 63% of those using automated checks or a monitoring threshold report AI that passed evaluations and then failed in the past 12 months. Of those relying only on people to notice, 34% do. The gap is statistically significant and holds among in-house respondents alone, at 59% against 29%.
The likeliest reading is that organizations without automated monitoring miss more of their failures, so a low failure count can mean a team is not looking. The survey cannot rule out another explanation: Organizations that run more AI may both monitor more and fail more, and we did not ask how much AI each organization runs.
Bases: 52 respondents using automated checks or a monitoring threshold and 29 relying only on people, among the 83 whose organizations run pre-deployment evaluations and could say whether a failure happened. Two who answered only "Don't know" are left out.
We also asked which description best fits how the organization's live production monitoring for AI agents is set up. The table covers respondents whose organizations deploy autonomous agents.
Finding 4 — Live production monitoring for AI agents
Base: 102 respondents whose organizations deploy autonomous agents. One answer each.
Inline quality assertions were chosen by 26% of those respondents, against 27% in September, a difference that is not statistically significant. Ad-hoc review was chosen by 25%, up from 13%, a Directional change.
Finding 5. Respondents test AI with Anthropic's and OpenAI's own tools more than with any specialist platform
57% use Anthropic's Claude Console and 43% OpenAI's developer platform, against 15% for LangSmith, the most used specialist platform. Anthropic's console is the primary evaluation platform for 33%, against 15% for OpenAI's.
An evaluation platform runs test cases against an AI agent or LLM feature and records the results.
We asked which agent reliability or evaluation platforms the organization uses today, and which one is its primary platform.
Finding 5 — Evaluation platform
Base: 106 qualified respondents, all of whom named at least one platform. Several answers allowed for use, so that column sums past 100%; one answer for primary. Platforms used by fewer than five respondents are not shown. Shares of users naming a platform primary are shown for platforms with 30 or more users; each base is the platform's user count.
Anthropic's console is used by 57% of respondents and OpenAI's developer platform by 43%, a statistically significant gap. As the primary platform, Anthropic's console is named by 33% and OpenAI's platform by 15%. Custom in-house tooling is named primary by 21%, and its gap with Anthropic's console is not statistically significant.
We also asked whether the organization plans to adopt a new, additional or replacement platform in the next 12 months, and whether it adopted one in the past 12 months.
Finding 5 — Plan to adopt a new, additional or replacement platform
Base: 106 qualified respondents. One answer each.
Finding 5 — Adopted a new, additional or replacement platform in the past 12 months
Base: 106 qualified respondents. One answer each. The question is new in October.
Of respondents, 55% say their organizations plan to adopt a new, additional or replacement platform within 12 months, and 59% say their organizations adopted one in the past 12 months.
Finding 5 — Platforms considered (several answers allowed)
Base: 58 respondents whose organizations plan to adopt a platform within 12 months, the group that answered the question in October. Platforms considered by fewer than four are not shown.
Finding 5 — Factor that most influenced vendor selection
Base: 99 respondents who named a primary platform and chose a listed factor. Seven others wrote in another answer, such as "We don't have one." One answer each. September's report gave this question's figures for respondents who named a primary platform.
Ease of integration, evaluation accuracy and cost of evaluations were each named by 26% to 28%, and the gaps between them are not statistically significant.
Respondents rated their primary platform 3.81 out of 5 for overall satisfaction, 3.80 for ease of implementation and 3.83 for value for money. In September, respondents who named a primary platform rated overall satisfaction 4.13.
Base: 106 respondents who named a primary platform.
What changed since the September Pulse
September's figures cover only respondents who would pass October's screen, so they differ slightly from the published September report. We asked the questions below in the same words in September and October. The rating question kept its wording but is now answered only by respondents who named a platform in use. We checked whether changes in the respondent mix could account for the changes. Consultants were 47% of September's qualified respondents and 24% of October's, and October's respondents work at larger organizations. Each Finding below holds among in-house respondents alone, and adjusting September's answers to October's mix of company sizes moves September's figures by up to about seven points. The seniority mix also changed: individual contributors were 41% of October's qualified respondents, against 16% in September. Adjusting September's answers to October's seniority mix moves September's figures for each Finding by up to about two points. The respondent profile gives both months' seniority mix.
Respondents' sources were not recorded in September. Among October respondents who did not come through the research panel, the changes in fully automated deployment are not statistically significant. We cannot establish from these surveys how much of any change reflects who answered rather than changes in what organizations do.
Findings: statistically significant changes
Findings: statistically significant changes
A Finding is a statistically significant change that also holds among in-house respondents alone and after September's answers are adjusted to October's company-size mix and, separately, its seniority mix. Bases, September then October: 89 and 102 for respondents whose organizations deploy autonomous agents; 90 and 106 for ratings.
Directional changes
Directional changes
Directional means the change meets the looser test described in the methodology, or is statistically significant overall but not among in-house respondents alone. A change that is statistically significant but not after the company-size or seniority adjustment is also Directional. Bases, September then October: 89 and 102 for respondents whose organizations deploy autonomous agents; 99 and 106 for all qualified respondents; 97 and 102 for respondents whose organizations run pre-deployment evaluations; 90 and 99 for the selection factor.
Not statistically significant
Not statistically significant
Not statistically significant means the change does not meet the threshold described in the methodology. Bases, September then October: 40 and 21, and 33 and 59, for the two policy groups; 97 and 102 for respondents whose organizations run pre-deployment evaluations. Other bases: 89 and 102 for respondents whose organizations deploy autonomous agents; 99 and 106 for all qualified respondents; 90 and 99 for the selection factor; 90 and 106 for ratings.
Platform use, two readings
Evaluation platform
Vendor usage is reported as two survey readings until a second month or a quarterly comparison confirms a change. Bases: 99 and 106 for use and primary; 68 and 58 for platforms considered. The methodology explains how margins of error are calculated. In September, 9% of respondents chose "No dedicated agent reliability or evaluation tools"; no October respondent chose it.
What we did not compare
We did not report the question on which limitation most reduces trust in automated evaluation. In September, 17% of respondents said they trust automated evaluation today. In October, only 25% of qualified respondents answered the question, so its figures would not describe the group the rest of this report covers.
Four questions were not in September's data or report. They asked how an organization would first find out about incorrect output and what happened in the most significant failure. They also asked what has to be true before allowing fully automated deployment, and whether the organization adopted a platform in the past 12 months.
The bottom line: Respondents using automated monitoring are nearly twice as likely to spot AI failures that slipped past testing
Among respondents whose organizations deploy autonomous AI agents, 24% let agents push some changes to production without human review, down from 47% in September. The share whose organizations do not expect to permit it rose to 40% from 24%.
Among respondents whose organizations deploy autonomous agents, run pre-deployment evaluations and could say, 76% of those letting agents push changes without review report a failure from AI that passed evaluations. Of respondents whose organizations require review, 46% report such a failure. Asked how they would first find out about incorrect AI output, 39% of respondents name neither automated checks nor a monitoring threshold, so a person has to notice the problem first. Respondents whose organizations monitor automatically are nearly twice as likely to report such failures, which suggests many organizations that rely on people are missing theirs.
In the next Pulse, VentureBeat Intelligence will measure how the share of respondents whose organizations let agents push changes without human review compares with October's 24%. VentureBeat Intelligence will also measure whether the gap in failures between respondents whose organizations let agents push changes without review and those that require review holds a third month.
Respondent profile
| Respondent type | Respondents |
|---|---|
| Works on AI for the organization that employs them | 76% (81) |
| Consultant who works on AI for client organizations | 24% (25) |
Base: 106 qualified respondents. One answer each.
| Organization size (main client, for consultants) | Respondents |
|---|---|
| 100–250 employees | 23% (24) |
| 251–500 employees | 11% (12) |
| 501–1,000 employees | 8% (8) |
| 1,001–5,000 employees | 20% (21) |
| 5,001–10,000 employees | 9% (10) |
| More than 10,000 employees | 29% (31) |
Base: 106 qualified respondents. One answer each.
| Job level | September | October |
|---|---|---|
| Individual contributor | 16% (16) | 41% (43) |
| Manager | 37% (37) | 30% (32) |
| C-suite | 18% (18) | 11% (12) |
| Director | 10% (10) | 10% (11) |
| VP | 5% (5) | 5% (5) |
| Partner or principal | 6% (6) | 1% (1) |
| Other | 7% (7) | 2% (2) |
Bases: 99 September and 106 October qualified respondents. One answer each.
Methodology
Who answered. VentureBeat Intelligence fielded this VB Pulse survey in October 2026. The survey received 117 responses, of which 106 qualified. Respondents qualified if they work on AI at an organization with 100 or more employees, or consult for one as their main client with day-to-day or partial access to its AI systems.
Who did not qualify. We excluded in-house respondents with no direct role in their organization's AI work or who do not work directly on AI systems. We also excluded analysts, academics and journalists, respondents at organizations with fewer than 100 employees, and one consultant who does not work directly on AI systems.
Consultants. Consultants answered about the client organization where they spent the most time on AI work.
Where respondents came from. Of qualified respondents, 58% came through a research panel, 8% through a VentureBeat email and 34% through a link with no source recorded or an unidentified one.
Who each figure covers. Each percentage is based on the respondents a question applies to, named in the note under each table. The first tables in Findings 1 and 3 show all 106 qualified respondents. The figures in the text on fully automated deployment and production monitoring cover respondents whose organizations deploy autonomous agents, and the figures on failures after passing evaluations leave out respondents whose organizations run no pre-deployment evaluations. One respondent said the organization already allows fully automated deployment for some changes but runs no pre-deployment evaluations; that respondent falls outside the Finding 2 comparison.
Routing. The question on platforms under consideration was shown to everyone in the questionnaire, but in October only respondents planning to adopt a platform answered it. September's report gave that question's figures for respondents planning to adopt a platform, so the two months cover the same group.
Several answers. The questions on platforms in use, platforms under consideration and how an organization would first find out about incorrect output allowed several answers. Their figures count people, not selections.
Small groups. Some groups in this report have fewer than 40 respondents. In Finding 2, respondents whose organizations let agents push changes without review number 21; respondents whose organizations require review number 59, split 25 and 34; and in-house respondents in the two groups number 14 and 46. Respondents whose organizations are engineering toward fully automated deployment number 34. Margins of error here are conventional calculations that assume a random sample, which this is not. On that basis, a result for a group of 21 could differ by about 21 percentage points either way, and for a group of 34 by about 17.
Tests. We describe a difference as statistically significant only when its test gives p below 0.05. Two percentages from separate groups use the two-proportion z-test, or Fisher's exact test when either group has fewer than 40 respondents. Two answers to a one-answer question use an exact binomial test, and two answers to a several-answer question use an exact McNemar test. A Directional change meets p below 0.10. Average ratings use a Mann-Whitney test. The tests treat every response as independent, although responses were not deduplicated by organization, so more than one respondent may represent the same organization.
Changes from September. Each change is labelled a Finding, Directional or not statistically significant. A Finding must meet the 95% significance threshold overall and among in-house respondents alone. The difference must also stay statistically significant after September's answers are adjusted to October's company-size mix, and separately to its seniority mix. The in-house check repeats the test without consultants; it does not test other changes in who answered. September's figures cover the 99 of the published September report's 103 qualified respondents who would pass October's screen, which also excludes in-house respondents with no direct role in AI work. Compared with the published September report, two labels change: the rise in ad-hoc review becomes Directional, and the fall in failures after passing evaluations becomes Directional.
Survey changes. In October, the 251–1,000 size band was split and the question on adopting a platform in the past 12 months was added. The primary-platform and rating questions were limited to respondents who named a platform in use.
Respondents chose to take this survey, and they come from a research panel and VentureBeat's readers. Because the sample is self-selected, the results describe these respondents and may not hold for all enterprises.
Published by VentureBeat Intelligence. © 2026 VentureBeat. All rights reserved.