PayPal is reporting an internal question-answer accuracy increase from 43% to 95%, but it didn’t change its underlying large language model (LLM). It added proprietary business context and kept it separate from the model that uses it.
The finance platform’s internal data science assistant already had access to vetted, high-quality data, said Ren Zhang, PayPal’s VP and global head of AI product, during VentureBeat’s Oct. 8 AI Impact Series discussion, but that didn't mean it could reliably answer business questions.
The assistant needed the definitions, query patterns and analytical procedures that PayPal’s employees use to turn data into answers. After adding that knowledge, Zhang said, accuracy reached 95%.
“The LLM is still the same,” she said.
The point of keeping it separate is to preserve that knowledge as models improve, rather than repeatedly reconstructing it around each new release. PayPal wants better models, but it does not want its accumulated business expertise bound to any particular one, Zhang said. The challenge extends to agents that do more than answer questions: PayPal is also deciding when they must consult trusted sources and when a human must approve their actions.
The knowledge that data access did not supply
The assistant is meant to answer business questions that typically reach PayPal’s data science team. For instance, when asked a question about “new accounts,” an agent might retrieve account records correctly but apply the wrong definition of “new.” The answer depends on business rules that must be supplied, not simply inferred from the availability of data.
PayPal addressed that gap through a knowledge layer with several components built to outlast the models used. One component is what Zhang called a “golden query”: a query specifying how to retrieve data for a particular purpose. Another is an analytical playbook specifying how to investigate a question, which could mean examining particular segments, products or behaviors, such as login failures and passkey use.
“The key thing is, we believe our IP is truly to build that knowledge layer,” Zhang said.
Keep the knowledge, change the model
Zhang said she deliberately designed the assistant to be model-agnostic. “If the model is getting better, I’ll just plug in and play,” Zhang said. “I should not change my knowledge.”
Zhang also described testing four or five models for PayPal’s separate, customer-facing commerce agent. While larger models offered strong capabilities, the agent initially had about 15 seconds of latency.
Switching to an 8-billion-parameter model improved latency significantly, but also reduced accuracy. Zhang’s team fine-tuned the smaller model to specialize in commerce. Accuracy improved, but it still fell short, she said.
PayPal then reduced its emphasis on extensive fine-tuning. Performance can be improved with customization, but the concern was how much ongoing work that improvement would require. Fine-tuning requires specialized expertise, and new model releases create pressure to repeat the process over and over.
“How many models have come out recently? One is better than the other,” Zhang noted. “If you train it too frequently, it's too costly, right?”
But that doesn’t mean abandoning fine-tuning altogether. The gains from customization have to be weighed against the cost of maintaining it as models and workloads change, she said.
Deciding when the agent uses what it knows
Keeping knowledge separate raises a second question: how the agent uses it.
PayPal is building a platform that gives agents access to enterprise knowledge while preserving procedures, checking proposed decisions against authoritative sources, and determining when an agent can act without human approval, said Deepika Khera, PayPal’s senior director of AI, during VentureBeat’s Oct. 28 Agentic Infrastructure Stack Summit.
Those controls are particularly important in incident response. Khera said PayPal has built a seven-agent workflow to help investigate alerts across more than 33,000 services. This workflow brings together information from Datadog, ServiceNow, and internal logs to distinguish meaningful signals from noise, assemble context, and analyze an incident.
Khera said the division of work reflects the roles agents perform, the context each needs, and the tools each is permitted to access. The number is specific to the task: A separate security review workflow uses as many as 18 agents.
Still, dividing the work does not remove the central risk: An incorrect diagnosis could lead to an incorrect fix.
“Every decision that is proposed is vetted against the source of truth,” Khera said. “That’s where our memory comes into play.”
Memory brings together certified data sources and standard operating procedures (SOPs), while policies and risk assessments provide additional checks. Many decisions still require human verification, Khera noted. PayPal is taking a cautious approach to autonomy.
A model may retrieve relevant information and still miss the procedure that makes it actionable. For instance, Khera described an internal HR question-answering tool that struggled because procedures were not represented in a way that supported accurate responses. This prompted the team to change how instructions were stored, preserving ordered steps together rather than breaking them into separate chunks.
PayPal has also found that accessing memory cannot always be left to an agent’s discretion. Khera described balancing Model Context Protocol (MCP) integrations with direct API calls. For some tool interactions, PayPal uses MCP when an agent needs to reason about which tool to select. But in an agentic software-development workflow, the team needed to ensure that memory was consulted at a particular step rather than leave that choice to the agent.
“For that flow, we had to force, ‘This is where I want to use the memory,’” she said.
PayPal is also experimenting with how memory changes as agents work. Episodic memory records sessions and pull-request (PR) outcomes, and a nightly distillation process examines experiences across users to identify lessons that can be stored to memory for future interactions. The goal is to turn experiences into reusable guidance, allowing subsequent work to benefit from what occurred before.
Portable knowledge still needs continuous testing
Even with business knowledge and workflow controls in place, test conditions may not resemble production, Zhang said. An evaluation can report strong performance against expected questions while overlooking the questions that actual users ask most frequently.
A system could perform correctly on nearly all of its test questions, but if users in production disproportionately ask the kinds of questions it gets wrong, the evaluation score can misrepresent their experience. “Your model is still the same,” Zhang said. “Your process is the same, but the distribution is completely different.”
The same gap shows up across enterprises. In VB Pulse's October Agentic Reliability and Evals survey, 42% of respondents whose organizations run pre-deployment evaluations reported that an AI agent or LLM feature had passed internal tests but subsequently failed within the preceding year. Of those reporting failures, 49% said their most significant incident reached customers.
Production monitoring does not necessarily close that gap, however. In the October survey, 39% of respondents identified no automated checks or monitoring alerts among the ways their organizations would first discover incorrect or low-quality AI output, relying instead on people to notice.
To better account for differences between testing and production, PayPal is performing broader offline simulations, synthetic question generation, and single-turn and multi-turn testing. Model-based judging alongside human assessment is also important, Zhang said: After deployment, employee comments help identify failures.
Zhang also reported more than 1,000 questions and an estimated 1,600 hours saved over 10 weeks with the internal assistant. But she cautioned against assuming that question volume alone demonstrates productivity.
Identifying failures is harder with consumer-facing systems, where users may not explain what went wrong. Repeated queries and abandonment can provide signals for further investigation, although those behaviors on their own do not prove an answer was incorrect.