Presented by Google Cloud
Malicious actors are using autonomous agents to find vulnerabilities and launch attacks with little human involvement. That means the time from discovery to exploitation is now too short for traditional methods like signature-based detection and patching to handle alone. Security teams need to spot anomalous behavior at machine speed.
These developments make the data layer more critical than ever. Security teams need to ingest enormous volumes of activity, including the actions of internal agents, and connect them to enterprise context in real time. Most data architectures weren't built for that type of work.
A new kind of security data
Threat intelligence today has to look deeper into what applications and agents are actually doing. Agentic systems add new data that has to be monitored, such as the actions and trajectories of the agents themselves. Because agents operate at machine speed, that data grows faster than anything security teams have dealt with in the past.
Every action an agent takes produces a log entry or trace, but simple log aggregation won’t tell you much on its own. Security teams need to tie those records to each agent's trajectory, so they can reconstruct a sequence of events to understand what happened and which systems were touched.
That gives threat intelligence a similar workload profile to observability, with massive-scale logs, continuous ingestion, stream processing and triage. The logs, traces and metrics that site reliability teams use to keep services running can also reveal signs of compromise or malicious intent.
The core requirements of a security data layer
Evaluating risk requires grounding raw telemetry in business context. An anomaly that affects a test environment is less urgent than one that touches a production database, and security teams need to be able to make that distinction in close to real time.
Making those decisions requires access to all the data, wherever it lives, and getting to it can't depend on a long migration project. That’s one reason we’re seeing the emergence of the concept of a “borderless lakehouse,” which gives security teams and agents direct access to data across multiple clouds, databases and SaaS tools without having to move it.
Open formats like Apache Iceberg are a big part of that shift. When security data sits in open formats, multiple engines can work on it, whether that's SQL running at scale or a Python program an agent writes on the fly. SIEMs are evolving too. To give security teams maximum interoperability and access to their data, SIEMs need to support open data platforms and work on federated data in place, without ingesting it first.
That flexibility brings its own security challenge. When data stays where it lives, so do the permissions governing it, and those permissions can’t be managed from a central place. Duplicating and syncing rules across every source doesn't scale to thousands of agents, and pushing authorization onto end users creates new risks, like prompt injection or leaked credentials. The answer is fine-grained controls, down to rows and columns, defined once and enforced at query time, including on open-format tables. Each query should carry the user's actual identity through short-lived, scoped credentials, so agents can’t overstep the authority of the person directing them.
When a threat signal shows up, the task moves from finding the right logs to analyzing the impact. Graph processing can map an isolated event to the assets around it and help security teams trace any lateral movement. A knowledge catalog adds another layer of context by giving agents and human investigators a map of the organization's data assets, so they know where to look.
Controlling cost at machine scale
Large language models are good at analyzing log entries, but running them across petabytes of security telemetry gets expensive fast. The data layer should narrow down what gets processed before the models begin their work.
That can happen in stages. Streaming queries watch events as they arrive and flag the suspicious ones. A second pass checks those against historical data to narrow the list further. Only then does an agent investigate.
Some foundation models are pretrained on billions of time series, such as network traffic or login activity over time, so they already know what normal patterns look like. That lets them flag unusual activity in the early stages without custom training. For simpler jobs, like sorting log entries into “dangerous” and “harmless,” the data platform can build a small, purpose-built model on the fly, which is cheaper than running every entry through a large one.
Agent-driven investigations also make demand unpredictable. When agents start hunting through historical logs, query volume can spike quickly and then drop back down just as fast. Hardware appliances and pre-provisioned cloud capacity weren't built for that pattern. The data platform needs serverless scaling that expands and shrinks as needed, so security teams pay only for what they use, down to the second.
Watching your own agents
Agents also create new security risks inside the enterprise. When a company hands work to an agent, it has to decide what that agent is allowed to touch, much as it would for a new employee.
Traditional software runs predictably, but agents don't. Their behavior is probabilistic, so they need their own identities, with limits on which data, documents and systems they can access, even when the person directing them has broader permissions.
Prompt injection makes those limits even more important. Whereas attackers once used SQL injection against databases, they can now hide instructions in data an agent reads. Whatever enters an agent's context can shape what it does next, so security teams need to control what each agent can read, write and send. Every external connection is a potential path for exfiltration.
Even an authorized agent can take a sequence of individually permissible actions that becomes dangerous in combination. But your own agents are also easier to watch. An attacker's agents show up only as anomalies, while yours leave a full record of what they did. Analyzing that record at machine scale is how security teams spot an agent that has gone off course.
From passive archive to active defense
As agents take on more of the work, the security analyst's job is changing. Instead of running queries and triaging alerts manually, analysts will spend more time defining what matters and reviewing what agents find, much as developers now review plans for agent-written code instead of reading every line. That only works if agents can show their work. Analysts need to see what data an agent examined, what steps it took and how it reached its conclusion.
That changes the role of the data layer in security. It used to be where events were stored and queried after the fact. It’s now part of how security teams understand those events as they're happening. Your data architecture must give your agents and security teams the speed, context and scale to stop an attack before it can do any harm.
To learn more, join our upcoming virtual event, Trusted Agents, Real Outcomes. Download this guide to securing autonomous environments with Google’s Agentic Data Cloud and building the foundation of a data platform for security with BigQuery.
Tomas Talius is VP of Engineering, Google BigQuery at Google Cloud.
Sponsored articles are content produced by a company that is either paying for the post or has a business relationship with VentureBeat, and they’re always clearly marked. For more information, contact sales@venturebeat.com.