Presented by Outshift by Cisco
IT leaders are now prioritizing generative AI and adopting business-critical use cases across the entire enterprise, from marketing, design and product development to data science, operations and sales. Outside the enterprise, it’s being used in what you might call humanity-critical cases, such as vaccine development and cancer detection, as well as environmental, social and governance initiatives like resource optimization and more.
But each of these use cases carries its own security risks -- especially when it comes to privacy and security, compliance, and potential loss of sensitive data and IP. And the risk will continue to grow.
“Organizations need to plan every generative AI project, in the enterprise and out in the world, not just for current risks, but with the future in mind,” says Vijoy Pandey, SVP at Outshift by Cisco. “At the same time, it’s imperative to strike the balance between application innovation and user trust, and prioritize privacy, authenticity and attribution.”
The unique risks of generative AI
Part of the challenge is that the risks of generative AI are unique. Consider the new sophistication of phishing attacks. The ability to create deep fakes and impersonate anyone over video, voice and text in order to cast out even more compelling bait has taken identity fraud to the next level. A fraudster could be trying to scam a company out of thousands of dollars -- but someone who looks and sounds like a company’s customer service agent could be targeting individuals.
Users are also out there blithely entering sensitive data into generative AI models in the wild. Generative AI likes to hang on to the data it receives, and any data that a system collects is used to train models. How does an organization protect sensitive information, or prevent the organization from violating privacy regulations in those cases? Organizations also need to stay on top of new regulations as they emerge in response to the potential risks of generative AI.
And while you might think the truly massive amounts of data that large language models (LLMs) require in order to operate -- as well as the vast amount of new data they create -- are the biggest security challenges, security paradigms exist for securing raw data and data platforms, as well as preventing data leaks.
Fraudsters can slip in anywhere along the gen AI pipeline to poison the model and tamper with its ability to make accurate predictions, deny service, or access the operating system.
The real issue, Pandey says, is the vulnerability of the gen AI pipeline. Fraudsters can slip in anywhere along the gen AI pipeline to poison the model and tamper with its ability to make accurate predictions, even make it cheerfully lie to users, deny service, access the operating system or embarrass you on social media. That can have profound impact both on customers, and on business users within the organization who might rely on a generative AI tool to help make pivotal decisions.
“We’re looking at issues like data poisoning attacks and model inversion attacks and more, and detection is the primary issue,” he explains. “We all rely on confidence intervals and a bar that we set for the entire pipeline to say yay or nay, whether I should trust the output or not. And if the attacks themselves are shifting towards compromising the pipeline, that bar or that confidence level might work against you.”
It’s astronomically difficult to detect that something has gone wrong immediately -- only over time do issues add up and the outline of the problem becomes clear. SecOps teams do have backup from known security frameworks like MITRE ATLAS and the OWASP Top 10, which are in the process of responding to generative AI security issues as they become known.
“But what we need to keep in mind is that generative AI is still new and still evolving, and security will need to keep pace,” Pandey says. “This will be a journey.”
IP security and the opaque box
The sophisticated outputs of generative AI are the result of those vast amounts of data the algorithm ingests, as well as the intricacy of those algorithms. Of course, the user at the other end of the prompt has no hope of tracing the decision pathway of any response they get, or discovering where the data came from. That leads directly to one of the bigger risks of using off-the-shelf models, whether that’s OpenAI or open-source models like Hugging Face or Mistral AI: the exposure of IP. The problem is both external (the data the model’s been trained on) and coming from inside the house (the data your users input).
"The question becomes, how can I secure my access to data, while protecting IP or any sensitive information from leaving the organization and making its way into that opaque box, and vice versa."
“The question becomes, how can I secure my access to data, while protecting IP or any sensitive information from leaving the organization and making its way into that opaque box, and vice versa,” Pandey says. “Am I inadvertently using some open source or some licensed content that made its way into that model that I should not be using without appropriate licenses in place? It's a two-way street.”
There’s also an issue with both recency and specificity, neither of which is baked into standard off-the-shelf foundation models -- the data is only as fresh as the date stamped on the box, and they aren’t tuned to your business environment or business use cases. That can directly impact retrieval-augmented generation (RAG), currently the number one way to add secure, up-to-the-moment business context -- and provide citations for users to verify and evaluate the model’s output.
Unlike pre-trained LLMs, RAG lets a model learn over time, minimizes or reduces hallucination, and keeps private data and IP internal.
RAG integrates information retrieval (or searching) into LLM text generation. It can both gather up real-time context and user-specific information and dip into a proprietary vector database or operational feature store, which can be standalone or embedded in the LLM application. Unlike pre-trained LLMs, RAG lets a model learn over time, minimizes or reduces hallucination, and keeps private data and IP internal.
“It's like asking a generalist to learn more about quantum physics and sending them to the library,” Pandey explains. “That process of going to the library, extracting those books and ingesting the information is basically RAG. There are challenges and it takes experimentation -- and it will change over time. But right now, it's a defacto way of customizing foundation models to an organization's use case without exposing your data.”
Protecting users and building for the future
“There is a distinct boundary drawn around generative AI use cases right now, but in a few years, that boundary will become blurred, and it will become pervasive in everything we build or consume,” Pandey says. “And that means when you’re looking to build for the future, you need to take on a zero-trust approach.”
Assume that anything and everything can go wrong in your entire pipeline: the data itself, the model from production to deployment, and unauthorized access to the data, and the final application.
That means assume that anything and everything can go wrong in your entire pipeline: the data itself, the model from production to deployment, and unauthorized access to the data, and the final application.
And there’s the human aspect: This is a brand-new technology, and the rules have not yet been set in stone. You might assume that everyone who touches the model has the potential to make mistakes, but there’s an increased possibility of inadvertently overlooking a vulnerability because it had not been an issue in the past. Documentation is key here, for forensic purposes should a violation occur, and to help prioritize security measures.
“Make sure that you are capturing the intent of what you want to do: cataloging the data sources, cataloging the models that are being used in production and that are being used to train and iterate on this pipeline,” Pandey explains. “Then catalogue the applications themselves, and categorize these applications based on how critical they are, and make sure that your security policies and your guardrails actually match that criticality.”
Of course, that security must be present at every layer of the stack all the way down to the infrastructure, because if one system or layer fails, defense in depth comes to the rescue -- and again, he emphasizes, the security process will be a journey.
"The way to tackle security in a generative AI environment is through stochastic processes -- building AI models that handle security in other models."
“This is where we need to invent new techniques,” he says. “If the attacks themselves are shifting towards compromising the pipeline, generating biased information or wrong decisions, the level of difficulty has shot skyward. The way to tackle security in a generative AI environment is through stochastic processes -- building AI models that handle security in other models, flagging issues in generated content when things go haywire.”
Generative AI and importance of user trust
Trust is a massive business KPI, Pandey says. Security directly impacts user trust, user experience and the success of a generative AI solution. If a consumer or a customer loses trust in your business, it has a profound revenue impact.
"An insecure application is an unavailable application, essentially nonexistent, and therefore any monetization or any business KPI from that application is a moot point."
“An insecure application is an unavailable application, essentially nonexistent, and therefore any monetization or any business KPI from that application is a moot point,” he explains. “The same thing can be said about a generative AI pipeline, model or application. If you have an insecure model, then it's an unavailable model -- you can't really build a business around it and monetize that model.”
On the flip side, if a consumer learns that their private data has been exposed or used inappropriately, which can happen in brand new ways in a gen AI world, their trust immediately vanishes forever -- and consumer trust in AI is already fairly shaky.
These two vectors that are happening at the same time, and coming to terms with the risks and finding solutions is going to take time.
“Assume a zero-trust mindset, build defense in depth, and assume that you need to consider the risks of an opaque box, as well as access to the internal pipeline itself, all the way from the data to the app itself,” he says. “And remember where you start today and where you'll be three years down the line is going to be very different because both the generative AI vector, as well as the security around it, are both evolving rapidly.”
Sponsored articles are content produced by a company that is either paying for the post or has a business relationship with VentureBeat, and they’re always clearly marked. For more information, contact sales@venturebeat.com