Secure by Design in the AI Era: Why Trustworthy Products Start With Security
Category: AI Security · Secure Software Engineering · Trustworthy Computing
AI has changed what we build. Applications that once had a simple path from user interface to API to database now have large language models (LLMs), autonomous agents, vector databases, plug-in tools and third-party model APIs sitting in the middle. Each new component adds capability. Each one also adds attack surface.
It is evident that in the AI era, security cannot be bolted on at the end. It must be a core feature from the first design decision. Let’s define what a "secure product" is, why AI makes trustworthy computing both harder and more necessary, and a practical checklist product teams can use.
1. What makes a product "secure"? Two sides of the CIA triad
The central focus of security is mostly described with the CIA triad: Confidentiality, Integrity and Availability. For a product or a software application, it helps to apply the triad to two different things.
Applies to | Confidentiality | Integrity | Availability |
Information | Data is seen only by those entitled to see it | Data is not altered, corrupted or poisoned | Data and services can be reached when needed. |
Product resources (models, APIs, compute, databases, tools) | Access is controlled and granted only to authorised users and services | Models, prompts, configurations and pipelines cannot be tampered with | Resources stay up and are not exhausted or abused |
So a secure product does two jobs:
1. It protects the information. It keeps that information confidential, accurate, and available when needed.
2. It controls its own resources. Only authorised users get access, and the integrity and availability of those resources are maintained.
A product that does only one of these is not secure. For instance: if you encrypt customer records perfectly but leave an unauthenticated inference endpoint open, attackers can still run up your compute bill, extract your model's behaviour, or use your system to attack others.
Security is a core requirement, not an extra
When security is a prime feature, it shapes the architecture, the data flows, the access model and the testing strategy. The result is a more robust product. When security is an afterthought, it becomes a patchwork of fixes on a design that was never meant to resist attack. That approach is costly, fragile, and usually incomplete.
Any product that wants to be called secure has to treat security as non-negotiable.
2. Assume the environment is hostile
In the AI era, products include APIs, agents, vector databases, model-serving infrastructure, orchestration frameworks and more. A key design principle follows:
Never assume the environment your product runs in is friendly, appropriate or well-behaved. Assume it is hostile and that someone is actively trying to exploit it.
In practice this means:
Every input is untrusted. This includes user prompts, uploaded documents, retrieved web pages, tool outputs, and even responses from other models.
Every component is a possible pivot point. If an agent can call tools, a compromised agent can misuse those tools.
Every integration widens the blast radius. Third-party models, plug-ins, and data sources bring their own weaknesses with them.
A product designed under this assumption has security built in from the start, as a core feature.
3. The need for trustworthy computing in AI
Adding LLMs, agentic models, and the tools to products and applications has widened the attack landscape a lot. Building an AI-enabled application is no longer the same as building a traditional one. Threats that barely existed a few years ago are now common:
Prompt injection. Direct or indirect instructions that hijack model behaviour, for example, hidden text in a document the model is asked to summarise.
Data leakage. Models revealing sensitive training data, system prompts, or other users' information.
Excessive agency. Agents given more permissions or tool access than the task needs, which turns one manipulated response into real-world actions.
Data and model poisoning. Corrupted training, fine-tuning or retrieval data that quietly changes outputs.
Supply-chain risk. Unverified model weights, datasets, libraries and plug-ins.
Resource abuse. Expensive inference endpoints used for denial-of-service or "denial-of-wallet" attacks.
Before an AI product can be called trustworthy, a lot of work is needed. Trustworthy AI products should be:
Reliable. They behave consistently and as advertised, even under unusual or adversarial input.
Resistant to attack. They are designed to withstand known exploitation techniques, not just to work in the "happy path".
Self-healing. They can detect anomalies, contain damage, and recover, for example by rolling back poisoned data, revoking compromised credentials, or degrading gracefully instead of failing open.
4. A checklist for secure AI products
Here are three questions every product team should be able to answer before, and long after, launch.
Checkbox 1: Can the product be considered a quality product?
A secure product is a quality product, and a quality product is a secure product. Security is part of quality, not something separate from it.
This does not mean the product must be perfect. Perfect security does not exist. It means:
Understand the context of use. Who uses the product, where, with what data, and under what threats? A product is "good enough" only relative to the environment it is expected to work in.
Build that context into the product. Threat models, access policies and safeguards should match the real deployment environment, not an idealised one.
Keep handling exploitation points in production. Security does not stop at release. Monitoring, logging, red-teaming, patching and incident response are what keep a product secure over time.
Checkbox 2: What edge do competitors have on security?
Security is a competitive differentiator.
Customers expect products to work as advertised without leaking their information. Every time they use or launch the product, they expect good service, with no privacy problems and no outages.
User testing is vital. This includes security-focused testing, abuse-case testing and privacy reviews to confirm that these expectations are actually met.
Security incidents make headlines fast. Breaches, data leaks and AI failures are attractive stories for the media, and the public and customers pay attention to them. Competitors can and do use these stories. When a product suffers a public security failure, existing and potential customers may move to a competitor that offers a more secure alternative.
In short, weak security is a technical risk and a market risk. Strong security builds trust, and trust keeps customers.
Checkbox 3: Does the product identify and address its possible points of vulnerability?
Traditional web applications have a well-defined architecture. Requests usually go from the GUI (Graphical User Interface) to the API (Application Programming Interface) to the database. With a path this predictable, it is fairly straightforward to put security controls at each layer: input validation at the interface, authentication and rate limiting at the API, and access control and encryption at the database.
AI components make this architecture much more complex. Inputs are no longer structured form fields. They are free-form natural language, and the system's responses come from probabilistic model inference. The same input may produce different outputs, and an attacker's instructions can be hidden inside content that looks harmless. On top of that, AI systems often:
Retrieve content from external sources (retrieval-augmented generation),
Call tools and APIs on the user's behalf (agents),
Store and search embeddings in vector databases, and
Chain several models together.
Every one of these is a new point of vulnerability. A secure AI product therefore needs to map its full data and control flow, including prompts, retrieved context, tool calls, model outputs and memory, and put controls at each stage. Examples:
Input and context controls. Separate trusted instructions from untrusted content. Filter and label retrieved data.
Output handling. Treat model output as untrusted before it is shown to users, run as code, or passed to another system.
Least privilege for agents. Scope tool permissions narrowly, and require human approval for high-impact actions.
Strong identity and access control. Apply it to model endpoints, vector stores and training pipelines.
Monitoring and anomaly detection. Watch prompts, responses and tool usage.
Rate limiting and quotas. Protect availability and cost.
Supply-chain verification. Check models, datasets and dependencies.
Established frameworks can help structure this work, including the OWASP Top 10 for LLM Applications, the NIST AI Risk Management Framework, and Secure-by-Design principles.
5. Key takeaways
Security comes first. Products designed with security as a core feature are more robust than those that add it later.
Apply CIA to both sides. Protect information and control the product's own resources.
Assume hostility. In the AI era, design for an environment that is actively trying to exploit you.
Trustworthy means reliable, attack-resistant and self-healing.
Secure equals quality. Perfection is impossible, but context-aware, continuously maintained security is achievable.
Security is a market advantage. Customers, the media and competitors all notice when it fails.
AI changes the architecture. Natural-language inputs and probabilistic outputs add new vulnerability points that must be mapped and controlled.
As AI becomes part of every product, the organisations that earn lasting trust will be the ones that treat security as a foundation, not a feature to add later.
Have thoughts on building secure AI systems, or want to collaborate with our group on secure AI systems research?
Comments