Your app processes medical records, client contracts, or employee HR data. The model provider cannot store those prompts, train on them, or route them through servers in a jurisdiction you haven't approved. That requirement is not theoretical: it's the first question your legal team will ask before sign-off, and the answer determines which models you can actually use.
The short version: this is not a model problem. It is a deployment architecture problem. Four viable architectures exist in 2026, each with a different cost, control, and compliance trade-off.
What "privacy-first" actually means for an LLM
Four compliance requirements tend to appear together but mean different things:
HIPAA BAA: if your app touches protected health information, you need a signed Business Associate Agreement from your AI provider before processing a single prompt. No BAA means no HIPAA. Period.
Zero data retention (ZDR): the provider discards the prompt and output immediately after returning the response. No logs, no 7-day buffer, nothing. Harder to get than a BAA, and not available from every provider for every model.
SOC 2 Type II: an independent auditor verified the provider's security controls over a period of months (not just at a point in time). This matters for enterprise sales and fintech/legal/HR apps.
Data residency: prompts never leave a specific geography, or your own infrastructure. Covers both EU GDPR requirements and use cases where "no cloud at all" is the answer.
Path 1: Managed API with ZDR
The fastest path to compliance without buying hardware. Both OpenAI and Anthropic now offer HIPAA BAAs and zero data retention, but the details differ in ways that affect which model you can actually deploy.
OpenAI API: HIPAA BAAs became available in February 2026. Zero data retention is available for eligible API endpoints and requires case-by-case approval through enterprise sales. One significant carve-out: the web search tool is not HIPAA-eligible unless configured in offline/cache-only mode on a ZDR-enabled project. GPT-5.5 and GPT-4o are both available under these terms once approved. See OpenAI's pricing page{:target="_blank" rel="noopener"} for current token rates.
Anthropic Claude API: data retention dropped to 7 days in September 2025, and ZDR is available for most models. The exception that matters: Claude Fable 5 and Claude Mythos 5 require 30-day retention with no ZDR option. If HIPAA or strict data minimization is a hard requirement, you are limited to earlier models. Claude Sonnet 4.6 and Claude Opus 4.8 are both ZDR-eligible. Anthropic offers HIPAA BAAs at the enterprise tier. See Anthropic's pricing page{:target="_blank" rel="noopener"} for current rates.
Consumer subscriptions (ChatGPT Plus at $20/mo, Claude Pro at $20/mo) carry no HIPAA coverage. Any team member pasting patient data into a consumer chat interface is creating a compliance gap regardless of subscription tier.
Path 2: SOC 2 and FedRAMP cloud
Google Vertex AI / Gemini: if your compliance checklist goes beyond HIPAA to include FedRAMP High, ISO 27001/27017/27018, or SOC 1/2/3, Google Vertex AI is the strongest managed cloud option in 2026. The full certification stack passed reassessment in Q2 2025. Enterprise contracts include HIPAA BAAs and data processing addenda with ZDR-equivalent commitments. Data region controls let you pin processing to US, EU, or other configurable geographies.
Gemini 2.5 Pro on Vertex AI runs roughly $1.25-2.50/M input tokens at standard tier. The consumer tier (Gemini Advanced via Google One AI Premium at $19.99/mo) carries no enterprise compliance contracts and should not be used for regulated data. Verify current Vertex AI pricing at Google's pricing page{:target="_blank" rel="noopener"}.
For healthcare organizations already running on Google Cloud and for government workloads that require FedRAMP, Vertex AI is the clear choice.
Path 3: EU data sovereignty
Mistral AI: the only major frontier model provider headquartered in the EU, with servers in France. That makes it the natural starting point for EU companies with GDPR data residency requirements, where the "model provider's servers" question has a clear EU answer.
Mistral also offers containerized self-hosted deployment: Le Chat Enterprise can be deployed on private infrastructure, private cloud, or Mistral's service with full data control. Mistral OCR 4, released June 2026, ships as a containerized deployment specifically designed for regulated enterprises. Specific SOC 2 or HIPAA certifications are not publicly documented as of June 2026, so if US certification frameworks are a hard requirement, Mistral alone does not satisfy them.
For EU teams that need API access with EU hosting at reasonable cost, Mistral Small 3.1 at $0.20/M input is worth considering alongside Gemini Flash. Our roundup of fast, budget models under $1/M tokens covers it alongside the other options in that tier. See Mistral's pricing page{:target="_blank" rel="noopener"} for the full model lineup.
Path 4: Full self-hosted
When the answer to "where does this data go?" must be "nowhere outside our network," self-hosted is the only real answer. No model provider BAA is needed because no protected data reaches the provider; the model runs on your own hardware.
Our guide to open-source LLMs for on-prem deployment covers the hardware tiers in detail. The short version:
- RTX 4090 or Mac M2 Max (24GB VRAM): Mistral 7B at Q4 quantization (~4GB VRAM), Phi-4 Mini at Q4 (~2GB). Good for classification, extraction, and short-form generation. Hardware cost: $1,500-2,000.
- A100 80GB: Llama 4 Scout (109B MoE, ~17B active parameters) at Q4_K_M fits in 55-61GB. Quality starts to compete with managed API options. Rented H100 at $2-3/hour is the entry point without CapEx.
- Two A100s or H100s: Qwen 2.5 72B (Apache 2.0) at Q4 fits in 40-45GB. DeepSeek V3 (MIT license) at Q4 for frontier-class output with zero data egress.
Inference tooling: Ollama and vLLM are the two standard options. LiteLLM in front of either gives you an OpenAI-compatible endpoint, so your existing application code requires no changes.
Initial setup cost for a mid-sized business runs $10,000-15,000. The break-even point against API pricing depends on volume; our on-prem post works through the exact math.
Comparison
| Provider | ZDR | SOC 2 | HIPAA BAA | On-prem | Data location |
|---|---|---|---|---|---|
| OpenAI API | Yes (approval required) | Yes | Yes (since Feb 2026) | No | US |
| Anthropic API | Yes (most models) | Not published | Yes | No | US |
| Google Vertex AI | Yes (enterprise) | Yes (Type II + FedRAMP) | Yes | No (managed) | Configurable |
| Mistral AI | Yes (self-hosted) | Not published | Not published | Yes | EU |
| Self-hosted OSS | Yes | Your stack | Your responsibility | Yes | Your infrastructure |
Subscription vs API
| Provider | API pricing | Consumer subscription | BAA available |
|---|---|---|---|
| OpenAI | Per-token (GPT-5.5 ~$2.50/M input) | Plus $20/mo, Pro $200/mo | API only, enterprise tier |
| Anthropic | Per-token (Sonnet 4.6 ~$3/M input) | Pro $20/mo, Max $100-200/mo | API only, enterprise tier |
| Per-token (Gemini 2.5 Pro ~$1.25-2.50/M) | Gemini Advanced $19.99/mo | Vertex AI enterprise only | |
| Mistral | Per-token, self-hosted quoted separately | Le Chat Pro ~€14/mo | Not published |
The consumer subscriptions exist for different people than the compliance use case. Enterprise compliance contracts are separate sales conversations with custom pricing. Do not assume the subscription price is the cost: it isn't, and the compliance coverage is also different.
When to choose which path
- US startup needing HIPAA, no infra: OpenAI API with ZDR + BAA, or Anthropic API on Sonnet 4.6 or Opus 4.8. Fastest path to documented compliance.
- Government or defense: Google Vertex AI. FedRAMP High is a hard requirement in most federal procurement and Vertex AI is the only option in this list that satisfies it.
- EU company under GDPR: Mistral self-hosted for maximum data locality, or Google Vertex AI with EU data region configured.
- Maximum control, high prompt volume: self-hosted on your own hardware. The hardware investment pays back when volume makes API billing more expensive than GPU depreciation.
You can test all the managed-API options above through the LLMTest proxy from a single endpoint, useful for benchmarking response quality across providers before signing an enterprise contract. Start free at LLMTest to compare without setting up separate API keys for each provider.