Trialling
Self-hosted LLMs on Azure
Last reviewed
One Azure VM with an A100 serves open-weight models behind a single OpenAI-compatible API, with no public IP. The setup is simple; keeping it that way means no changes by hand.
How the pieces fit
| Layer | What I use | Why |
|---|---|---|
| Access | Azure Bastion, Entra login on Linux | No public IP on any VM. SSH with Entra accounts, one group per VM, no local users to hand out |
| Compute | One NC A100 v4 GPU VM, one small management VM | The GPU VM only runs containers; people work from the management VM |
| Runtime | Docker Swarm on a single node, Portainer | Stacks, secrets and rolling updates without running Kubernetes for one machine |
| Models | vLLM with gpt-oss-20b and Gemma 4 12B, gpt-5.1 in Azure AI Foundry | Two open-weight models share the one GPU; Foundry adds a hosted model over a private endpoint |
| API | LiteLLM | One OpenAI-compatible endpoint for all models, a personal account and API key per person |
| Operations | Git repo with one stack per service, one pipeline per stack, Key Vault | A commit deploys the stack; the pipeline copies secrets from Key Vault into Docker secrets |
The Azure side (VMs, Key Vault, Foundry, role assignments) is Bicep in Deployment Stacks, promoted from dev to prod through pull requests. The stacks deploy through Azure DevOps Pipelines.
A request takes this path:
- A client calls LiteLLM with its own key, over TLS with a certificate from the internal CA.
- LiteLLM picks the model by name and forwards to the matching vLLM container on the same VM.
- Or it forwards to gpt-5.1 in Foundry through the private endpoint.
Getting in through Bastion
Without a public IP, the way in is a Bastion tunnel from my machine to the management VM, then SSH with a port forward to LiteLLM.
Open the tunnel (Bastion Standard SKU or higher, native client support on):
az network bastion tunnel --name <bastion-name> --resource-group <resource-group> \
--target-resource-id <vm-resource-id> --resource-port 22 --port 2222
In a second terminal, log in with Entra and forward a local port to LiteLLM:
az ssh vm --ip 127.0.0.1 --port 2222 -- -L 8443:llm.example.internal:443
Call the API through the forward, with the real hostname:
curl --resolve llm.example.internal:8443:127.0.0.1 https://llm.example.internal:8443/health/liveliness
Keep the hostname
https://127.0.0.1:8443 fails the certificate check. Use the name on the certificate and point it at 127.0.0.1: --resolve for curl, a hosts entry for the browser. The client also needs the internal root CA.
Where it breaks
- Changes in the Portainer UI. An upgrade clicked in the UI was overwritten by the next pipeline run and ended in a crash loop.
lateston vLLM. Every redeploy can pull an untested version and a 30 GB image. Pin the tag.docker system prune -af --volumesin cron. It deletes every image without a running container. If a service is down for a minute when it runs, the model image is gone.- Docker and ufw. Ports published by Docker bypass ufw. Filter in the
DOCKER-USERchain or in the network firewall. - One GPU, two models. Each vLLM container gets a fixed share of GPU memory (
--gpu-memory-utilization). Adding a third model means taking memory from the others.
Nothing changed by hand on the VM
No docker service update, no UI upgrades, no cron jobs, no edits in /etc. The next pipeline run overwrites it, or nobody knows it exists. The rules live in a repo skill that Claude Code and GitHub Copilot both read.
Links
- Azure Bastion: native client —
az network bastion tunneland SSH from your own terminal. - Entra login for Linux VMs — Roles,
az ssh vm, troubleshooting. - NC A100 v4 series — GPU VM sizes with A100 cards.
- LiteLLM proxy — Gateway config, virtual keys, users and the admin UI.
- vLLM — Serving open-weight models with an OpenAI-compatible server.
- Docker Swarm mode — Stacks, services and secrets on one or more nodes.
- Docker packet filtering and firewalls — Why published ports skip ufw, and
DOCKER-USER. - Portainer docs — Managing Swarm stacks from a web UI.
- Azure AI Foundry private link — Private endpoints for Foundry resources.
- Azure Key Vault — Secrets, keys and access control.
- gpt-oss-20b — Model card and weights.
- Gemma 4 — Model sizes, memory requirements, QAT checkpoints.
History
- 2 Oct 2026 Added as Trialling.