What you end up with
/opt/ollama/compose.yml
ollama ollama/ollama:0.40.6 127.0.0.1:11434 models in the ollama volume
open-webui ghcr.io/open-webui/… 127.0.0.1:3000 chat, users, document upload
browser ──► Traefik (HTTPS + login) ──► open-webui ──► ollama ──► GPU
Without a GPU, Ollama runs on the CPU — an 8B model at Q4 gives on the order of 10 tokens per second on dual-channel DDR5 (How much memory a model needs): fine for reading, not for agent work.
Steps
Pick a model for your memory
To try it out, a computer without a GPU, with 8 GB of RAM and 20 GB of free disk, is enough: the
ollama/ollamaand Open WebUI images are 3.8 and 1.6 GB compressed, more once unpacked. Models up to 4B fit:qwen3:1.7bis 1.4 GB,qwen3:4bis 2.6 GB. They answer slowly, but the API and the browser chat work the same as on a big machine.For real work you need a GPU. NVIDIA: compute capability 5.0 and up, driver 550 or newer (570 for 5.0–6.2). AMD: the ROCm 7 driver; Ollama runs cards without it through Vulkan. Disk: an SSD — an 8B model is 5.2 GB, 14B is 9.3 GB, 32B is 20 GB, and you usually keep several.
The whole model has to fit in VRAM along with the context. A rough guide (details — How much memory a model needs):
VRAM What fits at Q4 with an 8–16K context 8 GB 7–8B 12–16 GB 12–14B 24 GB 27–32B 48 GB and up 70B On a Mac with Apple Silicon, VRAM is shared with the system: with 32 GB, about 20–24 GB is available for a model.
Current models and their sizes are at
ollama.com/library. The examples below useqwen3:8b(5.2 GB at Q4, a window of up to 40K).Install Ollama and Open WebUI in Docker
On a server with NVIDIA, install the NVIDIA Container Toolkit first, or the container won’t see the GPU:
docker run --rm --gpus all ubuntu nvidia-smi # you should see a table with the card# /opt/ollama/compose.yml services: ollama: image: ollama/ollama:0.40.6 restart: unless-stopped ports: - "127.0.0.1:11434:11434" # only from this server: Ollama has no password environment: OLLAMA_CONTEXT_LENGTH: "16384" # context window; bigger means more memory OLLAMA_KEEP_ALIVE: "30m" # how long to keep the model in memory after a request volumes: - ollama:/root/.ollama # downloaded models, gigabytes deploy: resources: reservations: devices: - driver: nvidia count: all capabilities: [gpu] open-webui: image: ghcr.io/open-webui/open-webui:main restart: unless-stopped ports: - "127.0.0.1:3000:8080" environment: OLLAMA_BASE_URL: "http://ollama:11434" volumes: - open-webui:/app/backend/data depends_on: [ollama] volumes: ollama: open-webui:cd /opt/ollama && docker compose up -dOn your own computer without Docker: the installer from
ollama.comfor Windows and macOS; on Linux, theinstall.shscript from the same place, which sets up anollamasystemd service. Read the script first, then run it.Download a model and check the speed
docker compose exec ollama ollama pull qwen3:8b docker compose exec ollama ollama run qwen3:8b --verbose "How does docker compose down differ from stop?" # …answer… # eval rate: … tokens/s docker compose exec ollama ollama pseval rateis the generation speed: from 20 tokens per second it reads comfortably; an agent needs 50 or more. Inollama ps, thePROCESSORcolumn:100% GPUmeans the model is entirely in VRAM; something like40%/60% CPU/GPUmeans it didn’t fit, and speed drops several times.Command What it does ollama listdownloaded models and their sizes ollama pswhat’s in memory now and where: GPU or CPU ollama show qwen3:8bparameters, quant, window, license ollama rm qwen3:8bdelete a model from disk ollama stop qwen3:8bunload it from memory right away Call the API
Ollama’s own API:
curl -s http://127.0.0.1:11434/api/chat -d '{ "model": "qwen3:8b", "messages": [{"role": "user", "content": "Two lines: what is swap?"}], "stream": false, "options": {"num_ctx": 16384} }' | jq -r '.message.content'The OpenAI-compatible one is
http://127.0.0.1:11434/v1/chat/completions. Any program that works with OpenAI and lets you change the address (base_url) works with the local model without code changes; the key can be any non-empty string.Open the web UI through Traefik
Open WebUI listens on
127.0.0.1:3000. To the outside, only through Traefik with HTTPS (Traefik from scratch): theproxynetwork, router labels, port8080on the service. The first user to register becomes the administrator — register right after launch, then turn off open registration in the settings.
How to check
From another computer, against the server’s public address:
curl -m 5 http://203.0.113.10:11434/api/tags
# curl: (28) … timed out or curl: (7) Failed to connect … — closed, as it should be
If JSON with a list of models comes back, the port is open to everyone — go back to step 2. Then on the server itself,
curl -s http://127.0.0.1:11434/api/tags | jq '.models[].name' should show qwen3:8b.
If it doesn’t work
It answers at one word per second
- Cause
- The model didn’t fit in VRAM and partly runs on the CPU:
ollama psshowsCPU/GPUinstead of100% GPU. - Fix
- A smaller model or a lower quant (How much memory a model needs), a smaller
OLLAMA_CONTEXT_LENGTH; close browsers and games that hold VRAM.
A long document — the model “forgot” the beginning
- Cause
- Ollama’s context window is a setting and by default is smaller than the model can handle; whatever doesn’t fit is silently cut. Ollama picks the window by VRAM: under 24 GiB — 4K, 24–48 GiB — 32K, more — 256K.
- Fix
- Raise
OLLAMA_CONTEXT_LENGTHornum_ctxin the request; memory for the context grows with it (How much memory a model needs).
The container doesn’t see the GPU, everything runs on the CPU
- Cause
- The NVIDIA Container Toolkit isn’t installed, or compose lacks the
deploy.resources.reservations.devicesblock. - Fix
- Install the toolkit, check with
docker run --rm --gpus all ubuntu nvidia-smi, and add the block from step 2.
Port 11434 is closed in ufw but answers from outside
- Cause
- Docker published the port as
11434:11434— on all interfaces, bypassing theinputchain. - Fix
- Publish on
127.0.0.1:11434:11434(A host-only port goes on 127.0.0.1, Docker ports bypass the input chain).