basforge

Search titles, text and the “If it doesn’t work” cases.

    RU

    A model on your own hardware: Ollama, an API on port 11434 and a web UI

    Ollama downloads an open model, runs it on a GPU or CPU and serves an API on port 11434 — including one compatible with OpenAI. Next to it goes Open WebUI, a chat in the browser. Your data goes nowhere, but the model is weaker than cloud flagships and is limited by memory. Ollama has no password: don’t open the port to the outside.

    What you end up with

    /opt/ollama/compose.yml
      ollama      ollama/ollama:0.40.6        127.0.0.1:11434   models in the ollama volume
      open-webui  ghcr.io/open-webui/…       127.0.0.1:3000    chat, users, document upload
    browser ──► Traefik (HTTPS + login) ──► open-webui ──► ollama ──► GPU

    Without a GPU, Ollama runs on the CPU — an 8B model at Q4 gives on the order of 10 tokens per second on dual-channel DDR5 (How much memory a model needs): fine for reading, not for agent work.

    Steps

    1. Pick a model for your memory

      To try it out, a computer without a GPU, with 8 GB of RAM and 20 GB of free disk, is enough: the ollama/ollama and Open WebUI images are 3.8 and 1.6 GB compressed, more once unpacked. Models up to 4B fit: qwen3:1.7b is 1.4 GB, qwen3:4b is 2.6 GB. They answer slowly, but the API and the browser chat work the same as on a big machine.

      For real work you need a GPU. NVIDIA: compute capability 5.0 and up, driver 550 or newer (570 for 5.0–6.2). AMD: the ROCm 7 driver; Ollama runs cards without it through Vulkan. Disk: an SSD — an 8B model is 5.2 GB, 14B is 9.3 GB, 32B is 20 GB, and you usually keep several.

      The whole model has to fit in VRAM along with the context. A rough guide (details — How much memory a model needs):

      VRAM What fits at Q4 with an 8–16K context
      8 GB 7–8B
      12–16 GB 12–14B
      24 GB 27–32B
      48 GB and up 70B

      On a Mac with Apple Silicon, VRAM is shared with the system: with 32 GB, about 20–24 GB is available for a model.

      Current models and their sizes are at ollama.com/library. The examples below use qwen3:8b (5.2 GB at Q4, a window of up to 40K).

    2. Install Ollama and Open WebUI in Docker

      On a server with NVIDIA, install the NVIDIA Container Toolkit first, or the container won’t see the GPU:

      docker run --rm --gpus all ubuntu nvidia-smi    # you should see a table with the card
      # /opt/ollama/compose.yml
      services:
        ollama:
          image: ollama/ollama:0.40.6
          restart: unless-stopped
          ports:
            - "127.0.0.1:11434:11434"        # only from this server: Ollama has no password
          environment:
            OLLAMA_CONTEXT_LENGTH: "16384"   # context window; bigger means more memory
            OLLAMA_KEEP_ALIVE: "30m"         # how long to keep the model in memory after a request
          volumes:
            - ollama:/root/.ollama           # downloaded models, gigabytes
          deploy:
            resources:
              reservations:
                devices:
                  - driver: nvidia
                    count: all
                    capabilities: [gpu]
      
        open-webui:
          image: ghcr.io/open-webui/open-webui:main
          restart: unless-stopped
          ports:
            - "127.0.0.1:3000:8080"
          environment:
            OLLAMA_BASE_URL: "http://ollama:11434"
          volumes:
            - open-webui:/app/backend/data
          depends_on: [ollama]
      
      volumes:
        ollama:
        open-webui:
      cd /opt/ollama && docker compose up -d

      On your own computer without Docker: the installer from ollama.com for Windows and macOS; on Linux, the install.sh script from the same place, which sets up an ollama systemd service. Read the script first, then run it.

    3. Download a model and check the speed

      docker compose exec ollama ollama pull qwen3:8b
      docker compose exec ollama ollama run qwen3:8b --verbose "How does docker compose down differ from stop?"
      # …answer…
      # eval rate:            … tokens/s
      docker compose exec ollama ollama ps

      eval rate is the generation speed: from 20 tokens per second it reads comfortably; an agent needs 50 or more. In ollama ps, the PROCESSOR column: 100% GPU means the model is entirely in VRAM; something like 40%/60% CPU/GPU means it didn’t fit, and speed drops several times.

      Command What it does
      ollama list downloaded models and their sizes
      ollama ps what’s in memory now and where: GPU or CPU
      ollama show qwen3:8b parameters, quant, window, license
      ollama rm qwen3:8b delete a model from disk
      ollama stop qwen3:8b unload it from memory right away
    4. Call the API

      Ollama’s own API:

      curl -s http://127.0.0.1:11434/api/chat -d '{
        "model": "qwen3:8b",
        "messages": [{"role": "user", "content": "Two lines: what is swap?"}],
        "stream": false,
        "options": {"num_ctx": 16384}
      }' | jq -r '.message.content'

      The OpenAI-compatible one is http://127.0.0.1:11434/v1/chat/completions. Any program that works with OpenAI and lets you change the address (base_url) works with the local model without code changes; the key can be any non-empty string.

    5. Open the web UI through Traefik

      Open WebUI listens on 127.0.0.1:3000. To the outside, only through Traefik with HTTPS (Traefik from scratch): the proxy network, router labels, port 8080 on the service. The first user to register becomes the administrator — register right after launch, then turn off open registration in the settings.

    How to check

    From another computer, against the server’s public address:

    curl -m 5 http://203.0.113.10:11434/api/tags
    # curl: (28) … timed out  or  curl: (7) Failed to connect …  — closed, as it should be

    If JSON with a list of models comes back, the port is open to everyone — go back to step 2. Then on the server itself, curl -s http://127.0.0.1:11434/api/tags | jq '.models[].name' should show qwen3:8b.

    If it doesn’t work

    It answers at one word per second

    Cause
    The model didn’t fit in VRAM and partly runs on the CPU: ollama ps shows CPU/GPU instead of 100% GPU.
    Fix
    A smaller model or a lower quant (How much memory a model needs), a smaller OLLAMA_CONTEXT_LENGTH; close browsers and games that hold VRAM.

    A long document — the model “forgot” the beginning

    Cause
    Ollama’s context window is a setting and by default is smaller than the model can handle; whatever doesn’t fit is silently cut. Ollama picks the window by VRAM: under 24 GiB — 4K, 24–48 GiB — 32K, more — 256K.
    Fix
    Raise OLLAMA_CONTEXT_LENGTH or num_ctx in the request; memory for the context grows with it (How much memory a model needs).

    The container doesn’t see the GPU, everything runs on the CPU

    Cause
    The NVIDIA Container Toolkit isn’t installed, or compose lacks the deploy.resources.reservations.devices block.
    Fix
    Install the toolkit, check with docker run --rm --gpus all ubuntu nvidia-smi, and add the block from step 2.