Skip to content
me.lasindu.com
All writing

Running a local LLM with Ollama and Open WebUI

, 6 min read

Running a language model on your own hardware keeps your prompts on your own machine, has no per-token cost and works offline. The trade-off is that a small local model won't match the best hosted models, and its speed depends entirely on your hardware.

Ollama is the easiest place to start. It downloads and runs models with a single command, and it provides an API that your own code can call. Open WebUI adds a browser-based chat interface.

This guide works on almost any PC, with or without a dedicated GPU. There's a section at the end for running it on a TrueNAS server.

What your hardware can run

Memory is what limits you most. A model needs to fit in memory to run at a usable speed, and the fastest place for it is your GPU's video memory (VRAM).

  • Dedicated GPU: the best experience. Ollama supports NVIDIA cards with compute capability 5.0 or newer, and AMD cards through ROCm v7 or Vulkan.
  • Apple Silicon: works well, using Metal. The GPU shares the system's memory, so a Mac with plenty of memory can run larger models.
  • CPU only: works, but it's much slower. It's fine for small models, short prompts and experiments, but less so for long conversations.

If a model doesn't fit entirely in VRAM, Ollama splits it between the GPU and CPU. It still runs, but speed drops sharply.

Install Ollama

On Windows and macOS, use the installer. On Linux, run the install script. This pipes a script from the internet into your shell, so you may want to read it first.

curl -fsSL https://ollama.com/install.sh | sh

Then download and start your first model:

ollama run qwen3:4b

The first run downloads about 2.5 GB. Once you see a prompt, ask it something, and type /bye when you're done.

Choose a model

As a rough guide, leave a few GB of free memory on top of the model's download size. A model also needs more memory as its context window grows. These are reasonable places to start, with sizes from the Ollama library:

  • Up to 8 GB of memory: qwen3:4b (2.5 GB), or gemma3:1b (806 MB) for very limited machines.
  • 8 to 12 GB: qwen3:8b (5.2 GB).
  • 12 to 16 GB: gemma4:12b (about 8 GB).
  • 16 to 24 GB: gpt-oss:20b (14 GB).
  • 24 GB or more: gemma4:26b (16 to 19 GB) or qwen3:30b (19 GB).

New models arrive often, so check the library for current options. Each tag page lists its size, and licences vary between models, so check that too if you're building something commercial.

Avoid tags ending in -cloud if you want everything to stay local, because those run on Ollama's servers. Setting OLLAMA_NO_CLOUD=1 turns off cloud models and web search altogether.

Check that it's fast enough

Two commands tell you most of what you need to know. While a model is loaded, run:

ollama ps

The PROCESSOR column shows where it's running. 100% GPU is what you want, and a split like 48%/52% CPU/GPU means the model doesn't fit in VRAM. Try a smaller model or a smaller context window if it's too slow.

To measure speed, add --verbose:

ollama run qwen3:4b --verbose

After each answer, it prints statistics. The eval rate is how many tokens per second it generated, and you can use it to compare models on your own hardware.

The context window trap

The context window is how much text the model can consider at once, including your prompt, the conversation so far and any documents. Ollama's default depends on your VRAM: 4k tokens below 24 GiB, 32k from 24 to 48 GiB and 256k above that. This applies even if the model's page lists a much bigger window.

4k tokens is fine for short chats, but it's small for long documents or coding, and the model may not have the whole document in view. If answers seem to ignore earlier content, this is the first thing to check.

To raise the limit on Linux, edit the service:

sudo systemctl edit ollama

Then add the following and restart it with sudo systemctl restart ollama:

[Service]
Environment="OLLAMA_CONTEXT_LENGTH=16384"

On the desktop apps, use the context length slider in the settings. Larger windows use more memory, so check ollama ps afterwards to confirm that the model still fits on the GPU.

Add a chat interface

Open WebUI gives you a familiar chat interface, with saved conversations and support for several users. It's easiest to run with Docker. Create a compose.yaml file:

services:
  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    ports:
      - "3000:8080"
    volumes:
      - open-webui:/app/backend/data
    extra_hosts:
      - host.docker.internal:host-gateway
    environment:
      - WEBUI_SECRET_KEY=replace-this
    restart: unless-stopped

volumes:
  open-webui:

Replace the key with the output of openssl rand -hex 32 and keep it the same afterwards, otherwise recreating the container logs everyone out. Start it with docker compose up -d and open http://localhost:3000. The first account you create becomes the administrator, and sign-ups then turn off.

On Linux, one more step is needed. Ollama only listens on 127.0.0.1 by default, which a container can't reach. Add this to the same service override as before, then restart Ollama:

[Service]
Environment="OLLAMA_HOST=0.0.0.0:11434"

This has a security cost. Ollama's API doesn't require a login, so anyone who can reach port 11434 can use your models. Use your firewall to block that port from other machines, and never forward it to the internet. The TrueNAS setup below avoids the problem by not publishing the port at all.

Call it from your own code

Ollama's API runs on port 11434, so you can test it with curl:

curl http://localhost:11434/api/chat -d '{
  "model": "qwen3:4b",
  "messages": [{"role": "user", "content": "Why is the sky blue?"}],
  "stream": false
}'

It also has an OpenAI-compatible endpoint, so many existing tools and libraries work by changing the base URL:

from openai import OpenAI

# The library needs a key, but Ollama ignores its value.
client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")

response = client.chat.completions.create(
    model="qwen3:4b",
    messages=[{"role": "user", "content": "Why is the sky blue?"}],
)
print(response.choices[0].message.content)

Running it on TrueNAS

TrueNAS is an appliance, so don't run the install script on the host. Use containers instead. In this setup, Open WebUI talks to Ollama over their private Docker network, so port 11434 is never published.

Create tank/apps/ollama and tank/apps/open-webui datasets with the Apps preset. Then go to Apps > Discover > Install via YAML, name the app local-llm and paste this:

services:
  ollama:
    image: ollama/ollama
    container_name: ollama
    environment:
      - OLLAMA_NO_CLOUD=1
    volumes:
      - /mnt/tank/apps/ollama:/root/.ollama
    restart: unless-stopped

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: open-webui
    depends_on:
      - ollama
    ports:
      - "3000:8080"
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
      - WEBUI_SECRET_KEY=replace-this
    volumes:
      - /mnt/tank/apps/open-webui:/app/backend/data
    restart: unless-stopped

Once it's running, download a model from System > Shell:

sudo docker exec -it ollama ollama pull qwen3:4b

Then open http://your-server-ip:3000. If you've followed the apps guide, you can also reach it over Tailscale.

Adding a GPU

For an NVIDIA card, go to Apps > Configuration > Settings and turn on Install NVIDIA Drivers. TrueNAS 25.10 only supports Turing and newer cards, which means RTX or GTX 16-series onwards, so many older cards won't work. Get your GPU's ID:

nvidia-smi -L

Then add it to the ollama service, and add its ID to the environment list as well:

    environment:
      - OLLAMA_NO_CLOUD=1
      - NVIDIA_VISIBLE_DEVICES=GPU-your-id
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              capabilities: [gpu]
              device_ids: ["GPU-your-id"]

For an AMD card, use the ollama/ollama:rocm image and pass through /dev/kfd and /dev/dri as devices. Run sudo docker exec ollama ollama ps with a model loaded to confirm that it's using the GPU.

Sharing a server with your NAS

Models compete for memory with ZFS and your other apps. By default, Ollama keeps a model loaded for 5 minutes after its last use, then releases the memory. On a CPU-only NAS, stick to small models and expect slower replies, and check that your other services stay responsive while a model is running.

Model sizes and setup details were checked against the Ollama, Open WebUI and TrueNAS documentation in November 2026. Local models move quickly, so check the Ollama library for newer options.