Begin
Introduction
I recently got a Framework Desktop with an AMD Strix Halo and have been playing with local AI. I had bad experiences with the official Docker containers — they crashed regularly — so I decided to build my own.
This article is not a complete guide. It shows how I set up open-webui with llama-swap and a self-built llama.cpp, both on ROCm (Strix Halo) and CPU-only.
Building llama.cpp for rocm
For the GPU build I use .devops/rocm.Dockerfile from the llama.cpp repository. Before it builds, the COPY destination needs a fix so the binaries end up in an explicit directory:
-COPY --from=build /app/full/llama /app/full/llama-cli /app/full/llama-completion /app
+COPY --from=build /app/full/llama /app/full/llama-cli /app/full/llama-completion /app/
WORKDIR /app
@@ -137,7 +138,7 @@ FROM base AS server
ENV LLAMA_ARG_HOST=0.0.0.0
-COPY --from=build /app/full/llama /app/full/llama-server /app
+COPY --from=build /app/full/llama /app/full/llama-server /app/
docker build --target server\
--build-arg ROCM_DOCKER_ARCH=gfx1151 \
-t llama-cpp:rocm-gfx1151 \
-f .devops/rocm.Dockerfile .
Building llama.cpp for Intel E5-2680
For the CPU-only fallback I use .devops/cpu.Dockerfile from the llama.cpp repository. The E5-2680 does not support AVX2, so the build is tuned for ivybridge:
index cb92343d6..daee9ead3 100644
--- a/.devops/cpu.Dockerfile
+++ b/.devops/cpu.Dockerfile
@@ -33,8 +33,16 @@ COPY . .
COPY --from=web /app/tools/ui/dist tools/ui/dist
RUN if [ "$TARGETARCH" = "amd64" ] || [ "$TARGETARCH" = "arm64" ]; then \
- cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=OFF -DLLAMA_BUILD_TESTS=OFF -DGGML_BACKEND_DL=ON -DGGML_CPU_ALL_VARIANTS=ON; \
- else \
+ cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=OFF -DLLAMA_BUILD_TESTS=OFF -DGGML_BACKEND_DL=OFF -DGGML_CPU_ALL_VARIANTS=OFF \
+ -DGGML_AVX=ON \
+ -DGGML_AVX2=OFF \
+ -DGGML_FMA=OFF \
+ -DGGML_F16C=ON \
+ -DGGML_BMI2=OFF \
+ -DGGML_LTO=ON \
+ -DCMAKE_C_FLAGS="-march=ivybridge -mtune=ivybridge" \
+ -DCMAKE_CXX_FLAGS="-march=ivybridge -mtune=ivybridge"; \
+ else \
echo "Unsupported architecture"; \
exit 1; \
fi && \
@@ -104,7 +112,7 @@ ENTRYPOINT ["/app/tools.sh"]
### Light, CLI only
FROM base AS light
-COPY --from=build /app/full/llama /app/full/llama-cli /app/full/llama-completion /app
+COPY --from=build /app/full/llama /app/full/llama-cli /app/full/llama-completion /app/
WORKDIR /app
@@ -115,7 +123,7 @@ FROM base AS server
ENV LLAMA_ARG_HOST=0.0.0.0
-COPY --from=build /app/full/llama /app/full/llama-server /app
+COPY --from=build /app/full/llama /app/full/llama-server /app/
WORKDIR /app
Creating llama-swap starter container
llama-swap spawns a llama.cpp container per model and tears it down again when the model is not used. For that it needs the Docker CLI inside the container, which I add to the official image:
FROM ghcr.io/mostlygeek/llama-swap:v250-rocm-b10450
USER root
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
docker.io \
&& rm -rf /var/lib/apt/lists/*
COPY llama-docker.sh /usr/local/bin/llama-docker
RUN chmod +x /usr/local/bin/llama-docker
services:
llama-swap:
build:
context: ./
dockerfile: Dockerfile
container_name: llama-swap
restart: unless-stopped
volumes:
- ./config.yaml:/config.yaml:ro
- ./llama-docker.sh:/usr/local/bin/llama-docker:ro
- /var/run/docker.sock:/var/run/docker.sock
networks:
- ollama-net
environment:
- DOCKER_HOST=unix:///var/run/docker.sock
command:
- "--config"
- "/config.yaml"
networks:
ollama-net:
external: true
In my setup the ollama-net network is managed by nginx; all services attach to it.
The config.yaml lists every model together with the command llama-swap uses to start it. To keep it in sync with the models on disk I wrote a small generator: it walks the model directories (following symlinks), groups split models, matches mmproj files, and writes the config.yaml that the llama-swap container mounts:
#!/usr/bin/env python3
import argparse
import re
import sys
from pathlib import Path
import yaml
# ============================================================
# Regex
# ============================================================
SPLIT_RE = re.compile(
r"^(?P<base>.+)-(?P<part>\d{5})-of-(?P<total>\d{5})\.gguf$",
re.IGNORECASE,
)
MMPROJ_RE = re.compile(
r"^mmproj(?:[-_.].*)?\.gguf$",
re.IGNORECASE,
)
# ============================================================
# GGUF search
# ============================================================
def find_ggufs(root: Path):
"""
Recursively searches root for GGUF files.
Supports:
model.gguf
model.gguf -> ../../blobs/<hash>
as well as:
qwen -> /mnt/models/qwen
Important:
For symlinks, the name of the symlink counts.
A link like:
Qwen.gguf -> ../../blobs/abc123
is therefore recognized as a GGUF, even though the
target does not have a .gguf extension.
The logical path below the search directory is
always returned.
"""
visited = set()
def walk(physical_dir: Path, logical_dir: Path):
try:
real_dir = physical_dir.resolve(strict=True)
except OSError as exc:
print(
f"WARNING: Cannot resolve directory: "
f"{physical_dir}: {exc}",
file=sys.stderr,
)
return
# Prevent symlink loops.
try:
stat = real_dir.stat()
directory_id = (stat.st_dev, stat.st_ino)
except OSError:
directory_id = str(real_dir)
if directory_id in visited:
return
visited.add(directory_id)
try:
entries = list(physical_dir.iterdir())
except OSError as exc:
print(
f"WARNING: Cannot read directory: "
f"{physical_dir}: {exc}",
file=sys.stderr,
)
return
for entry in entries:
logical_entry = logical_dir / entry.name
try:
# ------------------------------------------------
# Symlink
# ------------------------------------------------
if entry.is_symlink():
# A symlink with .gguf in its name is
# treated as a GGUF.
if entry.name.lower().endswith(".gguf"):
try:
target = entry.resolve(strict=True)
if not target.is_file():
print(
f"WARNING: GGUF symlink does not "
f"point to a file: {entry}",
file=sys.stderr,
)
continue
except OSError as exc:
print(
f"WARNING: Broken GGUF symlink: "
f"{entry}: {exc}",
file=sys.stderr,
)
continue
yield logical_entry
continue
# Follow symlinks to directories.
try:
target = entry.resolve(strict=True)
if target.is_dir():
yield from walk(
target,
logical_entry,
)
except OSError as exc:
print(
f"WARNING: Cannot follow symlink "
f"{entry}: {exc}",
file=sys.stderr,
)
continue
# ------------------------------------------------
# Regular directory
# ------------------------------------------------
if entry.is_dir():
yield from walk(
entry,
logical_entry,
)
continue
# ------------------------------------------------
# Regular GGUF file
# ------------------------------------------------
if (
entry.is_file()
and entry.name.lower().endswith(".gguf")
):
yield logical_entry
except OSError as exc:
print(
f"WARNING: Cannot process {entry}: "
f"{exc}",
file=sys.stderr,
)
yield from walk(root, root)
# ============================================================
# Determine file type
# ============================================================
def parse_gguf(path: Path):
"""
Determines whether a GGUF file is:
- a main model
- a split main model
- an mmproj
"""
name = path.name
# Check mmproj first
if MMPROJ_RE.match(name):
return {
"type": "mmproj",
"model_id": None,
"part": None,
"total": None,
}
# Split model
match = SPLIT_RE.match(name)
if match:
return {
"type": "model_split",
"model_id": match.group("base"),
"part": int(match.group("part")),
"total": int(match.group("total")),
}
# Regular model
return {
"type": "model",
"model_id": path.stem,
"part": None,
"total": None,
}
# ============================================================
# Collect models
# ============================================================
def collect_files(roots):
"""
Collects all logical GGUF paths first.
"""
files = []
for root in roots:
if not root.exists():
print(
f"ERROR: Directory does not exist: {root}",
file=sys.stderr,
)
continue
if not root.is_dir():
print(
f"ERROR: Not a directory: {root}",
file=sys.stderr,
)
continue
print(f"Searching in: {root}")
for path in find_ggufs(root):
files.append(path)
return files
def collect_models(files):
"""
Groups main models and split models.
mmproj files are collected separately.
"""
models = {}
mmprojs = []
for path in files:
info = parse_gguf(path)
# --------------------------------------------------------
# mmproj
# --------------------------------------------------------
if info["type"] == "mmproj":
mmprojs.append(path)
continue
model_id = info["model_id"]
# --------------------------------------------------------
# Regular model
# --------------------------------------------------------
if info["type"] == "model":
entry = models.setdefault(
model_id,
{
"type": "model",
"files": [],
},
)
entry["files"].append(path)
# --------------------------------------------------------
# Split model
# --------------------------------------------------------
elif info["type"] == "model_split":
entry = models.setdefault(
model_id,
{
"type": "split",
"total": info["total"],
"parts": {},
},
)
# Different split count
if entry["total"] != info["total"]:
entry.setdefault(
"errors",
[],
).append(
(
"different split count: "
f"{entry['total']} vs "
f"{info['total']}"
)
)
part = info["part"]
if part in entry["parts"]:
entry.setdefault(
"duplicates",
[],
).append(
(
part,
entry["parts"][part],
path,
)
)
else:
entry["parts"][part] = path
return models, mmprojs
# ============================================================
# mmproj assignment
# ============================================================
def find_mmproj_for_model(model_path: Path, mmprojs):
"""
Tries to assign an mmproj to a model unambiguously.
Strategy:
1. mmproj in the same directory
2. If exactly one mmproj exists there:
-> use it
3. If several exist:
-> try to match by model name
4. If still ambiguous:
-> None + warning
"""
model_dir = model_path.parent
candidates = [
path
for path in mmprojs
if path.parent == model_dir
]
if not candidates:
return None
# --------------------------------------------------------
# Exactly one mmproj
# --------------------------------------------------------
if len(candidates) == 1:
return candidates[0]
# --------------------------------------------------------
# Multiple mmproj files:
# Try to match by model name.
# --------------------------------------------------------
model_name = model_path.stem.lower()
scored = []
for candidate in candidates:
candidate_name = candidate.stem.lower()
score = 0
# Common form:
#
# Qwen2.5-VL-7B-mmproj-BF16.gguf
#
# or:
#
# Qwen2.5-VL-7B-mmproj-F16.gguf
if "mmproj" in candidate_name:
prefix = candidate_name.split("mmproj", 1)[0]
prefix = prefix.rstrip("-_.")
if prefix and prefix in model_name:
score += 100
# Model name inside the mmproj name
if model_name in candidate_name:
score += 100
if score > 0:
scored.append(
(
score,
candidate,
)
)
if len(scored) == 1:
return scored[0][1]
if len(scored) > 1:
scored.sort(
key=lambda x: x[0],
reverse=True,
)
# Unambiguous best match
if (
len(scored) == 1
or scored[0][0] > scored[1][0]
):
return scored[0][1]
print(
f"\nWARNING: Multiple mmproj files for "
f"{model_path}:",
file=sys.stderr,
)
for candidate in candidates:
print(
f" {candidate}",
file=sys.stderr,
)
print(
" -> No automatic assignment.",
file=sys.stderr,
)
return None
def attach_mmprojs(models, mmprojs):
"""
Assigns mmproj files to the main models.
"""
used_mmprojs = set()
for model_id, model in models.items():
# --------------------------------------------------------
# Normal
# --------------------------------------------------------
if model["type"] == "model":
files = model["files"]
if not files:
continue
# If a model is found multiple times, use the first one.
model_path = files[0]
mmproj = find_mmproj_for_model(
model_path,
mmprojs,
)
if mmproj:
model["mmproj"] = mmproj
used_mmprojs.add(mmproj)
# --------------------------------------------------------
# Split
# --------------------------------------------------------
elif model["type"] == "split":
parts = model["parts"]
if not parts:
continue
first_part = min(parts)
model_path = parts[first_part]
mmproj = find_mmproj_for_model(
model_path,
mmprojs,
)
if mmproj:
model["mmproj"] = mmproj
used_mmprojs.add(mmproj)
# --------------------------------------------------------
# Report unassigned mmproj files
# --------------------------------------------------------
for mmproj in mmprojs:
if mmproj not in used_mmprojs:
print(
f"\nWARNING: unassigned mmproj:",
file=sys.stderr,
)
print(
f" {mmproj}",
file=sys.stderr,
)
# ============================================================
# Validate models
# ============================================================
def validate_models(models):
"""
Validates main and split models.
"""
result = []
for model_id in sorted(
models,
key=str.lower,
):
model = models[model_id]
# --------------------------------------------------------
# Regular model
# --------------------------------------------------------
if model["type"] == "model":
files = model["files"]
if len(files) > 1:
print(
f"\nWARNING: Model found multiple times: "
f"{model_id}",
file=sys.stderr,
)
for file in files:
print(
f" {file}",
file=sys.stderr,
)
result.append(
{
"model_id": model_id,
"path": files[0],
"type": "model",
"mmproj": model.get("mmproj"),
}
)
# --------------------------------------------------------
# Split model
# --------------------------------------------------------
elif model["type"] == "split":
total = model["total"]
parts = model["parts"]
expected = set(
range(1, total + 1)
)
found = set(parts)
missing = sorted(
expected - found
)
if missing:
print(
f"\nWARNING: Incomplete "
f"split model: {model_id}",
file=sys.stderr,
)
print(
f" Expected: 1-{total}",
file=sys.stderr,
)
print(
f" Missing: "
f"{', '.join(map(str, missing))}",
file=sys.stderr,
)
if model.get("errors"):
for error in model["errors"]:
print(
f"\nWARNING: {model_id}: {error}",
file=sys.stderr,
)
if model.get("duplicates"):
for part, first, second in model["duplicates"]:
print(
f"\nWARNING: Duplicate split file "
f"{model_id}, part {part}:",
file=sys.stderr,
)
print(
f" {first}",
file=sys.stderr,
)
print(
f" {second}",
file=sys.stderr,
)
if not parts:
continue
# llama.cpp gets the first split.
first_part = min(parts)
result.append(
{
"model_id": model_id,
"path": parts[first_part],
"type": "split",
"parts": len(parts),
"total": total,
"mmproj": model.get("mmproj"),
}
)
return result
def make_container_name(model_name: str) -> str:
name = model_name
if name.lower().endswith(".gguf"):
name = name[:-5]
# Docker-compatible name
name = re.sub(r"[^a-zA-Z0-9_.-]", "-", name)
return f"llama-{name}"
# ============================================================
# Container paths
# ============================================================
def container_path(path: Path, roots):
"""
Converts a logical host path to a container path.
Example:
/srv/llama/models/qwen/model.gguf
->
/models/qwen/model.gguf
"""
path = path.absolute()
candidates = []
for root in roots:
root = root.absolute()
try:
relative = path.relative_to(root)
candidates.append(
(
len(root.parts),
relative,
)
)
except ValueError:
continue
if not candidates:
raise ValueError(
f"File is not under a search directory:\n"
f" File: {path}\n"
f" Search directories:\n"
+ "\n".join(
f" {root}"
for root in roots
)
)
_, relative = max(
candidates,
key=lambda x: x[0],
)
return Path("/models") / relative
# ============================================================
# Unique model IDs
# ============================================================
def make_unique_model_ids(models):
"""
Prevents duplicate IDs in the YAML.
"""
used = {}
for model in models:
model_id = model["model_id"]
if model_id not in used:
used[model_id] = 1
model["config_id"] = model_id
else:
used[model_id] += 1
model["config_id"] = (
f"{model_id}-{used[model_id]}"
)
# ============================================================
# llama-swap Config
# ============================================================
def create_config(models, roots):
config = {
"models": {}
}
for model in models:
model_id = model["config_id"]
model_path = container_path(
model["path"],
roots,
)
command = (
"/usr/local/bin/llama-docker "
f"'{model_path}'"
)
# --------------------------------------------------------
# mmproj
# --------------------------------------------------------
if model.get("mmproj"):
mmproj_path = container_path(
model["mmproj"],
roots,
)
command += (
f" --mmproj '{mmproj_path}'"
)
# command += " ${PORT}"
config["models"][model_id] = {
"cmd": command,
"proxy": f"http://{model_id}:8080",
"cmdStop": f"docker stop {model_id}"
}
return config
# ============================================================
# Summary
# ============================================================
def print_summary(models, roots):
print()
print("=" * 80)
print(
f"Found models: {len(models)}"
)
print("=" * 80)
for model in models:
model_path = container_path(
model["path"],
roots,
)
if model["type"] == "split":
split_info = (
f" [split "
f"{model['parts']}/{model['total']}]"
)
else:
split_info = ""
print()
print(
f"{model['config_id']}{split_info}"
)
print(
f" Model: {model['path']}"
)
print(
f" -> {model_path}"
)
if model.get("mmproj"):
mmproj_path = container_path(
model["mmproj"],
roots,
)
print(
f" mmproj: {model['mmproj']}"
)
print(
f" -> {mmproj_path}"
)
else:
print(
" mmproj: -"
)
print()
print("=" * 80)
# ============================================================
# Main
# ============================================================
def main():
parser = argparse.ArgumentParser(
description=(
"Generates a llama-swap config.yaml "
"from GGUF models. "
"Supports symlinks, blob files, "
"split GGUFs, and mmproj."
)
)
parser.add_argument(
"directories",
nargs="+",
metavar="DIR",
help=(
"Directories to search. "
"Symlinks are followed."
),
)
parser.add_argument(
"-o",
"--output",
default="config.yaml",
metavar="FILE",
help=(
"Ausgabedatei "
"(Standard: config.yaml)"
),
)
parser.add_argument(
"--dry-run",
action="store_true",
help=(
"Only search and display, "
"do not write the config."
),
)
args = parser.parse_args()
roots = [
Path(directory)
.expanduser()
.absolute()
for directory in args.directories
]
print("GGUF Model Scanner")
print("-" * 80)
# --------------------------------------------------------
# Search for files
# --------------------------------------------------------
files = collect_files(roots)
if not files:
print(
"\nNo GGUF files found.",
file=sys.stderr,
)
sys.exit(1)
# --------------------------------------------------------
# Separate models and mmproj
# --------------------------------------------------------
raw_models, mmprojs = collect_models(
files
)
# --------------------------------------------------------
# Assign mmprojs
# --------------------------------------------------------
attach_mmprojs(
raw_models,
mmprojs,
)
# --------------------------------------------------------
# Validate models
# --------------------------------------------------------
models = validate_models(
raw_models
)
if not models:
print(
"\nNo valid models found.",
file=sys.stderr,
)
sys.exit(1)
make_unique_model_ids(
models
)
# --------------------------------------------------------
# Output
# --------------------------------------------------------
print_summary(
models,
roots,
)
if args.dry_run:
print(
"Dry run: config.yaml was "
"not written."
)
return
# --------------------------------------------------------
# YAML
# --------------------------------------------------------
config = create_config(
models,
roots,
)
output = (
Path(args.output)
.expanduser()
)
try:
with output.open(
"w",
encoding="utf-8",
) as file:
yaml.safe_dump(
config,
file,
allow_unicode=True,
sort_keys=False,
default_flow_style=False,
)
except OSError as exc:
print(
f"ERROR: Could not write {output}: "
f"{exc}",
file=sys.stderr,
)
sys.exit(1)
print(
f"Config written: {output}"
)
if __name__ == "__main__":
main()
llama-docker.sh
llama-swap calls this script for every model. It spawns a llama.cpp container with the correct model file (and the mmproj, if there is one) and tears it down afterwards:
#!/bin/sh
set -eu
# ------------------------------------------------------------
# Arguments
#
# Usage:
#
# llama-docker MODEL [PORT]
# llama-docker MODEL --mmproj MMPROJ [PORT]
# ------------------------------------------------------------
usage() {
echo "Usage:" >&2
echo " $0 MODEL [PORT]" >&2
echo " $0 MODEL --mmproj MMPROJ [PORT]" >&2
}
if [ "$#" -lt 1 ]; then
usage
exit 1
fi
MODEL="$1"
shift
MMPROJ=""
PORT="8080"
if [ "$#" -ge 1 ] && [ "$1" = "--mmproj" ]; then
# Remaining arguments: --mmproj MMPROJ [PORT]
if [ "$#" -lt 2 ]; then
echo "ERROR: --mmproj requires a path." >&2
exit 1
fi
MMPROJ="$2"
shift
shift
if [ "$#" -ge 1 ]; then
PORT="$1"
shift
fi
elif [ "$#" -ge 1 ]; then
PORT="$1"
shift
fi
if [ "$#" -ne 0 ]; then
echo "ERROR: Unexpected arguments: $*" >&2
exit 1
fi
# ------------------------------------------------------------
# Validation
# ------------------------------------------------------------
if [ -z "$MODEL" ]; then
echo "ERROR: No model specified." >&2
exit 1
fi
case "$PORT" in
''|*[!0-9]*)
echo "ERROR: Port must be a number: $PORT" >&2
exit 1
;;
esac
# ------------------------------------------------------------
# Container name
# ------------------------------------------------------------
NAME="$(basename "$MODEL")"
NAME="${NAME%.gguf}"
# Docker container names must not contain
# problematic characters.
NAME="$(printf '%s' "$NAME" | sed -E 's/-[0-9]{5}-of-[0-9]{5}$//')"
CONTAINER_NAME="${NAME}"
# ------------------------------------------------------------
# Output
# ------------------------------------------------------------
echo "Starting llama.cpp:"
echo " Model: $MODEL"
if [ -n "$MMPROJ" ]; then
echo " mmproj: $MMPROJ"
fi
echo " Port: $PORT"
echo " Container: $CONTAINER_NAME"
echo
# ------------------------------------------------------------
# Docker
# ------------------------------------------------------------
if [ -n "$MMPROJ" ]; then
exec docker run \
--rm \
--name "$CONTAINER_NAME" \
--network ollama-net \
--device=/dev/kfd \
--device=/dev/dri \
--ipc=host \
-v /data/llama/:/models:ro \
llama-cpp:rocm-gfx1151 \
-m "$MODEL" \
--mmproj "$MMPROJ" \
-ngl 999 \
--host 0.0.0.0 \
--port "$PORT"
else
exec docker run \
--rm \
--name "$CONTAINER_NAME" \
--network ollama-net \
--device=/dev/kfd \
--device=/dev/dri \
--ipc=host \
-v /data/llama/:/models:ro \
llama-cpp:rocm-gfx1151 \
-m "$MODEL" \
-ngl 999 \
--host 0.0.0.0 \
--port "$PORT"
fi
For CPU-only setups the script can be used without the /dev/kfd and /dev/dri passthrough — it just targets the CPU image instead:
@@ -96,11 +96,9 @@ if [ -n "$MMPROJ" ]; then
--rm \
--name "$CONTAINER_NAME" \
--network ollama-net \
- --device=/dev/kfd \
- --device=/dev/dri \
--ipc=host \
-v /data/llama/:/models:ro \
- llama-cpp:rocm-gfx1151 \
+ llama-cpp:cpu \
-m "$MODEL" \
--mmproj "$MMPROJ" \
-ngl 999 \
@@ -114,11 +112,9 @@ else
--rm \
--name "$CONTAINER_NAME" \
--network ollama-net \
- --device=/dev/kfd \
- --device=/dev/dri \
--ipc=host \
-v /data/llama/:/models:ro \
- llama-cpp:rocm-gfx1151 \
+ llama-cpp:cpu \
-m "$MODEL" \
-ngl 999 \
--host 0.0.0.0 \
The corresponding starter container is built from the CPU image:
FROM ghcr.io/mostlygeek/llama-swap:cpu
USER root
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
docker.io \
&& rm -rf /var/lib/apt/lists/*
COPY llama-docker.sh /usr/local/bin/llama-docker
RUN chmod +x /usr/local/bin/llama-docker
Build and start the container:
docker-compose build
docker-compose up
NGINX for reverse proxy (optional for direct access)
In my setup nginx also fronts llama-swap, so the web UI is reachable from the LAN with TLS instead of hitting port 8080 directly. Add the following server block to default.conf:
server {
keepalive_timeout 1d;
send_timeout 1d;
client_body_timeout 1d;
client_header_timeout 1d;
proxy_connect_timeout 1d;
proxy_read_timeout 1d;
proxy_send_timeout 1d;
fastcgi_connect_timeout 1d;
fastcgi_read_timeout 1d;
fastcgi_send_timeout 1d;
memcached_connect_timeout 1d;
memcached_read_timeout 1d;
memcached_send_timeout 1d;
listen 443 ssl;
listen [::]:443 ssl;
server_name llama-swap.lan;
ssl_certificate /etc/letsencrypt/live/llama-swap.lan/fullchain.pem; # if <-- ssl is available
ssl_certificate_key /etc/letsencrypt/live/llama-swap.lan/privkey.pem; # if <-- ssl is available
location / {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
proxy_pass http://llama-swap:8080/;
proxy_buffering off;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
client_max_body_size 20M;
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root /usr/share/nginx/html;
}
}
Configuring openwebui
In open-webui, create a connection under Connections to http://llama-swap:8080/v1 with the llama.cpp provider and verify it. Afterwards the models can be selected within open-webui.
Caveats
Mounting the Docker socket into the llama-swap container gives it full control over the host's Docker daemon. Treat that container like root on the host.
Some models spit out incomprehensible nonsense. Why? No idea, probably the context is full. Example (GLM-4.5-Air-Q4_0):
Reasoning
Der `docker run`-Befehl, das Sie ausgeführt haben, enthält mehrere Argumente, die möglicherweise "nVIDIA's nvidia-docker plugin." NVIDIA's nVIDIA's nVIDIA's nVIDIA's nVIDIA's nVIDIA's nVIDIA's nVIDIA's nVIDIA's nVIDIA's nVIDIA's nVIDIA's nVIDIA's nVIDIA's nVIDIA's nVIDIA's nVIDIA's nVIDIA's nVIDIA's nVIDIA's nVIDIA's
Links
- https://github.com/mostlygeek/llama-swap
