Run llama.cpp as an always-on local AI server on Linux

House unit

One GPU, Vulkan llama-server, house 35B. Replace the binary, GGUF, and User= login. systemd.service documents Type, ExecStart, Restart, and User.

# Example only — not a measured local desktop service.
# /etc/systemd/system/llama-server.service
# Copy house flags from /tutorials/serve-qwen. Replace the binary and GGUF paths.
[Unit]
Description=llama-server house 35B on loopback
After=network-online.target

[Service]
Type=simple
User=YOUR_LOGIN
ExecStart=/path/to/llama.cpp/build-vk/bin/llama-server -m /path/to/nvme/Qwen3.6-35B-A3B-UD-Q5_K_XL.gguf -ngl 99 -c 262144 --cache-type-k f16 --cache-type-v f16 --jinja -np 1 --host 127.0.0.1 --port 8080
Restart=on-failure

[Install]
WantedBy=multi-user.target

sudo install -m 0644 llama-server.service /etc/systemd/system/llama-server.service
sudo systemctl daemon-reload
sudo systemctl enable --now llama-server
systemctl status llama-server

journalctl -u llama-server -f
curl -s http://127.0.0.1:8080/health
curl -s http://127.0.0.1:8080/v1/models

Zero to always-on

1. 1 · Cited llama.cpp + systemd · not a measured service card

   Get tokens on loopback before you write a unit

   Ubuntu 24.04+ bare metal is the measured AMDai path. Build llama.cpp Vulkan, load Qwen3.6-35B-A3B-UD-Q5_K_XL.gguf on one R9700, and confirm curl http://127.0.0.1:8080/v1/models from /tutorials/serve-qwen. systemd does not fix a binary that never generated. House flags: -ngl 99 -c 262144 --cache-type-k f16 --cache-type-v f16 --jinja --host 127.0.0.1 --port 8080. llama.cpp first. Q6/Q5/Q4 recommended; Q3 is a labeled estimate. Q1/Q2 stay off.

   • Ubuntu setup
   • Serve Qwen
   • llama.cpp
   • R9700 measured page

2. 2 · Cited llama.cpp + systemd · not a measured service card

   Write one systemd unit for the house 35B

   Drop a Type=simple unit under /etc/systemd/system/llama-server.service. systemd.service documents Type, ExecStart, Restart, and User. Run as the login that already reaches /dev/dri — not root. Restart=on-failure. The ExecStart line is the serve-qwen command plus -np 1 for a single-user lane (default llama-server auto-slots copy idle KV on each new task). This is an example, not a measured local desktop service. Two-card split or combo units stay on /setup/multi-gpu.

   • Serve Qwen
   • Multi-GPU systemd notes
   • Tuning flags

3. 3 · Cited llama.cpp + systemd · not a measured service card

   daemon-reload, enable --now, then status

   systemctl daemon-reload after you install the file. systemctl enable --now llama-server starts it and keeps it across reboot. systemctl status llama-server is the first check. Those verbs are from the systemctl man page. Not a measured local desktop service.

   • Ubuntu setup
   • llama.cpp

4. 4 · Cited llama.cpp + systemd · not a measured service card

   journalctl and GET /health

   journalctl -u llama-server -f is the unit log (journalctl man page). llama-server README: GET /health is a public health check — no API key. /v1/health also works. Then curl /v1/models on 127.0.0.1:8080. Overnight 35B numbers are llama-bench cards, not this service.

   • Serve Qwen
   • Benchmarks
   • Run cards

5. 5 · Cited llama.cpp + systemd · not a measured service card

   Both RAM lines for the house 35B

   Where context and offload appear, AMDai prints Minimum system RAM needed and RAM needed at full context. Overnight UD-Q5 one-GPU peaks are 37.32 GB at 4K and 42.9 GB at 256K — measured llama-bench on the local desktop, Linux GB. Round Minimum system RAM needed to a 64 GB kit for this file on one R9700. Q3 stays a labeled estimate. Do not invent a systemd RAM number.

   • Calculator
   • R9700 measured page
   • Flash RAM offload
   • Benchmarks

6. 6 · Cited llama.cpp + systemd · not a measured service card

   Keep 127.0.0.1; LAN needs --api-key

   llama-server README: --host defaults to 127.0.0.1. That is the house bind. Phones and LAN clients are opt-in: --host 0.0.0.0 plus --api-key KEY (or LLAMA_API_KEY / --api-key-file). GET /health stays public even with a key. Do not publish 0.0.0.0 without a key. Open WebUI, Aider, and Hermes on the same box use http://127.0.0.1:8080/v1.

   • Open WebUI
   • Coding assistant
   • Connect a harness
   • Hermes

7. 7 · Cited llama.cpp + systemd · not a measured service card

   ufw and an SSH tunnel — unofficial extras

   If the unit stays on 127.0.0.1, ufw is not what exposes /v1. A LAN bind can use Ubuntu ufw to allow one subnet to port 8080 — unofficial for llama.cpp, cited from the noble ufw man-page. Prefer an OpenSSH local forward (ssh -L) so the GPU box never leaves loopback. ssh(1) documents -L. Unofficial for llama.cpp.

   • Docker / Podman
   • Connect a harness

8. 8 · Cited llama.cpp + systemd · not a measured service card

   Swap the GGUF with a restart

   Edit ExecStart (or systemctl edit for a drop-in), daemon-reload, then systemctl restart llama-server. journalctl -u llama-server -e confirms the new file. Do not run a second llama-server on :8080. A second model on a second card is /setup/multi-gpu (:8081), not a second copy of this unit.

   • Multi-GPU llama.cpp
   • House models
   • Quants

9. 9 · Cited llama.cpp + systemd · not a measured service card

   Attach a harness after /health is ok

   Point Hermes, Open WebUI, Aider, Cline, or Continue at http://127.0.0.1:8080/v1 only after /health and /v1/models work. Decode numbers on this site are Ubuntu llama-bench, not a harness tok/s. Stack Lab remains the Ubuntu companion — packages are not published yet.

   • Harnesses
   • Connect a harness
   • Open WebUI
   • Coding assistant
   • Stack Lab

10. 10 · Cited llama.cpp + systemd · not a measured service card

    When it breaks

    status=1/FAILURE: ExecStart path, missing GGUF, or the login cannot open /dev/dri. Empty /v1/models: still loading, or you curled the wrong port. GPU idle: you are on CPU — confirm the Vulkan binary, not systemd. Permission denied: groups, not a new unit flag. Start from /troubleshoot. Cited ROCm rows live on /rocm-problems.

    • Troubleshoot
    • Error paste
    • ROCm problems
    • FAQ
    • Glossary

LAN, ufw, and SSH

Opt-in only. --api-key is from the llama-server README. ufw and ssh -L are unofficial for llama.cpp.

# Opt-in LAN — not the house default.
# llama-server README: --host default is 127.0.0.1; --api-key KEY (or LLAMA_API_KEY).
# GET /health is public (no API key check). /v1/health also works.
ExecStart=... --host 0.0.0.0 --port 8080 --api-key YOUR_KEY

# Unofficial for llama.cpp — Ubuntu ufw. Only if you bound a LAN address.
sudo ufw status
# sudo ufw allow from 192.168.0.0/16 to any port 8080 proto tcp
# Copy the subnet you actually use. Do not open 0.0.0.0/0.

# Unofficial for llama.cpp — OpenSSH local forward.
# Keeps llama-server on 127.0.0.1 on the GPU box.
ssh -L 8080:127.0.0.1:8080 user@gpu-box

sudo systemctl edit llama-server
# or edit ExecStart, then:
sudo systemctl daemon-reload
sudo systemctl restart llama-server
journalctl -u llama-server -e

RAM lines (Ubuntu-measured)

Minimum system RAM needed and RAM needed at full context for the house 35B. Those peaks are llama-bench on the local desktop, not this unit.

Open in Stack Lab

Stack Lab remains Ubuntu-first. Packages are not published yet and there is no public download.

Sources

• llama.cpp — llama-server README (--host, --api-key, GET /health)
• llama.cpp repository
• systemd.service — Type, ExecStart, Restart, User
• systemctl — daemon-reload, enable, start, restart, status
• journalctl — unit logs
• Ubuntu man-pages — ufw (unofficial for llama.cpp) — unofficial for llama.cpp
• OpenSSH ssh(1) — -L local forward (unofficial for llama.cpp) — unofficial for llama.cpp