Blog

Anything with write access to a host and a language model in its decision path needs three things before it is allowed to run: a stop, a record, and a human. Not because the model is malicious — because it is confident.

1. A stop that works when the model is stuck

systemctl stop ai-ops.service
# and the socket, or the next request spawns it again
systemctl disable --now ai-ops.socket

Test the stop. A unit that ignores SIGTERM because it is blocked in a model call is not stoppable; set TimeoutStopSec=15 and make sure SIGKILL lands.

2. A record of what it did, not what it said

Log the commands, the exit codes, and the files touched. Not the reasoning — reasoning is not reproducible, effects are.

logger -t ai-ops "apply unit=$unit rc=$rc files=$(echo "$files" | wc -l)"

3. A human who can say no

The cheapest version: the tool writes a diff and stops. Somebody reads it and runs it. The expensive version, and the one people build first, is a rollback path for a change nobody reviewed.

Automation should be able to do the boring 90% and be structurally unable to do the dangerous 10% on its own.

That is not a limitation of the model. It is a design decision about the host, and it survives the next model release.

The most useful thing I built this year is not an agent. It is a loop with two of them: one writes, one reads and refuses to ship. The refusal is the product.

The shape

coder   : reads the task, writes the code, RUNS it, reports real output
reviewer: read-only, same repo, must end with VERDICT: SHIP | BLOCK

The reviewer has no write access — proven with fingerprints before and after, not promised. It reads the diff, the tests, and the output of the tests. It is allowed to be pedantic about three things and nothing else: does it run, does it do what was asked, is there a test that would fail if the code were wrong.

Why two and not one

A single agent that both writes and judges will pass its own work, because it has already decided what “passing” means. The value comes from the information barrier, not from the second model being smarter — you can run the same weights on both sides and still get the benefit.

Exit codes, not adjectives

for round in 1 2 3; do
  coder_round "$round"
  verdict=$(reviewer_round "$round")
  [ "$verdict" = "SHIP" ] && exit 0
done
exit 1

The loop exits zero only on SHIP. Anything else — three rounds of blocking, a crashed reviewer, a timeout — is a failure you can see in CI, which is the only place a verdict should ever be read.

What it caught

On a 2,400-line application it found a GET handler that ignored its own error parameter, a cookie marked Secure only behind a proxy header, and a password stash with no expiry. All three were mine. None of them were found by asking a model to “review this code”.

bpftrace gives you facts about a running kernel that no log will ever contain. The model’s job is not to run it — that stays you — but to turn the output into a sentence you can act on.

Start with a question, not a tool

“Which process is opening files under /etc at three in the morning?” is a question. The tool follows:

bpftrace -e 'tracepoint:syscalls:sys_enter_openat
  /str(args->filename) == "/etc/*"/
  { printf("%s %d %s\n", comm, pid, str(args->filename)); }'

What the model adds

It clusters. Twenty thousand identical lines are one fact, and the interesting output is the tail: the one process that appears twice at 03:12 with a different parent. Ask for that, explicitly:

Here are counts per (comm, filename). Name the three entries that do not share a parent process with the rest, and say nothing else.

Narrow asks get narrow answers. “Analyse this” gets a paragraph of plausible nothing.

The verification step

ps -o pid,ppid,cmd -p <pid>
systemctl status <unit> --no-pager | head -5

Two commands that either confirm the sentence or kill it. If they kill it, the model was not wrong about the data — it was wrong about the cause, and that is the failure mode worth reading for.

A journal is the most honest thing on a Linux host and the worst thing to paste into a model. It contains tokens, internal hostnames, mail addresses, sometimes an entire environment. The pipeline below redacts first, then summarises, and keeps a hash so you can still correlate.

Redact before you send, not after

journalctl -u myapp --since "1 hour ago" --no-pager \
  | sed -E 's/([A-Za-z0-9_]*(TOKEN|SECRET|PASSWORD|KEY)[A-Za-z0-9_]*=)[^ ]+/\1[redacted]/g' \
  | sed -E 's/[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+/[email]/g' \
  | sed -E 's/\b[0-9]{1,3}(\.[0-9]{1,3}){3}\b/[ip]/g' \
  | sha256sum

The hash is the point. You send the redacted text, keep the hash in your notes, and when the model says “the third line is the cause” you can prove which file it came from without sending the file.

Allow-list the shape, not the content

Instead of pattern-matching every possible secret — you will miss one — reduce the log to lines that match a small grammar of what you care about:

grep -E 'error|fail|denied|refused|timeout|OOM|Segmentation' \
  | cut -c1-240 | sort | uniq -c | sort -rn | head -40

Forty deduplicated lines, no arguments, no payloads. That is a prompt a small model can handle, and a log nobody can leak.

Keep the model out of the decision

The model proposes; systemctl status disposes. If the summary cannot be checked with one command, it is not a summary, it is a rumour.

A 32 GB machine with no GPU runs a useful model. Not the useful model — the other one. Here is the arithmetic I actually use, after running this on a Xeon E3-1275 v5 with DDR3 and a fan that never spins up.

What fits, in one table

• 7–8B at Q4_K_M — around 5 GB resident, ~18 tok/s. Reads logs, classifies failures, drafts playbooks. This is the workhorse.
• 14B at Q4_K_M — around 9 GB, ~7 tok/s. Noticeably better at YAML and at not inventing flags. Still comfortable.
• 32B at Q4_K_M — around 19 GB, ~2 tok/s. Fine for a nightly batch, unusable interactively.
• Anything at Q8 — double the memory for a quality gain you will not measure on a shell command.

The rule that matters

Leave RAM for the page cache. A model that evicts the filesystem cache to stay resident makes the whole host slower, and you will blame the storage.

ollama run qwen3:8b
# watch RSS while it answers, not while it loads
watch -n1 'ps -o rss=,cmd= -C ollama | sort -rn | head -3'

Quantisation is not a religion

Q4_K_M is the default because it is the point where the quality curve flattens and the memory curve does not. If a task fails at Q4 and succeeds at Q6, the task is out of scope — a bigger quantisation is a way of not fixing the prompt.

If the answer has to be right, the model is the second opinion. The machine is the first.

Measure throughput once, on your own hardware, with the prompt you actually send. Every published tok/s number was measured on somebody else’s box, with a shorter prompt, in a cooler room.