Run a Local LLM on Apple Silicon with MLX (vs llama.cpp): The Setup That Actually Works (2026)

Run a Local LLM on Apple Silicon with MLX (vs llama.cpp): The Setup That Actually Works (2026)

Want to run a local LLM on Apple Silicon with MLX — and know when llama.cpp is the better call? This is the setup that actually works in 2026: every command below was run on a real M-series Mac, not copied from a blog that never shipped.

If you want the complete version (full troubleshooting table, vision-model serving, and a commercial-use license), it's in The MLX Deploy Playbook. Use code **LAUNCH40** for 40% off.

Why MLX (and when llama.cpp instead)

MLX is Apple's array framework with unified-memory-native execution. On Apple Silicon it consistently serves **~45–58 tok/s** on 4-bit 27–31B models versus **~38–48 tok/s** for llama.cpp GGUF on the same hardware, and it exposes a drop-in OpenAI-compatible server.

- **Use MLX** when: you want max throughput on Apple Silicon, an OpenAI-compatible endpoint, and Python-native integration.

- **Use llama.cpp** when: you need GGUF portability across non-Apple hardware or ultra-low-RAM quantization (Q2/Q3).

The five-minute setup

# create a clean venv, install a known-good pair, serve a 32B model
PYBIN=/opt/homebrew/opt/[email protected]/bin/python3.12
DIR=$HOME/mlx-serve; mkdir -p "$DIR"; cd "$DIR"
"$PYBIN" -m venv .venv
.venv/bin/python -m pip install -q "mlx-lm==0.31.3" "transformers==5.6.0"
env -u PYTHONPATH NO_PROXY="*" .venv/bin/python -m mlx_lm.server \
  --host 127.0.0.1 --port 8081 \
  --model mlx-community/Qwen2.5-32B-Instruct-4bit

Any OpenAI client now works against `http://127.0.0.1:8081/v1`:

curl http://127.0.0.1:8081/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"local","messages":[{"role":"user","content":"Say hi"}]}'

Model picks by RAM (4-bit)

| RAM | Chat / coding | Vision |

|-----|---------------|--------|

| 16 GB | Qwen2.5-7B, Gemma-2-9B | Qwen2-VL-7B |

| 32 GB | Qwen2.5-14B / 32B | Qwen2-VL-7B |

| 64 GB+ | Qwen2.5-32B, Gemma-2-27B | Qwen2-VL-72B (4-bit) |

The three silent failures nobody documents

These look like MLX bugs but aren't:

1. **`ImportError` on start** — your shell injects a `PYTHONPATH` that shadows the venv's `numpy`/`pydantic`. Prefix every command with `env -u PYTHONPATH`.

2. **Weight download stalls at 0 bytes** — the Hugging Face LFS CDN (`cdn-lfs.huggingface.co`) is blocked on some networks even though the API host works. Route through the Xet bridge CDN: `env -u PYTHONPATH NO_PROXY="*" hf download mlx-community/Qwen2.5-32B-Instruct-4bit`.

3. **Browser can't connect to the server** — you bound to `0.0.0.0`. Browsers can *listen* on `0.0.0.0` but cannot *dial* it as a destination. Always bind `127.0.0.1`.

Keep it running across reboots (launchd)

A login-scoped `launchd` agent keeps the server alive. The trap: launchd does **not** inherit your interactive shell's `PYTHONPATH` or venv, so third-party imports vanish. Give it the venv's own interpreter and an explicit clean environment:

<key>ProgramArguments</key>
<array>
  <string>/Users/YOU/mlx-serve/.venv/bin/python</string>
  <string>-m</string><string>mlx_lm.server</string>
  <string>--host</string><string>127.0.0.1</string>
  <string>--port</string><string>8081</string>
  <string>--model</string><string>mlx-community/Qwen2.5-32B-Instruct-4bit</string>
</array>
<key>EnvironmentVariables</key>
<dict><key>NO_PROXY</key><string>*</string></dict>
<key>KeepAlive</key><true/>
<key>RunAtLoad</key><true/>

If a `KeepAlive` server ignores your kill, respawn it with `launchctl kickstart -k gui/$(id -u)/com.you.mlx-serve`.

What the full playbook adds

The free steps above get you a working server. The MLX Deploy Playbook adds the complete 60-second troubleshooting table, vision-model (VLM) serving with a local browser chat page, launchd persistence that survives logout, and a commercial-use license for client and product work — including every silent failure that cost hours. Use code **LAUNCH40** for 40% off.

References

- MLX — Apple's machine learning framework for Apple Silicon

- mlx-lm — the OpenAI-compatible local LLM server used above