AI Model (Local or Groq)

The AI features — the AI Assistant (/ask, /summarize), ticket AI triage, AI moderation, and code review — all talk to one OpenAI-compatible chat endpoint. By default that's a model running on your own server, so no data leaves the machine. If you can't run a model, you can use Groq's hosted API instead.

Setup	What you need	Data leaves your server?
Local model (recommended)	A server that can run a 7B model (see hardware)	No
Groq only	A GROQ_API_KEY	Yes, every AI request
Local, with Groq fallback	Both	Only requests the local model fails on

1. Install Ollama

Ollama runs the model and exposes an OpenAI-compatible API.

curl -fsSL https://ollama.com/install.sh | sh

Pull the model ArkenBot is set up for:

ollama pull qwen2.5:7b-instruct

Test it:

curl -s http://127.0.0.1:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"qwen2.5:7b-instruct","messages":[{"role":"user","content":"Say hi"}],"max_tokens":10}'

You should get a JSON response with a choices array.

2. Keep it on localhost and limit its CPU

Ollama listens on 127.0.0.1:11434 by default, which is what you want — the bot reaches it on the same machine, and nothing outside should.

Running a model on the same server as the bot can starve the bot of CPU, so cap it. On systemd-based Linux, create an override:

sudo systemctl edit ollama

Paste:

[Service]
CPUQuota=400%
Nice=15
Environment="OLLAMA_KEEP_ALIVE=30m"

Then apply it:

sudo systemctl daemon-reload
sudo systemctl restart ollama

CPUQuota=400% limits the model to four cores. Nice=15 lets the bot win any contention. OLLAMA_KEEP_ALIVE=30m keeps the model in memory between requests, so the first AI call after a quiet period doesn't pay a full reload.

3. Point the bot at it

These are the defaults, so you only need to set them if you changed something:

LLM_API_URL=http://127.0.0.1:11434/v1/chat/completions
LLM_MODEL=qwen2.5:7b-instruct

To turn all AI features off, set LLM_DISABLED=true.

4. Optional: Groq fallback

If the local model fails for a request — it's down, too slow, or returns an error — ArkenBot can retry that same request on Groq. Add a key to .env:

GROQ_API_KEY=gsk_...
GROQ_MODEL=openai/gpt-oss-20b   # optional; this is the default

Groq only sees requests the local model failed on. Requests that were cancelled by the bot itself are never sent there. If you run with no local model at all, set only GROQ_API_KEY: the bot will try the local endpoint first, fail to connect, and then use Groq.

Then restart the bot:

pm2 restart bot

Hardware

The model runs on CPU if you don't have a GPU, and CPU inference is much slower. On an 8-core server these were the measured times for a single call with the 7B model:

Call	Time
Trivial completion	about 2 seconds
AI moderation check	about 6 seconds
Full ticket triage	about 20–26 seconds

Those are fine for background work, but slow for anything a member is waiting on. A GPU makes a large difference. If the local model is consistently too slow, the Groq fallback will pick up those requests once they time out.

Troubleshooting

• /ask says AI is unavailable — the local endpoint isn't reachable and no Groq key is set. Check systemctl status ollama and the curl test in step 1.
• Triage or moderation keeps timing out — the model is starved or cold. Check the CPU limit in step 2 and that OLLAMA_KEEP_ALIVE is set.
• Requests go to Groq — the local model is failing. Check pm2 logs bot for the error from the local endpoint.

---

Next: Troubleshooting →