Troubleshooting

This guide covers common issues you may encounter with botserver and their solutions.

Quick Diagnostics

Check Overall Status

# View all service status
./botserver status

# Check specific service
./botserver status llm
./botserver status tables
./botserver status vault

View Logs

# All logs
tail -f botserver-stack/logs/*.log

# Specific service
tail -100 botserver-stack/logs/llm.log
tail -100 botserver-stack/logs/postgres.log
tail -100 botserver-stack/logs/vault.log

# With filtering
grep -i error botserver-stack/logs/*.log
grep -i "failed\|error\|panic" botserver-stack/logs/*.log

System Resources

# Memory usage
free -h

# Disk space
df -h botserver-stack/

# Process list
ps aux | grep -E "llama|postgres|minio|vault|valkey"

# Open ports
ss -tlnp | grep LISTEN

Startup Issues

Bootstrap Fails

Symptom: ./botserver fails during initial setup

Common Causes & Solutions:

Port already in use

# Find what's using the port
lsof -i :9000
lsof -i :5432

# Kill conflicting process
kill -9 <PID>

# Or change port in config

Insufficient disk space

# Check available space
df -h

# Clean up old installers
rm -rf botserver-installers/*.old

# Clean logs
rm -f botserver-stack/logs/*.log.old

Download failure

# Clear cache and retry
rm -rf botserver-installers/component-name*
./botserver bootstrap

# Manual download
curl -L -o botserver-installers/file.zip "URL"

Permission denied

# Fix permissions
chmod +x botserver
chmod -R u+rwX botserver-stack/

Vault Won’t Start

Symptom: Vault fails to initialize or unseal

Solutions:

First-time setup failed

# Reset Vault completely
rm -rf botserver-stack/data/vault/*
rm -f botserver-stack/conf/vault/init.json
./botserver bootstrap

Vault is sealed

# Check seal status
curl http://localhost:8200/v1/sys/seal-status

# Unseal manually
./botserver unseal

Lost unseal keys

# Check init.json exists
cat botserver-stack/conf/vault/init.json

# If lost, must reset Vault (DATA LOSS)
./botserver reset vault

Database Won’t Start

Symptom: PostgreSQL fails to start

Solutions:

Corrupted data directory

# Check PostgreSQL logs
tail -50 botserver-stack/logs/postgres.log

# Try recovery
./botserver-stack/bin/tables/bin/pg_resetwal -f botserver-stack/data/tables/

Port conflict

# Check if another PostgreSQL is running
lsof -i :5432

# Stop system PostgreSQL
sudo systemctl stop postgresql

Incorrect permissions
```
chmod 700 botserver-stack/data/tables/
```

Service Issues

LLM Server Not Responding

Symptom: Requests to port 8081/8082 fail

Solutions:

Check if running

pgrep llama-server
curl -k https://localhost:8081/health

Model not found

# Verify model exists
ls -la botserver-stack/data/llm/

# Re-download model
./botserver update llm

Out of memory

# Check memory usage
free -h

# Use smaller model or reduce context
# Edit config.csv:
# llm-server-ctx-size,2048

GPU issues

# Check CUDA
nvidia-smi

# Fall back to CPU
# Edit config.csv:
# llm-server-gpu-layers,0

Restart LLM server

pkill llama-server
./botserver start llm

Drive (MinIO) Issues

Symptom: File uploads/downloads fail

Solutions:

Check MinIO status

curl http://localhost:9000/minio/health/live

Credential issues

# Verify credentials from Vault
./botserver show-secret drive

# Test with mc client
mc alias set local http://localhost:9000 ACCESS_KEY SECRET_KEY
mc ls local/

Disk full

df -h botserver-stack/data/drive/

# Clean old versions
mc rm --recursive --force local/bucket/.minio.sys/

Cache (Valkey) Issues

Symptom: Session errors, slow responses

Solutions:

Check Valkey status

./botserver-stack/bin/cache/valkey-cli ping
# Expected: PONG

Memory issues

./botserver-stack/bin/cache/valkey-cli info memory

# Flush cache if needed
./botserver-stack/bin/cache/valkey-cli FLUSHALL

Connection refused

# Check if running
pgrep valkey-server

# Restart
./botserver restart cache

Directory (Zitadel) Issues

Symptom: Login fails, authentication errors

Solutions:

Check Zitadel logs

tail -100 botserver-stack/logs/zitadel.log

Database connection

# Zitadel uses PostgreSQL
psql $DATABASE_URL -c "SELECT 1;"

Certificate issues

# Regenerate certificates
./botserver regenerate-certs

Connection Issues

Cannot Connect to Database

Error: connection refused or authentication failed

Solutions:

Verify DATABASE_URL

echo $DATABASE_URL
# Should be: postgres://user:pass@localhost:5432/dbname

Check PostgreSQL is running

pgrep postgres
./botserver status tables

Test connection
```
psql $DATABASE_URL -c "SELECT 1;"
```

Check pg_hba.conf

cat botserver-stack/conf/tables/pg_hba.conf
# Ensure local connections are allowed

SSL/TLS Certificate Errors

Error: certificate verify failed or SSL handshake failed

Solutions:

Regenerate certificates
```
./botserver regenerate-certs
```

Check certificate validity

openssl x509 -in botserver-stack/conf/system/certificates/api/server.crt -noout -dates

Skip verification (development only)
```
curl -k https://localhost:8081/health
```

Network Timeouts

Error: Requests timeout after waiting

Solutions:

Check DNS resolution
```
nslookup api.botserver.local
```
Verify firewall rules
```
sudo ufw status
sudo iptables -L
```
Check service is listening
```
ss -tlnp | grep 8080
```

Performance Issues

Slow Response Times

Solutions:

Check system resources
```
top -b -n 1 | head -20
iostat -x 1 3
```

Database performance

psql $DATABASE_URL -c "SELECT * FROM pg_stat_activity;"

# Vacuum database
psql $DATABASE_URL -c "VACUUM ANALYZE;"

LLM performance

# Reduce context size
# config.csv: llm-server-ctx-size,2048

# Use GPU layers
# config.csv: llm-server-gpu-layers,35

Enable caching

# Verify cache is working
./botserver-stack/bin/cache/valkey-cli info stats

High Memory Usage

Solutions:

Identify memory hogs
```
ps aux --sort=-%mem | head -10
```

Reduce LLM memory

# Use quantized model (Q3_K_M instead of F16)
# Reduce context: llm-server-ctx-size,1024
# Reduce batch: llm-server-batch-size,256

Limit PostgreSQL memory

# Edit postgresql.conf
shared_buffers = 256MB
work_mem = 64MB

High Disk Usage

Solutions:

Find large files

du -sh botserver-stack/*
du -sh botserver-stack/data/*

Clean logs

truncate -s 0 botserver-stack/logs/*.log

Clean old installers

# Keep only latest versions
ls -la botserver-installers/
rm botserver-installers/old-*

Prune drive storage

mc rm --recursive --older-than 30d local/bucket/

Update Issues

Component Update Failed

Symptom: Update command fails or service won’t start after update

Solutions:

Clear cache and retry

rm botserver-installers/component-name*
./botserver update component-name

Checksum mismatch

# Verify checksum
sha256sum botserver-installers/file.zip

# Compare with 3rdparty.toml
grep sha256 3rdparty.toml | grep component

# Update checksum if release changed

Rollback to previous version

# If old version cached
ls botserver-installers/

# Restore old binary
cp botserver-installers/old-version.zip /tmp/
unzip /tmp/old-version.zip -d botserver-stack/bin/component/

Database Migration Failed

Solutions:

Check migration status
```
./botserver migrate --status
```
Run migrations manually
```
./botserver migrate
```
Rollback migration
```
./botserver migrate --rollback
```

Reset from backup

pg_restore -c -d $DATABASE_URL backup.dump

Common Error Messages

Error	Cause	Solution
`connection refused`	Service not running	Start the service
`permission denied`	File permissions	`chmod +x` on binary
`address already in use`	Port conflict	Kill conflicting process
`out of memory`	Insufficient RAM	Reduce model/context size
`no such file or directory`	Missing binary/config	Re-run bootstrap
`certificate verify failed`	SSL issues	Regenerate certificates
`authentication failed`	Wrong credentials	Check Vault secrets
`disk quota exceeded`	Disk full	Clean logs/old files
`too many open files`	ulimit too low	`ulimit -n 65536`
`connection timed out`	Network/firewall	Check firewall rules

Getting Help

Collect Diagnostics

# Generate diagnostic report
./botserver diagnose > diagnostics-$(date +%Y%m%d).txt

# Include in bug reports:
# - botserver version
# - OS and architecture
# - Error messages
# - Relevant logs

Debug Logging

# Enable verbose logging
RUST_LOG=debug ./botserver

# Trace level (very verbose)
RUST_LOG=trace ./botserver

Community Support

GitHub Issues: github.com/GeneralBots/botserver/issues
Documentation: docs.generalbots.ai

General Bots Documentation

Troubleshooting

Quick Diagnostics

Check Overall Status

View Logs

System Resources

Startup Issues

Bootstrap Fails

Vault Won’t Start

Database Won’t Start

Service Issues

LLM Server Not Responding

Drive (MinIO) Issues

Cache (Valkey) Issues

Directory (Zitadel) Issues

Connection Issues

Cannot Connect to Database

SSL/TLS Certificate Errors

Network Timeouts

Performance Issues

Slow Response Times

High Memory Usage

High Disk Usage

Update Issues

Component Update Failed

Database Migration Failed

Common Error Messages

Getting Help

Collect Diagnostics

Debug Logging

Community Support

See Also

Keyboard shortcuts

General Bots Documentation