LLM Providers

General Bots supports multiple Large Language Model (LLM) providers, both cloud-based services and local deployments. This guide helps you choose the right provider for your use case.

Overview

LLMs are the intelligence behind General Bots’ conversational capabilities. You can configure:

• Cloud Providers — External APIs (OpenAI, Anthropic, Google, etc.)
• Local Models — Self-hosted models via llama.cpp
• Hybrid — Use local for simple tasks, cloud for complex reasoning

Cloud Providers

OpenAI (GPT Series)

The most widely known LLM provider, offering the GPT-5 flagship model.

Configuration (config.csv):

name,value
llm-provider,openai
llm-model,gpt-5

Strengths:

• Most advanced all-in-one model
• Excellent general knowledge
• Strong code generation
• Good instruction following

Considerations:

• API costs can add up
• Data sent to external servers
• Rate limits apply

Anthropic (Claude Series)

Known for safety, helpfulness, and extended thinking capabilities.

Configuration (config.csv):

name,value
llm-provider,anthropic
llm-model,claude-sonnet-4.5

Strengths:

• Extended thinking mode for multi-step tasks
• Excellent at following complex instructions
• Strong coding abilities
• Better at refusing harmful requests

Considerations:

• Premium pricing
• Newer provider, smaller ecosystem

Google (Gemini & Vertex AI)

Google’s multimodal AI models with strong reasoning capabilities. General Bots natively supports both the public AI Studio API and Enterprise Vertex AI.

Configuration for AI Studio (Public API):

name,value
llm-provider,google
llm-model,gemini-1.5-pro
llm-url,https://generativelanguage.googleapis.com
llm-key,AIza...

Configuration for Vertex AI (Enterprise):

name,value
llm-provider,vertex
llm-model,gemini-1.5-pro
llm-url,https://us-central1-aiplatform.googleapis.com
llm-key,~/.vertex.json

Note: The bots will handle the Google OAuth2 JWT authentication internally if you provide the path or the raw JSON to a Service Account.

Strengths:

• Largest context window (2M tokens)
• Native multimodal (text, image, video, audio)
• Vertex AI support enables enterprise VPC/IAM integration

Considerations:

• Different endpoints for public vs enterprise deployments

xAI (Grok Series)

Integration with real-time data from X platform.

Configuration (config.csv):

name,value
llm-provider,xai
llm-model,grok-4

Strengths:

• Real-time data access from X
• Strong research and analysis
• Good for trend analysis

Considerations:

• Newer provider
• X platform integration focus

Groq

Ultra-fast inference using custom LPU hardware. Offers open-source models at high speed.

Configuration (config.csv):

name,value
llm-provider,groq
llm-model,llama-4-scout

Strengths:

• Fastest inference speeds (500+ tokens/sec)
• Competitive pricing
• Open-source models
• Great for real-time applications

Considerations:

• Rate limits on free tier
• Models may be less capable than GPT-5/Claude

Mistral AI

European AI company offering efficient, open-weight models.

Configuration (config.csv):

name,value
llm-provider,mistral
llm-model,mixtral-8x22b

Strengths:

• European data sovereignty (GDPR)
• Excellent code generation
• Open-weight models available
• Competitive pricing
• Proficient in multiple languages

Considerations:

• Smaller context than competitors
• Less brand recognition

DeepSeek

Known for efficient, capable models with exceptional reasoning.

Configuration (config.csv):

name,value
llm-provider,deepseek
llm-model,deepseek-r3
llm-server-url,https://api.deepseek.com

Strengths:

• Extremely cost-effective
• Strong reasoning (R1 model)
• Rivals proprietary leaders in performance
• Open-weight versions available (MIT/Apache 2.0)

Considerations:

• Data processed in China
• Newer provider

Amazon Bedrock

AWS managed service for foundation models, supporting Claude, Llama, Titan, and others.

Configuration (config.csv):

name,value
llm-provider,bedrock
llm-model,anthropic.claude-3-5-sonnet-20240620-v1:0
llm-url,https://bedrock-runtime.us-east-1.amazonaws.com/model/anthropic.claude-3-5-sonnet-20240620-v1:0/invoke
llm-key,YOUR_BEDROCK_API_KEY

Strengths:

• Native AWS integration
• Enterprise-grade security
• Multiple model families in one API

Azure OpenAI

Enterprise-grade deployment of OpenAI models hosted on Microsoft Azure.

Configuration (config.csv):

name,value
llm-provider,azureclaude
llm-model,gpt-4o
llm-url,https://YOUR_RESOURCE.openai.azure.com/openai/deployments/YOUR_DEPLOYMENT/chat/completions?api-version=2024-02-15-preview
llm-key,YOUR_AZURE_API_KEY

Strengths:

• High enterprise compliance (HIPAA, SOC2)
• Azure VNet integration
• Guaranteed provisioned throughput available

Cerebras

Ultra-fast inference powered by Wafer-Scale Engine hardware, specifically tuned for open-source models like Llama.

Configuration (config.csv):

name,value
llm-provider,cerebras
llm-model,llama3.1-8b
llm-url,https://api.cerebras.ai/v1/chat/completions
llm-key,YOUR_CEREBRAS_API_KEY

Strengths:

• Highest tokens-per-second available
• Excellent for real-time agent loops

Zhipu AI (GLM)

High-capability bilingual models (English/Chinese) directly competing with state-of-the-art global models.

Configuration (config.csv):

name,value
llm-provider,glm
llm-model,glm-4
llm-url,https://open.bigmodel.cn/api/paas/v4/chat/completions
llm-key,YOUR_ZHIPU_API_KEY

Strengths:

• Excellent bilingual performance
• Large context windows (up to 1M)

Local Models

Run models on your own hardware for privacy, cost control, and offline operation.

Setting Up Local LLM

General Bots uses llama.cpp server for local inference:

name,value
llm-provider,local
llm-server-url,http://localhost:8081
llm-model,DeepSeek-R3-Distill-Qwen-1.5B

Recommended Local Models

For High-End GPU (24GB+ VRAM)

For Mid-Range GPU (12-16GB VRAM)

For Small GPU or CPU (8GB VRAM or less)

Model Download URLs

Add models to installer.rs data_download_list:

#![allow(unused)]
fn main() {
// Qwen3 14B - Recommended for mid-range GPU
"https://huggingface.co/Qwen/Qwen3-14B-GGUF/resolve/main/qwen3-14b-q4_k_m.gguf"

// DeepSeek R1 Distill - For CPU or minimal GPU
"https://huggingface.co/unsloth/DeepSeek-R3-Distill-Qwen-1.5B-GGUF/resolve/main/DeepSeek-R3-Distill-Qwen-1.5B-Q4_K_M.gguf"

// GPT-oss 20B - Good balance for agents
"https://huggingface.co/openai/gpt-oss-20b-GGUF/resolve/main/gpt-oss-20b-q4_k_m.gguf"

// Gemma 3 27B - For quality local inference
"https://huggingface.co/google/gemma-3-27b-it-GGUF/resolve/main/gemma-3-27b-it-q4_k_m.gguf"
}

Embedding Models

For vector search, you need an embedding model:

name,value
embedding-provider,local
embedding-server-url,http://localhost:8082
embedding-model,bge-small-en-v1.5

Recommended embedding models:

Hybrid Configuration

Use different models for different tasks:

name,value
llm-provider,anthropic
llm-model,claude-sonnet-4.5
llm-fast-provider,groq
llm-fast-model,llama-3.3-70b
llm-fallback-provider,local
llm-fallback-model,DeepSeek-R3-Distill-Qwen-1.5B
embedding-provider,local
embedding-model,bge-small-en-v1.5

Model Selection Guide

By Use Case

By Budget

Configuration Reference

config.csv Parameters

All LLM configuration belongs in config.csv, not environment variables:

API Keys

API keys are stored in Vault, not in config files or environment variables:

# Store API key in Vault
vault kv put gbo/llm/openai api_key="sk-..."
vault kv put gbo/llm/anthropic api_key="sk-ant-..."
vault kv put gbo/llm/google api_key="AIza..."

Reference in config.csv:

name,value
llm-provider,openai
llm-model,gpt-5
llm-api-key,vault:gbo/llm/openai/api_key

Security Considerations

Cloud Providers

• API keys stored in Vault, never in config files
• Consider data residency requirements (EU: Mistral)
• Review provider data retention policies
• Use separate keys for production/development

Local Models

• All data stays on your infrastructure
• No internet required after model download
• Full control over model versions
• Consider GPU security for sensitive deployments

Performance Optimization

Caching

Enable semantic caching to reduce API calls:

name,value
llm-cache-enabled,true
llm-cache-ttl,3600
llm-cache-similarity-threshold,0.92

Batching

For bulk operations, use batch APIs when available:

name,value
llm-batch-enabled,true
llm-batch-size,10

Context Management

Optimize context window usage with episodic memory:

name,value
episodic-memory-enabled,true
episodic-memory-threshold,4
episodic-memory-history,2
episodic-memory-auto-summarize,true

See Episodic Memory for details.

Troubleshooting

Common Issues

API Key Invalid

• Verify key is stored correctly in Vault
• Check if key has required permissions
• Ensure billing is active on provider account

Model Not Found

• Check model name spelling
• Verify model is available in your region
• Some models require waitlist access

Rate Limits

• Implement exponential backoff
• Use caching to reduce calls
• Consider upgrading API tier

Local Model Slow

• Check GPU memory usage
• Reduce context size
• Use quantized models (Q4 instead of F16)

Logging

Enable LLM logging for debugging:

name,value
llm-log-requests,true
llm-log-responses,false
llm-log-timing,true

2025 Model Comparison

Next Steps

• config.csv Reference — Complete configuration guide
• Secrets Management — Vault integration
• Semantic Caching — Cache configuration
• NVIDIA GPU Setup — GPU configuration for local models