Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

[llm] — language model

Settings for the language model used for extraction, summarization, classification, and query answering.

[llm]
provider = "local"               # "local" (llama.cpp) or "api" (OpenAI-compatible)

Provider selection

ValueWhat it does
"local"Embedded llama.cpp inference. No network, no API key. Downloads a GGUF model on first run.
"api"OpenAI-compatible HTTP API. Requires api_key (or env var).

You can mix: see Embedding section for the combinations.

Local provider settings

[llm]
provider = "local"
model_file = "gemma-4-E2B-it-Q8_0.gguf"   # GGUF filename (auto-downloaded if missing)
models_dir = "llm-mem-data/models"         # where to put downloaded models
gpu_layers = 0                            # 0 = CPU only; set to 20+ for GPU
context_size = 16644                      # context window in tokens
cpu_threads = 0                           # 0 = auto-detect
auto_download = true                      # fetch from HuggingFace on first run
cache_model = true                        # cache in ~/.cache/llm-mem/models
cache_dir = ""                            # custom cache dir (default: ~/.cache/llm-mem/models)
llm_timeout_secs = 120                    # completion timeout
use_grammar = false                       # grammar-constrained sampling (faster, less flexible)
proxy_url = ""                            # proxy for downloads (overrides HTTPS_PROXY)

Vision (local)

If you have a vision-capable GGUF model + mmproj file:

vision_enabled = true
mmproj_file = ""                          # empty = auto-detect mmproj-F16.gguf for Gemma 4 E2B
vision_prompt_template = "Describe this image in detail, focusing on what would make it searchable in a personal memory system. What objects, people, text, colors, scene elements, actions, and contextual cues are visible? What domain or topic does this image relate to? Be specific and concrete."

When vision_enabled = true, ingested images (PNG/JPEG/GIF/WebP) get an AI-generated description stored alongside the original.

API provider settings

[llm]
provider = "api"
api_url = "https://api.openai.com/v1"
api_key = ""                              # or set LLM_MEM_LLM_API_KEY / OPENAI_API_KEY
model = "gpt-4o-mini"

Common api_url values:

ServiceURL
OpenAIhttps://api.openai.com/v1
OpenRouterhttps://openrouter.ai/api/v1
Anthropicuse api_dialect = "anthropic" with the Anthropic endpoint
Ollamahttp://localhost:11434/v1
llama-serverhttp://localhost:8080/v1
LM Studiohttp://localhost:1234/v1

Dialects

api_dialect = "openai-chat"   # one of: openai-chat, openai-completion, anthropic, ollama-chat, ollama-completion, custom

openai-chat (default) works for any OpenAI-compatible chat completions endpoint. Use anthropic for Anthropic, ollama-* for Ollama native APIs, custom for fully custom request/response shapes.

Custom dialect

api_dialect = "custom"

[llm.custom_dialect]
endpoint_path = "/v1/generate"
request_body_template = '{"model": "{{model}}", "prompt": "{{prompt}}", "max_tokens": {{max_tokens}}, "temperature": {{temperature}}}'
response_content_pointer = "/choices/0/text"

Placeholders: {{prompt}}, {{model}}, {{temperature}}, {{max_tokens}}. response_content_pointer is a JSON pointer to the text in the response.

Request format

request_format = "auto"        # "auto" (default), "rig", or "raw"
  • auto — try rig-core first, fall back to raw HTTP on 422 errors (most compatible)
  • rig — always use rig-core (may 422 on strict backends)
  • raw — always use raw HTTP with plain strings (bypasses rig-core)

Structured output

use_structured_output = true   # JSON schema validation (API only)
structured_output_retries = 2  # retries on validation failure

The server asks the API for JSON schema-validated output, retrying if the LLM returns malformed JSON. Disable for backends that don’t support it.

Generation settings (both providers)

temperature = 0.7              # 0.0 = deterministic, 2.0 = very creative
max_tokens = 4096              # max tokens per completion
max_concurrent_requests = 1    # 0 = unlimited, 1 = serial (safest)
strip_tags = ["think"]         # XML tags to strip from LLM output (e.g. "think", "reason")

Advanced / batching

batch_size = 10                # items per batch request
batch_max_tokens = 3000        # tokens per batch request (must be ≤ max_tokens)
batch_timeout_secs = 120       # base timeout for batch calls
batch_timeout_multiplier = 1.0 # timeout multiplier

The server batches independent LLM calls (e.g. classifying multiple memories at once) to reduce API overhead.

Next