Memories

Memories / Documentation

On this page

Search, embeddings and retrieval-augmented generation

Memories supports keyword and semantic retrieval over authorized ordinary memory and knowledge cards. Document ingestion accepts UTF-8 text and Markdown and creates a normal encrypted knowledge card. PDF/DOCX/binary file extraction is not automatic; convert them to text before explicit ingestion. Private content, credentials and file assets are excluded from retrieval.

Quick start

Use the same home/config for the CLI and running daemon. Restart the daemon after upgrading or changing model configuration. Initialize/unlock as usual, then:

memories rag ingest VAULT_ID notes.md --title "Research notes"
memories memory search VAULT_ID "Chinese English keywords"
memories rag search VAULT_ID "keywords" --mode bm25 --match-mode all
memories rag model-install
memories rag index VAULT_ID
memories rag search VAULT_ID "Find similar meaning" --mode vector
memories rag search VAULT_ID "Find evidence" --mode hybrid --limit 8
memories rag status VAULT_ID
memories rag ask VAULT_ID "What does the evidence say?"

rag ask requires a generation URL in the daemon configuration. To use lexical RAG without an embedding model, add --mode bm25; the generation URL is still required. memory_search remains lexical. Advanced search returns document chunks with asset/revision IDs, stable chunk hashes, original UTF-8 byte offsets, scores and citation labels. [S1] identifies a source within one result; use its id and revision fields for ordinary memory read, not the chunk hash.

Chinese BM25 and chunks

Chinese and mixed-language text are tokenized with Jieba. Titles, bodies and safe metadata participate in FTS5 BM25 ranking, weighted 5/1/2. any (default) combines keywords; all requires every query term; phrase requires the original phrase. Strict all/phrase constraints also filter vector/hybrid candidates. Queries are escaped, not interpreted as arbitrary FTS operators.

Text is split near sentence/paragraph boundaries into chunks of up to 768 Unicode characters, normally overlapping by 128 characters. Returned offsets address the original UTF-8 body. Search is bounded to 100 requested results, 64 lexical terms and 1,024 query bytes. Basic memory search groups chunks by document; advanced RAG retrieval returns individual chunks. --cards-only narrows retrieval.

Indexes are reused between queries. A vault mutation, sync or policy change invalidates its cached index; locking/restarting clears all indexes and vectors. They are not persisted as plaintext or added to encrypted sync payloads. Limits per cached vault are 10,000 chunks, 16 MiB of source text/metadata and 32 MiB of vectors; at most two vaults are cached. Large libraries still face the underlying whole-state persistence limits described in performance.

In-process EmbeddingGemma

The default embedding backend is Google's EmbeddingGemma 300M Q4 via native ONNX inference. It runs inside the Memories daemon, with no Ollama, Python or separate inference service required. The application includes the inference engine with ONNX telemetry explicitly disabled before session creation; the model/tokenizer files (about 209 MiB) are installed once by the app:

memories rag model-install

The installer downloads a pinned onnx-community/embeddinggemma-300m-ONNX revision, verifies SHA-256, supports ranged downloads/retries, and never replaces an existing mismatching file. Existing verified files are reused. The default directory is models/embeddinggemma under the selected Memories home. Supply an existing directory with rag.local_model_dir; it must contain the Q4 ONNX graph, its external data and the tokenizer/config files. Restart the daemon after changing model files. Weights are governed separately by the Gemma Terms of Use and Prohibited Use Policy; the installer writes a notice and upstream model card.

[rag]
local_model_dir = "./models/embeddinggemma"
local_threads = 4
max_tokens = 2048
# Optional mirror of the exact pinned files:
# model_download_base_url = "https://your-mirror.example/embeddinggemma"

Relative model paths are resolved beside the config file. Lower local_threads limits CPU use; max_tokens bounds tokenizer input and can truncate long inputs when set below the chunk's token count. Embeddings are 768-dimensional normalized vectors. The model uses separate query/document task prefixes. rag index warms the index; vector/hybrid retrieval also computes missing embeddings on demand. A five-minute operation budget is checked between steps; completed batches can be reused on retry while the vault/model remain unchanged and unlocked.

Customer-supplied model URLs

Gemini Embedding is an online API, not the offline EmbeddingGemma model. Add rag.embedding to select a remote provider; omit it to keep embeddings local. Use full operation URLs and configure the model expected by your service:

[rag.embedding]
protocol = "gemini"
url = "https://generativelanguage.googleapis.com/v1beta/models/gemini-embedding-001:embedContent"
model = "gemini-embedding-001"
dimensions = 768
api_key_env = "GEMINI_API_KEY"

[rag.generation]
protocol = "gemini"
url = "https://generativelanguage.googleapis.com/v1beta/models/YOUR_MODEL:generateContent"
model = "YOUR_MODEL"
api_key_env = "GEMINI_API_KEY"

Set the named API-key environment variable in the daemon's environment. Do not put key values in TOML, URLs or command arguments. Gemini uses x-goog-api-key; protocol = "compatible" uses Bearer authentication and common {model,input} embedding / chat-completion JSON formats. The compatible embedding response must contain indexed data entries with embedding arrays; generation returns choices[0].message.content. Keys are optional for local services. Remote URLs require HTTPS; HTTP is accepted for loopback only. Optional embedding dimensions is sent as Gemini outputDimensionality or compatible dimensions, checked on receipt and bound into the vector-space key. Credentials, query strings and fragments are not accepted in operation URLs.

There is no implicit public model endpoint. Configuring a remote embedding URL allows eligible document chunks to be sent there. Generation sends only the selected chunks and question; no stored credential/private/file content enters this pipeline. These configuration settings belong to the daemon, not individual MCP calls. See security.

Hybrid search and reranking

Vector mode uses cosine similarity. Hybrid mode merges BM25 and vector rankings with reciprocal-rank fusion (k=60). Default lightweight reranking boosts term coverage and phrase matches. It does not claim to be a learned cross-encoder. Configure a learned reranker service when desired:

[rag.rerank]
protocol = "compatible"
url = "https://your-reranker.example/rerank"
model = "YOUR_RERANKER"
api_key_env = "RERANK_API_KEY"

The request is {model, query, documents, top_n}. Its response must contain results with a unique valid index and finite relevance_score for every candidate. Unknown/duplicate entries fail instead of silently reordering the wrong document. --no-rerank skips both optional reranking stages.

Proxies, timeouts and retries

[rag.http]
use_env_proxy = true
proxy_url = "http://127.0.0.1:7890"
retries = 2
backoff_ms = 500
connect_timeout_seconds = 10
timeout_seconds = 60
download_timeout_seconds = 900

Explicit proxies override environment proxy discovery. HTTP, HTTPS, SOCKS5 and SOCKS5h URLs are supported. Authenticated proxy values can be supplied through the proxy environment; TOML proxy URLs cannot contain credentials. Loopback model/mirror endpoints always connect directly. TLS verification stays enabled.

retries is the number of extra attempts (0-5). Model API connection/read failures, 429 and 500/502/503/504 can retry with exponential backoff; Retry-After seconds are honored up to 30 seconds. Authentication/validation errors, unsafe redirects and malformed model output are not retried. Model API redirects are disabled. Download redirects require HTTPS and every final file is hash-checked. The download timeout is a per-file budget. Retrying generation can incur extra provider billing. Authorization/cancellation is rechecked between attempts.

Answers, citations and MCP

memory_retrieve exposes scoped retrieval to MCP clients; memory_ask calls the configured generation service. Both preserve the grant's vault and ordinary-data restrictions. They cannot select an arbitrary endpoint or access a credential. Device-only vaults reject operations that would send data to non-loopback model endpoints; local inference and explicitly configured loopback services work. The older memory_search and memory_read tools continue to work. The bundled Memories skill documents memory_retrieve and memory_ask for AI clients. After upgrading, replace an older installed skill with memories inject all --force.

Generation receives source text as untrusted data and has no application tools. The model must return structured answer/citation JSON, with citations such as [S1] that actually exist in the result. Invalid or invented IDs are rejected. When there is no evidence, no generation call is made and the result marks insufficient_evidence. Citation validation checks references, not whether every claim is factually supported; review important answers against their sources.

Lock, expired/revoked grants and changed source state invalidate in-flight work and suppress late results. An already started remote request cannot be recalled. No model-backed RAG interface is added to the mobile FFI in this release; mobile ordinary search receives the Chinese/BM25 improvements and can use BM25 retrieval.