AgentSkillsCN

vector-memory-hack

利用 TF-IDF 与 SQLite，为 AI 代理的记忆文件提供快速语义搜索功能。使代理能够在启动任务前，即时检索 MEMORY.md 或任意 Markdown 文档中的相关上下文。当代理需要：(1) 在开始任务之前快速找到相关背景信息；(2) 高效搜索大型记忆文件；(3) 无需逐行阅读文件，即可快速定位特定规则或决策；(4) 以语义相似度搜索替代传统的关键词匹配。相较于重量级的嵌入式模型，此方法轻量且无需外部依赖，搜索耗时低于 10 毫秒。

SKILL.md
--- frontmatter
name: vector-memory-hack
description: Fast semantic search for AI agent memory files using TF-IDF and SQLite. Enables instant context retrieval from MEMORY.md or any markdown documentation. Use when the agent needs to (1) Find relevant context before starting a task, (2) Search through large memory files efficiently, (3) Retrieve specific rules or decisions without reading entire files, (4) Enable semantic similarity search instead of keyword matching. Lightweight alternative to heavy embedding models - zero external dependencies, <10ms search time.

Vector Memory Hack

Ultra-lightweight semantic search for AI agent memory systems. Find relevant context in milliseconds without heavy dependencies.

Why Use This?

Problem: AI agents waste tokens reading entire MEMORY.md files (3000+ tokens) just to find 2-3 relevant sections.

Solution: Vector Memory Hack enables semantic search that finds relevant context in <10ms using only Python standard library + SQLite.

Benefits:

  • •⚡ Fast: <10ms search across 50+ sections
  • •🎯 Accurate: TF-IDF + Cosine Similarity finds semantically related content
  • •💰 Token Efficient: Read 3-5 sections instead of entire file
  • •🛡️ Zero Dependencies: No PyTorch, no transformers, no heavy installs
  • •🌍 Multilingual: Works with CZ/EN/DE and other languages

Quick Start

1. Index your memory file

bash
python3 scripts/vector_search.py --rebuild

2. Search for context

bash
# Using the CLI wrapper
vsearch "backup config rules"

# Or directly
python3 scripts/vector_search.py --search "backup config rules" --top-k 5

3. Use results in your workflow

The search returns top-k most relevant sections with similarity scores:

code
1. [0.288] Auto-Backup System
   Script: /root/.openclaw/workspace/scripts/backup-config.sh
   ...

2. [0.245] Security Rules
   Never send emails without explicit user consent...

How It Works

code
MEMORY.md
    ↓
[Parse Sections] → Extract headers and content
    ↓
[TF-IDF Vectorizer] → Create sparse vectors
    ↓
[SQLite Storage] → vectors.db
    ↓
[Cosine Similarity] → Find top-k matches

Technology Stack:

  • •Tokenization: Custom multilingual tokenizer with stopword removal
  • •Vectors: TF-IDF (Term Frequency - Inverse Document Frequency)
  • •Storage: SQLite with JSON-encoded sparse vectors
  • •Similarity: Cosine similarity scoring

Commands

Rebuild Index

bash
python3 scripts/vector_search.py --rebuild

Parses MEMORY.md, computes TF-IDF vectors, stores in SQLite.

Incremental Update

bash
python3 scripts/vector_search.py --update

Only processes changed sections (hash-based detection).

Search

bash
python3 scripts/vector_search.py --search "your query" --top-k 5

Statistics

bash
python3 scripts/vector_search.py --stats

Integration for Agents

Required step before every task:

bash
# Agent receives task: "Update SSH config"
# Step 1: Find relevant context
vsearch "ssh config changes"

# Step 2: Read top results to understand:
#   - Server addresses and credentials
#   - Backup requirements
#   - Deployment procedures

# Step 3: Execute task with full context

Configuration

Edit these variables in scripts/vector_search.py:

python
MEMORY_PATH = Path("/path/to/your/MEMORY.md")
VECTORS_DIR = Path("/path/to/vectors/storage")
DB_PATH = VECTORS_DIR / "vectors.db"

Customization

Adding Stopwords

Edit the stopwords set in _tokenize() method for your language.

Changing Similarity Metric

Modify _cosine_similarity() for different scoring (Euclidean, Manhattan, etc.)

Batch Processing

Use rebuild() for full reindex, update() for incremental changes.

Performance

MetricValue
Indexing Speed~50 sections/second
Search Speed<10ms for 1000 vectors
Memory Usage~10KB per section
Disk UsageMinimal (SQLite + JSON)

Comparison with Alternatives

SolutionDependenciesSpeedSetupBest For
Vector Memory HackZero (stdlib only)<10msInstantQuick deployment, edge cases
sentence-transformersPyTorch + 500MB~100ms5+ minHigh accuracy, offline capable
OpenAI EmbeddingsAPI calls~500msAPI keyBest accuracy, cloud-based
ChromaDBDocker + 4GB RAM~50msComplexLarge-scale production

When to use Vector Memory Hack:

  • •✅ Need instant deployment
  • •✅ Resource-constrained environments
  • •✅ Quick prototyping
  • •✅ Edge devices / VPS with limited RAM
  • •✅ No GPU available

When to use heavier alternatives:

  • •Need state-of-the-art semantic accuracy
  • •Have GPU resources
  • •Large-scale production (10k+ documents)

File Structure

code
vector-memory-hack/
├── SKILL.md                  # This file
└── scripts/
    ├── vector_search.py      # Main Python module
    └── vsearch               # CLI wrapper (bash)

Example Output

bash
$ vsearch "backup config rules" 3

Search results for: 'backup config rules'

1. [0.288] Auto-Backup System
   Script: /root/.openclaw/workspace/scripts/backup-config.sh
   Target: /root/.openclaw/backups/config/
   Keep: Last 10 backups
   
2. [0.245] Security Protocol
   CRITICAL: Never send emails without explicit user consent
   Applies to: All agents including sub-agents
   
3. [0.198] Deployment Checklist
   Before deployment:
   1. Run backup-config.sh
   2. Validate changes
   3. Test thoroughly

Troubleshooting

"No sections found"

  • •Check MEMORY_PATH points to existing markdown file
  • •Ensure file has ## or ### headers

"All scores are 0.0"

  • •Rebuild index: python3 scripts/vector_search.py --rebuild
  • •Check vocabulary contains your search terms

"Database locked"

  • •Wait for other process to finish
  • •Or delete vectors.db and rebuild

License

MIT License - Free for personal and commercial use.

Created by: OpenClaw Agent (@mig6671)
Published on: ClawHub
Version: 1.0.0