Semantic Search Technique

Purpose

Decision framework and execution guide for using semantic search effectively in CodeCompass.

When to Use Semantic Search

✅ Use Semantic Search When

1. Concept-based Queries

• •"Find code that handles payment processing"
• •"Where do we validate email addresses?"
• •"Show me error handling patterns"

2. Intent-based Queries

• •"How is user authentication implemented?"
• •"What code calculates shipping costs?"
• •"Find business rules for order approval"

3. Cross-language/Cross-file

• •Searching across PHP, TypeScript, config files
• •Pattern discovery across multiple modules
• •Finding similar implementations

4. Fuzzy/Exploratory

• •User doesn't know exact class/function names
• •Exploring unfamiliar codebase
• •"Code that does something like X"

5. Natural Language

• •"Show me all database migrations"
• •"Find controllers that handle file uploads"
• •"Where are API rate limits defined?"

❌ Use Grep/Glob When

1. Exact Matches

• •"Find class named PaymentController"
• •"Where is processPayment function defined?"
• •"Find all imports of UserService"

2. Syntax Patterns

• •"Find all functions starting with get"
• •"Show me all @Injectable() decorators"
• •"Find TypeScript interfaces"

3. Performance Critical

• •Quick lookups in known files
• •Repeated searches in tight loops
• •When you know exact location

4. Structural Queries

• •"Find all .ts files in src/modules"
• •"List all test files"
• •"Show directory structure"

Execution Guide

Step 1: Formulate Effective Query

❌ Bad Queries (too vague):

• •"payment"
• •"code"
• •"function"

✅ Good Queries (specific context):

• •"business logic for processing customer payments and updating order status"
• •"validation rules for user email and password requirements"
• •"error handling patterns for database connection failures"

Why: More context = better semantic matching

Formula:

[Action/Purpose] for [Specific Entity] with [Context/Constraints]

Examples:

• •"Extract business capabilities from Yii2 controllers"
• •"Validation logic for user registration with email verification"
• •"Database migration patterns for schema versioning"

Step 2: Verify Indexing

Before searching, ensure codebase is indexed:

# Check if indexed
curl http://localhost:8081/v1/schema

# Should show collections like:
# - CodeContext
# - AtlasCode

If not indexed:

codecompass batch:index <path-to-codebase>

Step 3: Execute Search

codecompass search:semantic "business logic for payment processing"

Alternative (if using as library):

const results = await searchService.semanticSearch({
  query: "business logic for payment processing",
  limit: 10,
  certainty: 0.7 // Minimum relevance score
});

Step 4: Interpret Results

Check relevance scores:

• •>0.8: Highly relevant (exact match)
• •0.7-0.8: Good match (related)
• •0.6-0.7: Moderate match (possibly relevant)
• •<0.6: Weak match (may be noise)

Verify context:

• •Does the returned code actually match intent?
• •Are results from expected modules?
• •Multiple related files found (good signal)
• •Or isolated random matches (refine query)

Step 5: Refine if Needed

Too many results (>50):

• •Add more specific context to query
• •Increase certainty threshold
• •Add domain constraints ("in authentication module")

Too few results (<3):

• •Broaden query (less specific)
• •Lower certainty threshold
• •Check if area is actually indexed
• •Try related terms/synonyms

Wrong results:

• •Rephrase query with different terminology
• •Add negative constraints
• •Try breaking into multiple specific queries

Behind the Scenes

Architecture

Query Text
    ↓
Ollama Embedding (mxbai-embed-large)
    ↓
1024-dimensional vector
    ↓
Weaviate Vector Search (cosine similarity)
    ↓
Ranked Results

Key Components

From .ai/capabilities.json:

• •Module: search, vectorizer, weaviate
• •Embedding: Ollama mxbai-embed-large (1024 dimensions)
• •Vector DB: Weaviate with HNSW indexing
• •Collections: CodeContext, AtlasCode

Configuration (from .env):

EMBEDDING_SERVICE=ollama
OLLAMA_EMBEDDING_MODEL=mxbai-embed-large
OLLAMA_URL=http://localhost:11434
CODECOMPASS_WEAVIATE_URL=http://localhost:8081

Advanced Patterns

Pattern 1: Multi-Query Exploration

For complex questions, break into multiple searches:

# Instead of:
"authentication and authorization and session management"

# Do:
codecompass search:semantic "user authentication login process"
codecompass search:semantic "authorization and access control"
codecompass search:semantic "session management and tokens"

Pattern 2: Iterative Refinement

# 1. Broad search
codecompass search:semantic "payment processing"

# 2. Review results, identify specific module
# 3. Narrow search
codecompass search:semantic "payment gateway integration in PaymentController"

# 4. Pinpoint implementation
codecompass search:semantic "Stripe API call for processing credit cards"

Pattern 3: Cross-Domain Search

Search across different aspects:

# Code implementation
codecompass search:semantic "email validation logic"

# Tests
codecompass search:semantic "test cases for email validation"

# Configuration
codecompass search:semantic "email service configuration"

Common Pitfalls

❌ Pitfall 1: Searching Before Indexing

Symptom: No results or error Solution: Run codecompass batch:index first

❌ Pitfall 2: Too Vague Queries

Symptom: Returns everything or nothing useful Solution: Add specific context and intent

❌ Pitfall 3: Expecting Exact Matches

Symptom: "Why didn't it find function processPayment?" Reason: Semantic search is for concepts, not exact names Solution: Use grep for exact matches

❌ Pitfall 4: Ignoring Relevance Scores

Symptom: Reading irrelevant results Solution: Filter by score >0.7, ignore weak matches

❌ Pitfall 5: Single Query for Complex Questions

Symptom: Poor results for multi-faceted questions Solution: Break into multiple targeted queries

Decision Tree

┌─────────────────────────────────────┐
│ I need to find code that...         │
└─────────────────────────────────────┘
              ↓
        ┌─────────┐
        │ Know    │ Exact class/function name?
        │ exact   │
        │ name?   │
        └─────────┘
          ↙     ↘
        YES      NO
         ↓        ↓
    Use Grep  ┌─────────┐
              │ Concept │ Searching by meaning/purpose?
              │ search? │
              └─────────┘
                ↙     ↘
              YES      NO
               ↓        ↓
          Semantic  ┌─────────┐
          Search    │ Pattern │ Looking for code pattern?
                    │ match?  │
                    └─────────┘
                      ↙     ↘
                    YES      NO
                     ↓        ↓
                Use Glob  Use both
                          (Glob + Semantic)

Performance Considerations

Speed

• •Grep: Milliseconds (fast, synchronous)
• •Semantic Search: 100-500ms (embedding + vector search)

Tradeoff: Semantic is slower but finds conceptually related code

Token Cost (Embeddings)

• •Each query → 1 embedding generation
• •Ollama local → No API cost
• •But consumes local compute

Scaling

• •Small codebase (<1K files): Either method fine
• •Medium codebase (1K-10K files): Semantic search advantage grows
• •Large codebase (>10K files): Semantic search essential

Integration with Other Tools

With Yii2 Analysis

# 1. Analyze Yii2 project
codecompass analyze:yii2 <path>

# 2. Index results
codecompass batch:index <path>

# 3. Explore with semantic search
codecompass search:semantic "Yii2 controller actions for user management"

With Requirements Extraction

# 1. Extract requirements
codecompass requirements:extract

# 2. Search extracted requirements
codecompass search:semantic "business rules for order validation"

With Weaviate Direct Query

# Alternative: Query Weaviate GraphQL API directly
curl -X POST http://localhost:8081/v1/graphql \
  -H "Content-Type: application/json" \
  -d '{
    "query": "{
      Get {
        CodeContext(
          nearText: { concepts: [\"payment processing\"] }
          limit: 10
        ) {
          content
          filePath
        }
      }
    }"
  }'

Related Skills

• •0-discover-capabilities.md - How to discover modules
• •analyze-yii2-project.md - Uses semantic search in workflow

Related Modules

From .ai/capabilities.json:

• •search - SearchService, IntegratedSearchService
• •vectorizer - Ollama embedding generation
• •weaviate - Vector database client
• •indexing - File indexing pipeline

Remember: Semantic search finds code by meaning, not by name. Choose the right tool for the job.