CrawlChat Knowledge Collection Setup Guide

This guide covers creating and syncing a knowledge collection (web source) in CrawlChat.

Prerequisites

Follow setup-dev skill first to have the dev server running.

Create a Knowledge Group

1. Navigate to Knowledge page

Open: http://localhost:5173/knowledge

2. Create a new group

Click "Add group" and fill in:

• •Name: Docs (or your preferred name)
• •URL: https://docs.crawlchat.app (or your target website)
• •Source Type: Web

Click "Create" to add the group.

3. Start fetching

On the knowledge page, find your group and click "Refetch it" to start crawling.

Common Issues & Fixes

Issue 1: MongoDB Unique Constraint Error

Error:

PrismaClientKnownRequestError: Unique constraint failed on the constraint: `ScrapeItem_knowledgeGroupId_sourcePageId_key`

Fix: Drop the unique index from MongoDB:

docker exec crawlchat-local-database-1 mongosh --eval 'db.getSiblingDB("crawlchat").getCollection("ScrapeItem").dropIndex("ScrapeItem_knowledgeGroupId_sourcePageId_key")'

Issue 2: Source-sync OOM / Connection Pool Exhausted

Symptoms:

• •sorry, too many clients already (pgvector)
• •Process killed by OOM killer

Fix: Reduce worker concurrency in source-sync/src/worker.ts:

// Change from:
concurrency: 4,

// To:
concurrency: 1,

Issue 3: OpenRouter API Key Missing

Error: Embedding API calls fail

Fix: Add your OpenRouter API key to .env:

OPENROUTER_API_KEY=sk-or-v1-your-key-here

Get a free key from: https://openrouter.ai

Issue 4: Embedding API Response Error

Error:

Cannot read properties of undefined (reading '0')

Cause: The embedding API response validation is missing in packages/indexer/src/earth-indexer.ts.

Fix: Add null-safe check in makeEmbedding():

const data = await response.json();
if (!data?.data?.[0]?.embedding) {
  throw new Error(`Embedding API error: ${JSON.stringify(data)}`);
}
return data.data[0].embedding;

Useful Commands

Check embeddings in pgvector

docker exec crawlchat-local-pgvector-1 psql -U postgres -d crawlchat -c "SELECT COUNT(*) FROM earth_embeddings;"

Restart dev server

cd /root/.openclaw/workspace/crawlchat
pkill -9 -f "tsx"
npm run dev:core

Restart only front (if server crashes)

cd /root/.openclaw/workspace/crawlchat/front
npm run dev

Troubleshooting