FireCrawl Known Pitfalls

Overview

Common mistakes and anti-patterns when integrating with FireCrawl.

Prerequisites

•Access to FireCrawl codebase for review
•Understanding of async/await patterns
•Knowledge of security best practices
•Familiarity with rate limiting concepts

Pitfall #1: Synchronous API Calls in Request Path

❌ Anti-Pattern

typescript

// User waits for FireCrawl API call
app.post('/checkout', async (req, res) => {
  const payment = await firecrawlClient.processPayment(req.body);  // 2-5s latency
  const notification = await firecrawlClient.sendEmail(payment);   // Another 1-2s
  res.json({ success: true });  // User waited 3-7s
});

✅ Better Approach

typescript

// Return immediately, process async
app.post('/checkout', async (req, res) => {
  const jobId = await queue.enqueue('process-checkout', req.body);
  res.json({ jobId, status: 'processing' });  // 50ms response
});

// Background job
async function processCheckout(data) {
  const payment = await firecrawlClient.processPayment(data);
  await firecrawlClient.sendEmail(payment);
}

Pitfall #2: Not Handling Rate Limits

❌ Anti-Pattern

typescript

// Blast requests, crash on 429
for (const item of items) {
  await firecrawlClient.process(item);  // Will hit rate limit
}

✅ Better Approach

typescript

import pLimit from 'p-limit';

const limit = pLimit(5);  // Max 5 concurrent
const rateLimiter = new RateLimiter({ tokensPerSecond: 10 });

for (const item of items) {
  await rateLimiter.acquire();
  await limit(() => firecrawlClient.process(item));
}

Pitfall #3: Leaking API Keys

❌ Anti-Pattern

typescript

// In frontend code (visible to users!)
const client = new FireCrawlClient({
  apiKey: 'sk_live_ACTUAL_KEY_HERE',  // Anyone can see this
});

// In git history
git commit -m "add API key"  // Exposed forever

✅ Better Approach

typescript

// Backend only, environment variable
const client = new FireCrawlClient({
  apiKey: process.env.FIRECRAWL_API_KEY,
});

// Use .gitignore
.env
.env.local
.env.*.local

Pitfall #4: Ignoring Idempotency

❌ Anti-Pattern

typescript

// Network error on response = duplicate charge!
try {
  await firecrawlClient.charge(order);
} catch (error) {
  if (error.code === 'NETWORK_ERROR') {
    await firecrawlClient.charge(order);  // Charged twice!
  }
}

✅ Better Approach

typescript

const idempotencyKey = `order-${order.id}-${Date.now()}`;

await firecrawlClient.charge(order, {
  idempotencyKey,  // Safe to retry
});

Pitfall #5: Not Validating Webhooks

❌ Anti-Pattern

typescript

// Trust any incoming request
app.post('/webhook', (req, res) => {
  processWebhook(req.body);  // Attacker can send fake events
  res.sendStatus(200);
});

✅ Better Approach

typescript

app.post('/webhook',
  express.raw({ type: 'application/json' }),
  (req, res) => {
    const signature = req.headers['x-firecrawl-signature'];
    if (!verifyFireCrawlSignature(req.body, signature)) {
      return res.sendStatus(401);
    }
    processWebhook(JSON.parse(req.body));
    res.sendStatus(200);
  }
);

Pitfall #6: Missing Error Handling

❌ Anti-Pattern

typescript

// Crashes on any error
const result = await firecrawlClient.get(id);
console.log(result.data.nested.value);  // TypeError if missing

✅ Better Approach

typescript

try {
  const result = await firecrawlClient.get(id);
  console.log(result?.data?.nested?.value ?? 'default');
} catch (error) {
  if (error instanceof FireCrawlNotFoundError) {
    return null;
  }
  if (error instanceof FireCrawlRateLimitError) {
    await sleep(error.retryAfter);
    return this.get(id);  // Retry
  }
  throw error;  // Rethrow unknown errors
}

Pitfall #7: Hardcoding Configuration

❌ Anti-Pattern

typescript

const client = new FireCrawlClient({
  timeout: 5000,  // Too short for some operations
  baseUrl: 'https://api.firecrawl.com',  // Can't change for staging
});

✅ Better Approach

typescript

const client = new FireCrawlClient({
  timeout: parseInt(process.env.FIRECRAWL_TIMEOUT || '30000'),
  baseUrl: process.env.FIRECRAWL_BASE_URL || 'https://api.firecrawl.com',
});

Pitfall #8: Not Implementing Circuit Breaker

❌ Anti-Pattern

typescript

// When FireCrawl is down, every request hangs
for (const user of users) {
  await firecrawlClient.sync(user);  // All timeout sequentially
}

✅ Better Approach

typescript

import CircuitBreaker from 'opossum';

const breaker = new CircuitBreaker(firecrawlClient.sync, {
  timeout: 10000,
  errorThresholdPercentage: 50,
  resetTimeout: 30000,
});

// Fails fast when circuit is open
for (const user of users) {
  await breaker.fire(user).catch(handleFailure);
}

Pitfall #9: Logging Sensitive Data

❌ Anti-Pattern

typescript

console.log('Request:', JSON.stringify(request));  // Logs API key, PII
console.log('User:', user);  // Logs email, phone

✅ Better Approach

typescript

const redacted = {
  ...request,
  apiKey: '[REDACTED]',
  user: { id: user.id },  // Only non-sensitive fields
};
console.log('Request:', JSON.stringify(redacted));

Pitfall #10: No Graceful Degradation

❌ Anti-Pattern

typescript

// Entire feature broken if FireCrawl is down
const recommendations = await firecrawlClient.getRecommendations(userId);
return renderPage({ recommendations });  // Page crashes

✅ Better Approach

typescript

let recommendations;
try {
  recommendations = await firecrawlClient.getRecommendations(userId);
} catch (error) {
  recommendations = await getFallbackRecommendations(userId);
  reportDegradedService('firecrawl', error);
}
return renderPage({ recommendations, degraded: !recommendations });

Instructions

Step 1: Review for Anti-Patterns

Scan codebase for each pitfall pattern.

Step 2: Prioritize Fixes

Address security issues first, then performance.

Step 3: Implement Better Approach

Replace anti-patterns with recommended patterns.

Step 4: Add Prevention

Set up linting and CI checks to prevent recurrence.

Output

•Anti-patterns identified
•Fixes prioritized and implemented
•Prevention measures in place
•Code quality improved

Error Handling

Issue	Cause	Solution
Too many findings	Legacy codebase	Prioritize security first
Pattern not detected	Complex code	Manual review
False positive	Similar code	Whitelist exceptions
Fix breaks tests	Behavior change	Update tests

Examples

Quick Pitfall Scan

bash

# Check for common pitfalls
grep -r "sk_live_" --include="*.ts" src/        # Key leakage
grep -r "console.log" --include="*.ts" src/     # Potential PII logging

Resources

Quick Reference Card

Pitfall	Detection	Prevention
Sync in request	High latency	Use queues
Rate limit ignore	429 errors	Implement backoff
Key leakage	Git history scan	Env vars, .gitignore
No idempotency	Duplicate records	Idempotency keys
Unverified webhooks	Security audit	Signature verification
Missing error handling	Crashes	Try-catch, types
Hardcoded config	Code review	Environment variables
No circuit breaker	Cascading failures	opossum, resilience4j
Logging PII	Log audit	Redaction middleware
No degradation	Total outages	Fallback systems