API Provider Testing Skill

Overview

Guidelines for testing LLM API providers when adding or modifying support in Caret.

Test Categories

1. Basic Conversation Test

• •Simple prompt → response verification
• •Streaming chunk reception
• •Token usage reporting

Example:

const response = await handler.createMessage("System prompt", [
  { role: 'user', content: 'Hello' }
])
// Verify: text chunks + usage chunk

2. Tool Calling Test

Core verification for agentic capabilities.

Test Flow:

1. •Send message with tool definitions
2. •Verify tool_calls chunk received
3. •Verify function name and arguments
4. •Send tool result back
5. •Verify final text response

Provider-Specific Formats:

3. Timeout Test

• •Use AbortController with configurable timeout
• •Default: 60 seconds recommended
• •Verify timeout error is thrown correctly

4. Error Handling Test

• •Invalid API key → 401/403 error
• •Rate limiting → 429 error with retry
• •Empty response → specific error message

Test Script Template

Location: scripts/test-{provider}-api.js

// Load .env
const fs = require('fs')
const path = require('path')
const envPath = path.resolve(__dirname, '../.env')
if (fs.existsSync(envPath)) {
  fs.readFileSync(envPath, 'utf-8').split('\n').forEach((line) => {
    const match = line.match(/^([^=]+)=(.*)$/)
    if (match && !process.env[match[1]]) {
      process.env[match[1]] = match[2]
    }
  })
}

// Test structure
async function testBasicConversation() { ... }
async function testToolCalling() { ... }
async function testToolResultFlow() { ... }
async function testTimeout() { ... }

async function main() {
  const results = {
    basic: await testBasicConversation(),
    toolCall: await testToolCalling(),
    toolFlow: await testToolResultFlow(),
  }
  console.log('Results:', results)
}

Existing Test Scripts

Key Files

Checklist for New Provider

1. • Basic conversation works
2. • Streaming chunks parse correctly
3. • Tool calling triggers tool_calls type chunk
4. • Tool result → final response flow works
5. • Timeout with AbortController works
6. • Error messages are clear
7. • Token usage reported correctly
8. • Integration test added to __tests__/

Mirroring Policy

.agents/와 .users/는 1:1 미러링 구조입니다.

• •이 파일 수정 시 .users/skills/api-provider-testing/SKILL.md도 동일하게 업데이트
• •.agents/는 영어(토큰 효율), .users/는 사용자/팀 언어(상세 설명)
• •참조: assets/agents_template/AGENTS.md의 Key Principles