AgentSkillsCN

data-import-workflow

从外部数据源到本地数据库，管理武器与模组数据的全生命周期

SKILL.md
--- frontmatter
name: data-import-workflow
description: Managing the lifecycle of weapon and mod data from external sources to the local database

Data Import Workflow Skill

Use this skill when working on or running the data import pipeline.

Scope

  • •Running the importer to populate or refresh data
  • •Extending the importer to handle new fields or types
  • •Debugging import failures

Running the Importer

CommandUse Case
task importer:startFetch fresh data from tarkov.dev API and update the database.
task importer:start:use-cacheUse locally cached JSON files (faster, avoids API rate limits).
task importer:start:cache-onlyFetch from API and update local cache, but skip database writes.

Import Behavior

  1. •Purge: Existing weapons, mods, trader offers, and optimum builds are deleted.
  2. •Fetch: Data is retrieved from tarkov.dev GraphQL API (or local cache).
  3. •Transform: External schema is mapped to internal models structs.
  4. •Persist: Data is written to the database within a transaction.
  5. •Invalidate: optimum_builds are purged since they depend on the imported data.

Extending the Importer

When adding new fields or types:

  1. •Update GraphQL Query: Modify internal/tarkovdev/schemas/queries.graphql.
  2. •Regenerate Client: Run task tarkovdev:regenerate.
  3. •Update Models: Add new fields to structs in internal/models/.
  4. •Update Mapping: Adjust conversion logic in internal/importers/.
  5. •Update SQL: Modify Upsert functions in internal/models/ to include new columns.

Troubleshooting

  • •Rate Limiting: Use --use-cache for repeated runs during development.
  • •Schema Mismatch: Run task tarkovdev:get-schema then task tarkovdev:regenerate.
  • •Connection Issues: Verify POSTGRES_HOST in .env and ensure the database is running.