AgentSkillsCN

multi-dataset-unification-pipeline

当用户请求从Hugging Face或其他类似资源库中查找、分析并整合多组数据集，使之统一为一种标准化格式时，该技能会全流程地完成数据集的搜寻、结构化分析与格式规范的解析，将各类数据集格式（ToolACE、Glaive、XLAM等）转换为统一的规范模式，并最终合并为单一输出文件。关键触发条件包括“数据集统一”“格式转换”“数据集合并”“Hugging Face数据集”“工具调用数据集”“JSONL输出”，或当用户提供了诸如“unified_format.md”之类的格式规范文档时。

SKILL.md
--- frontmatter
name: multi-dataset-unification-pipeline
description: When the user requests to find, analyze, and merge multiple datasets from Hugging Face or similar repositories into a unified format. This skill handles the complete pipeline searching for datasets, examining their structure, parsing format specifications, converting diverse dataset formats (ToolACE, Glaive, XLAM, etc.) into a standardized schema, and merging them into a single output file. Key triggers include requests involving 'dataset unification', 'format conversion', 'merge datasets', 'Hugging Face datasets', 'tool-calling datasets', 'JSONL output', or when users provide a format specification document like 'unified_format.md'.

Instructions

1. Understand the Request

  • •Identify the target datasets (names, IDs, or search queries).
  • •Locate and read any provided format specification (e.g., unified_format.md).
  • •Clarify the scope: number of entries per dataset, output file name/location.

2. Search and Validate Datasets

  • •Use huggingface-dataset_search to find each dataset.
  • •Use huggingface-hub_repo_details to get metadata and confirm availability.
  • •Note the dataset size and structure.

3. Analyze Dataset Structures

  • •Load a sample from each dataset using datasets.load_dataset.
  • •Examine the column names and a few sample entries.
  • •Identify the native format (e.g., ToolACE uses system/conversations, Glaive uses conversations/tools, XLAM uses query/answers/tools).

4. Convert to Unified Format

  • •Always refer to the provided unified_format.md (or equivalent) for the target schema.
  • •Key Rules:
    • •conversation_id: Format as {source_short_name}_{index} (e.g., toolace_0).
    • •messages: List of messages with role (user/assistant/tool), content, and optionally tool_calls or tool_call_id.
    • •tool_calls: For assistant messages, include id (format tool_call_{n}), name, arguments.
    • •tools: List of normalized tool definitions.
    • •Remove system messages if they only contain tool instructions.
    • •If assistant only made tool calls, set content to null.
    • •Do not add tool return results or assistant replies not in the original data.
  • •Use the bundled scripts/convert_and_merge.py for reliable, deterministic conversion of ToolACE, Glaive, and XLAM formats.
  • •For new dataset formats, write a custom converter following the patterns in the script.

5. Merge and Output

  • •Limit entries per dataset as requested (default: first 500).
  • •Merge all converted entries into a single list.
  • •Write to a JSONL file in the workspace (default: unified_tool_call.jsonl).
  • •Verify the output: count entries, check file size, validate format against the specification.

6. Finalize

  • •Provide a summary: datasets found, entries converted, output location.
  • •Optionally, show a sample entry from each source for validation.
  • •Confirm the task is complete.