AgentSkillsCN

zireael-unicode-text

实现确定性的 UTF-8 编码、语素分割、宽度策略以及换行原语。

SKILL.md
--- frontmatter
name: zireael-unicode-text
description: Implement deterministic UTF-8, grapheme segmentation, width policy, and wrapping primitives.
metadata:
  short-description: Unicode correctness + policies

When to use

Use this skill when implementing or modifying:

  • •UTF-8 decoder behavior
  • •grapheme segmentation rules
  • •width measurement policy (CJK/emoji/VS/ZWJ)
  • •wrapping and tab expansion

Source of truth

  • •docs/modules/UNICODE_TEXT.md — module specification
  • •docs/VERSION_PINS.md — Unicode 15.1.0, emoji-wide policy, invalid UTF-8 policy

Pinned policies (locked)

  • •Unicode version: 15.1.0
  • •Emoji width: wide (2 columns)
  • •Invalid UTF-8: emit U+FFFD, consume 1 byte, continue

Implementation requirements

Must provide deterministic primitives:

  • •UTF-8 decode (invalid sequences handled deterministically)
  • •grapheme-safe ops (never split a cluster)
  • •column width measurement + wrapping

Keep Unicode code OS-header-free (src/unicode/ + src/util/ only).

Invalid UTF-8 handling

  • •Never read past buffer bounds
  • •Make progress deterministically (consume at least 1 byte)
  • •Return replacement U+FFFD for invalid sequences
  • •Mark result as invalid for caller inspection

Testing

Unit tests:

  • •UTF-8 valid/invalid sequences
  • •width edge cases (emoji, ZWJ, VS16)
  • •grapheme boundaries
  • •wrapping positions
  • •tab expansion

Fuzz tests:

  • •UTF-8 decoder: no crash/hang on arbitrary bytes
  • •Grapheme iterator: always makes progress