IO Utilities Guide

This skill provides comprehensive guidance for using the IO utilities in speedy_utils.

When to Use This Skill

Use this skill when you need to:

• •Read and write data in various formats (JSON, JSONL, Pickle, CSV, TXT).
• •Efficiently process large JSONL files with streaming and multi-threading.
• •Automatically handle file compression (gzip, bz2, xz, zstd).
• •Load data based on file extension automatically.
• •Serialize Pydantic models and other objects easily.

Prerequisites

• •speedy_utils installed.
• •Optional dependencies for specific features:
  • •orjson: For faster JSON parsing.
  • •zstandard: For .zst file support.
  • •pandas: For CSV/TSV loading.
  • •pyarrow: For faster CSV reading with pandas.

Core Capabilities

Fast JSONL Processing (fast_load_jsonl)

• •Streams data line-by-line for memory efficiency.
• •Supports automatic decompression.
• •Uses orjson if available for speed.
• •Supports multi-threaded processing for large files.
• •Shows progress bar with tqdm.

Universal Loading (load_by_ext)

• •Detects file type by extension.
• •Supports glob patterns (e.g., data/*.json) and lists of files.
• •Uses parallel processing for multiple files.
• •Supports memoization via do_memoize=True.

Serialization (dump_json_or_pickle, load_json_or_pickle)

• •Unified interface for JSON and Pickle.
• •Handles Pydantic models automatically.
• •Creates parent directories if they don't exist.

Usage Examples

Example 1: Streaming Large JSONL

Read a large compressed JSONL file line by line.

from speedy_utils import fast_load_jsonl

# Iterates lazily, low memory usage
for item in fast_load_jsonl('large_data.jsonl.gz', progress=True):
    process(item)

Example 2: Loading Any File

Load a file without worrying about the format.

from speedy_utils import load_by_ext

data = load_by_ext('config.json')
df = load_by_ext('data.csv')
items = load_by_ext('dataset.pkl')

Example 3: Parallel Loading

Load multiple files in parallel.

from speedy_utils import load_by_ext

# Returns a list of results, one for each file
all_data = load_by_ext('logs/*.jsonl')

Example 4: Dumping Data

Save data to disk, creating directories as needed.

from speedy_utils import dump_json_or_pickle

data = {"key": "value"}
dump_json_or_pickle(data, 'output/processed/result.json')

Guidelines

1. •

   Prefer JSONL for Large Datasets:

   • •Use fast_load_jsonl for datasets that don't fit in memory.
   • •It handles compression transparently, so keep files compressed (.jsonl.gz or .jsonl.zst) to save space.
2. •

   Use load_by_ext for Scripts:

   • •When writing scripts that might accept different input formats, use load_by_ext to be flexible.
3. •

   Error Handling:

   • •fast_load_jsonl has an on_error parameter (raise, warn, skip) to handle malformed lines gracefully.
4. •

   Performance:

   • •Install orjson for significantly faster JSON operations.
   • •load_by_ext uses pyarrow engine for CSVs if available, which is much faster.

Limitations

• •Memory Usage: load_by_ext loads the entire file into memory. Use fast_load_jsonl for streaming.
• •Glob Expansion: load_by_ext with glob patterns loads all matching files into memory at once (in a list). Be careful with massive datasets.