pyKT Dataset Preparation

Name: dataset-prep
Rating: 87
Author: L-yifan

Prepare and validate datasets for Knowledge Tracing experiments with pyKT framework.

Quick Start

Check Preprocessing Status

bash

python scripts/preprocess.py --status --data-dir ../data

Preprocess a Dataset

bash

# Single dataset
python scripts/preprocess.py --dataset assist2009 --pykt-path ../pykt-toolkit

# Multiple datasets
python scripts/preprocess.py --dataset assist2009 assist2015 ednet

# All available datasets
python scripts/preprocess.py --all

Validate Dataset

bash

# Validate entire dataset directory
python scripts/validate_dataset.py --dir ../data/assist2009

# Validate specific file
python scripts/validate_dataset.py --file ../data/assist2009/data.txt --type raw
python scripts/validate_dataset.py --file ../data/assist2009/train_valid_sequences.csv --type csv

Scripts

`preprocess.py`

Wrapper for pyKT's preprocessing pipeline with batch support.

Option	Description
`--dataset NAME`	Dataset(s) to preprocess
`--all`	Process all available datasets
`--status`	Show preprocessing status only
`--data-dir PATH`	Data directory location
`--pykt-path PATH`	pykt-toolkit installation path
`--min-seq-len N`	Minimum sequence length (default: 3)
`--maxlen N`	Maximum sequence length (default: 200)
`--kfold N`	Number of CV folds (default: 5)
`--list`	List supported datasets with download URLs

`validate_dataset.py`

Validate data format and report statistics.

Option	Description
`--file PATH`	Single file to validate
`--dir PATH`	Dataset directory to validate
`--type raw/csv`	File type (auto-detected)
`--json`	Output results as JSON

Supported Datasets

Dataset	Type	Description
assist2009	Q+C	ASSISTments 2009-2010 Math
assist2012	Q+C	ASSISTments 2012-2013
assist2015	C	ASSISTments Skill Builder
assist2017	Q+C	ASSISTments Competition
algebra2005	Q+C	KDD Cup Algebra
bridge2algebra2006	Q+C	KDD Cup Bridge to Algebra
statics2011	C	Andes Physics
nips_task34	Q+C	Eedi Education Challenge
ednet	Q+C	TOEIC English (Riiid)
junyi2015	Q+C	Junyi Academy K-12 Math
slepemapy	Q+C	Geography
poj	C	Programming Judge

Type: Q+C = Questions + Concepts, C = Concepts only

References

•Data format details: See references/data_format.md
•Dataset configurations: See references/datasets_config.md

Workflow

•Download raw data from source (see --list for URLs)
•Place in pykt-toolkit/data/{dataset_name}/
•Preprocess with preprocess.py
•Validate with validate_dataset.py
•Train models using pyKT