Education Data Context

This skill provides critical context for interpreting data from the Urban Institute Education Data Portal. Education data has source-specific limitations that can significantly affect analysis validity.

Why Data Context Matters

• •Source-specific limitations: Each data source (CCD, IPEDS, CRDC, etc.) has unique constraints
• •Missing values have meaning: Codes like -1, -2, -3 indicate specific conditions, not random missingness
• •Definitions change over time: Variable definitions, categories, and coding schemes evolve
• •State comparisons require caution: State-level data often cannot be directly compared
• •Citation is required: The ODC Attribution License mandates proper citation

Reference File Structure

Quick Context (This Skill)

Deep-Dive Source Skills (Comprehensive Documentation)

For comprehensive understanding beyond the quick context files above, load the dedicated data source skill:

When to load deep-dive skills:

• •Need to understand data collection methodology in detail
• •Analyzing historical trends and need to know about definition changes
• •Encountering data quality issues that require deeper investigation
• •Writing documentation or reports that require precise methodology descriptions

Decision Trees

What data source did I pull from?

What endpoint did you use?
├─ schools/ccd/* → Read ./references/ccd-context.md
│   └─ Need more depth? → Load education-data-source-ccd skill
├─ school-districts/* → Read ./references/ccd-context.md
│   └─ Need more depth? → Load education-data-source-ccd skill
├─ schools/crdc/* → Read ./references/crdc-context.md
│   └─ Need more depth? → Load education-data-source-crdc skill
├─ schools/edfacts/* → Read ./references/edfacts-context.md
│   └─ Need more depth? → Load education-data-source-edfacts skill
├─ schools/meps/* → Load education-data-source-meps skill
├─ college-university/ipeds/* → Read ./references/ipeds-context.md
│   └─ Need more depth? → Load education-data-source-ipeds skill
├─ college-university/scorecard/* → Read ./references/scorecard-context.md
│   └─ Need more depth? → Load education-data-source-scorecard skill
├─ college-university/fsa/* → Load education-data-source-fsa skill
├─ college-university/nacubo/* → Load education-data-source-nacubo skill
├─ college-university/eada/* → Load education-data-source-eada skill
├─ college-university/pseo/* → Load education-data-source-pseo skill
├─ school-districts/saipe/* → Load education-data-source-saipe skill
└─ Multiple sources → Read ./references/data-relationships.md first

How do I interpret missing values?

What value do you see?
├─ In a CATEGORICAL column (grade, race, sex)?
│   └─ These use integer encoding, NOT coded missing values!
│       ├─ grade = -1 means Pre-K (NOT missing!)
│       ├─ race = 1-7 (NOT WH, BL, HI strings)
│       └─ sex = 1-2 (NOT M, F strings)
├─ In a NUMERIC column (enrollment, FTE, counts)?
│   ├─ -1 → Missing/not reported (treat as NULL)
│   ├─ -2 → Not applicable (exclude from that variable's analysis)
│   └─ -3 → Suppressed for privacy (cannot recover)
├─ null/blank?
│   └─ Source matters:
│       ├─ CCD, CRDC, EDFacts → Should use -1/-2/-3 codes
│       └─ Scorecard, MEPS, NACUBO → Use native nulls
├─ Ranges (e.g., "10-20") → EDFacts suppression bounds
└─ Unsure → Check source-specific reference file

What are the limitations?

What type of analysis are you doing?
├─ Cross-state comparison
│   ├─ K-12 assessments → INVALID (states not comparable)
│   ├─ K-12 other metrics → Check state reporting consistency
│   └─ College data → Generally valid (federal definitions)
├─ Time series
│   ├─ Check for definition changes
│   ├─ Check for ID changes (schools/districts merge/split)
│   └─ Check COVID-19 impact (2020-2021)
├─ Subgroup analysis
│   ├─ Check suppression rates
│   ├─ Smaller groups = more suppression
│   └─ Cannot impute suppressed values accurately
└─ Graduate outcomes
    ├─ IPEDS → First-time full-time only
    └─ Scorecard → Title IV recipients only

Universal Data Caveats

Portal Integer Encoding System

CRITICAL: The Education Data Portal uses integer codes, not string labels, for categorical variables. This applies to all sources.

Demographic Variable Encodings

Race codes:

Grade codes:

SEMANTIC TRAP - Grade -1: In CCD enrollment data, grade = -1 means Pre-Kindergarten, NOT missing data. This is a common source of errors. Missing data in enrollment uses the separate coded value system (-1/-2/-3) only for numeric fields like enrollment counts, not for the grade categorical variable.

# WRONG - filters out Pre-K students!
df = df.filter(pl.col("grade") >= 0)

# RIGHT - Pre-K students have grade = -1
pre_k = df.filter(pl.col("grade") == -1)
k_12 = df.filter(pl.col("grade").is_between(0, 12))
total = df.filter(pl.col("grade") == 99)

Variable Names Are Lowercase

Portal variable names are lowercase, not the uppercase names from original NCES documentation:

• •enrollment not MEMBER or ENROLLMENT
• •grade not GRADE
• •fips not FIPS or STATE

Missing Value Codes

IMPORTANT: Coded values (-1/-2/-3) apply to numeric measure columns (enrollment counts, FTE, etc.), NOT to categorical identifier columns like grade, race, or sex. Those use the integer encoding system above.

Missing Data Handling Varies by Source:

Important: Filter coded values BEFORE calculating statistics:

# WRONG - includes coded values in mean
df["enrollment"].mean()

# RIGHT - exclude coded missing values
df.filter(pl.col("enrollment") >= 0)["enrollment"].mean()

Year Definitions

• •year refers to the FALL of the academic year
• •year=2020 means the 2020-21 school year
• •Graduation rates use cohort entry year (cohort started 4-6 years prior)
• •Finance data may use fiscal year (varies by institution)

Suppression

Data is suppressed to protect student privacy:

• •Small cell sizes: Typically fewer than 5-10 students
• •Affects disaggregated data: Race, disability, gender breakdowns
• •More suppression in smaller schools: Rural areas most affected
• •Cannot be imputed accurately: Do not attempt to recover
• •Complementary suppression: Other cells may be suppressed to prevent calculation

State Reporting Variation

State education agencies interpret federal definitions differently:

• •Dropout definitions vary (CCD covers grades 7-12, CPS covers 10-12)
• •Average daily attendance calculated differently by state law
• •Discipline categories interpreted inconsistently
• •Missing data tends to cluster by state

Data Quality Checklist

Before analyzing any Education Data Portal data:

• • Check coded values: Filter out -1, -2, -3 before calculations
• • Understand year definition: Fall of academic year vs. cohort year
• • Note suppression rates: Calculate % suppressed by variable
• • Check definition changes: Compare codebooks across years
• • Verify identifier consistency: NCES IDs can change when schools/districts merge
• • Document state anomalies: Note any state-specific reporting issues
• • Check coverage: Not all schools appear in all sources
• • Consider COVID-19: 2020-2021 data may not be comparable to prior years

Quick Coverage Check

# Check missingness and suppression by state
df.group_by("fips").agg([
    pl.col("variable").filter(pl.col("variable") == -1).count().alias("missing"),
    pl.col("variable").filter(pl.col("variable") == -3).count().alias("suppressed"),
    pl.col("variable").count().alias("total")
])

Citation Requirements

Full Citation Format

Use for publications, reports, and formal documents:

[Dataset name(s)], Education Data Portal (Version X.X.X), 
Urban Institute, accessed [Month DD, YYYY], 
https://educationdata.urban.org/documentation/, 
made available under the ODC Attribution License.

Example:

Common Core of Data (CCD) School Directory, Education Data Portal 
(Version 0.20.0), Urban Institute, accessed January 15, 2026, 
https://educationdata.urban.org/documentation/, 
made available under the ODC Attribution License.

Short Citation Format

Use for visualizations, dashboards, and space-constrained contexts:

Source: [Dataset name(s)], Education Data Portal v.X.X.X, 
Urban Institute, ODC-By License.

Example:

Source: CCD School Directory, Education Data Portal v.0.20.0, 
Urban Institute, ODC-By License.

License Terms

License: Open Data Commons Attribution License (ODC-By) v1.0

Key requirements:

• •Must attribute the Urban Institute as data source
• •Must indicate if data was modified
• •May use for any purpose including commercial
• •May redistribute with attribution

Notification

Email educationdata@urban.org with any published work using the data. This helps the Urban Institute track usage and improve the portal.

Quick Reference: Source-Specific Caveats

What Each Source Covers

Data Lag Reference

Data availability lags behind the current year. As of January 2026:

Always verify year availability before building pipelines. Use mirror discovery endpoints (see mirrors.yaml) or filter downloaded data to confirm which years are present. See education-data-query skill for mirror-based fetch patterns.

Common Analysis Mistakes

DO NOT:

1. •

   Compare state assessment scores across states (EDFacts)

   • •Each state has different tests and cut scores
2. •

   Use IPEDS graduation rates to represent all students

   • •Only tracks first-time, full-time students
3. •

   Assume Scorecard earnings represent all graduates

   • •Only covers Title IV aid recipients
4. •

   Calculate statistics without filtering coded values

   • •-1, -2, -3 are not zeros; they corrupt calculations
5. •

   Compare 2020-2021 data to prior years without noting COVID

   • •Testing waivers, discipline changes, enrollment shifts
6. •

   Merge data across years assuming stable identifiers

   • •Schools and districts merge, split, and change IDs

DO:

1. •Check suppression rates before disaggregating
2. •Use within-state comparisons for assessment data
3. •Document all data limitations in your analysis
4. •Verify identifier stability for longitudinal analyses
5. •Cite the data source properly

Cross-References

• •Variable definitions: Load education-data-explorer skill to understand what variables measure
• •Query assistance: Load education-data-query skill to re-fetch data with different parameters
• •Joining data: Read ./references/data-relationships.md for identifier mappings
• •Deep source context: Load the appropriate education-data-source-* skill for comprehensive methodology, historical changes, and detailed variable definitions
• •Source-specific gotchas: Load the relevant education-data-source-* skill for variable name mappings, data lags, and endpoint-specific behaviors

Topic Index