Data Check

Inspect and validate a data file before analysis. Report findings with suggested fixes. Do not auto-fix.

When to Use

• •Before any analysis or visualization on a new data file
• •When parsing errors or unexpected values appear
• •When switching to a different dataset mid-session
• •User says "check", "inspect", "what's in this", "profile" a data file

When NOT to Use

• •File has already been checked in this session and nothing changed
• •User just wants to read a specific value (use Read tool directly)

Workflow

Step 1: Raw File Inspection

Before loading into R/Stata, check the raw file:

file <filename>                          # encoding detection
head -c 500 <filename> | cat -v          # hidden chars: ^M (CR), BOM, non-UTF8
wc -l <filename>                         # row count sanity check
head -3 <filename>                       # peek at delimiter and header

Flag these issues:

Step 2: Load and Summarize

R-first. Use Stata only when the file is .dta and project context is Stata-based.

For CSV files:

df <- read.csv("file.csv", stringsAsFactors = FALSE)

For Stata .dta files:

library(haven)
df <- read_dta("file.dta")

Report this table:

Then per column:

Column          Type        Missing   Unique   Example Values
─────────────────────────────────────────────────────────────
age             numeric     12 (2%)   45       18, 25, 34, 67, 89
gender          labelled    0 (0%)    3        1=Male, 2=Female, 3=Other
weight_var      numeric     0 (0%)    847      0.23, 1.05, 2.11

Step 3: Stata .dta Specific Checks

When the file is .dta, also report:

1. •Variable labels: Show all variable descriptions from Stata metadata
2. •Value labels: For each labelled variable, show the label-to-value mapping
3. •Survey weight variables: Flag any variables with names matching *weight*, *wt*, pw, fw, iw or that have pweight/fweight characteristics
4. •Stata extended missings: Check for .a through .z missing types (preserved by haven as tagged NAs)

Step 4: Coded Missing Values

Scan ALL numeric columns for values that are likely coded missings, not real data:

Detection method: For each numeric column, check if values 97-99 (or -1, -9) appear AND the variable's main range is much lower (e.g., 1-5 scale with some 98s).

Report as:

POSSIBLE CODED MISSINGS:
  income_cat: values {97, 98, 99} found (n=45). Main range: 1-12. Likely coded missing.
  age: value {99} found (n=2). Main range: 18-95. Ambiguous - could be real age.

Step 5: Light Profiling

After the structural check:

1. •Numeric columns: Correlation matrix (top 10 strongest pairs if many columns)
2. •Categorical columns: Frequency tables (top 10 levels per variable, flag if >50 unique levels)
3. •Flag: Numeric columns stored as character (likely parsing issue), zero-variance columns, duplicate rows

Step 6: Report Summary

Present a concise summary with three sections:

Clean: Variables ready for analysis (no issues)

Needs attention: Variables with potential coded missings, encoding issues, or type problems. Include suggested fix code but do NOT execute.

# SUGGESTED FIX for income_cat coded missings:
df$income_cat[df$income_cat %in% c(97, 98, 99)] <- NA

Notes: Observations about the data structure (survey weights detected, panel structure, etc.)

Quick Reference

Common Mistakes