Arguments

$ARGUMENTS - Optional dataset name to filter (e.g., hcrl_sa). If empty, check all datasets.

Execution Steps

•

Check SLURM job queue

bash

squeue -u $USER --format="%.10i %.20j %.8T %.10M %.6D %.15R" 2>&1

•

Check experiment checkpoints for all datasets (or filtered dataset)

bash

# List all completed stages
for ds in hcrl_ch hcrl_sa set_01 set_02 set_03 set_04; do
  echo "=== $ds ==="
  ls -lh experimentruns/$ds/*/best_model.pt 2>/dev/null || echo "  (no checkpoints)"
done

•
Check for recent SLURM errors in log directory
bash
```
ls -lt slurm_logs/*.err 2>/dev/null | head -10
```

•

If dataset specified via $ARGUMENTS, show detailed status:

bash

ls -la experimentruns/$ARGUMENTS/*/best_model.pt 2>/dev/null
ls -la experimentruns/$ARGUMENTS/*/config.json 2>/dev/null
ls -la experimentruns/$ARGUMENTS/*/metrics.json 2>/dev/null

Output Summary

Provide a concise status report:

Dataset	Stage	Status	Last Updated
hcrl_sa	teacher_autoencoder	complete/missing	timestamp
hcrl_sa	teacher_curriculum	complete/missing	timestamp
hcrl_sa	teacher_fusion	complete/missing	timestamp
...	...	...	...

Useful Follow-up Commands

bash

# Watch job queue
watch -n 5 'squeue -u $USER'

# Follow specific SLURM log
tail -f slurm_logs/<jobid>-<rule>.err

# Check Snakemake DAG status
snakemake -s pipeline/Snakefile --summary