Pandas Best Practices

Expert guidelines for Pandas development, focusing on data manipulation, analysis, and efficient DataFrame operations.

Code Style and Structure

• •Write concise, technical responses with accurate Python examples
• •Prioritize reproducibility in data analysis workflows
• •Use functional programming; avoid unnecessary classes
• •Prefer vectorized operations over explicit loops
• •Use descriptive variable names reflecting data content
• •Follow PEP 8 style guidelines

DataFrame Creation and I/O

• •Use pd.read_csv(), pd.read_excel(), pd.read_json() with appropriate parameters
• •Specify dtype parameter to ensure correct data types on load
• •Use parse_dates for automatic datetime parsing
• •Set index_col when the data has a natural index column
• •Use chunksize for reading large files incrementally

Data Selection

• •Use .loc[] for label-based indexing
• •Use .iloc[] for integer position-based indexing
• •Avoid chained indexing (e.g., df['col'][0]) - use .loc or .iloc instead
• •Use boolean indexing for conditional selection: df[df['col'] > value]
• •Use .query() method for complex filtering conditions

Method Chaining

• •Prefer method chaining for data transformations when possible
• •Use .pipe() for applying custom functions in a chain
• •Chain operations like .assign(), .query(), .groupby(), .agg()
• •Keep chains readable by breaking across multiple lines

Data Cleaning and Validation

Missing Data

• •Check for missing data with .isna() and .info()
• •Handle missing data appropriately: .fillna(), .dropna(), or imputation
• •Use pd.NA for nullable integer and boolean types
• •Document decisions about missing data handling

Data Quality Checks

• •Implement data quality checks at the beginning of analysis
• •Validate data types with .dtypes and convert as needed
• •Check for duplicates with .duplicated() and handle appropriately
• •Use .describe() for quick statistical overview

Type Conversion

• •Use .astype() for explicit type conversion
• •Use pd.to_datetime() for date parsing
• •Use pd.to_numeric() with errors='coerce' for safe numeric conversion
• •Utilize categorical data types for low-cardinality string columns

Grouping and Aggregation

GroupBy Operations

• •Use .groupby() for efficient aggregation operations
• •Specify aggregation functions with .agg() for multiple operations
• •Use named aggregation for clearer output column names
• •Consider .transform() for broadcasting results back to original shape

Pivot Tables and Reshaping

• •Use .pivot_table() for multi-dimensional aggregation
• •Use .melt() to convert wide to long format
• •Use .pivot() to convert long to wide format
• •Use .stack() and .unstack() for hierarchical index manipulation

Performance Optimization

Memory Efficiency

• •Use categorical data types for low-cardinality strings
• •Downcast numeric types when appropriate
• •Use pd.eval() and .eval() for large expression evaluation

Computation Speed

• •Use vectorized operations instead of .apply() with row-wise functions
• •Prefer built-in aggregation functions over custom ones
• •Use .values or .to_numpy() for NumPy operations when faster

Avoiding Common Pitfalls

• •Avoid iterating with .iterrows() - use vectorized operations
• •Don't modify DataFrames while iterating
• •Be aware of SettingWithCopyWarning - use .copy() when needed
• •Avoid growing DataFrames row by row - collect in list and create once

Time Series Operations

• •Use DatetimeIndex for time series data
• •Leverage .resample() for time-based aggregation
• •Use .shift() and .diff() for lag operations
• •Use .rolling() and .expanding() for window calculations

Merging and Joining

• •Use .merge() for SQL-style joins
• •Specify how parameter: 'inner', 'outer', 'left', 'right'
• •Use validate parameter to check join cardinality
• •Use .concat() for stacking DataFrames

Key Conventions

• •Import as import pandas as pd
• •Use snake_case for column names when possible
• •Document data sources and transformations
• •Keep notebooks reproducible with clear cell execution order