CI/CD Pipelines

GitHub Actions Workflows

Core Concepts

• •Workflows: YAML files defining automation processes
• •Events: Triggers for workflow execution (push, pull_request, schedule)
• •Jobs: Collections of steps that run on the same runner
• •Steps: Individual tasks within a job
• •Actions: Reusable components for common tasks

Best Practices

• •

  Workflow Organization

  • •Use separate workflows for different concerns (CI, CD, release)
  • •Use workflow templates for consistency
  • •Use composite actions for reusable steps
  • •Use reusable workflows for shared pipeline logic
  • •Use matrix strategy for testing across multiple configurations
• •

  Job Optimization

  • •Use caching for dependencies and build artifacts
  • •Use parallel jobs for faster execution
  • •Use job dependencies for sequential execution
  • •Use artifacts for passing data between jobs
  • •Use self-hosted runners for custom environments
• •

  Security

  • •Use secrets for sensitive data
  • •Use environments for deployment protection
  • •Use required reviews for critical deployments
  • •Use OIDC for cloud provider authentication
  • •Use branch protection rules

GitHub Actions Examples

name: CI/CD Pipeline

on:
  push:
    branches: [main, develop]
  pull_request:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        node-version: [14.x, 16.x, 18.x]
    steps:
      - uses: actions/checkout@v3
      - name: Use Node.js ${{ matrix.node-version }}
        uses: actions/setup-node@v3
        with:
          node-version: ${{ matrix.node-version }}
          cache: 'npm'
      - run: npm ci
      - run: npm test
      - run: npm run build

  deploy:
    needs: test
    runs-on: ubuntu-latest
    if: github.ref == 'refs/heads/main'
    steps:
      - uses: actions/checkout@v3
      - name: Deploy to production
        run: |
          # Deployment commands

GitLab CI/CD Pipelines

Core Concepts

• •Pipelines: Series of jobs executed in stages
• •Stages: Logical grouping of jobs
• •Jobs: Individual tasks that execute scripts
• •Runners: Agents that execute jobs
• •Artifacts: Files passed between jobs

Best Practices

• •

  Pipeline Configuration

  • •Use .gitlab-ci.yml for pipeline definition
  • •Use stages for logical job grouping
  • •Use only/except rules for job execution control
  • •Use rules for more complex conditions
  • •Use needs for job dependencies
• •

  Job Optimization

  • •Use artifacts for passing data between jobs
  • •Use cache for dependency caching
  • •Use parallel jobs for faster execution
  • •Use retry for transient failures
  • •Use timeout for job duration limits
• •

  Security

  • •Use CI/CD variables for sensitive data
  • •Use protected variables for protected branches
  • •Use masked variables for hiding values
  • •Use environments for deployment control
  • •Use approval rules for critical deployments

GitLab CI/CD Examples

stages:
  - test
  - build
  - deploy

test:
  stage: test
  image: node:18
  script:
    - npm ci
    - npm test
  parallel:
    matrix:
      - NODE_VERSION: [14, 16, 18]
  cache:
    paths:
      - node_modules/

build:
  stage: build
  image: node:18
  script:
    - npm ci
    - npm run build
  artifacts:
    paths:
      - dist/

deploy:
  stage: deploy
  image: node:18
  script:
    - npm install -g serverless
    - serverless deploy
  only:
    - main
  when: manual

Jenkins Pipelines

Core Concepts

• •Pipelines: Groovy scripts defining automation
• •Stages: Logical grouping of steps
• •Steps: Individual operations
• •Agents: Nodes where pipeline runs
• •Credentials: Secure storage for secrets

Best Practices

• •

  Pipeline Design

  • •Use Declarative Pipeline syntax
  • •Use shared libraries for reusable code
  • •Use pipeline stages for logical organization
  • •Use when conditions for conditional execution
  • •Use post sections for cleanup and notifications
• •

  Agent Management

  • •Use labels for agent selection
  • •Use Docker agents for consistent environments
  • •Use Kubernetes agents for dynamic scaling
  • •Use agent none for lightweight pipelines
  • •Use stash/unstash for passing data
• •

  Security

  • •Use credentials binding for secrets
  • •Use credential scopes for access control
  • •Use approval steps for manual gates
  • •Use input steps for user interaction
  • •Use role-based strategy for permissions

Jenkins Pipeline Examples

pipeline {
    agent any
    
    stages {
        stage('Test') {
            parallel {
                stage('Node 14') {
                    agent { docker { image 'node:14' } }
                    steps {
                        sh 'npm ci'
                        sh 'npm test'
                    }
                }
                stage('Node 16') {
                    agent { docker { image 'node:16' } }
                    steps {
                        sh 'npm ci'
                        sh 'npm test'
                    }
                }
            }
        }
        
        stage('Build') {
            agent { docker { image 'node:18' } }
            steps {
                sh 'npm ci'
                sh 'npm run build'
                archiveArtifacts artifacts: 'dist/**'
            }
        }
        
        stage('Deploy') {
            when {
                branch 'main'
            }
            steps {
                input message: 'Deploy to production?', ok: 'Deploy'
                sh 'npm install -g serverless'
                sh 'serverless deploy'
            }
        }
    }
    
    post {
        always {
            cleanWs()
        }
        success {
            emailext subject: 'Build Success',
                body: 'The build completed successfully.',
                to: 'team@example.com'
        }
        failure {
            emailext subject: 'Build Failed',
                body: 'The build failed.',
                to: 'team@example.com'
        }
    }
}

Azure DevOps Pipelines

Core Concepts

• •Pipelines: YAML or GUI-defined automation
• •Stages: Major divisions in a pipeline
• •Jobs: Units of work within a stage
• •Tasks: Pre-built units of work
• •Agents: Machines that run jobs

Best Practices

• •

  Pipeline Configuration

  • •Use YAML pipelines for version control
  • •Use templates for reusable pipeline logic
  • •Use parameters for pipeline customization
  • •Use variables for configuration values
  • •Use stages for environment separation
• •

  Job Optimization

  • •Use parallel jobs for faster execution
  • •Use artifacts for passing data between jobs
  • •Use caching for dependency caching
  • •Use matrix strategy for multiple configurations
  • •Use demands for agent selection
• •

  Security

  • •Use secret variables for sensitive data
  • •Use variable groups for organization
  • •Use key vault for secret management
  • •Use environment checks for deployment control
  • •Use approval gates for manual intervention

Azure DevOps Pipeline Examples

trigger:
- main
- develop

pool:
  vmImage: 'ubuntu-latest'

stages:
- stage: Test
  jobs:
  - job: Test
    strategy:
      matrix:
        Node_14:
          node_version: 14.x
        Node_16:
          node_version: 16.x
        Node_18:
          node_version: 18.x
    steps:
    - task: UseNode@1
      inputs:
        versionSpec: $(node_version)
    - script: |
        npm ci
        npm test
      displayName: 'Install dependencies and run tests'

- stage: Build
  dependsOn: Test
  jobs:
  - job: Build
    steps:
    - task: UseNode@1
      inputs:
        versionSpec: '18.x'
    - script: |
        npm ci
        npm run build
      displayName: 'Build application'
    - publish: dist
      artifact: dist

- stage: Deploy
  dependsOn: Build
  condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/main'))
  jobs:
  - deployment: Deploy
    environment: 'production'
    strategy:
      runOnce:
        deploy:
          steps:
          - script: |
              npm install -g serverless
              serverless deploy
            displayName: 'Deploy to production'

Build and Test Strategies

Build Strategies

• •Incremental Builds: Build only changed components
• •Parallel Builds: Build multiple components simultaneously
• •Matrix Builds: Build across multiple configurations
• •Cached Builds: Use cache for faster builds
• •Docker Builds: Use Docker for consistent build environments

Test Strategies

• •Unit Tests: Test individual components in isolation
• •Integration Tests: Test component interactions
• •End-to-End Tests: Test complete user flows
• •Performance Tests: Test system performance
• •Security Tests: Test for security vulnerabilities

Quality Gates

• •Code Coverage: Minimum code coverage threshold
• •Linting: Code style and quality checks
• •Static Analysis: Security and vulnerability scanning
• •Dependency Scanning: Check for vulnerable dependencies
• •License Compliance: Check for license compliance

Deployment Patterns

Blue-Green Deployment

• •Concept: Maintain two identical production environments
• •Benefits: Zero downtime, instant rollback
• •Implementation: Switch traffic between blue and green environments
• •Best Practices: Use load balancers for traffic switching, test green environment before switching

Canary Deployment

• •Concept: Gradually roll out changes to a subset of users
• •Benefits: Reduced risk, gradual feedback
• •Implementation: Route small percentage of traffic to new version
• •Best Practices: Monitor metrics closely, have rollback plan ready

Rolling Deployment

• •Concept: Replace instances one at a time
• •Benefits: Gradual rollout, minimal downtime
• •Implementation: Update instances in batches
• •Best Practices: Use health checks, monitor for failures

Feature Flags

• •Concept: Enable/disable features without deployment
• •Benefits: Gradual rollout, instant rollback
• •Implementation: Use feature flag management system
• •Best Practices: Keep flags temporary, clean up old flags

Immutable Infrastructure

• •Concept: Replace rather than modify infrastructure
• •Benefits: Consistency, easier rollback, no configuration drift
• •Implementation: Use infrastructure as code, replace instances on updates
• •Best Practices: Version all infrastructure, use golden images

Pipeline Orchestration

Pipeline Dependencies

• •Sequential Execution: Jobs run in sequence
• •Parallel Execution: Jobs run simultaneously
• •Conditional Execution: Jobs run based on conditions
• •Artifact Passing: Pass data between jobs
• •Environment Promotion: Promote through environments

Pipeline Triggers

• •Push Triggers: Run on code push
• •Pull Request Triggers: Run on pull requests
• •Schedule Triggers: Run on schedule
• •Manual Triggers: Run on demand
• •Webhook Triggers: Run on webhook events

Pipeline Notifications

• •Email Notifications: Send email on pipeline events
• •Slack Notifications: Send Slack messages on pipeline events
• •Microsoft Teams Notifications: Send Teams messages on pipeline events
• •Custom Webhooks: Send custom webhook notifications
• •Status Badges: Display pipeline status as badges

Pipeline Security

Secrets Management

• •Secret Variables: Store secrets as pipeline variables
• •Secret Stores: Use secret stores (Vault, AWS Secrets Manager)
• •OIDC Authentication: Use OIDC for cloud provider authentication
• •Key Vault: Use key vault for secret management
• •Secret Rotation: Rotate secrets regularly

Access Control

• •Branch Protection: Protect branches from direct pushes
• •Pull Request Reviews: Require reviews for merges
• •Environment Approvals: Require approvals for deployments
• •Role-Based Access: Use RBAC for pipeline access
• •Audit Logging: Enable audit logging for compliance

Supply Chain Security

• •Dependency Scanning: Scan for vulnerable dependencies
• •Container Scanning: Scan container images for vulnerabilities
• •SBOM Generation: Generate software bill of materials
• •Code Signing: Sign artifacts for integrity
• •Policy Enforcement: Enforce policies with tools like OPA