Screenenv Integration Guide

Screenenv is a Playwright-based tool that runs in Kubernetes Jobs to record browser-based demos. This skill explains how to use it within the demo-creator pipeline.

What is Screenenv?

Screenenv is a headless browser recording tool from HuggingFace that:

•Runs Playwright Python scripts in a sandboxed environment
•Records the screen to high-quality video
•Executes in Kubernetes Jobs for isolation and resource management

Important: Screenenv is NOT an MCP server. You don't call it directly via tools. Instead, you write Playwright scripts that screenenv will execute.

Architecture

code

┌─────────────────────────────────────────────────────────────┐
│  Phase 1: Script Development (detailed-script agent)        │
│  - Agent writes Playwright Python script                    │
│  - Uses domain knowledge of the app                          │
│  - Tools: Read, Write, Grep, Bash                           │
└─────────────────┬───────────────────────────────────────────┘
                  │
                  ▼
         script.py saved to .demo/{demo_id}/
                  │
                  ▼
┌─────────────────────────────────────────────────────────────┐
│  Phase 2: Recording (record-demo agent)                     │
│  - Uses ScreenenvJobManager Python utility                   │
│  - Creates Helm release with screenenv-job chart            │
│  - Passes script.py to K8s Job                              │
└─────────────────┬───────────────────────────────────────────┘
                  │
                  ▼
┌─────────────────────────────────────────────────────────────┐
│  Phase 3: Execution (Kubernetes Job)                        │
│  - Screenenv runs in k8s Pod                                │
│  - Executes Playwright script in headless Chrome            │
│  - Records screen at 1920x1080                              │
│  - Saves video to PVC                                       │
└─────────────────────────────────────────────────────────────┘

Writing Playwright Scripts for Screenenv

When developing demo scripts (in the detailed-script agent), write standard Playwright Python code:

Template Structure

python

"""
Demo Script: {Feature Name}
Generated by: demo-creator detailed-script agent
"""

from playwright.sync_api import sync_playwright
import time


def run_demo(page):
    """Execute the demo script."""

    # Scene 1: Navigate to Feature
    print("Scene 1: Navigate to Feature")
    page.goto("http://localhost:3000/drugs")
    page.wait_for_load_state("networkidle")
    time.sleep(2)

    # Verify page loaded
    assert page.locator('input[placeholder*="Search"]').is_visible()
    page.screenshot(path="scene_1.png")

    # Scene 2: Interact with UI
    print("Scene 2: Apply Filters")
    page.click('button:has-text("Filter")')
    time.sleep(0.5)

    page.fill('input[name="search"]', "EGFR")
    time.sleep(0.4)

    page.click('button[type="submit"]')
    time.sleep(1.5)

    page.screenshot(path="scene_2.png")


def setup():
    """Run setup commands before demo (optional)."""
    import subprocess

    # Example: Seed test data
    subprocess.run([
        "kubectl", "exec", "-n", "your-namespace",
        "deployment/backend", "--",
        "python", "scripts/seed_demo_data.py"
    ], check=True)


def teardown():
    """Run cleanup commands after demo (optional)."""
    import subprocess

    subprocess.run([
        "kubectl", "exec", "-n", "your-namespace",
        "deployment/backend", "--",
        "python", "scripts/cleanup_demo_data.py"
    ], check=True)


def main():
    """Main entry point - screenenv executes this."""
    setup()

    with sync_playwright() as p:
        browser = p.chromium.launch(headless=True)
        context = browser.new_context(
            viewport={"width": 1920, "height": 1080},
            record_video_dir="./recordings",
            record_video_size={"width": 1920, "height": 1080}
        )
        page = context.new_page()

        try:
            run_demo(page)
        finally:
            context.close()
            browser.close()
            teardown()


if __name__ == "__main__":
    main()

Key Playwright APIs

Navigation:

python

page.goto("http://localhost:3000/path")
page.wait_for_load_state("networkidle")
page.wait_for_url("**/drugs")

Clicking:

python

page.click('button:has-text("Submit")')
page.locator('.submit-btn').click()
page.click('button[aria-label="Close"]')

Typing:

python

page.fill('input[name="search"]', "text")
page.type('input', "text", delay=100)  # With typing delay

Assertions:

python

assert page.locator('.result').is_visible()
page.wait_for_selector('.result', timeout=5000)

Screenshots:

python

page.screenshot(path="scene_1.png")
page.locator('.specific-element').screenshot(path="element.png")

Waits:

python

time.sleep(2)  # Explicit wait (use for timing)
page.wait_for_timeout(2000)  # Playwright wait
page.wait_for_selector('.element')  # Wait for element

Selector Strategy (Priority Order)

•Text-based (most resilient): page.click('button:has-text("Submit")')
•Placeholder: page.fill('input[placeholder="Search"]', text)
•Aria-label: button[aria-label="Close"]
•Test ID: [data-testid="submit-btn"]
•CSS classes (last resort, fragile)

Using ScreenenvJobManager

The record-demo agent uses the Python utility to manage K8s Jobs:

python

import sys
sys.path.append("plugins/demo-creator")
from utils.screenenv_job import ScreenenvJobManager, create_and_run_recording
from utils.manifest import Manifest

# Load manifest
manifest = Manifest("{demo_id}")
manifest.load()

# Get script path
script_path = manifest.get_file_path("script.py")

# Option 1: Convenience function (recommended)
result = create_and_run_recording(
    demo_id=manifest.demo_id,
    script_url=f"http://demo-script-server/scripts/{manifest.demo_id}.py",
    output_path=manifest.get_file_path("raw_recording.mp4"),
    target_url="http://localhost:3000",
    cleanup=True
)

# Option 2: Manual control
manager = ScreenenvJobManager(
    namespace="infra",
    helm_chart_path="k8s/infra/charts/screenenv-job",
    context="k3d-local"
)

# Create job
job_result = manager.create_job(
    demo_id=manifest.demo_id,
    script_url=f"http://demo-script-server/scripts/{manifest.demo_id}.py",
    target_url="http://localhost:3000",
    resolution="1920x1080",
    frame_rate="30"
)

# Wait for completion
wait_result = manager.wait_for_completion(
    demo_id=manifest.demo_id,
    poll_interval=5,
    max_wait=600
)

# Retrieve recording
if wait_result["status"] == "completed":
    success = manager.retrieve_recording(
        demo_id=manifest.demo_id,
        output_path=manifest.get_file_path("raw_recording.mp4")
    )

    if success:
        print("✅ Recording retrieved")

    # Cleanup
    manager.cleanup_job(manifest.demo_id)

Kubernetes Deployment

The screenenv Helm chart is at k8s/infra/charts/screenenv-job/.

Key Configuration

Parameter	Default	Description
`demoId`	-	Unique demo identifier
`scriptUrl`	-	URL to Playwright script
`targetUrl`	`http://localhost:3000`	App base URL
`resolution`	`1920x1080`	Video resolution
`frameRate`	`30`	Video frame rate
`image.repository`	`ghcr.io/huggingface/screenenv`	Docker image

Job Lifecycle

•Creation: Helm install creates Job
•Execution: Pod starts, runs Playwright script
•Recording: Video saved to PVC at /recordings/{demo_id}/raw_recording.mp4
•Completion: Job status becomes "succeeded"
•Retrieval: kubectl cp to extract video
•Cleanup: Helm uninstall (automatic after 10 minutes)

Debugging

Check Job Status

bash

kubectl get job screenenv-{demo_id} -n infra --context k3d-local

View Logs

bash

kubectl logs job/screenenv-{demo_id} -n infra --context k3d-local

Check Recording

bash

# Get pod name
POD=$(kubectl get pods -n infra -l job-name=screenenv-{demo_id} \
  --context k3d-local -o jsonpath='{.items[0].metadata.name}')

# Check file exists
kubectl exec -n infra $POD --context k3d-local -- \
  ls -lh /recordings/{demo_id}/

Common Issues

"Script not found" Error

The scriptUrl must be accessible from inside the K8s cluster. For local development:

•Use a ConfigMap to embed the script
•Or run a simple HTTP server in the cluster
•Or mount the script via PVC

Recording is Empty/Corrupt

•Check script actually runs: Add print() statements
•Verify browser launches: Check logs for Chromium errors
•Ensure app is accessible from Pod: Test with kubectl exec ... -- curl http://localhost:3000

Job Never Completes

•Check timeout: Default is 10 minutes via --wait --timeout 10m
•Review logs for hangs: Look for page.wait_for_selector() that never resolves
•Verify selectors work: Test script locally with headed browser first

Best Practices

•Test locally first: Run Playwright script on your machine with headless=False to verify selectors
•Use resilient selectors: Prefer text-based and aria-label over CSS classes
•Add generous waits: Network conditions in k8s may be slower
•Keep scripts simple: Each scene should be 5-10 seconds max
•Verify assertions: Use assert to catch UI changes that break the script
•Clean up resources: Always call manager.cleanup_job() after retrieval

Example Workflow (detailed-script agent)

python

# 1. Load context
import sys
sys.path.append("plugins/demo-creator")
from utils.manifest import Manifest

manifest = Manifest("{demo_id}")
manifest.load()

# Read outline
with open(manifest.get_file_path("outline.md")) as f:
    outline = f.read()

# 2. Write Playwright script based on outline
# (Use domain knowledge of the app, not screenenv MCP calls)
script_content = """
from playwright.sync_api import sync_playwright
import time

def run_demo(page):
    # Based on outline, write the actual interactions
    page.goto("http://localhost:3000/drugs")
    # ... etc
"""

# 3. Save script
with open(manifest.get_file_path("script.py"), "w") as f:
    f.write(script_content)

# 4. Update manifest
manifest.complete_stage(2, {
    "script_path": "script.py",
    "estimated_duration_seconds": 30
})

print("✅ Stage 2 complete: Playwright script created")

Adapting Existing E2E Tests for Demos

If you have existing Playwright pytest files, you can use them as a starting point for demo scripts. Read the test, understand the flow, and adapt it for demo purposes.

What to Keep from Tests

•Working selectors - The test has already figured out how to find elements
•The user flow - The sequence of actions represents a real user journey
•URL paths - Navigation targets are correct

What to Change for Demos

Test Pattern	Demo Adaptation
`page.fill('[name="q"]', "text")`	`page.type('[name="q"]', "text", delay=100)` - visible typing
No delays between actions	Add `time.sleep(1-2)` for pacing
`expect(...).to_be_visible()`	Remove or minimize assertions
Complex error handling	Simple happy-path only
Fast execution	Deliberate, watchable pacing

Example Adaptation

Original test:

python

def test_search(self, page):
    page.goto("/drugs")
    page.fill('[name="q"]', "aspirin")
    page.click('button[type="submit"]')
    expect(page.locator(".results")).to_be_visible()

Demo script:

python

def run_demo(page):
    print("Scene 1: Navigate to search")
    page.goto("http://localhost:3000/drugs")
    time.sleep(2)

    print("Scene 2: Search for aspirin")
    page.type('[name="q"]', "aspirin", delay=100)
    time.sleep(0.5)
    page.click('button[type="submit"]')
    page.wait_for_selector(".results")
    time.sleep(2)

Summary

•✅ Screenenv runs Playwright scripts in K8s Jobs
•✅ You write standard Playwright Python code
•✅ ScreenenvJobManager handles K8s orchestration
•✅ Recording happens asynchronously in isolated Pods
•✅ Existing E2E tests can be adapted for demos
•❌ Screenenv is NOT an MCP server you call directly
•❌ Don't create .mcp.json for screenenv in plugins

When to use this skill:

•When writing demo scripts (detailed-script agent)
•When setting up recording jobs (record-demo agent)
•When debugging recording failures
•When understanding the demo pipeline architecture