[SIP-178] Proposal for Migrating Superset E2E Test Framework from Cypress to Playwright

Motivation

Superset’s current E2E suite, built on Cypress, has shown increasing issues with flakiness, slowness, and high maintenance overhead. Additionally, changes in Cypress’s licensing model and ecosystem direction have raised concerns about its long-term viability. Instead of simply porting the test suite to a new tool, this SIP starts by defining what constitutes a good E2E testing framework and uses that lens to evaluate our options.

Proposed Change

We propose to fully replace Cypress with Playwright for all E2E test coverage in Superset. This is not a trial; it is a deliberate transition based on our evaluation criteria.

Criteria for Effective E2E Testing

We define good E2E test infrastructure as one that satisfies the following:

# Criterion Description
1 Ease of local execution Tests should be easy to run and debug locally with minimal setup
2 Stability under UI changes Framework should support smart auto-waiting and reduce flake-prone code
3 Maintainable syntax Tests should be written in modern syntax with good async/await and modular support
4 Actively maintained OSS project The framework should be open source and well-supported
5 CI scalability Supports parallel execution, test retries, and trace artifacts
6 Upgrade stability Semantic versioning with limited breaking changes
7 Test tooling Provides useful tools for authoring (codegen), debugging (trace viewer), and analysis

Evaluation: Cypress vs. Playwright

Criterion Cypress Playwright
Local execution ✅ Good GUI, but slower reloads ✅ Fast CLI runs, VS Code debugging integration
Test stability ⚠️ Manual waits often required ✅ Smart auto-waiting built-in
Syntax & DX ⚠️ Uses older .then() chaining, harder to debug ✅ Modern async/await, better readability
Ecosystem health ⚠️ Increasingly commercialized, some core features behind login ✅ Fully OSS, actively maintained by Microsoft
CI performance ⚠️ Parallelism is paywalled ✅ Native support for parallelism, retries, trace output
Codegen/tooling ⚠️ Cypress Studio unstable and gated ✅ Codegen and trace viewer included by default
Upgrade cost ⚠️ Breaking changes common between versions ✅ Rare breaking changes, semver-respecting

New or Changed Public Interfaces

None. This change affects only internal test infrastructure. No updates to models, REST endpoints, dashboards, visualizations, React components, or CLI behavior.

New dependencies

Package Source Maintained? License
@playwright/test npm Actively maintained Apache-2.0

Playwright is a well-maintained open-source project under the Apache-2.0 license, backed by Microsoft.

Migration Plan and Compatibility

Step 1: Audit and Cleanup

  • Review all existing Cypress E2E test cases
  • Remove any tests that do not represent true end-to-end workflows
  • Open PRs to relocate non-E2E tests (e.g., UI logic, visual validations) to appropriate test layers or repositories (e.g., unit/component)

Step 2: Parallel Execution & Stability Validation

  • Configure both Playwright and Cypress to run concurrently in CI during the transition
  • Gather data on test pass/fail patterns, false positives/negatives, flake rate, and runtime
  • Use this data to validate the stability of Playwright before full switch-over

Step 3: Framework Setup

  • Integrate Playwright test runner into the codebase
  • Add base utilities (e.g., login helpers, selectors, test fixtures)
  • Set up GitHub Actions for Playwright runs, including test retries, trace collection, and artifacts
  • Provide developer documentation for local test running and debugging

Step 4: Migrate Core E2E Coverage

  • Port the following high-value workflows:
  • Explore (chart creation, control panel interactions, save flow)
  • Dashboard (load, edit, add chart, save)
  • SQL Lab (query execution, preview, save query)
  • Authentication (login/logout)

Step 5: Expand Remaining E2E Test Coverage

  • Migrate additional E2E tests that meet our defined criteria
  • Ensure tagging (e.g., smoke, sanity) is preserved where appropriate
  • Replace Cypress-specific utilities with Playwright equivalents

Step 6: Deprecate Cypress

  • Remove Cypress dependencies and test files
  • Remove related CI jobs and scripts
  • Enforce Playwright for all new E2E contributions going forward

Rejected Alternatives

Option Reason for Rejection
Stay on Cypress High maintenance cost, flakiness, paywalled features, aging syntax
TestCafe / WebdriverIO / Nightwatch Poor tooling, limited adoption, or complex setup
Hybrid frameworks Increases maintenance burden and contributor confusion

References

Comment From: rusackas

I love it! Between this and the (currently optional) playwright thumbnails, I love that we're removing flakiness, removing legacy platforms, and consolitating on tooling that people like :)

Comment From: msyavuz

This is great! Cypress was a major pain point so switching to Playwright might be the best dev-ex improvement yet.

Comment From: LevisNgigi

Awesome, cypress was quite a pain to get it to run locally.Between the browser version issues and random permission stuff, it really makes you work for it.

Comment From: EnxDev

I'm absolutely in favor of this; It's often quite challenging to determine the root cause when tests fail in CI but not locally, or worse, they fail differently in local runs. Improving the reliability and debuggability of the test environment would be a significant help for both contributors and maintainers.

Comment From: hainenber

Oh, I can put into use the old PR from the closet again :D

Comment From: rusackas

@sadpandajoe Let's open a DISCUSS thread and get this ball in motion :D

Comment From: eschutho

@sadpandajoe This is a great proposal! Pretty typical question, but do we know the cost of running Cypress and Playwright simultaneously (as in time to run tests, size of containers, etc, not monetary cost) and can we outline a timeframe to removing Cypress so that we don't have the old tech-debt around for longer than needed?

Comment From: sadpandajoe

So this wouldn't be big bang. I think the best way to do it is probably to do one test file at a time. Make the Cypress one still required and the Playwright one optional. We'd probably run them in parallel for say a week, a max two and if everything looks the same, we'll deprecate the Cypress one by deleting the file and then making the Playwright one required.

Comment From: sadpandajoe

Just in case you'd want a full project plan:

Apache Superset: Cypress to Playwright Migration Project

Project Overview

Migrate end-to-end testing from Cypress to Playwright while maintaining test coverage and reliability. The migration will be conducted in phases to allow both frameworks to run in parallel during the transition period.

Current State Analysis

Cypress Setup

  • Location: superset-frontend/cypress-base/
  • Version: Cypress 11.2.0
  • Test Count: ~50+ test files
  • Coverage: Dashboard, charts, SQL Lab, authentication, datasets
  • CI Parallelization: 6-way split using custom Python script (scripts/cypress_run.py)
  • Features:
  • Code coverage with @cypress/code-coverage
  • Visual testing with Applitools Eyes
  • Custom commands and utilities
  • Docker database setup for tests

Key Files

  • superset-frontend/cypress-base/cypress.config.ts - Main configuration
  • scripts/cypress_run.py - CI parallelization orchestration
  • .github/workflows/superset-e2e.yml - GitHub Actions workflow
  • docker/docker-entrypoint-initdb.d/cypress-init.sh - Database setup

Phase Breakdown

Development Principles

  • YAGNI (You Aren't Gonna Need It): Build only what's immediately needed
  • DRY (Don't Repeat Yourself): Reuse existing Superset test infrastructure
  • KISS (Keep It Simple, Stupid): Prefer simple solutions over complex ones
  • Iterative: Each phase builds on proven success from previous phase

Phase 1: Get One Test Working End-to-End (Week 1)

Goal: Minimal viable Playwright setup with one auth test running locally and in CI

YAGNI Principle: Build only what's needed to get one test working

Tasks

  • [x] 1.1 Minimal Setup
  • Add Playwright to superset-frontend/package.json
  • Create superset-frontend/playwright.config.ts (Chrome only initially)

  • Create superset-frontend/tests/auth/login.spec.ts (port simplest Cypress auth test)

  • [ ] 1.2 Essential Utilities (Only What's Needed)

  • Create basic login helper function
  • Add baseURL configuration pointing to localhost:8088

  • Setup minimal TypeScript configuration

  • [x] 1.3 CI Integration

  • Add Playwright to existing .github/workflows/superset-e2e.yml
  • Use same database and setup as Cypress (DRY principle)
  • Run single auth test alongside Cypress tests
  • Add basic artifact collection for failures

Success Criteria (KISS - Keep It Simple): - ✅ One auth test runs locally: npx playwright test auth/login - ✅ Same test passes in GitHub Actions CI - ✅ Test uses existing Superset test infrastructure (no duplication) - ✅ Takes < 1 day to implement

Phase 2: Expand Testing Infrastructure (Week 2)

Goal: Add necessary infrastructure for migrating more tests

Build on Phase 1 success, add only what's proven needed

Tasks

  • [ ] 2.1 Test Structure (Based on Phase 1 learnings)
  • Create tests/ directory structure as needed
  • Add utilities discovered during auth test implementation

  • Setup page object models only if reuse is obvious

  • [ ] 2.2 CI Optimization

  • Add Playwright parallelization (native, not custom script)
  • Configure browser caching
  • Setup proper test result reporting

  • Add 2-3 more simple tests to validate infrastructure

  • [ ] 2.3 Developer Experience

  • Add Makefile targets based on actual usage patterns
  • Document basic Playwright commands
  • Setup debugging configuration

Success Criteria: - Multiple test types working (auth + list view + simple dashboard) - Playwright runs faster than equivalent Cypress tests - Clear path for migrating remaining tests - Developer documentation exists for common tasks

Phase 3: Systematic Test Migration (Weeks 3-8)

Goal: Systematically migrate tests from Cypress to Playwright

Migration Strategy

Priority Order: 1. Authentication tests (simple, foundational) 2. List view tests (charts, dashboards, datasets) 3. Form interactions and modals 4. Dashboard functionality 5. Chart/explore functionality 6. SQL Lab functionality 7. Complex integration tests

Tasks by Priority

  • [ ] 3.1 Authentication & Navigation (Week 3)
  • cypress/e2e/auth/ → playwright/tests/auth/
  • Basic login/logout functionality

  • Menu navigation and permissions

  • [ ] 3.2 List Views (Week 4)

  • cypress/e2e/chart_list/ → playwright/tests/chart_list/
  • cypress/e2e/dashboard_list/ → playwright/tests/dashboard_list/
  • cypress/e2e/dataset/dataset_list.test.ts → playwright/tests/dataset/

  • Filtering, sorting, search functionality

  • [ ] 3.3 Database & Connections (Week 4)

  • cypress/e2e/database/ → playwright/tests/database/

  • Modal interactions, form submissions

  • [ ] 3.4 Dashboard Functionality (Week 5-6)

  • cypress/e2e/dashboard/ → playwright/tests/dashboard/
  • Dashboard loading, editing, filters

  • Chart interactions within dashboards

  • [ ] 3.5 Chart Explore Interface (Week 6-7)

  • cypress/e2e/explore/ → playwright/tests/explore/
  • Chart creation, visualization types
  • Advanced analytics, annotations

  • Control panels and configurations

  • [ ] 3.6 SQL Lab (Week 7-8)

  • cypress/e2e/sqllab/ → playwright/tests/sqllab/
  • Query execution, result handling

  • Tab management, saved queries

  • [ ] 3.7 Visualization Tests (Week 8)

  • cypress/e2e/explore/visualizations/ → playwright/tests/visualizations/
  • Chart-specific testing (big_number, table, pie, etc.)

Per-Test Migration Process

  1. Analysis: Review Cypress test for custom commands and dependencies
  2. Conversion: Rewrite using Playwright APIs and patterns
  3. Enhancement: Add cross-browser testing where beneficial
  4. Validation: Compare results with Cypress baseline
  5. Documentation: Update test descriptions and comments

Success Criteria: - Each migrated test passes consistently - No increase in flakiness compared to Cypress equivalent - Performance improvements in test execution time

Phase 4: Validation & Optimization (Weeks 7-9)

Goal: Validate Playwright reliability and optimize performance

Tasks

  • [ ] 4.1 Parallel Execution Monitoring
  • Run both Cypress and Playwright in CI for all PRs

  • Compare via CI logs and GitHub test reports:

    • Test execution times
    • Flakiness rates (retry counts)
    • Resource usage (CI job duration)
    • Coverage metrics (existing tools)

  • [ ] 4.2 Performance Optimization

  • Optimize Playwright configuration for fastest execution
  • Implement parallel test execution best practices
  • Configure browser reuse and context management

  • Tune timeouts and retry settings

  • [ ] 4.3 Cross-Browser Testing

  • Enable Firefox and Safari testing where appropriate
  • Add mobile viewport testing for responsive features

  • Configure browser-specific test exclusions

  • [ ] 4.4 Developer Experience

  • Create debugging guides and best practices
  • Setup VS Code integration and extensions
  • Document common patterns and utilities

Success Criteria: - Playwright tests run 30-50% faster than Cypress - Zero increase in false positives/negatives - Team comfortable with Playwright tooling

Phase 5: Cypress Sunset (Weeks 9-10)

Goal: Remove Cypress infrastructure and complete migration

Tasks

  • [ ] 5.1 Final Migration
  • Convert any remaining Cypress-specific tests
  • Handle edge cases and complex integrations

  • Migrate Applitools visual tests or find alternatives

  • [ ] 5.2 Cleanup

  • Remove Cypress dependencies from package.json
  • Delete superset-frontend/cypress-base/ directory
  • Remove Cypress GitHub Actions workflow

  • Clean up Docker configuration

  • [ ] 5.3 Documentation

  • Update CLAUDE.md and other contributor guides
  • Create Playwright testing best practices document
  • Archive Cypress-specific documentation

  • Update README and development setup guides

  • [ ] 5.4 Training & Knowledge Transfer

  • Conduct team training sessions on Playwright
  • Create troubleshooting guides
  • Document common migration patterns

Success Criteria: - No Cypress dependencies remain - All tests migrated and passing - Team fully trained on Playwright - Documentation complete and accurate

Risk Mitigation

Technical Risks

  • Test Coverage Gaps: Maintain both frameworks during transition
  • Performance Regression: Benchmark and optimize continuously
  • Team Adoption: Provide training and clear migration guides
  • CI/CD Stability: Gradual rollout with rollback capability

Mitigation Strategies

  • Keep Cypress as backup during entire migration
  • Create comprehensive test result comparisons
  • Implement feature flags for Playwright adoption
  • Maintain detailed migration documentation

Success Metrics

Performance

  • [ ] Test execution time reduced by 30-50%
  • [ ] CI pipeline stability maintained or improved
  • [ ] Resource usage (memory/CPU) optimized

Quality

  • [ ] Zero increase in test flakiness
  • [ ] Maintained or improved test coverage
  • [ ] Cross-browser testing coverage added

Developer Experience

  • [ ] Reduced test maintenance overhead
  • [ ] Improved debugging capabilities
  • [ ] Better local development experience

Timeline

Total Duration: 9-10 weeks - Phase 1: Week 1 (Get One Test Working - MVP) - Phase 2: Week 2 (Expand Infrastructure Based on Learnings) - Phase 3: Weeks 3-7 (Migration) - Phase 4: Weeks 7-9 (Validation) - Phase 5: Weeks 9-10 (Cleanup)

Resources

Documentation

Key Benefits Expected

  • Native Parallelization: Remove custom Python orchestration script
  • Cross-Browser Testing: Chrome, Firefox, Safari support out of the box
  • Better Debugging: Trace viewer, inspector, and screenshot tools
  • Mobile Testing: Device emulation capabilities
  • Improved Reliability: Better wait mechanisms and retry logic