[SIP] Proposal for Migrating Superset E2E Test Framework from Cypress to Playwright
Motivation
Superset’s current E2E suite, built on Cypress, has shown increasing issues with flakiness, slowness, and high maintenance overhead. Additionally, changes in Cypress’s licensing model and ecosystem direction have raised concerns about its long-term viability. Instead of simply porting the test suite to a new tool, this SIP starts by defining what constitutes a good E2E testing framework and uses that lens to evaluate our options.
Proposed Change
We propose to fully replace Cypress with Playwright for all E2E test coverage in Superset. This is not a trial; it is a deliberate transition based on our evaluation criteria.
Criteria for Effective E2E Testing
We define good E2E test infrastructure as one that satisfies the following:
| # | Criterion | Description |
|---|---|---|
| 1 | Ease of local execution | Tests should be easy to run and debug locally with minimal setup |
| 2 | Stability under UI changes | Framework should support smart auto-waiting and reduce flake-prone code |
| 3 | Maintainable syntax | Tests should be written in modern syntax with good async/await and modular support |
| 4 | Actively maintained OSS project | The framework should be open source and well-supported |
| 5 | CI scalability | Supports parallel execution, test retries, and trace artifacts |
| 6 | Upgrade stability | Semantic versioning with limited breaking changes |
| 7 | Test tooling | Provides useful tools for authoring (codegen), debugging (trace viewer), and analysis |
Evaluation: Cypress vs. Playwright
| Criterion | Cypress | Playwright |
|---|---|---|
| Local execution | ✅ Good GUI, but slower reloads | ✅ Fast CLI runs, VS Code debugging integration |
| Test stability | ⚠️ Manual waits often required | ✅ Smart auto-waiting built-in |
| Syntax & DX | ⚠️ Uses older .then() chaining, harder to debug |
✅ Modern async/await, better readability |
| Ecosystem health | ⚠️ Increasingly commercialized, some core features behind login | ✅ Fully OSS, actively maintained by Microsoft |
| CI performance | ⚠️ Parallelism is paywalled | ✅ Native support for parallelism, retries, trace output |
| Codegen/tooling | ⚠️ Cypress Studio unstable and gated | ✅ Codegen and trace viewer included by default |
| Upgrade cost | ⚠️ Breaking changes common between versions | ✅ Rare breaking changes, semver-respecting |
New or Changed Public Interfaces
None. This change affects only internal test infrastructure. No updates to models, REST endpoints, dashboards, visualizations, React components, or CLI behavior.
New dependencies
| Package | Source | Maintained? | License |
|---|---|---|---|
| @playwright/test | npm | Actively maintained | Apache-2.0 |
Playwright is a well-maintained open-source project under the Apache-2.0 license, backed by Microsoft.
Migration Plan and Compatibility
Step 1: Audit and Cleanup
- Review all existing Cypress E2E test cases
- Remove any tests that do not represent true end-to-end workflows
- Open PRs to relocate non-E2E tests (e.g., UI logic, visual validations) to appropriate test layers or repositories (e.g., unit/component)
Step 2: Parallel Execution & Stability Validation
- Configure both Playwright and Cypress to run concurrently in CI during the transition
- Gather data on test pass/fail patterns, false positives/negatives, flake rate, and runtime
- Use this data to validate the stability of Playwright before full switch-over
Step 3: Framework Setup
- Integrate Playwright test runner into the codebase
- Add base utilities (e.g., login helpers, selectors, test fixtures)
- Set up GitHub Actions for Playwright runs, including test retries, trace collection, and artifacts
- Provide developer documentation for local test running and debugging
Step 4: Migrate Core E2E Coverage
- Port the following high-value workflows:
- Explore (chart creation, control panel interactions, save flow)
- Dashboard (load, edit, add chart, save)
- SQL Lab (query execution, preview, save query)
- Authentication (login/logout)
Step 5: Expand Remaining E2E Test Coverage
- Migrate additional E2E tests that meet our defined criteria
- Ensure tagging (e.g., smoke, sanity) is preserved where appropriate
- Replace Cypress-specific utilities with Playwright equivalents
Step 6: Deprecate Cypress
- Remove Cypress dependencies and test files
- Remove related CI jobs and scripts
- Enforce Playwright for all new E2E contributions going forward
Rejected Alternatives
| Option | Reason for Rejection |
|---|---|
| Stay on Cypress | High maintenance cost, flakiness, paywalled features, aging syntax |
| TestCafe / WebdriverIO / Nightwatch | Poor tooling, limited adoption, or complex setup |
| Hybrid frameworks | Increases maintenance burden and contributor confusion |