To run the Python tests:
pytest --cov=ipywidgets ./python/ipywidgets
This will run the test suite using
karma with ‘debug’ level logging.
Visual Regression Tests#
ipywidgets uses the Galata framework for visual regression testing. Galata provides a high level API to programmatically interact with the JupyterLab UI, and tools for taking screenshots and generating test reports.
UI tests are written in TypeScript and run with the Playwright test runner. The test suites are located in the ui-tests/tests directory.
The main test suite uploads a notebook to JupyterLab, runs it cell by cell and captures a screenshot of the cell outputs. The cell outputs correspond to widgets of different types. The cell outputs captures are then compared to the reference snapshots to detect any visual regression. The test report (diffs, result, videos) is uploaded to GitHub as an artifact and accessible from the GitHub Actions page in
Running Tests Locally#
First install the dependencies:
cd ui-tests yarn install
Galata needs to connect to a running instance of JupyterLab 3 to run UI tests. First launch JupyterLab and keep it running in a terminal window.
# in ui-tests directory yarn start
Then run the
# in the ui-tests directory yarn test
You can pass additional arguments to
playwright by appending parameters to the command. For example to run the test in headed mode,
yarn test --headed.
Checkout the Playwright Command Line Reference for more information about the available command line options.
Adding new UI tests#
New test suites can be added to the ui-tests/tests directory. Their names need to end with
.test.ts. You can see some additional example test suites in the JupyterLab repo. If the tests in new suites are doing visual regression tests or HTML source regression tests then you also need to add their reference images to the
Reference Image Captures#
When doing visual regression tests, it is important to use reference images that were generated in the same environment. Otherwise, even though the same browser is used for testing, there might be minor differences that could cause visual regression tests to fail.
When adding a new visual regression test, first make sure your tests pass locally on your development environment, with a reference snapshots generated in your dev environment. You can generate new reference snapshots by running
To update the snapshots:
push the new changes to the branch
wait for the CI check to complete
go to the artifacts section and download the
extract the archive
-snapshotsdirectories to replace the existing ones
commmit and push the changes