Unit Tests

To run the Python tests:

pytest --cov=ipywidgets ./python/ipywidgets

To run the Javascript tests in each package directory:

yarn test

This will run the test suite using karma with ‘debug’ level logging.

Visual Regression Tests

ipywidgets uses the Galata framework for visual regression testing. Galata provides a high level API to programmatically interact with the JupyterLab UI, and tools for taking screenshots and generating test reports.

UI tests are written in TypeScript and run with the Playwright test runner. The test suites are located in the ui-tests/tests directory.

The main test suite uploads a notebook to JupyterLab, runs it cell by cell and captures a screenshot of the cell outputs. The cell outputs correspond to widgets of different types. The cell outputs captures are then compared to the reference snapshots to detect any visual regression. The test report (diffs, result, videos) is uploaded to GitHub as an artifact and accessible from the GitHub Actions page in Artifacts section.

Running Tests Locally

First install the dependencies:

cd ui-tests
yarn install

Galata needs to connect to a running instance of JupyterLab 3 to run UI tests. First launch JupyterLab and keep it running in a terminal window.

# in ui-tests directory
yarn start

Then run the test script:

# in the ui-tests directory
yarn test

You can pass additional arguments to playwright by appending parameters to the command. For example to run the test in headed mode, yarn test --headed.

Checkout the Playwright Command Line Reference for more information about the available command line options.

Adding new UI tests

New test suites can be added to the ui-tests/tests directory. Their names need to end with .test.ts. You can see some additional example test suites in the JupyterLab repo. If the tests in new suites are doing visual regression tests or HTML source regression tests then you also need to add their reference images to the -snapshots directories.

Reference Image Captures

When doing visual regression tests, it is important to use reference images that were generated in the same environment. Otherwise, even though the same browser is used for testing, there might be minor differences that could cause visual regression tests to fail.

When adding a new visual regression test, first make sure your tests pass locally on your development environment, with a reference snapshots generated in your dev environment. You can generate new reference snapshots by running yarn test:update.

To update the snapshots:

• push the new changes to the branch

• wait for the CI check to complete

• go to the artifacts section and download the ipywidgets-updated-snapshots archive

• extract the archive

• copy the -snapshots directories to replace the existing ones

• commmit and push the changes