Unit Tests

To run the Python tests:

pytest --cov=ipywidgets ipywidgets

To run the Javascript tests in each package directory:

yarn test

This will run the test suite using karma with ‘debug’ level logging.

Visual Regression Tests

ipywidgets uses Galata framework for visual regression testing. Galata provides a high level API to interact with JupyterLab UI programmatically and tools for capturing, comparison and report generation.

Galata UI tests are written using typescript and jest. ipywidgets UI test suites are located in ui-tests/tests directory.

ui-tests/tests/widgets.test.ts test suite uploads a notebook into JupyterLab, runs it cell by cell and takes image captures of cell outputs. Cell outputs of this notebook are widgets of different types. Then, cell output captures are compared to reference images to detect any visual regressions. Test output (diffs, result report) generated by Galata are uploaded to GitHub artifact storage and accessible from GitHub Actions page in Artifacts section.

Running Tests Locally

First install the dependencies:

cd ui-tests
yarn install

Galata needs to connect to a running instance of JupyterLab 3 to run UI tests. First launch JupyterLab and keep it running in a terminal window.

# in ui-tests directory
yarn run start-jlab

Alternatively, you can use the command below to launch JupyterLab which is equivalent to start-jlab script:

jupyter lab --expose-app-in-browser --no-browser --ServerApp.token='' --ServerApp.password='' --ServerApp.disable_check_xsrf='True'

Then you can run the test script which in turn launches Galata to run UI tests:

# in the ui-tests directory
yarn run test

You can pass additional arguments to Galata by appending parameters to the command as in yarn run test -- --no-headless. For a full list of Galata command-line options see https://github.com/jupyterlab/galata#command-line-options.

Adding new UI tests

New test suites can be added into ui-tests/tests directory. Their names need to end with .test.ts. You can see some additional example test suites in Galata repo. If tests in new suites are doing visual regression tests or HTML source regression tests then you also need to add their reference images / HTML files into ui-tests/tests/reference-output directory.

Reference Image Captures

When doing visual regression tests, it is important to use reference images that were generated in the same environment. Otherwise, even though the same browser is used for testing, there might be minor differences in image renderings generated that could cause visual regression tests to fail.

When adding a new visual regression test, first make sure your tests pass locally on your development environment, with a reference image generated in your dev environment. You can use images captured by Galata as reference images. They will be saved as part of test output, under ui-tests/test-output/test/screenshots. However, you shouldn’t push these references images generated in your development environment to github. Instead, have the new regression tests run and fail by GitHub Actions first, then download the artifacts from GitHub and use the captures generated in GitHub testing environment as the reference images and push those in a separate commit.