To run the Python tests:
pytest --cov=ipywidgets ipywidgets
This will run the test suite using
karma with ‘debug’ level logging.
Visual Regression Tests¶
ipywidgets uses Galata framework for visual regression testing. Galata provides a high level API to interact with JupyterLab UI programmatically and tools for capturing, comparison and report generation.
ui-tests/tests/widgets.test.ts test suite uploads a notebook into JupyterLab, runs it cell by cell and takes image captures of cell outputs. Cell outputs of this notebook are widgets of different types. Then, cell output captures are compared to reference images to detect any visual regressions. Test output (diffs, result report) generated by Galata are uploaded to GitHub artifact storage and accessible from GitHub Actions page in
Running Tests Locally¶
First install the dependencies:
cd ui-tests yarn install
Galata needs to connect to a running instance of JupyterLab 3 to run UI tests. First launch JupyterLab and keep it running in a terminal window.
# in ui-tests directory yarn run start-jlab
Alternatively, you can use the command below to launch JupyterLab which is equivalent to
jupyter lab --expose-app-in-browser --no-browser --ServerApp.token='' --ServerApp.password='' --ServerApp.disable_check_xsrf='True'
Then you can run the
test script which in turn launches Galata to run UI tests:
# in the ui-tests directory yarn run test
You can pass additional arguments to Galata by appending parameters to the command as in
yarn run test -- --no-headless. For a full list of Galata command-line options see https://github.com/jupyterlab/galata#command-line-options.
Adding new UI tests¶
New test suites can be added into ui-tests/tests directory. Their names need to end with
.test.ts. You can see some additional example test suites in Galata repo. If tests in new suites are doing visual regression tests or HTML source regression tests then you also need to add their reference images / HTML files into ui-tests/tests/reference-output directory.
Reference Image Captures¶
When doing visual regression tests, it is important to use reference images that were generated in the same environment. Otherwise, even though the same browser is used for testing, there might be minor differences in image renderings generated that could cause visual regression tests to fail.
When adding a new visual regression test, first make sure your tests pass locally on your development environment, with a reference image generated in your dev environment. You can use images captured by Galata as reference images. They will be saved as part of test output, under ui-tests/test-output/test/screenshots. However, you shouldn’t push these references images generated in your development environment to github. Instead, have the new regression tests run and fail by GitHub Actions first, then download the artifacts from GitHub and use the captures generated in GitHub testing environment as the reference images and push those in a separate commit.