While not mandatory for most pull requests, we ask that performance related PRs include a benchmark in order to clearly depict the use-case that is being optimized for.
In this section we will review how to setup the benchmarks,
and three commands
asv run and
Begin by installing airspeed velocity
in your development environment. Prior to installation, be sure to activate your
development environment, then if using
venv you may install the requirement with:
source napari-dev/bin/activate pip install asv
If you are using conda, then the command:
conda activate napari-dev conda install asv
is more appropriate. Once installed, it is useful to run the command:
To let airspeed velocity know more information about your machine.
Writing a benchmark¶
To write benchmark, add a file in the
napari/_benchmarks directory which
contains a class with one
setup method and at least one method prefixed
time_ method should only contain code you wish to benchmark.
Therefore it is useful to move everything that prepares the benchmark scenario
setup method. This function is called before calling a
method and its execution time is not factored into the benchmarks.
Take for example the
import numpy as np import napari from qtpy.QtWidgets import QApplication class ViewImageSuite: """Benchmarks for viewing images in the viewer.""" def setup(self): app = QApplication.instance() or QApplication() np.random.seed(0) self.data = np.random.random((512, 512)) self.viewer = None def teardown(self): self.viewer.window.close() def time_view_image(self): """Time to view an image.""" self.viewer = napari.view_image(self.data)
Here, the creation of the image is completed in the
setup method, and not
included in the reported time of the benchmark.
It is also possible to benchmark features such as peak memory usage. To learn
more about the features of
asv, please refer to the official
airspeed velocity documentation.
Testing the benchmarks locally¶
Prior to running the true benchmark, it is often worthwhile to test that the code is free of typos. To do so, you may use the command:
asv dev -b TransformSuite
TransformSuite above will be run once in your current environment
to test that everything is in order.
Running your benchmark¶
The command above is fast, but doesn’t test the performance of the code
adequately. To do that you may want to run the benchmark in your current
environment to see the performance of your change as you are developing new
features. The command
asv run -E existing will specify that you wish to run
the benchmark in your existing environment. This will save a significant amount
of time since building napari can be a time consuming task:
asv run -E existing -b TransformSuite
Comparing results to master¶
Often, the goal of a PR is to compare the results of the modifications in terms
speed to a snapshot of the code that is in the master branch of the
napari repository. The command
asv continuous is of help here:
asv continuous master -b TransformSuite
This call will build out the environments specified in the
file and compare the performance of the benchmark between your current commit
and the code in the master branch.
The output may look something like:
$ asv continuous master -b TransformSuite · Creating environments · Discovering benchmarks ·· Uninstalling from conda-py3.7-cython-numpy1.15-scipy ·· Installing 544c0fe3 <benchmark_docs> into conda-py3.7-cython-numpy1.15-scipy. · Running 4 total benchmarks (2 commits * 2 environments * 1 benchmarks) [ 0.00%] · For napari commit 37c764cb <benchmark_docs~1> (round 1/2): [...] [100.00%] ··· ...ansform.TransformSuite.time_hough_line 33.2±2ms BENCHMARKS NOT SIGNIFICANTLY CHANGED.
In this case, the differences between HEAD and master are not significant enough for airspeed velocity to report.
To profile a particular benchmark in napari you can run
asv profile benchmark_qt_viewer.QtViewerSuite.time_create_viewer -g snakeviz --python=same
benchmark_qt_viewer is the file name,
QtViewerSuite is the test suite class name,
time_create_viewer is the test method.
To profile a particular parameterized benchmark you can run
asv profile "benchmark_image_layer.Image2DSuite.time_create_layer\(512\)" -g snakeviz --python=same
benchmark_image_layer is the file name,
Image2DSuite is the test suite class name,
time_to_create_layer is the test method and
512 is a valid parameter input to the test method.
Note that we in both these cases we have sent the output of the profiling to snakeviz which you can pip install with
pip install snakeviz
and we use
--python=same to profile against our current python environment.