NAP-9 — Multiple Views#

Authors:: Ashley Anderson <aandersoniii@chanzuckerberg.com>, Wouter-Michiel Vierdag, Lorenzo Gaifas
Created:: 2023-08-04
Status:: Draft
Type:: Standards Track

Definitions#

Note

This NAP was previously discussed as Multiple Canvases, but due to the term Canvas having a well-established and more restrictive meaning of what we aim to implement, we’re shifting to using the term View for a (hopefully) clearer distinction from existing terms. For a more thorough discussion of terminology, see alternative terminology.

In order to facilitate discussion - this NAP will use the following definitions.

Viewer - Currently maps basically 1:1 to the napari application main window, including canvas, dims sliders, layer list, layer controls, and dock widgets. Related is the ViewerModel, a class in napari that maintains the state of the Viewer.

Canvas - Currently usually used to refer to the central widget of the napari Viewer which renders the data from the current slice, as well as keeping track of all the visualisation scenegraph from vispy and various other napari-vispy interfaces.

View - New term introduced by this NAP in order to formalize and disentangle the previous notions of Canvas and Viewer. This NAP introduces a View class in napari with its own Layerlist, Dims, and Camera.

Layer - The base unit of the napari image data model. Currently, a ViewerModel maintains an ordered list of Layers that it may display on its View.

Layer Slice - A subset of data from a Layer, reduced to 2D or 3D (via slicing and projecting) for visualization. This concept was introduced in NAP-4 as part of the async slicing work, and was already implemented for most layer types in subsequent PRs (Image, Labels, Points, and Vectors).

Warning

VisPy (the only current rendering backend in napari) has its own specific definitions for some of these or related concepts, such as:

Canvas: the whole surface on which things can be rendered (and which can be embedded in the GUI as we do in napari)
ViewBox: a rectangular portion of a canvas where a Scene can be visualised
Scene: a collection of renderable objects and cameras that the canvas can use to render inside a ViewBox. In vispy, scenes are tree-like structures called scenegraphs composed by Nodes.

Where necessary to refer to these concepts in this NAP (or discussion), such concepts will be qualified accordingly (for example: “a VisPy Canvas”)

Abstract#

Current napari architecture supports a single canvas/camera/view per viewer. Simultaneously showing multiple views of the same (or different) data generally necessitates opening an entirely new napari viewer window or low-level work with Qt widgets and private napari APIs. This wastes resources (primarily memory) and complicates interaction. This NAP establishes a plan to implement a builtin system for opening, visualizing and interacting with multiple views within a single viewer.

We propose to achieve this in two parts that can be implemented independently:

Splitting of part of the current ViewerModel into a new View model holding a Layerlist, a Dims, and a Camera. ViewerModel will hold a list/Sequence of Views, allowing for multiple independent views.
Completion of the async LayerSlicer and SliceRequest/SliceResponse work for each layer type, and subsequent complete separation of layer slicing state from the layer models. This will allow to reuse layer objects between different views.

Motivation and Scope#

The ability to view n-D data from multiple perspectives (ortho-view), or different data from the same perspective (for example side-by-side segmentations) is a common feature request, and has proved useful in many other tools for data exploration and analysis. Here is a sampling of issues requesting support and discussing potential implementations:

#5348 Multicanvas viewer
#2338 Multicanvas API Thoughts
#760 Linked multicanvas support
#662 Linked 2D views
#561 multicanvas grid display for layers in Napari
#1478 Orthogonal viewer plugin

Several plugins and examples have been created to address these limitations, for example:

Significant overlap of issues ideas can also be found in NAP-3 and its surrounding discussions (#4419, #4684, #4734).

This document is intended to cover what #5348 refers to as “True Multicanvas”.

Providing native support in napari would allow developers to more easily create these experiences, enable interoperability between such plugins, and improve performance.

Out of Scope#

Improvements to VisPy to support multiple views of the same SceneGraph (sharing data, saving VRAM) - for relevant discussion start with vispy/#1992.
For now, view arrangement (for example: tiling behavior) will be handled in the viewer only (left to Qt or custom Qt widgets). Making this state (de)serializable is out of scope for this project, but may be relevant when implementing a “savable viewer state” feature.
Normalizing slice data for different layer types to a uniform protocol/system, though this may benefit in the course of this work.
Specific UI implementations will be explored as part of this work, but UX and UI will likely be formalized later. Since this NAP was first drafted, a lot of work and discussion a on UX and UI has already been carried on in #5348, which may be included in this NAP or developed later depending on discussion.
Supporting alternative frontend (Qt) and backend (Vispy) frameworks. While this work should not make such tasks more difficult in the future, explicit consideration is out-of-scope until further progress is made in these areas.
Non-goals also in NAP-3 are related but also considered out-of-scope here
- Separation of rendering information (e.g: colormap) from the data (e.g: features) which currently both live on the layer model
- Window state restoration and “workspaces” (see #4227 for further discussion)

Requirements#

The application data model (ViewerModel + Layers) shall support multiple views.
The application shall natively display multiple views simultaneously.
- There shall be a minimum of one view (current status) per viewer.
Each view shall have independent:
- Layer list - or layerlist “subview” (see #Alternative single LayerList for details) (necessary for visualizing different data)
- Camera (necessary for viewing data from different POV)
- Dims model (necessary for viewing different slices/dimensions of the data)
The implementation should minimize changes to the existing public API.
The napari application (ViewerModel) shall maintain a concept of a single “active” (currently focused) view.
- Alternatively, there could be a “main” view that does not change (“main” and “active” could even be simultaneously supported).
- There will be no possibility of a viewer with no views.
Users shall be able to add, remove, and (eventually[1]) rearrange views.

Design Considerations & Decisions#

Part of this design document is intended to capture the desired behavior and prevent scope creep. At the extreme “multiple views” can be achieved with “multiple viewers”. Therefore we need to draw a line somewhere to differentiate a “view” from a “viewer”. [2]

An important consideration is to minimize breaking changes to the public napari API. While napari is still pre-1.0, there is already a healthy developing ecosystem of plugins, scripts, and users. Changes to the API may be necessary and should be made if they constitute improvements, but should be minimized and well documented.

In addition to maintaining the model-view-controller (MVC) architecture of napari, this proposal aims to maintain or improve decoupling of the UI framework (currently Qt), the visualization library (currently VisPy), and the napari core code.

Grid mode vs Multi-View#

The napari viewer currently has a grid mode that can be activated to distribute all layers in a rectangular grid of vispy ViewBoxes according to a few simple parameters (mainly shape and stride) (as of time of writing, the viewbox-based implementation in #7870 is not yet merged, but will be soon).

This feature has some theoretical and functional overlap with the multi-view described in this NAP, but with a few key differences:

grid mode allows no control over individual viewbox size and placement except through grid shape and stride. Viewboxes cannot be reordered, removed, or added.
grid mode does not create views with independent dims and cameras. The same Camera and Dims are used to control all viewboxes.
grid mode does not have fine-grained control of which layers are displayed in which viewbox. Layers are simply distributed based on the order of the layerlist and according to the stride.

These limitations dramatically simplify the usability of grid mode (one button click is usually all that’s needed for most use cases). On the other hand, multi-view as described in this NAP offers greater control over every aspect described above, at the cost of a more complicate API and GUI accessibility.

In this NAP, we assume that grid mode and multiple views remain separate features. For discussion about why (and whether they should instead be merged into a single feature) see #Alternative: Multi-View grid mode.

Implementation#

Part 1: View model#

Introduce a minimally disruptive _views attribute on the ViewerModel, and implement a View model to hold a Layerlist, a Dims, and a Camera (as well as some related concepts and models).
using a SelectableEventedList for _views lets us easily introduce the concept of an active View. This, together with some simple dispatch of current viewer elements to the active view, will make this part of the implementation a drop-in replacement.
At this stage, only the active view will be displayed. This will leave the GUI identical to before. When a new view becomes active, the contents of the QtViewer and vispy canvas are simply swapped to be linked with the new layerlist, camera, and dims.
Similarly, existing events and event callbacks will connected to the active view only.
Add APIs to add, remove and select views

The concept of an “active” view will work in service of minimizing API changes. This will allow existing APIs on the main Viewer/ViewerModel to remain and simply delegate to the active view.

Additionally, by providing access to the components of the active view through the ViewerModel, we can also seamlessly transition to a multiview system without breaking any high level API.

Currently, the ViewerModel (when stripped down to its barebones) contains the following fields and functionality:

class ViewerModel:
    camera: Camera
    cursor: Cursor
    dims: Dims
    grid: GridCanvas
    layers: LayerList
    help: str
    status: Union[str, dict]
    tooltip: Tooltip
    theme: str
    title: str
    _overlays: EventedDict[str, Overlay]
    _layer_slicer: _LayerSlicer

With this NAP and the above considerations, we expect to separate the above components roughly as follows:

class View:
    camera: Camera
    cursor: Cursor
    dims: Dims
    grid: GridCanvas
    layers: LayerList
    _overlays: EventedDict[str, Overlay]
    _layer_slicer: _LayerSlicer

class ViewerModel:
    _views: SelectableEventedList[View]
    help: str
    status: Union[str, dict]
    tooltip: Tooltip
    theme: str
    title: str

    @property
    def camera(self):
        return self._views.active.camera

    [...]

TODO

Discuss whether the LayerSlicer should be singleton on the viewer or one per view.

Part 2: Decouple slicing state from layer models#

Important

The following assumes that we work from the top down. However, this refactoring work is very large, encompassing a large portion of the layer code.

Therefore, it might be better to simple rewrite the layer classes from the ground up (as has been discussed and suggested in many other occasions). If that option were to be chosen, a new NAP will likely be in order.

following the existing implementations, add SliceResponse classes for each layer type (currently remaining: Shapes, Surface, Tracks).
move all layer slicing state from Layers to the SliceResponse objects living in the Views. This includes base attributes like _slice_input, _data_slice and so on, as well as layer-specific properties and methods such as all the Points._view_* attributes. One way of handling this is to add a (private?) mapping of layers to slice responses on the View, which is updated by the LayerSlicer when slicing occurs:

class View:
    ...
    _layer_slices: EventedDict[Layer, SliceResponse]

On the vispy side, VispyLayer subclasses will now hold references to their Layer (for rendering information) as well as a SliceResponse (for data).
If we retain a singleton LayerSlicer, it will need to update the sync and async callbacks to pass a view parameter where the resulting sliced data will be stored, rather than storing it on the layer itself. Slice task cancellation logic will need to be revisited accordingly.
- async callback: LayerSlicer._on_slice_done
- sync callback: Layer._slice_dims

Note

some layers include “selection” information (Points, Shapes, and Tracks). These will be considered as Layer-level concepts, and will not be view-dependent but global for that specific layer object.

Note

Consider to codify a protocol (typing.Protocol) for Response classes.

Table 1 - Layer attributes and methods that hold, manipulate, or depend on slice data. For brevity, some implied attributes are left out (such as _thumbnail and _thumbnail_shape since the public thumbnail is listed). These attributes will either be moved from the Layer onto the SliceResponse objects, moved to mouse or shortcut callbacks (Shapes._is_moving), rendered obsolete (update_draw), refactored to not use slicing state (get_value), or require further discussion (cursor).

Note

A question mark is next to attributes and methods that need particular discussion

Layer Class	Attributes
Base/all subclasses	`_slice_input`
	`_update_dims`
	`_data_slice`
	`corner_pixels`
	`thumbnail` ?
	`cursor` ?
	`set_view_slice`
	`_set_view_slice`
	`_make_slice_input`
	`_update_slice_response`
	`_get_value`
	`get_`
	`loaded` ?
	`_update_draw`
	`_get_value`
	`_get_value_3d`
	`_get_value_ray`
Image, Labels	`_data_view`
	`_data_level`
Points	all `_view_` and `__view` properties
	`interaction_box` ?
	`_set_drag_start`
Surface	all `_view_` and `__view` properties
Vectors	all `_view_` and `__view` properties
Tracks	all `_view_` and `__view` properties
	`_current_displayed_dims`
Shapes	`_data_dict`
	`_data_view`
	all `_*_box` properties
	all `_drag_*` properties
	`_is_moving`
	`_is_selecting`
	`_is_creating`
	`_fixed_aspect`
	`_fixed_index`
	`_update_properties`
	`_allow_thumbnail_update`
	`_vertex_size`
	`_rotation_handle_length`

Part 3: GUI and UX#

In this last step we want to unlock the ability to visualize multiple views at the same time. This requires changes to our _qt and _vispy modules (and any other potential future backend for GUI and rendering).

UX design#

Implementation#

QtViewer will be changed to hold a mapping (or list?) of Views to VispyCanvases (with the latter likely being renamed to VispyView for consistency)
Methods, shortcuts, and menus/buttons must be added to allow showing/hiding, reordering, adding/removing, and selecting views.

Backward Compatibility#

Maintaining the proxy API on the viewer via the concepts of a main and/or active view will make this work mostly backward-compatible, though there will inevitably be some breaking changes. There will likely be significant breaking changes to private APIs. For example if plugins are attempting to access slice data directly from a layer instance, it may no longer be as expected. If this is a large burden, it too may be mitigated by delegating from the layer to the slice corresponding to the main or active view.

Future Work#

The goal of this NAP is to cover the main architectural changes to enable multi-view work. Future work is expected in

user experience, design, and GUI implementation details
consistent, ergonomic, and documented public APIs for advanced interaction with multiple views.
development of frequently requested features dependent on this work (e.g.: orthoview)

Alternatives#

Terminology#

Other names for View that were discussed are:

Canvas: does not convey the independent dims and camera
Viewport: used similarly in other rendering software, conveys canvas + camera
Portal: more evocative of camera + dims, but no established meaning
Realm: same as above, but a stronger implication of “separate space” which might be undesirable

View is very similar to viewport but shorter and less jargony, so it seems the best candidate so far.

Single LayerList#

An alternative to each View holding a separate layerlist would be to have a single, centralized layerlist held by the ViewerModel, while each View only allows to independently set the visibility of each layer within that specific View. From the user perspective this makes it a bit easier to keep track of layers (as they are all always available in the layerlist) and to “transfer” layers across Views (by simply toggling visibility instead of moving them between lists). On the flip side, this makes working with large layerlists and several views more cumbersome from the GUI as the layerlist gets more crowded.

On the implementation side, this would require deprecating Layer.visible since layer visibility would now be a property of the View. This might accessible via something like a set of indices View.visible_layers = {0, 1}. This could be completely transparent via the GUI, but would be significantly more cumbersome programmatically.

A significant advantage of using multiple layerlists over a single one is that it allows us to proceed with part 1 (and potentially part 3 without depending on part 2, by initially disallowing layers to live in myultiple views (and thus having multiple slicing states).

TODO

discuss other differences

Multi-View grid mode#

Grid mode is currently distinct from multi-view. However, the two could be merged by implementing grid-mode as a one-button activation of a preset multi view (similar to how we plan to implement ortho-view). This would require the following:

Generate and NxM grid of views based on the grid shape and stride
link all view Cameras together
link all view Dims together
distribute layers in the various view layerlists according to stride
add callbacks so that modifying stride and grid shape will redistribute layers
add callbacks so adding/removing/reordering layers causes Views to change accordingly. This can only be done if the single layerlist alternative is used, otherwise users will have to manually access individual layerlists.
potentially allow destroying all views when clicking the grid mode button again (this might break any manual modifications made by the user)
potentially hide all unnecessary controls, such as redundant dims, redundant layerlist, etc, which (depending on UI implementation) might take up precious screen real estate

It’s unclear/undecided how the above should behave if at any point the user “breaks” the assumptions of the grid mode (e.g: reorders views manually, unlinks dims, then adds new layers, etc).

While this alternative unified two code branches into a single one, the downsides for grid mode users are significant. With separate features, users of grid-mode who want more control can easily move to a multiview approach when needed, while retaining the one-click simplicity of grid mode.

Additionally, differently from this NAP’s proposal, the implementation described above would disallow enabling grid mode in a canvas while retaining other canvases in “normal” mode. Refer to #5348 for extensive discussion about this.

Users can open multiple napari viewers#

Using multiple napari viewers does not satisfy the core user needs for multiple views when processing or manipulating data. Multiple viewers also wastes system resources as viewers do not communicate or share memory, as well as wasting screen real estate by duplicating widgets unnecessarily.

Leave multiview to plugins and custom widgets#

Unifying an implementation and (eventually) providing a stable multi-view API will save work for plugin authors, and allow more plugins to interoperate.

Implement slices using shallow Layer copies#

This is a good and reasonable alternative to the proposed implementation, and is how the multiple viewer widgets example is implemented. This implementation also makes it easier to configure rendering/appearance per-view (layer visibility, colormap, etc.). However this implementation relies more on careful bookkeeping than data modeling. If this is desired functionality, layer data should be fully separated from the data view (slice and view state). Ultimately this implementation is similar to that proposed in this NAP, and could be considered along a continuum of separating layer data, slice data, and rendering configuration.

Discussion#

#5348 Multicanvas viewer
- This is the most recent and thorough discussion multi-view prior to this NAP

Copyright#

This document is dedicated to the public domain with the Creative Commons CC0 license [3]. Attribution to this source is encouraged where appropriate, as per CC0+BY [4].

NAP-9 — Multiple Views#

Definitions#

Abstract#

Motivation and Scope#

Out of Scope#

Requirements#

Design Considerations & Decisions#

Grid mode vs Multi-View#

Related Work#

Implementation#

Part 1: View model#

Part 2: Decouple slicing state from layer models#

Part 3: GUI and UX#

UX design#

Implementation#

Backward Compatibility#

Future Work#

Alternatives#

Terminology#

Single LayerList#

Multi-View grid mode#

Users can open multiple napari viewers#

Leave multiview to plugins and custom widgets#

Implement slices using shallow Layer copies#

Discussion#

Copyright#