Surface with multiple textures

This example demonstrates one possible method for displaying a 3D surface with multiple textures.

Thanks to Emmanuel Reynaud and Luis Gutierrez for providing the gorgeous coral model for this demo. You can find the data on FigShare: https://zenodo.org/records/13380203

More information on the methods used to generate this model can be found in L. Gutierrez-Heredia, C. Keogh, E. G. Reynaud, Assessing the Capabilities of Additive Manufacturing Technologies for Coral Studies, Education, and Monitoring. Front. Mar. Sci. 5 (2018), doi:10.3389/fmars.2018.00278.

A bit about 3D models

A standard way to define a 3D model (mesh, or Surface in napari) is by listing vertices (3D point coordinates) and faces (triplets of vertex indices - each face is a triangle in 3D space). Meshes are often stored in “Wavefront” (.obj) files, which may have companion material (.mtl) files that describe some shading properties (base color, shinyness, etc.) for different parts of the model.

In some cases, the color of a vertex is given by a single point value that is then colormapped on the fly (vertex_values). In other cases, each vertex or face may be assigned a specific color (vertex_colors). These methods are demonstrated in Surface with texture and vertex_colors.

In the case of “photorealistic” models, the color of each vertex is instead determined by mapping a vertex to a point in an image called a texture using 2D texture coordinates in the range [0, 1]. The color of each individual pixel is smoothly interpolated (sampled) on the fly from the texture (the GPU makes this interpolation very fast).

Napari does not (yet) support models with multiple textures or materials. If the textures don’t overlap, you can display them on separate meshes as shown in this demo. If the textures do overlap, you may instead be able to combine the textures as images. This relies on textures having the same texture coordinates, and may require resizing the textures to match each other.

Tags: visualization-nD

import os

import matplotlib.pyplot as plt
import pooch
from vispy.io import imread, read_mesh

import napari

Download the model

download = pooch.DOIDownloader(progressbar=True)
doi = '10.5281/zenodo.13380203'
tmp_dir = pooch.os_cache('napari-surface-texture-example')
os.makedirs(tmp_dir, exist_ok=True)
data_files = {
    'mesh': 'PocilloporaDamicornisSkin.obj',
    # "materials": "PocilloporaVerrugosaSkinCrop.mtl",  # not yet supported
    'Texture_0': 'PocilloporaDamicornisSkin_Texture_0.jpg',
    'GeneratedMat2': 'PocilloporaDamicornisSkin_GeneratedMat2.png',
}
print(f'downloading data into {tmp_dir}')
for file_name in data_files.values():
    if not (tmp_dir / file_name).exists():
        print(f'downloading {file_name}')
        download(
            f'doi:{doi}/{file_name}',
            output_file=tmp_dir / file_name,
            pooch=None,
        )
    else:
        print(f'using cached {tmp_dir / file_name}')

downloading data into /home/runner/.cache/napari-surface-texture-example
downloading PocilloporaDamicornisSkin.obj

  0%|                                              | 0.00/93.8M [00:00<?, ?B/s]
  0%|                                      | 18.4k/93.8M [00:00<13:47, 113kB/s]
  0%|                                      | 57.3k/93.8M [00:00<08:19, 188kB/s]
  0%|                                       | 136k/93.8M [00:00<04:49, 324kB/s]
  0%|                                       | 289k/93.8M [00:00<02:44, 567kB/s]
  1%|▏                                     | 612k/93.8M [00:00<01:26, 1.08MB/s]
  1%|▍                                    | 1.25M/93.8M [00:00<00:45, 2.05MB/s]
  3%|▉                                    | 2.53M/93.8M [00:01<00:23, 3.96MB/s]
  5%|██                                   | 5.08M/93.8M [00:01<00:11, 7.71MB/s]
  9%|███▎                                 | 8.41M/93.8M [00:01<00:07, 11.7MB/s]
 13%|████▋                                | 11.8M/93.8M [00:01<00:05, 14.5MB/s]
 15%|█████▋                               | 14.5M/93.8M [00:01<00:05, 15.1MB/s]
 18%|██████▊                              | 17.3M/93.8M [00:01<00:04, 15.9MB/s]
 22%|████████                             | 20.4M/93.8M [00:02<00:04, 16.8MB/s]
 25%|█████████▍                           | 23.8M/93.8M [00:02<00:03, 18.0MB/s]
 29%|██████████▌                          | 26.8M/93.8M [00:02<00:03, 18.2MB/s]
 32%|███████████▊                         | 30.0M/93.8M [00:02<00:03, 18.7MB/s]
 35%|████████████▉                        | 32.9M/93.8M [00:02<00:03, 18.4MB/s]
 39%|██████████████▎                      | 36.3M/93.8M [00:02<00:02, 19.2MB/s]
 42%|███████████████▋                     | 39.8M/93.8M [00:03<00:02, 19.9MB/s]
 46%|████████████████▉                    | 43.1M/93.8M [00:03<00:02, 20.0MB/s]
 50%|██████████████████▎                  | 46.5M/93.8M [00:03<00:02, 20.4MB/s]
 53%|███████████████████▋                 | 50.0M/93.8M [00:03<00:02, 20.7MB/s]
 57%|█████████████████████▏               | 53.5M/93.8M [00:03<00:01, 21.0MB/s]
 61%|██████████████████████▌              | 57.1M/93.8M [00:03<00:01, 21.4MB/s]
 65%|███████████████████████▉             | 60.5M/93.8M [00:04<00:01, 21.2MB/s]
 68%|█████████████████████████▏           | 63.9M/93.8M [00:04<00:01, 21.2MB/s]
 71%|██████████████████████████▎          | 66.8M/93.8M [00:04<00:01, 20.1MB/s]
 75%|███████████████████████████▋         | 70.3M/93.8M [00:04<00:01, 20.3MB/s]
 79%|█████████████████████████████        | 73.8M/93.8M [00:04<00:00, 20.8MB/s]
 82%|██████████████████████████████▏      | 76.6M/93.8M [00:04<00:00, 19.7MB/s]
 85%|███████████████████████████████▌     | 80.1M/93.8M [00:05<00:00, 20.2MB/s]
 89%|████████████████████████████████▉    | 83.6M/93.8M [00:05<00:00, 20.5MB/s]
 92%|██████████████████████████████████   | 86.5M/93.8M [00:05<00:00, 19.7MB/s]
 96%|███████████████████████████████████▍ | 89.9M/93.8M [00:05<00:00, 20.1MB/s]
100%|████████████████████████████████████▉| 93.5M/93.8M [00:05<00:00, 20.6MB/s]
  0%|                                              | 0.00/93.8M [00:00<?, ?B/s]
100%|██████████████████████████████████████| 93.8M/93.8M [00:00<00:00, 449GB/s]
downloading PocilloporaDamicornisSkin_Texture_0.jpg

  0%|                                              | 0.00/17.3M [00:00<?, ?B/s]
  0%|                                      | 12.3k/17.3M [00:00<02:36, 111kB/s]
  0%|                                      | 45.1k/17.3M [00:00<01:39, 173kB/s]
  1%|▎                                      | 116k/17.3M [00:00<00:56, 303kB/s]
  1%|▌                                      | 253k/17.3M [00:00<00:32, 528kB/s]
  3%|█▏                                     | 512k/17.3M [00:00<00:18, 924kB/s]
  6%|██▎                                  | 1.07M/17.3M [00:00<00:09, 1.79MB/s]
 13%|████▋                                | 2.16M/17.3M [00:01<00:04, 3.44MB/s]
 25%|█████████▎                           | 4.34M/17.3M [00:01<00:01, 6.66MB/s]
 42%|███████████████▌                     | 7.25M/17.3M [00:01<00:00, 10.2MB/s]
 57%|████████████████████▉                | 9.80M/17.3M [00:01<00:00, 11.9MB/s]
 74%|███████████████████████████▌         | 12.9M/17.3M [00:01<00:00, 14.0MB/s]
 96%|███████████████████████████████████▍ | 16.6M/17.3M [00:01<00:00, 16.7MB/s]
  0%|                                              | 0.00/17.3M [00:00<?, ?B/s]
100%|█████████████████████████████████████| 17.3M/17.3M [00:00<00:00, 81.2GB/s]
downloading PocilloporaDamicornisSkin_GeneratedMat2.png

  0%|                                               | 0.00/120k [00:00<?, ?B/s]
  1%|▎                                     | 1.02k/120k [00:00<00:30, 3.94kB/s]
 13%|████▊                                 | 15.4k/120k [00:00<00:02, 43.7kB/s]
 42%|████████████████▎                      | 50.2k/120k [00:00<00:00, 112kB/s]
 90%|████████████████████████████████████▏   | 109k/120k [00:00<00:00, 203kB/s]
  0%|                                               | 0.00/120k [00:00<?, ?B/s]
100%|████████████████████████████████████████| 120k/120k [00:00<00:00, 721MB/s]

Load the model

Next, read the model data from the .obj file. Currently napari/vispy do not support reading material properties (.mtl files) nor separate texture and vertex indices (i.e. repeated vertices). Normal vectors read from the file are also ignored and re-calculated from the faces.

vertices, faces, _normals, texcoords = read_mesh(tmp_dir / data_files['mesh'])

Load the textures

This model comes with two textures: Texture_0 is generated from photogrammetry of the actual object, and GeneratedMat2 is a generated material to fill in parts of the model lacking photographic texture.

photo_texture = imread(tmp_dir / data_files['Texture_0'])
generated_texture = imread(tmp_dir / data_files['GeneratedMat2'])

This is what the texture images look like in 2D:

fig, axs = plt.subplots(1, 2)
axs[0].set_title(f'Texture_0 {photo_texture.shape}')
axs[0].imshow(photo_texture)
axs[0].set_xticks((0, photo_texture.shape[1]), labels=(0.0, 1.0))
axs[0].set_yticks((0, photo_texture.shape[0]), labels=(0.0, 1.0))
axs[1].set_title(f'GeneratedMat2 {generated_texture.shape}')
axs[1].imshow(generated_texture)
axs[1].set_xticks((0, generated_texture.shape[1]), labels=(0.0, 1.0))
axs[1].set_yticks((0, generated_texture.shape[0]), labels=(0.0, 1.0))
fig.show()

Create the napari layers

Next create two separate layers with the same mesh - once with each texture. In this example the texture coordinates happen to be the same for each texture, but this is not a strict requirement.

photo_texture_layer = napari.layers.Surface(
    (vertices, faces),
    texture=photo_texture,
    texcoords=texcoords,
    name='Texture_0',
)
generated_texture_layer = napari.layers.Surface(
    (vertices, faces),
    texture=generated_texture,
    texcoords=texcoords,
    name='GeneratedMat2',
)

Add the layers to a viewer

Finally, create the viewer and add the Surface layers. sphinx_gallery_thumbnail_number = 2

viewer = napari.Viewer(ndisplay=3)

viewer.add_layer(photo_texture_layer)
viewer.add_layer(generated_texture_layer)

viewer.camera.angles = (90.0, 0.0, -75.0)
viewer.camera.zoom = 75

if __name__ == '__main__':
    napari.run()