In this series "From Image to Results", explore various case studies explaining how to reach results from your demanding samples and acquired images in an efficient way. For each case study, we highlight different samples, imaging systems, and research questions.

In this sixth episode, we generate a cell-profiling workflow to segment and quantify cellular structures in 3D EM images.

Key Learnings:

• How to generate a cell-profiling workflow to segment and quantify cellular structures in 3D EM images
  
• How to quantify size and distributions for mitochondria, the nucleus, and nuclear pore regions
  

Focused ion beam scanning electron microscopy (FIB-SEM) is a powerful imaging tool that achieves resolution under 10 nm. Though it produces highly detailed 3D image volumes, one drawback is that it is difficult to use standard image processing segmentation algorithms to detect many cellular structures of interest. This is largely because FIB-SEM highlights the entirety of the cell, generating images dense with cellular features, structural edges, and varying pixel combinations. Due to this difficulty, quantitative analysis of FIB-SEM data often relies on manual drawing of features of interest on 2D slices of a 3D image volume. Though this manual approach can be used to identify and reconstruct 3D objects from the image volume, it is tedious and time-consuming.

Previous work has focused on moving beyond this reliance on manual annotation to segment cellular structures from FIB-SEM image volumes. Notably, Gunes Parlakgül et al. (2022)¹ recently took a deep-learning approach to identify mitochondria, nucleus, endoplasmic reticulum, and lipid droplets within FIB-SEM image volumes of liver cells. The resulting neural network models trained on these organelles resulted in step toward a comprehensive automated cell-profiler workflow for FIB-SEM image data, where the models could be used on multiple image volumes to efficiently quantify these organelles. Here, we take a similar deep-learning approach, with our goal being the development of a cell-profiling workflow that uses neural-network training and image analysis tools that are readily accessible to researchers and do not require coding.

In this study, we highlight the use the ZEISS arivis Cloud (previously known as APEER) online deep learning platform in combination with the ZEISS arivis Pro image analysis software toolkit to facilitate automated profiling based on both large and small structures within a FIB-SEM image of a HeLa cell. The original dataset was kindly provided by Anna Steyer and Yannick Schwab, EMBL, Heidelberg, Germany. We used ZEISS arivis Cloud to train neural networks that identify large organelles: mitochondria and the nucleus. Using ZEISS arivis Cloud, we manually draw a subset of the instances of these cellular features and train neural network models that successfully predict the remainder of the instances of cellular features within the FIB-SEM image volume. These arivis Cloud-trained models were then used first to infer mitochondria and the nucleus in arivis Vison4D. We then built ZEISS arivis Pro analysis pipelines to filter and improve the initial inferences into usable 3D segments.

Having defined mitochondria and nucleus, we used the measurement and visualization tools in ZEISS arivis Pro to examine the cytoplasmic organization of the HeLa cell. We noticed that, even though our images are of low resolution and quality compared to current state-of-the-art FIB-SEM, we could visualize the nuclear membrane and the nuclear pores and sought to develop a method to assess their distribution. Through a series of pipeline workflows in ZEISS arivis Pro we identify the nuclear pore complex (NPC) regions of the nuclear membrane. Our approach utilizes 3D operations that enable enhancement and segmentation of 3D spatially resolved NPC-associated objects in a way that would not be possible by segmenting each 2D plane separately within the image stack. These objects are reliable proxies for the NPCs in distribution analysis that be subsequently used to make 3D masks of individual NPCs for neural network trainings that respect the 3D nature of the data, which is necessary for accurate segmentation in cells.

Overall, this work highlights how the ZEISS arivis Cloud deep learning approach, when combined with the powerful 3D tools of ZEISS arivis Pro, enables 3D segmentation and measurements within FIB-SEM image sets.

Image Analysis Pipeline

In this study, we present novel approaches to efficiently segment sub-cellular structures from FIB-SEM imaging data. Using the ZEISS arivis Cloud platform to perform convolutional neural network training, along with the ZEISS arivis Pro image analysis software, we were able to both expedite the creation of objects representing cellular structures (mitochondria and nuclei) and use these structures to develop analysis pipelines to identify additional smaller structures (nuclear pore regions). Moreover, we took advantage of the arivis Pro Python API to extend the analytical capabilities of arivis Pro to measure the density of nuclear pore regions across the nucleus.

Our findings open new avenues for workflows utilizing a combination of traditional and deep learning algorithms, combined with prior biological knowledge. For instance, our approach of generating objects in the proximity of the NPCs can help identify nuclear pores in 3D regions, where the presence of a nuclear pore may be unclear from the plane-wise 2D analysis only. These 3D objects representing the nuclear pores can be used as ground truth for deep learning training of neural networks. Specifically, because these nuclear pore objects are 3D, varying XYZ planes of these 3D regions can be taken for the ground truth to train the network. We plan to augment these NPC annotations along numerous image axes, thereby multiplying the number of instances constituting the nuclear pore while preserving the structural pattern of this protein complex. This approach would not be possible using the ground truth annotations on individual 2D planes only. Here we demonstrate a successful application of a complex workflow, which once established, can be scaled up for the automatic segmentation and quantitative analysis and profiling in ZEISS arivis Hub.