Toronto AI Lab
This work considers gradient-based mesh optimization, where we iteratively optimize for a 3D surface mesh by representing it as the isosurface of a scalar field, an increasingly common paradigm in applications including photogrammetry, generative modeling, and inverse physics. Existing implementations adapt classic isosurface extraction algorithms like Marching Cubes or Dual Contouring; these techniques were designed to extract meshes from fixed, known fields, and in the optimization setting they lack the degrees of freedom to represent high-quality feature-preserving meshes, or suffer from numerical instabilities. We introduce FlexiCubes, an isosurface representation specifically designed for optimizing an unknown mesh with respect to geometric, visual, or even physical objectives. Our main insight is to introduce additional carefully-chosen parameters into the representation, which allow local flexible adjustments to the extracted mesh geometry and connectivity. These parameters are updated along with the underlying scalar field via automatic differentiation when optimizing for a downstream task. We base our extraction scheme on Dual Marching Cubes for improved topological properties, and present extensions to optionally generate tetrahedral and hierarchically-adaptive meshes. Extensive experiments validate FlexiCubes on both synthetic benchmarks and real-world applications, showing that it offers significant improvements in mesh quality and geometric fidelity.
FlexiCubes provides two significant advantages that allow for easy, efficient, and high-quality mesh optimization for various applications:
Grad.
Differentiation with respect to the mesh is well-defined, and gradient-based optimization converges effectively in practice.
Flexible. Mesh vertices can be individually and locally adjusted to fit surface features and find a high-quality mesh with a small number of elements.
To illustrate these points, we present a video comparison of different isosurfacing methods for mesh optimization. Starting from a sphere initialization, we optimize the shapes
towards the ground truth mesh (bottom-right) using a set of isosurfacing methods. DC and NDC lack gradient differentiation, leading to divergence during optimization. MC and MT lack the flexibility to adjust vertices locally, thus failing to capture sharp features or producing many sliver triangles.
Explore additional qualitative results below. Hover over each image to view the wireframe visuals.
Note that the grid resolution is \(64^3\) for MC and FlexiCubes. DMTet uses \(80^3\) tetrahedral grid to match the triangle count in output
meshes.
The differentiable isosurfacing technique DMTet is at the core of the recent work, nvdiffrec, which jointly optimizes shape, materials, and lighting from images. By simply replacing DMTet with FlexiCubes in the topology optimization step, leaving the remainder of the pipeline unmodified, we observe improved geometry reconstructions at equal triangle count.
The recent 3D generative model, GET3D, differentiably render a 3D representation into 2D images, and leverages a generative adversarial framework to synthesize 3D content using only 2D image supervision. We show FlexiCubes can serve as a plug-and-play differentiable mesh extraction module in a 3D generative model, and produce significantly improved mesh quality.
FlexiCubes can differentiably extract tetrahedral meshes. The uniform tesellation allows us to combine it with a differentiable physics simulation framework (gradSim) and a differentiable rendering pipeline (nvdiffrast) to jointly recover 3D shapes and physical parameters from multi-view videos. Here, we show a preliminary result - given a video sequence of a deforming object, we can recover a tetrahedral mesh of the rest pose, as well as material parameters that reproduce the motion under simulation.
Rather than fitting a single mesh in a reference pose, FlexiCubes also allows us to differentiably skin and deform the mesh via off-the-shelf skinning tools, and simultaneously optimize with respect to the entire animated sequence. End-to-end optimization over the entire animation helps re-distribute triangle density to avoid mesh stretching.
Our FlexiCubes representation is flexible enough that objectives and regularizers which depend on the extracted mesh itself can be directly evaluated with automatic differentiation and incorporated into gradient-based optimization. Here we apply a developability term on extracted mesh to encourages fabricability from panels. FlexiCubes has the freedom to fit the shape and satisfy the regularizer, whereas Marching Cubes does not.
@article{shen2023flexicubes,
author = {Shen, Tianchang and Munkberg, Jacob and Hasselgren, Jon and Yin, Kangxue and Wang, Zian
and Chen, Wenzheng and Gojcic, Zan and Fidler, Sanja and Sharp, Nicholas and Gao, Jun},
title = {Flexible Isosurface Extraction for Gradient-Based Mesh Optimization},
year = {2023},
issue_date = {August 2023},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
volume = {42},
number = {4},
issn = {0730-0301},
url = {https://doi.org/10.1145/3592430},
doi = {10.1145/3592430},
journal = {ACM Trans. Graph.},
month = {jul},
articleno = {37},
numpages = {16}
}
The authors are grateful to Aaron Lefohn and David I.W. Levin for their support and discussions during this research, as well as the anonymous reviewers for their valuable comments and feedback.
|
|
