Toronto AI Lab
NVIDIA Research
This work explores expanding the capabilities of large language models (LLMs) pretrained on text to generate 3D meshes within a unified model. This offers key advantages of (1) leveraging spatial knowledge already embedded in LLMs, derived from textual sources like 3D tutorials, and (2) enabling conversational 3D generation and mesh understanding. A primary challenge is effectively tokenizing 3D mesh data into discrete tokens that LLMs can process seamlessly. To address this, we introduce LLaMA-Mesh, a novel approach that represents the vertex coordinates and face definitions of 3D meshes as plain text, allowing direct integration with LLMs without expanding the vocabulary. We construct a supervised fine-tuning (SFT) dataset enabling pretrained LLMs to (1) generate 3D meshes from text prompts, (2) produce interleaved text and 3D mesh outputs as required, and (3) understand and interpret 3D meshes. Our work is the first to demonstrate that LLMs can be fine-tuned to acquire complex spatial knowledge for 3D mesh generation in a text-based format, effectively unifying the 3D and text modalities. LLaMA-Mesh achieves mesh generation quality on par with models trained from scratch while maintaining strong text generation performance.
Overview of our method. LLaMA-Mesh unifies text and 3D mesh in a uniform format by representing the numerical values of
vertex coordinates and face definitions of a 3D mesh as plain text. Our model is trained using text and 3D interleaved data in an end-to-end
manner. Therefore, our model can generate both text and 3D meshes in a unified model.
The following meshes are generated with text prompt "Create a 3D obj file using the following description: {caption}".
3D model of a workbench
Low poly chess piece
Polygonal windmill
3D rendering of a model chair
Model of a small barrel
3D rendering of a tower
Low poly model of a hammer
TV bench
TV
bunk
chair
fence
horse
knife
soda
toy car
tree
The following videos show the animation of the mesh generation process. We first generate all the vertices and all the faces sequentially.
Illustration of our 3D representation approach. We represents the vertex coordinates and face definitions of 3D meshes as plain text, allowing direct integration with LLMs without expanding the vocabulary. Left: A snippet of an OBJ file represented as plain text, containing vertex (v) and face (f) definitions. Right: The 3D object rendered from the OBJ file.
| Mesh Type | # Tokens | # Vertices | # Faces |
|---|---|---|---|
| With Quantization | 1280 | 64 | 96 |
Note that 3D mesh coordinates are typically stored as floating-point numbers which leads to long token sequences.
Above: To address this, we quantize the vertex coordinates into a fixed number (64) of bins.
Below: The original OBJ file represents vertex coordinates in decimal values, splitting a single coordinate into several language tokens.
| Mesh Type | # Tokens | # Vertices | # Faces |
|---|---|---|---|
| Without Quantization | 4607 | 64 | 96 |
@misc{wang2024llamameshunifying3dmesh,
title={LLaMA-Mesh: Unifying 3D Mesh Generation with Language Models},
author={Zhengyi Wang and Jonathan Lorraine and Yikai Wang and Hang Su and Jun Zhu and Sanja Fidler and Xiaohui Zeng},
year={2024},
eprint={2411.09595},
archivePrefix={arXiv},
primaryClass={cs.LG},
url={https://arxiv.org/abs/2411.09595},
}