Learning Deep Generative Models
for 3D Shape Structures
徐 凯
国防科技大学计算机学院
GAMES Web Seminar 2017.6.29
GAMES Webinar – June 29, 2017
Deep Generative Models recently …
2
[Radford et al. 2015]
[Isola et al. 2017]
GAMES Webinar – June 29, 2017
Even for 3D shapes …
3
3D-GAN [Wu et al. NIPS 2016]
GAMES Webinar – June 29, 2017
Generating 3D shapes
4
3D-GAN [Wu et al. NIPS 2016]
GRASS [Li et al. SIGGRAPH 2017]
GAMES Webinar – June 29, 2017
Today
What is special in generating 3D shapes?
What is a good generative model for 3D shapes?
What do we learn from building such a model?
5
Sponsored by
SA2015.SIGGRAPH.ORG
Generative Models
both for analysis and synthesis
GAMES Webinar – June 29, 2017
Generative models
For analysis:
To form a conditional probability density function
Example: Maximum a Posterior (MAP)
7
Observation
Parameters
Likelihood
Analysis-by-Synthesis
GAMES Webinar – June 29, 2017
Generative models
For synthesis
To model data distribution directly
Example: Variational Bayesian method
8
Sample from the likelihood
Likelihood
Latent variable
GAMES Webinar – June 29, 2017
Generate new for ?
Curiosity about unsupervised learning?
Better understanding / interpretation?
Applications?
9
Generate natural language
[Rajeswar et al. 2017]
Generate natural image
[Zhang et al. 2016]
GAMES Webinar – June 29, 2017
Generate new for 3D Content Creation
In Computer Graphics, 3D Content Creation
has been a hard-core challenge for long.
10
GAMES Webinar – June 29, 2017
Generate new for creativity support !
3D content creation calls for creativity
Automatic, diverse & creative
11
GAMES Webinar – June 29, 2017
Generate new for creativity support !
3D content creation calls for creativity
Automatic, diverse & creative
12
Sponsored by
SA2015.SIGGRAPH.ORG
3D Content Creation
GAMES Webinar – June 29, 2017
3D Content Creation
14
… finding powerful means to create 3D shapes is the key to
make graphics as ubiquitous as we had wanted it to be.
Jim Kajiya
Recipient of Steven Anson Coons Award
GAMES Webinar – June 29, 2017
Generating 3D shape variations
Two basic approaches
15
Deforming
Assembling
Preserving local details
Easier
GAMES Webinar – June 29, 2017
Automatic 3D shape synthesis
Two basic approaches
16
Deforming
Assembling
Preserving global structures
Much harder
GAMES Webinar – June 29, 2017
Our attempts over the past years
Style-content separation
17
Style
Content
[Xu et al. SIGGRAPH Asia 2010]
Part correspondence
Structure preservation:
GAMES Webinar – June 29, 2017
Our attempts over the past years
Structure-preserving variation driven by photos
18
[Xu et al. SIGGRAPH 2011]
+
Photograph Shape database
GAMES Webinar – June 29, 2017
Our attempts over the past years
Structure-preserving variation driven by photos
19
Photograph ControllersShape
[Xu et al. SIGGRAPH 2011]
GAMES Webinar – June 29, 2017
Our attempts over the past years
Structure-preserving variation driven by photos
20
[Xu et al. SIGGRAPH 2011]
GAMES Webinar – June 29, 2017
Our attempts over the past years
Structure-preserving variation driven by photos
21
Structure
optimization
Hard-coded rules
Structure preservation:
[Xu et al. SIGGRAPH 2011]
GAMES Webinar – June 29, 2017
Our attempts over the past years
Shape set evolution
22
Input set
The 15th generation
The 10th generation
The 5th generation
[Xu et al. SIGGRAPH 2012]
Part correspondence
Structure preservation:
GAMES Webinar – June 29, 2017
Our attempts over the past years
Structure blending
23
Part correspondence + hard-code rules
Structure preservation:
[Alhashim et al. SIGGRAPH 2014]
GAMES Webinar – June 29, 2017
Our new goal
Generating 3D shape variations
24
Structure preserving
No hard-coded rule
Completely
data-driven
Completely
unsupervised
Correspondence-free
GAMES Webinar – June 29, 2017
Our new goal
More generally, modelling the structure space
of 3D shapes ?
25
Geometry space of 3D shapes [Kilian et al. 2007]
GAMES Webinar – June 29, 2017
Our new goal
26
How about 3D shape structures?
More generally, modelling the structure space
of 3D shapes ?
✓ Structure preserving
✓ Completely data-driven
✓ Correspondence-free
Sponsored by
SA2015.SIGGRAPH.ORG
Modeling 3D Structure Space
GAMES Webinar – June 29, 2017
Our approach
Basic idea:
Learn a distribution that approximates the data
distribution of true 3D structures
Marginalize over a latent variable
28
Parameters
Likelihood
maximize
GAMES Webinar – June 29, 2017
Variational Bayesian formulation
29
maximize
maximize
should reconstruct
, given that it was
drawn from
Assuming ’s follow a
normal distribution
GAMES Webinar – June 29, 2017
Variational auto-encoder
30
Encoder
Decoder
maximize
Reconstruction loss KL divergence loss
GAMES Webinar – June 29, 2017
Variational auto-encoder
31
Encoder
Decoder
31
Enc
Enc Enc
GAMES Webinar – June 29, 2017
Variational auto-encoder
Encoder
Decoder
Encoder
Decoder
GAMES Webinar – June 29, 2017
Remaining issues
33
What would be a
good representation
for input structures?
How to encode /
decode 3D shape
structures?
How to deal with
sampling far
away from ?
Encoder
Decoder
How to measure the
quality of generated
structures?
GAMES Webinar – June 29, 2017
Remaining issues
34
What would be a
good representation
for input structures?
How to encode /
decode 3D shape
structures?
How to deal with
sampling far
away from ?
Encoder
Decoder
How to measure the
quality of generated
structures?
GAMES Webinar – June 29, 2017
3D shape representation for DL
Typically two methods
35
Volumetric representation
Good for visual classification & recognition
[Wu et al. 2015]
[Wu et al. 2016]
Multi-view rendered images
[Su et al. 2015]
[Su et al. 2015]
GAMES Webinar – June 29, 2017
3D shape representation for DL
Typically two methods
36
Volumetric representation Multi-view rendered images
Limitation: Oblivious to structure!
[Su et al. 2015]
[Wu et al. 2015]
[Wu et al. 2016]
[Su et al. 2015]
GAMES Webinar – June 29, 2017
Structure-oblivious 3D shape generation
37
[Wu et al. 2016] by MIT [Girdhar et al. 2016] by CMU
GAMES Webinar – June 29, 2017
Structure-aware representation
38
Input shape Box structure
reflection
rotation
connection
OBB extraction
GAMES Webinar – June 29, 2017
Structure-aware representation
Problems with box structure representation
Number of boxes varies from shape to shape
Not neural networks friendly
39
Volume
representation
Box representation
GAMES Webinar – June 29, 2017
Structure-aware representation
Problems with box structure representation
Number of boxes varies from shape to shape
Encode the whole structure into a fixed length code?
40
A fixed-length code
sampled from some
low-dimensional space
GAMES Webinar – June 29, 2017
Structure encoding / decoding
41
Refl. sym.
Refl. sym.
Adjacency
encoding
Symmetry
encoding
Recursive Neural Networks
[Socher et al. 2011]
GAMES Webinar – June 29, 2017
Recursive structure encoding
42
Shared weights across
different levels
A fixed-length root code
encodes the whole
structure
Different networks for
adjacency and symmetry
How to determine the
grouping order?
GAMES Webinar – June 29, 2017
Recursive auto-encoder
Self-reconstruction
43
RvNN decoder
RvNN encoder
-D
root code
GAMES Webinar – June 29, 2017
Recursive auto-encoder
44
Perceptually meaningful grouping leads to low
self-reconstruction error!
GAMES Webinar – June 29, 2017
Remaining issues
45
What would be a
good representation
for input structures?
How to encode /
decode 3D shape
structures?
How to deal with
sampling far
away from ?
Encoder
Decoder
How to measure the
quality of generated
structures?
GAMES Webinar – June 29, 2017
Sampling far away from
46
Encoder
Decoder
GAMES Webinar – June 29, 2017
Sampling far away from
47
Encoder
Decoder
?
GAMES Webinar – June 29, 2017
Adversarial training
VAE-GAN (Generative Adversarial Network) architecture
48
VAE
GAN
Encoder
Decoder or
Generator
Discriminator
Real box
structures
GAMES Webinar – June 29, 2017
Adversarial training
Benefit of VAE-GAN
49
VAE
GAMES Webinar – June 29, 2017
Synthesize geometry within a box
50
32D
part code
32x32x32 output
part volume
GAMES Webinar – June 29, 2017
Synthesize geometry within a box
51
32D
part code
32x32x32 output
part volume
GAMES Webinar – June 29, 2017
Synthesize geometry within a box
52
training
parts
part code
32 FC
32x32x32 input
part volume
32x32x32 output
part volume
96
256
384
256 256
384
256
96
6
32 FC
512 FC
512 FC
?
GAMES Webinar – June 29, 2017
Synthesize geometry within a box
53
Concatenated part code
GAMES Webinar – June 29, 2017
Synthesis results
54
GAMES Webinar – June 29, 2017
Interpolation results
55
GAMES Webinar – June 29, 2017
Today
What is special in generating 3D shapes?
What is a good generative model for 3D shapes?
What do we learn from building such a model?
56
GAMES Webinar – June 29, 2017
Discussion
What is a good representation for 3D shapes?
Depends on task
Visual recognition and classification?
Fusing 2D, 2.5D and 3D data?
Structure-aware tasks?
It may not be wise to learn everything from raw
data, despite the feature learning power of DNN
One never learns to understand natural language from
images of characters, but instead works with symbols.
For 3D shapes, graphics people should think
independently
57
Multi-view rendering
Volumetric representation
Part representation
GAMES Webinar – June 29, 2017
Discussion
What does our model learn?
A hierarchical organization of
part structure (minimizing self-
reconstruction error)
A good way to generate 3D
structure
Bottom-up generation
Creating parts and
Hierarchically grouping parts
This is exactly how a human
modeler creates a 3D model !
58
Refl. sym.
Refl. sym.
Refl. sym.
Refl. sym.
GAMES Webinar – June 29, 2017
Discussion
Generally, how to generate things?
Coarse-to-fine:
First generate coarse structure
Then generate fine details
May employ different representations and models
A guiding rule for designing a generative
model
59
GAMES Webinar – June 29, 2017
Discussion
The trade-off between plausibility and diversity
Plausible requires keeping close to input exemplars
Is there a definition on plausibility really?
Similarity against exemplars?
Diverse requires going further away
60
?
GAMES Webinar – June 29, 2017
Discussion
Is there a low-dim manifold of 3D shape
structures?
Not every in-between structure is functionally valid
However, they may reveal
The evolution of design in human brains
The exploration of design space in human brains
We cannot say we model the manifold of 3D
shape structures.
Our ongoing research …
61
GAMES Webinar – June 29, 2017
Acknowledgement
Reviewers
Jun Li is a visiting PhD student of University of Bonn,
supported by the CSC
Co-authors
Siddhartha Chaudhuri, Ersin Yumer, Hao Zhang and Leonidas Guibas
Help on data collection
Yifei Shi, Min Liu, Chengjie Niu and Yizhi Wang
Research grants
NSFC, NSERC, NSF
Google Focused Research Award, Gifts from the Adobe, Qualcomm
and Vicarious corporations.
62
Thank you!
Welcome to try - code & data
www.kevinkaixu.net
