Skip to main content

可微渲染 SoftRas 实践

SoftRas 是目前主流三角网格可微渲染器之一。

可微渲染通过计算渲染过程的导数,使得从单张图片学习三维结构逐渐成为现实。可微渲染目前被广泛地应用于三维重建,特别是人体重建、人脸重建和三维属性估计等应用中。

安装#

conda 安装 PyTorch 环境:

conda create -n torch python=3.8 -yconda activate torch
conda install pytorch torchvision torchaudio cudatoolkit=11.1 -c pytorch -c nvidia -y
conda activate torchpython - <<-EOFimport platformimport torchprint(f"Python : {platform.python_version()}")print(f"PyTorch: {torch.__version__}")print(f"  CUDA : {torch.version.cuda}")EOF
Python : 3.8.10PyTorch: 1.9.0  CUDA : 11.1

获取代码并安装:

git clone https://github.com/ShichenLiu/SoftRas.gitcd SoftRaspython setup.py install

可设 setup.py 镜像源:

cat <<-EOF > ~/.pydistutils.cfg[easy_install]index_url = http://mirrors.aliyun.com/pypi/simpleEOF

应用#

安装模型查看工具:

snap install ogre-meshviewer# 或snap install meshlab

渲染物体#

渲染测试:

CUDA_VISIBLE_DEVICES=0 python examples/demo_render.py

渲染结果:

对比前后模型:

ogre-meshviewer data/obj/spot/spot_triangulated.obj
ogre-meshviewer data/results/output_render/saved_spot.obj

Mesh 重建#

下载数据集:

bash examples/recon/download_dataset.sh

训练模型:

$ CUDA_VISIBLE_DEVICES=0 python examples/recon/train.py -eid reconLoading dataset: 100%|██████████████████████████| 13/13 [00:35<00:00,  2.74s/it]Iter: [0/250000]    Time 1.189  Loss 0.655  lr 0.000100 sv 0.000100Iter: [100/250000]  Time 0.464  Loss 0.405  lr 0.000100 sv 0.000100...Iter: [250000/250000]   Time 0.450  Loss 0.128  lr 0.000030 sv 0.000030

测试模型:

$ CUDA_VISIBLE_DEVICES=0 python examples/recon/test.py -eid recon \    -d 'data/results/models/recon/checkpoint_0250000.pth.tar'Loading dataset: 100%|██████████████████████████| 13/13 [00:03<00:00,  3.25it/s]Iter: [0/97]    Time 0.419      IoU 0.697=================================Mean IoU: 65.586 for class Airplane
Iter: [0/43]    Time 0.095      IoU 0.587=================================Mean IoU: 49.798 for class Bench
Iter: [0/37]    Time 0.089      IoU 0.621=================================Mean IoU: 68.975 for class Cabinet
Iter: [0/179]   Time 0.088      IoU 0.741Iter: [100/179] Time 0.083      IoU 0.772=================================Mean IoU: 74.224 for class Car
Iter: [0/162]   Time 0.086      IoU 0.565Iter: [100/162] Time 0.085      IoU 0.522=================================Mean IoU: 52.933 for class Chair
Iter: [0/26]    Time 0.094      IoU 0.681=================================Mean IoU: 60.553 for class Display
Iter: [0/55]    Time 0.087      IoU 0.526=================================Mean IoU: 45.751 for class Lamp
Iter: [0/38]    Time 0.086      IoU 0.580=================================Mean IoU: 65.626 for class Loudspeaker
Iter: [0/56]    Time 0.090      IoU 0.783=================================Mean IoU: 68.683 for class Rifle
Iter: [0/76]    Time 0.092      IoU 0.647=================================Mean IoU: 68.111 for class Sofa
Iter: [0/204]   Time 0.090      IoU 0.405Iter: [100/204] Time 0.087      IoU 0.435Iter: [200/204] Time 0.086      IoU 0.567=================================Mean IoU: 46.206 for class Table
Iter: [0/25]    Time 0.097      IoU 0.901=================================Mean IoU: 82.261 for class Telephone
Iter: [0/46]    Time 0.087      IoU 0.503=================================Mean IoU: 61.019 for class Watercraft
=================================Mean IoU: 62.287 for all classes

Mesh 重建:

# 获取 `softras_recon.py` 进 `examples/recon/`#   https://github.com/ikuokuo/start-3d-recon/blob/master/samples/softras_recon.py# 注释 `iou` 直接返回 0,位于 `examples/recon/models.py` `evaluate_iou()`
# 2D 图像重构 3D MeshCUDA_VISIBLE_DEVICES=0 python examples/recon/softras_recon.py \    -s '.' \    -d 'data/results/models/recon/checkpoint_0250000.pth.tar' \    -img 'data/car_64x64.png'
ogre-meshviewer data/car_64x64.obj

重建图像:

重建结果:

或重建 ShapeNet 数据集内图像:

# mesh recon images of ShapeNet datasetCUDA_VISIBLE_DEVICES=0 python examples/recon/softras_recon.py \    -s '.' \    -d 'data/results/models/recon/checkpoint_0250000.pth.tar' \    -imgs 'data/datasets/02958343_test_images.npz'

或使用 SoftRas 训练好的模型:

  • SoftRas trained with silhouettes supervision (62+ IoU): google drive
  • SoftRas trained with shading supervision (64+ IoU, test with --shading-model arg): google drive
  • SoftRas reconstructed meshes with color (random sampled): google drive

更多#