0%

3D Reconstruction with Meshroom

1. Meshroom

Photogrammetry is the science of making measurements from photographs. It infers the geometry of a scene from a set of unordered photographies or videos. Photography is the projection of a 3D scene onto a 2D plane, losing depth information. The goal of photogrammetry is to reverse this process.

The dense modeling of the scene is the result yielded by chaining two computer vision-based pipelines:
“Structure-from-Motion” (SfM) and “Multi View Stereo” (MVS).

2. Pipeline

  1. Feature extraction
  2. Goal: extract feature points
  3. Method: SIFT
  4. Image matching
  5. Goal: find images looking to the same area, then we can reduce computation
  6. Method: Vocabulary tree approach
  7. Feature matching
  8. Goal: match feature points between candidate image pairs
  9. Method: calculate distance and use RANSAC
  10. Structure from Motion (SFM)
  11. Goal: compute the pose of each images
  12. Methods:
  13. Fuse all matched feature point pairs
  14. Choose the best initial image pair
  15. Compute the fundamental matrix
  16. Using PnP to calculate the pose
  17. Filter the results using Bundle Adjustment
  18. Depth Maps Estimation
  19. Goal: get the depth value of each pixel
  20. Method: Semi-Global Matching
  21. Meshing
  22. Goal: create geometric surface representation of the scene
  23. Method:
  24. Fuse all depth maps into global octree
  25. Perform 3D Delaunay tetrahedralization (四面体化), then voting
  26. Graph cut Max-Flow, optimally cut the volume
  27. Texturing
  28. Goal: texture the generated mesh
  29. Method:
  30. Compute UV maps
  31. For each triangle, use the color of each candidate to estimate

3. Data Acquisition

Goal: Clear images from different angles

Camera setting

  • Fixed low ISO (low noise)
  • Fixed high shutter speed (less then 1/100s, no blur)
  • Fixed long focal length (everything is in focus)
  • Fixed white balance (stable image color for feature matching)
  • Enough lighting (so that we can keep low iso and high shutter speed)

Move:

  • Camera fixed, object move

  • will need a spin base

  • keep the background pure color

  • Camera move, object fixed

  • Shot around the object (360 degree)

  • In each spot, shot from 3 different heights

Illustration:

Whiteboard_20211214T151703

4. Results

2021-12-14-16-39-18

The final results of Meshroom is pretty unstable

  • In many cases, even high quality images dataset can not generate satisfied results
  • However, when the algorithm works, it can generate detailed mesh and texture.

5. Discussion

Why performs bad?

I think the main reason is the object itself. For the leaf object, it has a few features that make the algorithm doesn’t work

  • The leaf is so thin. It’s hard to generate a plane using point cloud
  • The leaf is semi-transparent, it will look differently from different angle
  • The leaf skin is so smooth that would generate reflection

How to solve the problem? (Possible solution)

  • Using laser to generate point cloud (to capture textureless feature)
  • Create great lighting system, try to avoid reflection and semi-transparent situation