ML擅长inverse problem

NeRF

image-20230911001950542
  • MLP的作用：用神经网络函数来拟合一个体素的“函数”，从而实现给出5D坐标时可以查询到颜色和Density值，也就为随后的体素渲染提供了输入

神经网络倾向于学习低频函数

  • 加入Positional encoding帮助维持高频信息的恢复：傅里叶变换？使输入掺入高频函数
  • Hierarchical sampling 加速

Instant-NGP

Read list:

【CSDN】Instant-NGP-环境搭建和概念解读

To be read:

Instant-ngp重要的加速手段：

  1. cuda加速
  2. 多分辨率Hash位置编码(跟NeRF的5Dinput相比，把输入特征化)
  3. 射线推进：指数步进、空白跳过、样本压缩

  • 空间哈希函数

NeuS

Read list:

To be read:

基于SDF

NeRF是专注于生成新视点图像而不是表面重建所以有明显噪声

Colmap

Differentiable rendering

Physically-Based Differentiable Rendering Tutorial at SIGGRAPH 2020

GAMES-101

Bli-Pong 着色？

着色？

BRDF

image-20230917171619422

The rendering equation

image-20230917171646433

  • What‘s pdf ?

Utah teapot

Cornell box

  • color bleading

为什么Path Tracing 可以设置为1条光线？

递归终止条件？设置等期望值的Russian Roulette

SPP(Samples per Pixel)

纯光源贡献无需设置概率；非光源贡献(反射光)才需要

材质==BRDF

微表面模型：远处看是材质，近处看是几何

利用纹理建模真实材质

Concepts

SfM：

SfM is short for Structure from Motion, which is a technique for reconstructing 3D models of scenes or objects from a set of 2D images taken from different viewpoints1. SfM is based on the principle of triangulation, which means finding the position of a point in 3D space by measuring its distance from two or more known points2. SfM can be used for various applications, such as 3D scanning, virtual reality, archaeology, and photogrammetry3.

To understand SfM, you need to learn some basic concepts and steps involved in the process. Here is a brief overview of the main steps of SfM4:

  • Image acquisition: This is the first step, where you capture a set of images of the scene or object you want to reconstruct. You need to make sure that the images have enough overlap and coverage, and that they are not blurry or distorted. You can use any camera or device that can take pictures, such as a smartphone, a DSLR, or a drone.
  • Feature detection and matching: This is the second step, where you find distinctive points or features in each image, and match them across different images. These features are usually corners, edges, or blobs that can be easily detected and described by algorithms. The matching process involves finding the best correspondences between features in different images based on their appearance and geometry. This step produces a set of feature matches that indicate which parts of the images belong to the same scene point.
  • Camera pose estimation: This is the third step, where you estimate the position and orientation of each camera relative to the scene. This is also known as camera calibration or camera registration. You can use various methods to estimate the camera pose, such as solving a system of equations, using optimization techniques, or applying geometric constraints. This step produces a set of camera poses that define the viewpoints from which the images were taken.
  • Sparse point cloud reconstruction: This is the fourth step, where you triangulate the feature matches to obtain a sparse set of 3D points that represent the scene structure. You can use various methods to triangulate the feature matches, such as linear algebra, projective geometry, or bundle adjustment. This step produces a sparse point cloud that gives an initial approximation of the scene shape.
  • Dense point cloud reconstruction: This is the fifth step, where you refine and densify the sparse point cloud by adding more 3D points that are not detected by features. You can use various methods to densify the point cloud, such as stereo matching, multi-view stereo, or depth estimation. This step produces a dense point cloud that gives a more detailed representation of the scene shape.
  • Mesh generation and texturing: This is the final step, where you convert the dense point cloud into a mesh or a surface that connects the 3D points with triangles or polygons. You can use various methods to generate the mesh, such as Delaunay triangulation, Poisson surface reconstruction, or marching cubes. You can also apply textures or colors to the mesh by projecting the images onto it. This step produces a textured mesh that gives a realistic visualization of the scene.

If you want to learn more about SfM and how to use it in practice, you can check out some online tutorials and resources that explain the theory and implementation of SfM in more depth. For example:

I hope this helps you understand SfM better. If you have any questions or feedback, feel free to ask me. 😊

面试之前的准备

仔细过一遍GAMES101，尤其联系nerf、instant-ngp、neus等工作

光场相机？

光线追踪？

思考：

  • instant-ngp使用multi-resolution hash encoding的优势在哪里？与对体素直接使用该trick有何不同？联想Plenoxels，读完后考虑NN真的是必要的吗？
  • nerf的优势，劣势?nerf的主要贡献在哪里？
  • instant-ngp是如何跳过空区域的
  • Open-World项目的method设计思路？