We added textures to the meshes in the real-time demo. The textures are built from the views of 60 cameras. This requires to compress 60 4Mpix images to less than 10 MB so that the NIC and GPU of the rendering machine do not get overloaded.
Therefore, the textures are clipped using the silhouette mask that we have available anyway, and compressed in a hardware format supported by the GPU (DXT1). Rate control is applied to make sure that the total amount of data sent to the rendering machine does not exceed a maximum budget. The image can be degraded to accommodate this budget.
A GLSL fragment shader has access to the 60 images + a depth image to decide about the color of the resulting pixel. Obviously we are happy to have a fast GPU to do this…
It is a bit surprising that the whole process is not noticeably slower than the version without textures.
The video below shows a test object (blue ball with giraffe on top that has too thin legs to be reconstructed properly with the bad calibration we used). On top is the window that gives most important stats on the process. Then the developer who moves around and expresses his joy after he worked around a nvidia driver bug.
For future work: there is still a bug with triangles that disappear. The pink patches highlight geometry for which no texture could be found (despite the 60 cams).
Authors: Matthijs, Mickael (for the awesome green reflection removal)
