3D Focus Image Stacks

We developed a novel multi-view image noise reduction algorithm for camera array using a data structure called 3D focus image stacks (3DFIS) with which the disparity map (from the target view) can be efficiently constructed, and multi-view denoising can be performed. The 3DFIS is constructed by shifting each of the views with respect to the target view by an amount that is related to the each camera position (s, t) and candidate disparity values d. The mathematical formulation of 3DFIS, denoted as F^d, can be expressed as the following equation:

Each surface point in the 3D space is projected onto each image, forming corresponding points that are separated by (sd, td). In other words, if a pixel (x, y) has the correct disparity value that equals d, then its corresponding points will be shifted to the same position in the 3DFIS F^d. Such a point is called an in-focus pixel. On the contrary, if they are shifted by some other amount, e.g. d’, then these points will be shifted to different positions in F^d’, causing an out-of-focus effect. This process can be illustrated in Fig. 1. A visual representation of the 3DFIS is shown in Fig. 2 by simply averaging each stack with respect to the 3^rd dimension.

Fig. 1 Illustration of 3DFIS construction using a three-view system.

Disparity = 5	Disparity = 6	Disparity = 10	Disparity = 14

Fig. 2 Visualization of 3D focus image stacks.

Disparity Map Estimation

Disparity map estimation is a preliminary but also crucial procedure prior to the actual denoising as it is the essential tool to extract the corresponding 3DFIS for each pixel during denoising process. Using the 3DFIS, we proposed a robust disparity map estimation algorithm with texture-based view selection and patch size variation strategy.

Our disparity estimation is based on local stereo matching algorithm, and a robust matching cost C for each pixel (x, y) and each candidate disparity d is proposed, as shown in the following equation:

where W(x, y) is a window centered at (x, y), N is the total number of pixels in W, and K is the number of views. This matching cost is robust to noise, but vulnerable to occlusion, and also generates artifacts in flat regions.

In response, we proposed a view selection scheme to handle occlusion problem, and a patch size variation scheme to reduce artifacts in flat regions. The key is to find the corresponding regions of occlusion and flat areas. A texture map is estimated from the 3DFIS, which is shown in Fig. 3, and we derived the size of patches L and number of views V from the texture values with pre-defined linear equations.

Fig. 3 (a) noisy image, (b) texture map

We applied the variable patch size L and number of views V back to the matching cost C, and the resulting disparity map shows great improvement over existing algorithms, as shown in Fig. 4.

Fig. 4 Comparison of disparity estimation methods: (a) Taniai et al.; (b) Lee et al.;
(c) Klaus et al.; (d) Miyata et al.; (e) Zhou et al.; (f) proposed.

Multi-view Denoising Algorithm

With the 3DFIS and disparity map, we can denoise the target image by searching for similar patches and performing denoising process on the patch groups. In order to improve the noise robustness of searching, we proposed a 3DFIS-based similarity that takes the entire patch volume instead of single patch into account. Fig. 5 shows the comparison of different searching metrics, with red boxes indicating reference patch and green boxes indicating similar patches.

Fig. 5 Similar patch searching using different metrics: (a) block matching (clean); (b) block matching (noisy); (c) Zhang et al.; (d) proposed.

In the meantime, for each pixel, instead of using the centered window all the way, we proposed a depth-guided adaptive window selection that can intelligently select the appropriate window that causes minimum match error. As indicated by Fig. 6, totally five candidate windows are considered for the reference pixel. Among them, window #2 and #4 are eliminated due to incorrect disparity, and window #1, which has smallest matching error within the remaining candidates, is chosen as the final window for the pixel.

Fig. 6 (a) sample pixel and five windows; (b)-(f) patch vectors across views and RMSE plots for j = 0, 1, 2, 3, 4.

Finally, after the similar patches are grouped, a low-rank minimization scheme is applied to remove the noise. Each pixel is covered by multiple denoised patches, and to determine the value of the pixel in the denoised image, we can take an average of all denoised patches that cover this pixel.

The comparison of our proposed denoising algorithm with other methods is shown in Table 1.

Table 1. Denoising performance (PSNR) compared with other methods (best – bold; second best – underlined).

References

S. Zhou, Z. Lou, Y. H. Hu, and H. Jiang, “Multiple view image denoising using 3D focus image stacks,” Computer Vision and Image Understanding, to appear, 2018.

M. Miyata, K. Kodama, and T. Hamamoto. “Fast multiple-view denoising based on image reconstruction by plane sweeping.” in Proc. IEEE Vis. Commun. Image Process., Dec. 2014, pp. 462-465.

L. Zhang, S. Vaddadi, H. Jin, and S. Nayar, “Multiple view image denoising,” in Proc. IEEE Conf. Comput. Vision Pattern Recognit., Jun. 2009, pp. 1542-1549.

S. Zhou, Y. H. Hu, and H. Jiang, “Patch-based multiple view image denoising with occlusion handling,” in Proc. IEEE Int. Conf. Accoustic, Speech, Signal Process. (ICASSP), 2017, pp. 1782-1786.

T. Taniai, Y. Matsushita, Y. Sato, and T. Naemura, “Continuous stereo matching using local expansion moves,” arXiv preprint arXiv:1603.08328, Mar. 2016.

S. Lee, J. H. Lee, J. Lim, and H. I. Suh, “Robust stereo matching using adaptive random walk with restart algorithm,” Image and Vision Computing, vol. 37, 2015, pp.1-11.

A. Klaus, M. Sormann, and K. Karner, “Segment-based stereo matching using belief propagation and a self-adapting dissimilarity measure,” in Proc. 18th IEEE Int. Conf. Pattern Recognit., vol. 3, 2006, pp. 15-18.

A. Buades, B. Coll, and J. Morel. “A non-local algorithm for image denoising,” in Proc. IEEE Conf. Comput. Vision Pattern Recognit., Jun. 2005, pp. 60-65.

K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian. “Image denoising by sparse 3-D transform-domain collaborative filtering,” IEEE Trans. Image Process., vol. 16, no. 8, 2007, pp. 2080-2095.

S. Gu, L. Zhang, W. Zuo, and X. Feng, “Weighted nuclear norm minimization with application to image denoising,” in Proc. IEEE Conf. Comput. Vision Pattern Recognit., 2014, pp. 2862-2869.

M. Maggioni, G. Boracchi, A. Foi, and K. Egiazarian, “Video denoising, deblocking, and enhancement through separable 4-D nonlocal spatiotemporal transforms,” IEEE Trans. Image Process., vol. 21, no. 9, 2012, pp. 3952-3966.