logo

SCIENTIA SINICA Informationis, Volume 48, Issue 3: 261-273(2018) https://doi.org/10.1360/N112017-00243

A luminance compensation method for fisheye video panorama stitching

More info
  • ReceivedJan 7, 2018
  • AcceptedJan 31, 2018
  • PublishedMar 16, 2018

Abstract

The fisheye camera has becoming the most popular panoramic video capturing device. In fact, the stitching for fisheye images has drawn more and more attention. To solve the artifacts resulting from the exposure difference of several cameras, this paper proposes a luminance compensation method based on an uneven sampling histogram. According to the transition of the spatial sampling rate with the latitude's increasing in panorama, the proposed method samples the overlapped region of two adjacent images to calculate the histograms, and matches the histograms one by one to balance the luminance. Moreover, in consideration of the influence of luminance distribution to panorama video quality, we present a method for adaptive reference image selection. The selected reference image based on the proposed method can improve the overall quality of the panorama effectively. Due to the consideration of the distortion characteristics of fisheye images, the proposed method does not require an accurate alignment, and it is also well-adapted to luminance variations. Experimental results show that the proposed method can reduce the time complexity for panorama video stitching significantly without obvious quality degeneration, and it is suitable for fisheye lens-based real-time panoramic video stitching.


Funded by

国家高技术研究发展计划(2015AA015903)


References

[1] Deng X M, Wu F C, Wu Y H, et al. Automatic spherical panorama generation with two fisheye images. In: Proceedings of the 7th World Congress on Intelligent Control and Automation, Chongqing, 2008. 5955--5959. Google Scholar

[2] Ho T, Budagavi M. Dual-fisheye lens stitching for 360-degree imaging. In: Proceedings of International Conference on Acoustics, Speech and Signal Processing, New Orleans, 2017. Google Scholar

[3] Liao W S, Hsieh T J, Liang W Y, et al. Real-time spherical panorama image stitching using OpenCL. In: Proceedings of International Conference on Computer Graphics and Virtual Reality, Las Vegas, 2011. 113--119. Google Scholar

[4] Xu Y, Zhou Q, Gong L. High-Speed Simultaneous Image Distortion Correction Transformations for a Multicamera Cylindrical Panorama Real-time Video System Using FPGA. IEEE Trans Circuits Syst Video Technol, 2014, 24: 1061-1069 CrossRef Google Scholar

[5] Reinhard E, Adhikhmin M, Gooch B, et al. Color transfer between images. IEEE Comput Graph Appl, 2001, 21: 34--41. Google Scholar

[6] Tian G Y, Gledhill D, Taylor D, et al. Colour correction for panoramic imaging. In: Proceedings of the 6th International Conference on Information Visualisation, London, 2002. 483--488. Google Scholar

[7] Brown M, Lowe D G. Automatic Panoramic Image Stitching using Invariant Features. Int J Comput Vision, 2007, 74: 59-73 CrossRef Google Scholar

[8] Kim Y N, Sim D G. Vignetting and illumination compensation for omni-directional image generation on spherical coordinate. In: Proceedings of the 16th International Conference on Artificial Reality and Telexistence--Workshops, Hangzhou, 2006. 413--418. Google Scholar

[9] Fecker U, Barkowsky M, Kaup A. Histogram-Based Prefiltering for Luminance and Chrominance Compensation of Multiview Video. IEEE Trans Circuits Syst Video Technol, 2008, 18: 1258-1267 CrossRef Google Scholar

[10] Yamamoto K, Oi R. Color correction for multi-view video using energy minimization of view networks. Int J Autom Comput, 2008, 5: 234-245 CrossRef Google Scholar

[11] Basu A, Licardie S. Alternative models for fish-eye lenses. Pattern Recognition Lett, 1995, 16: 433-441 CrossRef Google Scholar

[12] Salomon D. Transformations and Projections in Computer Graphics. Berlin: Springer, 2007. Google Scholar

[13] Ye Y, Alshina E, Boyce J. Algorithm descriptions of projection format conversion and video quality metrics in 360Lib version 54. In: Joint Video Exploration Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, Document: JVET-H1004, Macau, 2017. Google Scholar

[14] Szeliski R, Shum H Y. Creating full view panoramic image mosaics and environment maps. In: Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques. New York: ACM, 1997. 251--258. Google Scholar

[15] Sun Y, Lu A, Yu L. Ahg8: WS-PSNR for 360 video objective quality evaluation. In: Joint Video Exploration Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, Document: JVET-D0040, Chengdu, 2016. Google Scholar

[16] Chen J L, Liu J L, Ye J H, et al. Luminance and chrominance correction for multi-view video using overlapped local histogram matching. Chinese J Image Graph, 2007, 12: 1992--1999. Google Scholar

[17] Ruderman D L, Cronin T W, Chiao C C. Statistics of cone responses to natural images: implications for visual coding. J Opt Soc Am A, 1998, 15: 2036-2045 CrossRef ADS Google Scholar

  • Figure 1

    (Color online) Fisheye-based video caption and stitching system. (a) Frame for 6 GoPro cameras; (b) fisheye image; (c) equirectangular projection

  • Figure 2

    (Color online) Overlap region before and after equirectangular projection. (a) Origin image; (b) equirectangular projection result and the overlap region; (c) overlap region in the origin image; (d) histogram of the two regions

  • Figure 3

    Preprocessing of the fisheye image

  • Figure 4

    (Color online) (a) Weight distribution map of WS-PSNR; (b) density distribution of an equirectangular projection panorama image

  • Figure 6

    (Color online) Real-time panorama video stitching system

  • Figure 7

    (Color online) The luminance compensation results based on histogram statistics with (a) original images,protectłinebreak (b) corrected images

  • Figure 8

    Histograms of three input images. From left to right are img0, img1, img5. (a) Histogram of overlap region with previous image; (b) histogram of overlap region with next image

  • Figure 9

    (Color online) Different results of different reference images. (a) Selecting img5 (score=0.8527) as reference image; (b) Selecting img0 (score=1.5063) as reference image

  • Figure 10

    (Color online) Experiment results of referenced method and proposal method. (a) Ref. [5], YUV color space;protect łinebreak (b) proposal method

  • Table 1   Effectiveness and efficiency of luminance compensation with different min/max sampling intervals
    Index (min, max) Time (ms) PSNR WS-PSNR Index (min, max) Time (ms) PSNR WS-PSNR
    1 (1, 1024) 9.08 31.4385 31.5229 9 (2, 256) 10.05 34.2231 34.1798
    2 (1, 512) 9.93 31.9630 32.0129 10 (4, 256) 9.11 34.4783 34.4546
    3 (1, 256) 11.57 32.8647 32.8514 11 (8, 256) 8.24 35.7685 35.7989
    4 (1, 128) 13.30 33.9658 33.8869 12 (16, 256) 7.51 38.0210 37.9278
    5 (1, 64) 14.99 34.5430 34.4906 13 (32, 256) 6.97 37.1566 37.1456
    6 (1, 32) 17.54 35.7429 35.6996 14 (64, 256) 6.58 34.8895 35.1714
    7 (1, 16) 20.52 37.0278 36.8767 15 (128, 256) 6.38 34.4281 34.0717
    8 (1, 8) 24.31 39.6580 39.5388
  • Table 2   Experiment results for efficiency (100 frames)
    Method Time (ms)
    [5], YUV 2173
    Proposal method (without adaptive reference image) 1115
    Proposal method (with adaptive reference image) 1143

Copyright 2020 Science China Press Co., Ltd. 《中国科学》杂志社有限责任公司 版权所有

京ICP备18024590号-1