SCIENCE CHINA Information Sciences, Volume 63 , Issue 10 : 202301(2020) https://doi.org/10.1007/s11432-019-2734-y

Measuring quality of experience for 360-degree videos in virtual reality

More info
  • ReceivedJul 18, 2019
  • AcceptedNov 15, 2019
  • PublishedAug 19, 2020


In recent years, we witness dramatic growing attention in immersive media technologies like 360-degree videos and virtual reality (VR). However, measuring the quality-of-experience (QoE) for 360-degree VR videos is not a trivial task. Streaming such videos to head mounted displays (HMDs) is extremely bandwidth-demanding when compared to traditional 2D videos. In HTTP adaptive streaming, QoE tends to deteriorate significantly during fluctuating network conditions, which results in various bitrate changes and causes multiple stalling events during playback. Thus, understanding how the human visual system perceives 360-degree video with the effect of stalling and different bitrate levels becomes inevitable. In this paper, we investigate the impact of stalling on users QoE under different bitrate levels and the interaction between stalling event and bitrate level for 360-degree videos in VR. To aim this, we first build a 360-degree videos database by encoding videos in three different bitrate levels (1, 5, and 15 Mbps) with 4K resolutions ($3840\times1920$ pixels). We then simulate various stalling events in the videos and conduct a subjective experiment in a virtual reality environment to investigate the human responses. Finally, we use a Bayesian method to estimate and predict the QoE while measuring the quality drop owing to various stalling events and bitrate changes. Proposed solution and prediction results show a strong dependency between playback stalling and bitrate of 360-degree video in VR. Stalling always impacts the QoE of 360-degree videos, but the strength of this negative impact depends on the video bitrate level. The adverse effect of stalling events is more profound when bitrate level approaches to the high and low end, which is in close agreement with subjective opinion.


[1] Le Callet P, Möller S, Perkis A. Qualinet white paper on definitions of quality of experience (2012). European Network on Quality of Experience in Multimedia Systems and Services (COST Action IC 1003). version 1.2. 2013. Google Scholar

[2] Liu Y T, Yun S, Mao Y N, et al. A study on quality of experience for adaptive streaming service. In: Proceedings of IEEE International Conference on Communications Workshops (ICC), 2013. 682--686. Google Scholar

[3] Lewcio B, Belmudez B, Enghardt T, et al. On the way to high-quality video calls in future mobile networks. In: Proceedings of 2011 3rd International Workshop on Quality of Multimedia Experience (QoMEX), 2011. 43--48. Google Scholar

[4] Zare A, Aminlou A, Hannuksela M M, et al. HEVC-compliant tile-based streaming of panoramic video for virtual reality applications. In: Proceedings of Proceedings of the 24th ACM International Conference on Multimedia, 2016. 601--605. Google Scholar

[5] Qian F, Ji L, Han B, et al. Optimizing 360 video delivery over cellular networks. In: Proceedings of the 5th Workshop on All Things Cellular: Operations, Applications and Challenges, 2016. 1--6. Google Scholar

[6] Anwar M S, Wang J, Ullah A, Khan W, Li Z, and Ahmad S. User profile analysis for enhancing QoE of 360 panoramic video in virtual reality environment. In: Proceedings of 2018 International Conference on Virtual Reality and Visualization (ICVRV), 2018. 106--111. Google Scholar

[7] Zhang B, Zhao J, Yang S, et al. Subjective and objective quality assessment of panoramic videos in virtual reality environments. In: Proceedings of IEEE International Conference on Multimedia & Expo Workshops (ICMEW), 2017. 163--168. Google Scholar

[8] Li C, Xu M, Du X Z, et al. Bridge the gap between VQA and human behavior on omnidirectional video. 2018. 932--940. Google Scholar

[9] Schatz R, Sackl A, Timmerer C, et al. Towards subjective quality of experience assessment for omnidirectional video streaming. In: Proceedings of 2017 9th International Conference on Quality of Multimedia Experience (QoMEX), 2017. 1--6. Google Scholar

[10] Zou W, Yang F. Measuring quality of experience of novel 360-degree streaming video during stalling. In: Proceedings of International Conference on Communicatins and Networking in China, 2017. 417--424. Google Scholar

[11] Zhang W, Zou W, Yang F. The impact of stalling on the perceptual quality of http-based omnidirectional video streaming. In: Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2019. 4060--4064. Google Scholar

[12] Chao Chen , Lark Kwon Choi , de Veciana G. Modeling the time--varying subjective quality of HTTP video streams with rate adaptations.. IEEE Trans Image Process, 2014, 23: 2206-2221 CrossRef PubMed Google Scholar

[13] Singla A, Fremerey S, Robitza W, et al. Measuring and comparing qoe and simulator sickness of omnidirectional videos in different head mounted displays. In: Proceedings of 2017 9th International Conference on Quality of Multimedia Experience (QoMEX), 2017. 1--6. Google Scholar

[14] Huyen T, Ngoc N P, Pham C T, et al. A subjective study on QoE of 360 video for VR communication. In: Proceedings of 2017 IEEE 19th International Workshop on Multimedia Signal Processing (MMSP), 2017. 1--6. Google Scholar

[15] Tasaka S. Bayesian Hierarchical Regression Models for QoE Estimation and Prediction in Audiovisual Communications. IEEE Trans Multimedia, 2017, 19: 1195-1208 CrossRef Google Scholar

[16] Ahmad S, Li K, Amin A. A Multilayer Prediction Approach for the Student Cognitive Skills Measurement. IEEE Access, 2018, 6: 57470-57484 CrossRef Google Scholar

[17] Zhang Y, Wang Y, Liu F. Subjective Panoramic Video Quality Assessment Database for Coding Applications. IEEE Trans Broadcast, 2018, 64: 461-473 CrossRef Google Scholar

[18] Corbillon X, de Simone F, Simon G. 360-degree video head movement dataset. In: Proceedings of the 8th ACM on Multimedia Systems Conference, 2017. 199--204. Google Scholar

[19] Lo W-C, Fan C-L, Lee J, et al. 360 video viewing dataset in head-mounted virtual reality. In: Proceedings of the 8th ACM on Multimedia Systems Conference, 2017. 211--216. Google Scholar

[20] Wu C, Tan Z, Wang Z, et al. A dataset for exploring user behaviors in vr spherical video streaming. In: Proceedings of the 8th ACM on Multimedia Systems Conference, 2017. 193--198. Google Scholar

[21] Staelens N, Moens S, Van den Broeck W. Assessing Quality of Experience of IPTV and Video on Demand Services in Real-Life Environments. IEEE Trans Broadcast, 2010, 56: 458-466 CrossRef Google Scholar

[22] ITU-T P.910, subjective video quality assessment methods for multimedia applications. 2008. Google Scholar

[23] ITU-R BT, 500-12, recommendation: Methodology for the subjective assessment of the quality of television pictures. 1993. Google Scholar

[24] Yao S-H, Fan C-L, Hsu C-H. Towards quality-of-experience models for watching 360 videos in head-mounted virtual reality. In: Proceedings of 2019 11th International Conference on Quality of Multimedia Experience (QoMEX), 2019. 1--3. Google Scholar

  • Figure 1

    (Color online) Source sequence thumbnails. (a) Idaho Boat; (b) Animation 1; (c) Science Fiction HELP; protectłinebreak (d) Gyrocopter; (e) Snow driving; (f) Animation 2;. (g) Skiing; (h) Military parade; (i) Beach volleyball; (j) Rio Olympics; (k) Cockpit view; (l) Undersea; (m) Roller coaster; (n) Animation 3; (o) Surrounded by Elephants; (p) Project Soul.

  • Figure 2

    (Color online) PLCC and SRCC performance evaluation of individual subjects.

  • Figure 3

    (Color online) MOS of all compressed videos in the absence of stalling.

  • Figure 4

    (Color online) The posterior QoE with three different video bitrates in the absence of stalling (a), and in the presence of initial stalling (b), mid stalling (c), multiple stalling (d).

  • Figure 5

    (Color online) Prediction of overall QoE.

  • Figure 6

    (Color online) PLCC and SRCC performance evaluation of the average MOS and predicted values.

  • Table 1  

    Table 1The details of source 360-degree videos

    Index 360-degree videos name Description Frame rate (fps) SI/TI
    (a) Idaho Dinghy Boat Human, high motion 30 59.9/66.8
    (b) Animation 1 Outdoor scene 24 48.3/38.4
    (c) Science Fiction HELP Human, indoor 30 28.5/18.4
    (d) Skyhub Dubai, Gyrocopter Human, architecture 30 42.9/14.7
    (e) Snow driving Human, nature 25 54.7/22.4
    (f) Animation 2 Snake in the forest 24 49.1/0.9
    (g) Skiing Human, high motion 25 54.1/1.5
    (h) Military parade Human, outdoor 30 50.1/16.1
    (i) Beach volleyball Human, high motion 24 30.6/7.4
    (j) Rio Olympics Human, outdoor 24 56.8/15.5
    (k) Cockpit view Human, indoor 25 59.5/21.7
    (l) Undersea Human, nature 30 32.1/7.1
    (m) Roller coaster Human, high motion 30 78.3/48.1
    (n) Animation 3 Outdoor scene 24 64.7/56.2
    (o) Surrounded by Elephants Wild, nature 30 36.2/2.8
    (p) Project Soul Human, indoor 30 27.5/11.9

    Algorithm 1 Bitrate posterior QoE estimation

    Require:sets QoE, bitrates.

    Output:sets ${\rm~BR}_m$ $(m=1,2,3)$. Initialization:

    Iterations bitrate:

    foreach $i$ in bitrates

    foreach $j$ in QoE

    Calculate posterior of $j$ with respect to $i$;

    Add posterior to ${\rm~BR}_m$;

    end for

    return ${\rm~BR}_m$;

    Increment ${\rm~BR}_m$ to ${\rm~BR}_{m+1}$;

    end for

  • Table 2  

    Table 2Average MOS values for various stalling events under different bitrate levels

    Stalling event Bitrate (1 Mbps) Bitrate (5 Mbps) Bitrate (15 Mbps)
    No stalling 3.94 6.3 8.67
    Initial stalling 3.49 5.66 6.94
    Middle stalling 3.3 5.04 6.22
    Multiple stalling 2.79 4.38 5.44
    No stalling – initial stalling 0.45 0.64 1.73
    No stalling – middle stalling 0.64 1.26 2.45
    No stalling – multiple stalling 1.15 1.92 3.23

    Algorithm 2 Stalling experience posterior QoE estimation under different bitrate levels

    Require:sets ${\rm~BR}_1$, ${\rm~BR}_2$, ${\rm~BR}_3$, Stalling.

    Output:sets ${\rm~ST}_p$ $(p=1,2,3)$, ${\rm~ST}_q$ $(q=1,2,3)$, and ${\rm~ST}_r$ $(r=1,2,3)$. Initialization:

    Iterations of Stalling:

    foreach $a$ in Stalling

    foreach $b$ in ${\rm~BR}_1$

    Calculate posterior of $a$ with respect to $b$;

    Add posterior to ${\rm~ST}_p$;

    end for

    return ${\rm~ST}_p$;

    Increment ${\rm~ST}_p$ to ${\rm~ST}_{p+1}$;

    foreach $c$ in ${\rm~BR}_2$

    Calculate posterior of $a$ with respect to $c$;

    Add posterior to ${\rm~ST}_q$;

    end for

    return ${\rm~ST}_q$;

    Increment ${\rm~ST}_q$ to ${\rm~ST}_{q+1}$;

    foreach $d$ in ${\rm~BR}_3$

    calculate posterior of $a$ with respect to $d$;

    add posterior to ${\rm~ST}_r$;

    end for

    return ${\rm~ST}_r$;

    Increment ${\rm~ST}_r$ to ${\rm~ST}_{r+1}$;

    end for

  • Table 3  

    Table 3Performance comparsion of the proposed technique

    Model PLCC SRCC
    Model 1 0.6769 0.6861
    Model 2 0.7190 0.7158
    Model 3 0.7476 0.7205
    Model 4 0.7723 0.7205
    Model 5 0.7905 0.7430
    Model 6 0.7395 0.6414
    Model 7 0.7455 0.6560
    Proposed technique 0.8095 0.7954

Copyright 2020  CHINA SCIENCE PUBLISHING & MEDIA LTD.  中国科技出版传媒股份有限公司  版权所有

京ICP备14028887号-23       京公网安备11010102003388号