But what is the user experience of these video conferencing software under different network impairments (continuous and uniform packet loss of 0%, 10%, 20%, 30%, 40%, 50%)? What is the service level of these video conferencing software?
In this regard, intervisibility evaluation launched the "test-based video conferencing software evaluation": we selected two Dell notebooks, used Siburon Umetrix video testing system and Attero network destructor, and selected six mainstream video conferencing solutions represented by domestic and foreign Internet vendors, terminal and network equipment vendors and operators. Such as Zoom (version v5. 1. 1), Tencent's Tencent Conference (version v 6.5438+0.7. 0(483)), Ali's Nail (version 5.1.15-release.24). This paper mainly evaluates the video quality, video frame rate change, video congestion rate, voice quality and audio-video synchronization performance of video conferencing software in the same test environment.
The final test results show that among the six test softwares, cloud video experience is the best, followed by Zoom, and flying books and nailing need to be improved.
1
Video quality
Video quality is an important factor affecting user experience. Fig. 1 shows the performance of video quality MOS score given by Sibolux video test system under different network damage and packet loss rates. It can be seen that the greater the MOS score in the evaluation results, the better the subjective feeling of user experience.
Generally speaking, Zoom and cloud video support point-to-point video conference streaming mode in local area network, and the video quality in local area network is better than other applications. Zoom and cloud video in non-LAN will automatically switch to cloud server mode, and the image quality of Zoom will be greatly reduced, while cloud video is not far from peer-to-peer mode. Huawei WeLink, Feishu, Nail, Tencent Conference and Video Conference all go through the cloud server. Among them, Huawei WeLink has the best picture quality, followed by Tencent.
2
Comparison of video average frame rate
Video frame rate affects the smoothness of video playback, and its influence on subjective quality varies with video content. The more complex the video content, the higher the frame rate. As can be seen from Figure 2, the image quality of Shufei and Nail in the ideal environment is very poor, and neither software provides HD video options. Its video frame rate is not as good as other applications. Zoom has the best ability to resist packet loss at the user end, and the video quality gradually decreases with the increase of packet loss, and the frame rate can always be maintained at more than 25 frames. Cloud video and Tencent conference are the same in the case of 50% packet loss, and the frame rate is slightly lower, but it can also ensure the smoothness of the video.
three
Video jam comparison
There are three main factors that cause the video jam: the frame rate is too low, the upload is blocked, and the downlink is not good, so the video conference software needs to design a targeted optimization scheme. In the evaluation, Ubuntu Umetrix video test system makes a comparative analysis of video interference with each interference time exceeding 200ms user experience window. As can be seen from Figure 3, in terms of video fluency, Zoom performs very well in both peer-to-peer and cloud server modes, which can guarantee that it will not get stuck when the packet loss rate is 50%. China Mobile Cloud Video performs well in the peer-to-peer mode, but it lacks in the cloud server mode. However, the anti-packet loss ability of nail client is not good. When the packet loss rate reaches 30%, the video will get stuck.
four
Speech quality comparison
Voice quality directly affects the user's experience. The ITU-based POLQA algorithm provided by Sibron Umetrix video testing system is a commonly used evaluation method to evaluate voice quality through MOS score. The greater the MOS score, the better the user experience. As can be seen from Figure 4, whether in peer-to-peer mode or cloud server mode, the voice quality of Zoom and cloud video is almost perfect. Both applications support ultra-wideband voice, and the sound quality can be maintained at a good level in the case of 50% packet loss. Although the nail supports ultra-wideband voice, the sound quality is not as good as other applications under 20% packet loss, and the sound quality of the client network drops rapidly after a large number of packet losses.
five
Comparison of audio and video synchronization performance
Have any partners ever encountered the problem that the video picture and sound are out of sync during the video conference, and they will feel very uncomfortable at this time. By evaluating the difference of audio-video synchronization performance between 10%-50% packet loss and no packet loss based on Sperling GED fluency algorithm, it can be seen that in the case of 10%-50% packet loss, ByteDance Shu Fei has the best audio-video synchronization performance, followed by Tencent Conference, and cloud server mode amplification is acceptable.
Combined with the performance test of the above five dimensions, it is not difficult to see that each video software gives users different experiences in different modes.
In terms of video conference quality, cloud video has the best video conference quality, and it can provide the resolution of 1080p in an ideal network, which is obviously superior to others. Zoom has good quality in peer-to-peer mode, but the video quality in cloud server mode is average. In the cloud server mode, the video quality of Huawei WeLink and Tencent Conference is obviously better than Zoom, but the video fluency is not as good as Zoom. The video experience of flying books and nailing video conferences is relatively general.
Under the restriction of network environment, Zoom, Cloud Video, Huawei WeLink, Tencent Conference and Shu Fei can provide voice quality above the fixed-line level under various network conditions. However, in the case of poor network, the quality of the speech at the conference will be greatly reduced.