Post on 29-Apr-2023
1
Comparison of VoIP-QoE from Skype, LINE, Tango and Viber over 3G Networks in Thailand
Presented by:Presented by:
Asst. Prof. Dr. Pongpisit WuttidittachottiAsst. Prof. Dr. Pongpisit Wuttidittachotti Dept. of Data Communication and NetworkingDept. of Data Communication and Networking
Faculty of Information TechnologyFaculty of Information TechnologyKing Monkut’s University of Technology North Bangkok King Monkut’s University of Technology North Bangkok
Bangkok, ThailandBangkok, Thailand
2
CONTENTSCONTENTS• INTRODUCTION• VOIP–QOE • COMPARISON TECHNIQUE • RELATED WORKS• EXPERIMENT AND DATA GATHERING • RESULTS, ANALYSIS AND DISCUSSION• CONCLUSION AND FUTURE WORK• ACKNOWLEDGMENT• REFERENCES
3
IntroductionIntroduction• It has been mentioned about the statistics of users of VoIP applications as follows:
• Therefore, this study mainly presents a comparison of VoIP quality of popular applications in Thailand, Skype, LINE, Tango and Viber, to discover if there are significant differences or not.
• Thus, the results from this study is useful for users who need better voice quality from the free call services provided by social network applications, which is one contribution of this work.
Social Network Application with
VoIPApprox. registered users (Million) Remark
Skype 300 Q3/2013 [1]LINE 490 Thailand: 27 Million
users [2]Tango 200 [3]Viber 350 [4]
4
VOIP-QOEVOIP-QOE• With effects of IP network factors (e.g., loss, delay and jitter), end users perceive the quality of experience of VoIP, called VoIP-QoE, which may
be dissatisfied by the users. • In order to evaluate VoIP-QoE at the point of view
of users, Mean Opinion Score (MOS) has been accepted as a major metric [5-6].
• Originally, it has been recommend by ITU-T to obtain MOS from subjective tests with a group of subjects within a controlled environment [7].
• However, objective VoIP quality measurement methods become popular solutions.
• Perceptual Evaluation of Speech Quality (PESQ) is the most popular intrusive method when compared to
the others. • It provides very good performance with the
correlation between PESQ scores and the subjective scores of about 0.95 [8-10].
5
COMPARISON TECHNIQUE COMPARISON TECHNIQUE • A hypothesis test is a method to compare performance of at least 2 things, such as, MOS from different VoIP
applications. • It uses the gathered data to analyze and then to decide between a null hypothesis (H0), which describes that the null hypothesis is true, and an alternative
hypothesis (H1), which describes that the null hypothesis is false, for example: H0: MOS from Skype = MOS from LINE = MOS from TangoH1: MOS from Skype ≠ MOS from LINE ≠ MOS from Tango
• To accept or reject H0,it is necessary to determine for the p-value from the gathered data. If the p-value is < the significance level that is the breakpoint to accept or reject the null hypothesis (e.g., p-value = 0.05 with 95% confidence interval), H0 is rejected. • On the other hand, if the p-value is > the
significance level, H0 is accepted. • Two important statistical tools are ANOVA and t-test
[11].
6
RELATED WORKSRELATED WORKS• In 2006, Lisha and Junzhou reported that VoIP quality of Skype with GIPS iSAC codec provides good voice quality but it has been found that there were no significant differences when
compared to MSN Messenger [12]. • In 2006, Chiang et al. studied and found that
Skype provides better VoIP quality than MSN. Also, in cases of low bandwidth, high loss rate and facing NAT, Skype shows better performance
than MSN [13].• One year later, 2007, Chiang et al. studied VoIP
quality of Skype, Google Talk, Windows Live and Yahoo Messenger. It was observed that Yahoo Messenger seems slightly better than other
applications within the best scenario and medium jitters. However, it has been observed that
Windows Live is more robust to packet losses and high jitters [14].
7
RELATED WORKSRELATED WORKS• In 2009, Wu et al. evaluated Skype, Google Talk
and MSN Messenger focusing on different buffer size issue. It has been found that MSN Messenger provides the best performances in terms of buffer dimensioning to suit varying network conditions but Skype does not adjust its buffer size [15]. • It is consistent with the conclusion from one study by Dmytrienko et al. that has been presented in the same year [16], which was found that the highest communication quality was provided with Windows Live Messenger, and the lowest – with Yahoo Messenger, referring to jitter effects.
• In 2014, Wuttidittachotti et al. compared two popular social network services in Thailand, Facebook and LINE [17]. It has been found that
LINE tends to provide better quality than Facebook.
8
RELATED WORKSRELATED WORKS• For other social network applications, Casas et al.
presented the comparative study of YouTube and Facebook, two popular social sites, in mobile networks [18]. It has been found that YouTube QoE is sensitive to downlink-encoding bottlenecks highly but Facebook is robust to changing network conditions. However, Casas et al. did not study about voice quality of
those services. • Recently, Azfar et al. have presented very interesting
work with ten popular VoIP applications, including Skype, Google Talk, Viber and Tango [19]. It has been
discovered that voice traffics may not be not be encrypted in six tenth applications. Unfortunately, this work focused on security only, not about voice
quality. • From the previous works, it can be claimed that there
is room for comparison of VoIP quality from other modern social network applications/services, such as
LINE, Tango and Viber to Skype, which is the benchmark.
9
EXPERIMENT AND DATA GATHERING EXPERIMENT AND DATA GATHERING • The experiment was designed to conduct stationary tests
over 3G networks in the inner city of Bangkok. It presents the locations of 14 universities in Bangkok, Thailand, that have been selected for this study, due
to covering the area interest. • The experiment were
conducted by making calls from Skype, LINE, Tango and Viber on two
computers, focusing on voice quality only. While there are 5 3G operators, consisting of AIS, DTAC, TrueMove, TOT and CAT. All 3G networks were
connected using air cards and SIM cards.
10
EXPERIMENT AND DATA GATHERING EXPERIMENT AND DATA GATHERING • Using PESQ, speech samples were required. Thus,
the English speech samples were selected from the ITU-T website, including American and
British English [20].• Then, 14 rounds of experiments were conducted
at 14 locations, during February 2014. • Before testing with each network, the speed
tests were also tested with www.speedtest.or.th.
• In each round, based-on VoIP only, the experiment was started with Skype, then LINE, Tango and Viber respectively, over five 3G
networks randomly. In Each round, 4 reference English speech samples (2 male and 2 female) were played at the caller’s side and recorded as the degraded speech files at the destination
side.
11
EXPERIMENT AND DATA GATHERING EXPERIMENT AND DATA GATHERING • Besides, see the figure below, both sides are supposed to connect to the same base station of
3G network in each experiment because both devices were at the same location, in order to
minimize network factors, although it is different from the real practice that users
using VoIP usually are far apart.• After gathering degraded English speech samples
from all VoIP applications via five 3G networks, both reference speech samples and degraded speech samples were measured using PESQ at the VoIP Lab. in KMUTT, in order to
obtain MOS-LQO.
12
RESULTS, ANALYSIS AND DISCUSSIONRESULTS, ANALYSIS AND DISCUSSION: Speed Test Results: Speed Test Results
(a) Downlink speed results (b) Uplink speed results, where N = 126, 124, 140, 130 and 130 for AIS, DTAC, TrueMove, TOT, and CAT respectively.
13
RESULTS, ANALYSIS AND DISCUSSIONRESULTS, ANALYSIS AND DISCUSSION: VoIP Quality Results: VoIP Quality Results MOS-LQO from five 3G
networks (a) AIS, where SD = 0.85, 0.58, 0.58 and 0.56, and N = 88, 96, 96 and 96 for Skype, LINE, Tango and Viber respectively (b) DTAC, where SD = 0.88, 0.53, 0.61 and 0.64, and N = 96, 104, 104 and 104 for Skype, LINE, Tango and Viber respectively (c) TrueMove, where SD = 0.67, 0.60, 0.41 and 0.61, and all Ns = 112 for all VoIP applications (d) TOT, where SD = 0.74, 0.55, 0.67 and 0.90, and N = 104, 104, 96 and 104 for Skype, LINE, Tango and Viber respectively and (e) CAT, where SD = 0.83, 0.54, 0.53 and 0.53, and N = 112, 112, 104 and 112 for Skype, LINE, Tango and Viber respectively.
(a) (b) (c)
(d) (e)
14
RESULTS, ANALYSIS AND DISCUSSIONRESULTS, ANALYSIS AND DISCUSSION: Analyzed Results: Analyzed Results
15
CONCLUSION AND FUTURE WORKCONCLUSION AND FUTURE WORK•To compare VoIP quality from Skype and three modern VoIP applications, LINE, Tango and Viber, it can be concluded that Skype tends to provide the highest VoIP quality over high stabile 3G networks when compared to other modern VoIP applications, whereas LINE tends to provide the highest VoIP quality over lower stabile 3G networks. Therefore, Skype and LINE should be good VoIP solutions for users to use over 3G networks. •For future work, the modern VoIP applications should be investigated looking at their characteristics, codecs and algorithms inside. While investigation about QoS management within each 3G network would also be of interest. •Furthermore, this study has conducted stationary tests only. It is a challenge to study as future work about mobility in 3G networks.
16
AcknowledgmentAcknowledgmentThanks to Asst. Prof. Dr. Vajirasak Vanijja, Mr Tuul Triyason and the VoIP Lab., SIT, KMUTT for PESQ measurement tool and support. Finally, special thanks Mr. Gary Sherriff for editing.
17
ReferencesReferences1. http://blogs.skype.com/2013/08/28/skype-celebrates-a-decade-of-meaningful-conversations/ 2. http://www.techinasia.com/line-releases-regional-breakdowns-for-its-490m-registered-
users/3. http://www.businessinsider.com/tango-raises-280-million-from-alibaba-2014-3 4. http://techcrunch.com/2014/06/10/viber-hits-100-million-active-users/5. M. Adel et al., “Improved E-model for Monitoring Quality of Multi-Party VoIP
communications.” Proc. IEEE Globecom Workshops 2013, pp. 1180-1185, 2013.6. WZ. Li, J. Wang, CW. Xing, ZS. Fei and JM. Kuang, “A real-time QoE methodology for AMR
codec voice in mobile network,” Science China, Vol. 57, pp. 1-13, 2014.7. ITU-T Recommendation P.800, “Methods for subjective determination of transmission
quality”, 1996.8. ITU-T Recommendation P.8629. ITU-T Recommendation P.862.210.Ditech Networls, “Limitations of PESQ for Measuring Voice Quality in Mobile and VoIP
Networks,” White paper, Dec 2007.11.T. Daengsi, “VoIP Quality Measurement: Recommendation of MOS and Enhanced Objective
Measurement Method for Standard Thai Spoken Language.” Ph.D. Thesis, KMUTNB, 2012.12.G. Lisha and L. Junzhou, “Performance Analysis of a P2P-Based VoIP Software,” Proc.
AICT/ICIW 2006, pp. 11, 2006.13.WH. Chiang, WC. Xiao and CF. Chou, “A Performance Study of VoIP Applications: MSN vs.
Skype,” Proc. Multicomm 2006.14.B. Sat and B. Wah, “Evaluation of Conversational Voice Communication Quality of the
Skype, Google-Talk, Windows Live, and Yahoo Messenger VoIP Systems,” Proc. IEEE 9th MMSP 2007, pp. 135-138, 2007.
15.C.-C. Wu, K.T. Chen, C.-Y. Huang, and C.-L. Lei, “An Empirical Evaluation of VoIP Playout Buffer Dimensioning in Skype, Google Talk, and MSN Messenger,” Proc. NOSSDAV, pp. 97-102, 2009.
16.O. Dmytrienko, B. Bilodid and M. Ternovoy, “Comparative analysis of the VoIP Software,” Proc. CADSM 2009, pp. 24-28, 2009.
17.P. Wuttidittachotti, K. Yochanang, N. Chumkot, T. Triyason and T. Daengsi, “Quality of Experience of VoIP for Social Network Services: Facebook vs LINE over 3G Networks in North Bangkok,” Proc. 11thECTI-CON 2014, pp. 1-6, 2014.
18.P. Casas, A. Sackl, S. Egger, and R. Schatz, “YouTube & Facebook Quality of Experience in Mobile Broadband Networks,” Proc. IEEE Globecom Workshops, pp. 1269-1274, 2012.
19.A. Azfar, K.-K.R. Choo, L. Liu, “A study of ten popular Android mobile VoIP applications: Are the communications encrypted?” in Proc. of 47th Hawaii Int. Conf. System Sciences (HICSS) 2014, pp. 4858-4867, 2014.
20.http://www.itu.int/net/itu-t/sigdb/genaudio/AudioForm-g.aspx?val=10000501