1

Comparison of VoIP-QoE from Skype, LINE, Tango and Viber over 3G Networks in Thailand

Presented by:Presented by:

Asst. Prof. Dr. Pongpisit WuttidittachottiAsst. Prof. Dr. Pongpisit Wuttidittachotti Dept. of Data Communication and NetworkingDept. of Data Communication and Networking

Faculty of Information TechnologyFaculty of Information TechnologyKing Monkut’s University of Technology North Bangkok King Monkut’s University of Technology North Bangkok

Bangkok, ThailandBangkok, Thailand

2

CONTENTSCONTENTS• INTRODUCTION• VOIP–QOE • COMPARISON TECHNIQUE • RELATED WORKS• EXPERIMENT AND DATA GATHERING • RESULTS, ANALYSIS AND DISCUSSION• CONCLUSION AND FUTURE WORK• ACKNOWLEDGMENT• REFERENCES

3

IntroductionIntroduction• It has been mentioned about the statistics of users of VoIP applications as follows:

• Therefore, this study mainly presents a comparison of VoIP quality of popular applications in Thailand, Skype, LINE, Tango and Viber, to discover if there are significant differences or not.

• Thus, the results from this study is useful for users who need better voice quality from the free call services provided by social network applications, which is one contribution of this work.

Social Network Application with

VoIPApprox. registered users (Million) Remark

Skype 300 Q3/2013 [1]LINE 490 Thailand: 27 Million

users [2]Tango 200 [3]Viber 350 [4]

4

VOIP-QOEVOIP-QOE• With effects of IP network factors (e.g., loss, delay and jitter), end users perceive the quality of experience of VoIP, called VoIP-QoE, which may

be dissatisfied by the users. • In order to evaluate VoIP-QoE at the point of view

of users, Mean Opinion Score (MOS) has been accepted as a major metric [5-6].

• Originally, it has been recommend by ITU-T to obtain MOS from subjective tests with a group of subjects within a controlled environment [7].

• However, objective VoIP quality measurement methods become popular solutions.

• Perceptual Evaluation of Speech Quality (PESQ) is the most popular intrusive method when compared to

the others. • It provides very good performance with the

correlation between PESQ scores and the subjective scores of about 0.95 [8-10].

5

COMPARISON TECHNIQUE COMPARISON TECHNIQUE • A hypothesis test is a method to compare performance of at least 2 things, such as, MOS from different VoIP

applications. • It uses the gathered data to analyze and then to decide between a null hypothesis (H0), which describes that the null hypothesis is true, and an alternative

hypothesis (H1), which describes that the null hypothesis is false, for example: H0: MOS from Skype = MOS from LINE = MOS from TangoH1: MOS from Skype ≠ MOS from LINE ≠ MOS from Tango

• To accept or reject H0,it is necessary to determine for the p-value from the gathered data. If the p-value is < the significance level that is the breakpoint to accept or reject the null hypothesis (e.g., p-value = 0.05 with 95% confidence interval), H0 is rejected. • On the other hand, if the p-value is > the

significance level, H0 is accepted. • Two important statistical tools are ANOVA and t-test

[11].

6

RELATED WORKSRELATED WORKS• In 2006, Lisha and Junzhou reported that VoIP quality of Skype with GIPS iSAC codec provides good voice quality but it has been found that there were no significant differences when

compared to MSN Messenger [12]. • In 2006, Chiang et al. studied and found that

Skype provides better VoIP quality than MSN. Also, in cases of low bandwidth, high loss rate and facing NAT, Skype shows better performance

than MSN [13].• One year later, 2007, Chiang et al. studied VoIP

quality of Skype, Google Talk, Windows Live and Yahoo Messenger. It was observed that Yahoo Messenger seems slightly better than other

applications within the best scenario and medium jitters. However, it has been observed that

Windows Live is more robust to packet losses and high jitters [14].

7

RELATED WORKSRELATED WORKS• In 2009, Wu et al. evaluated Skype, Google Talk

and MSN Messenger focusing on different buffer size issue. It has been found that MSN Messenger provides the best performances in terms of buffer dimensioning to suit varying network conditions but Skype does not adjust its buffer size [15]. • It is consistent with the conclusion from one study by Dmytrienko et al. that has been presented in the same year [16], which was found that the highest communication quality was provided with Windows Live Messenger, and the lowest – with Yahoo Messenger, referring to jitter effects.

• In 2014, Wuttidittachotti et al. compared two popular social network services in Thailand, Facebook and LINE [17]. It has been found that

LINE tends to provide better quality than Facebook.

8

RELATED WORKSRELATED WORKS• For other social network applications, Casas et al.

presented the comparative study of YouTube and Facebook, two popular social sites, in mobile networks [18]. It has been found that YouTube QoE is sensitive to downlink-encoding bottlenecks highly but Facebook is robust to changing network conditions. However, Casas et al. did not study about voice quality of

those services. • Recently, Azfar et al. have presented very interesting

work with ten popular VoIP applications, including Skype, Google Talk, Viber and Tango [19]. It has been

discovered that voice traffics may not be not be encrypted in six tenth applications. Unfortunately, this work focused on security only, not about voice

quality. • From the previous works, it can be claimed that there

is room for comparison of VoIP quality from other modern social network applications/services, such as

LINE, Tango and Viber to Skype, which is the benchmark.

9

EXPERIMENT AND DATA GATHERING EXPERIMENT AND DATA GATHERING • The experiment was designed to conduct stationary tests

over 3G networks in the inner city of Bangkok. It presents the locations of 14 universities in Bangkok, Thailand, that have been selected for this study, due

to covering the area interest. • The experiment were

conducted by making calls from Skype, LINE, Tango and Viber on two

computers, focusing on voice quality only. While there are 5 3G operators, consisting of AIS, DTAC, TrueMove, TOT and CAT. All 3G networks were

connected using air cards and SIM cards.

10

EXPERIMENT AND DATA GATHERING EXPERIMENT AND DATA GATHERING • Using PESQ, speech samples were required. Thus,

the English speech samples were selected from the ITU-T website, including American and

British English [20].• Then, 14 rounds of experiments were conducted

at 14 locations, during February 2014. • Before testing with each network, the speed

tests were also tested with www.speedtest.or.th.

• In each round, based-on VoIP only, the experiment was started with Skype, then LINE, Tango and Viber respectively, over five 3G

networks randomly. In Each round, 4 reference English speech samples (2 male and 2 female) were played at the caller’s side and recorded as the degraded speech files at the destination

side.

11

EXPERIMENT AND DATA GATHERING EXPERIMENT AND DATA GATHERING • Besides, see the figure below, both sides are supposed to connect to the same base station of

3G network in each experiment because both devices were at the same location, in order to

minimize network factors, although it is different from the real practice that users

using VoIP usually are far apart.• After gathering degraded English speech samples

from all VoIP applications via five 3G networks, both reference speech samples and degraded speech samples were measured using PESQ at the VoIP Lab. in KMUTT, in order to

obtain MOS-LQO.

12

RESULTS, ANALYSIS AND DISCUSSIONRESULTS, ANALYSIS AND DISCUSSION: Speed Test Results: Speed Test Results

(a) Downlink speed results (b) Uplink speed results, where N = 126, 124, 140, 130 and 130 for AIS, DTAC, TrueMove, TOT, and CAT respectively.

13

RESULTS, ANALYSIS AND DISCUSSIONRESULTS, ANALYSIS AND DISCUSSION: VoIP Quality Results: VoIP Quality Results MOS-LQO from five 3G

networks (a) AIS, where SD = 0.85, 0.58, 0.58 and 0.56, and N = 88, 96, 96 and 96 for Skype, LINE, Tango and Viber respectively (b) DTAC, where SD = 0.88, 0.53, 0.61 and 0.64, and N = 96, 104, 104 and 104 for Skype, LINE, Tango and Viber respectively (c) TrueMove, where SD = 0.67, 0.60, 0.41 and 0.61, and all Ns = 112 for all VoIP applications (d) TOT, where SD = 0.74, 0.55, 0.67 and 0.90, and N = 104, 104, 96 and 104 for Skype, LINE, Tango and Viber respectively and (e) CAT, where SD = 0.83, 0.54, 0.53 and 0.53, and N = 112, 112, 104 and 112 for Skype, LINE, Tango and Viber respectively.

(a) (b) (c)

(d) (e)

14

RESULTS, ANALYSIS AND DISCUSSIONRESULTS, ANALYSIS AND DISCUSSION: Analyzed Results: Analyzed Results

15

CONCLUSION AND FUTURE WORKCONCLUSION AND FUTURE WORK•To compare VoIP quality from Skype and three modern VoIP applications, LINE, Tango and Viber, it can be concluded that Skype tends to provide the highest VoIP quality over high stabile 3G networks when compared to other modern VoIP applications, whereas LINE tends to provide the highest VoIP quality over lower stabile 3G networks. Therefore, Skype and LINE should be good VoIP solutions for users to use over 3G networks. •For future work, the modern VoIP applications should be investigated looking at their characteristics, codecs and algorithms inside. While investigation about QoS management within each 3G network would also be of interest. •Furthermore, this study has conducted stationary tests only. It is a challenge to study as future work about mobility in 3G networks.

16

AcknowledgmentAcknowledgmentThanks to Asst. Prof. Dr. Vajirasak Vanijja, Mr Tuul Triyason and the VoIP Lab., SIT, KMUTT for PESQ measurement tool and support. Finally, special thanks Mr. Gary Sherriff for editing.

17

ReferencesReferences1. http://blogs.skype.com/2013/08/28/skype-celebrates-a-decade-of-meaningful-conversations/ 2. http://www.techinasia.com/line-releases-regional-breakdowns-for-its-490m-registered-

users/3. http://www.businessinsider.com/tango-raises-280-million-from-alibaba-2014-3 4. http://techcrunch.com/2014/06/10/viber-hits-100-million-active-users/5. M. Adel et al., “Improved E-model for Monitoring Quality of Multi-Party VoIP

communications.” Proc. IEEE Globecom Workshops 2013, pp. 1180-1185, 2013.6. WZ. Li, J. Wang, CW. Xing, ZS. Fei and JM. Kuang, “A real-time QoE methodology for AMR

codec voice in mobile network,” Science China, Vol. 57, pp. 1-13, 2014.7. ITU-T Recommendation P.800, “Methods for subjective determination of transmission

quality”, 1996.8. ITU-T Recommendation P.8629. ITU-T Recommendation P.862.210.Ditech Networls, “Limitations of PESQ for Measuring Voice Quality in Mobile and VoIP

Networks,” White paper, Dec 2007.11.T. Daengsi, “VoIP Quality Measurement: Recommendation of MOS and Enhanced Objective

Measurement Method for Standard Thai Spoken Language.” Ph.D. Thesis, KMUTNB, 2012.12.G. Lisha and L. Junzhou, “Performance Analysis of a P2P-Based VoIP Software,” Proc.

AICT/ICIW 2006, pp. 11, 2006.13.WH. Chiang, WC. Xiao and CF. Chou, “A Performance Study of VoIP Applications: MSN vs.

Skype,” Proc. Multicomm 2006.14.B. Sat and B. Wah, “Evaluation of Conversational Voice Communication Quality of the

Skype, Google-Talk, Windows Live, and Yahoo Messenger VoIP Systems,” Proc. IEEE 9th MMSP 2007, pp. 135-138, 2007.

15.C.-C. Wu, K.T. Chen, C.-Y. Huang, and C.-L. Lei, “An Empirical Evaluation of VoIP Playout Buffer Dimensioning in Skype, Google Talk, and MSN Messenger,” Proc. NOSSDAV, pp. 97-102, 2009.

16.O. Dmytrienko, B. Bilodid and M. Ternovoy, “Comparative analysis of the VoIP Software,” Proc. CADSM 2009, pp. 24-28, 2009.

17.P. Wuttidittachotti, K. Yochanang, N. Chumkot, T. Triyason and T. Daengsi, “Quality of Experience of VoIP for Social Network Services: Facebook vs LINE over 3G Networks in North Bangkok,” Proc. 11thECTI-CON 2014, pp. 1-6, 2014.

18.P. Casas, A. Sackl, S. Egger, and R. Schatz, “YouTube & Facebook Quality of Experience in Mobile Broadband Networks,” Proc. IEEE Globecom Workshops, pp. 1269-1274, 2012.

19.A. Azfar, K.-K.R. Choo, L. Liu, “A study of ten popular Android mobile VoIP applications: Are the communications encrypted?” in Proc. of 47th Hawaii Int. Conf. System Sciences (HICSS) 2014, pp. 4858-4867, 2014.

20.http://www.itu.int/net/itu-t/sigdb/genaudio/AudioForm-g.aspx?val=10000501

18

Thank you very muchThank you very muchfor your attention.for your attention.