Learning social signals for predicting relevant search results

HAL Id: hal-02927196https://hal.archives-ouvertes.fr/hal-02927196

Submitted on 2 Sep 2020

HAL is a multi-disciplinary open accessarchive for the deposit and dissemination of sci-entific research documents, whether they are pub-lished or not. The documents may come fromteaching and research institutions in France orabroad, or from public or private research centers.

L’archive ouverte pluridisciplinaire HAL, estdestinée au dépôt et à la diffusion de documentsscientifiques de niveau recherche, publiés ou non,émanant des établissements d’enseignement et derecherche français ou étrangers, des laboratoirespublics ou privés.

2SRM: Learning social signals for predicting relevantsearch results

Ismail Badache

To cite this version:Ismail Badache. 2SRM: Learning social signals for predicting relevant search results. Web Intelligenceand Agent Systems, IOS Press, 2020, 18 (1), pp.15-33. �10.3233/WEB-200426�. �hal-02927196�

https://hal.archives-ouvertes.fr/hal-02927196

https://hal.archives-ouvertes.fr

Web Intelligence 0 (0) 1 1IOS Press

2SRM: Learning Social Signals for PredictingRelevant Search ResultsIsmail Badache ∗

Aix Marseille Univ, Université de Toulon, CNRS, LIS, Marseille, FranceAix Marseille Univ, INSPE, SFEREP, Marseille, France

Abstract. Search systems based on both professional meta-data (e.g., title, description, etc.) and social signals (e.g., like, com-ment, rating, etc.) from social networks is the trending topic in information retrieval (IR) field. This paper presents 2SRM (SocialSignals Relevance Model), an approach of IR which takes into account social signals (users’ actions) as an additional informa-tion to enhance a search. We hypothesize that these signals can play a role to estimate a priori social importance (relevance)of the resource (document). In this paper, we first study the impact of each such signal on retrieval performance. Next, somesocial properties such as popularity, reputation and freshness are quantified using several signals. The 2SRM combines the socialrelevance, estimated from these social signals and properties, with the conventional textual relevance. Finally, we investigatethe effect of the social signals on the retrieval effectiveness using state-of-the-art learning approaches. In order to identify themost effective signals, we adopt feature selection algorithms and the correlation between the signals. We evaluated the effective-ness of our approach on both IMDb (Internet Movie Databese) and SBS (Social Book Search) datasets containing movies andbooks resources and their social characteristics collected from several social networks. Our experimental results are statisticallysignificant, and reveal that incorporating social signals in retrieval model is a promising approach for improving the retrievalperformance.

Keywords: Social Signals, Social Properties, Social Information Retrieval, Criteria Evaluation, Learning Approaches

1. Introduction

Before web 2.0, user interactions were limited tocreation of links from one website to another one [50].Nowadays, social web has completely changed themanner in which people communicate and share in-formation on the web. It allows users to interact andproduce large masses of social signals. On Facebook,the Like and Share Buttons are viewed across almost10 million websites daily1. On Twitter, second mostpopular social network after Facebook, thanks to itsfunctionalities of tweet and retweet more than 150 mil-lion tweets were published just for the 2012 Olympicsgames2. Other types of functions such as endorsement,share, comment and rating allow users to interact withweb resources. Through these social actions, web re-

*Corresponding author. E-mail: [email protected]://zephoria.com/top-15-valuable-facebook-statistics/2https://blog.twitter.com/2012/olympic-and-twitter-records

sources could become popular by accumulating thecounts that people share such information, facilitateand help, users access novel information in convenientmanner.

While we witness some recent moves from big play-ers towards a more social information retrieval (suchas Google and Bing expansion of results with thoseliked by the users’ Facebook friends), the ways searchengines and/or web 2.0 applications exploit social sig-nals (if they ever do) are usually not disclosed. This pa-per describes an approach that exploits social networksor involve a collective intelligence process to help theuser satisfy an information need. Particularly, we focuson exploiting social signals to estimate the social im-portance (relevance) of the resource to a given query.The main research questions addressed in this paperare the following:

2405-6456/0-1900/$35.00 c© 0 – IOS Press and the authors. All rights reserved

2 Ismail Badache / Learning Social Signals for Predicting Relevant Search Results

1. Can these social signals help the search systemsfor guiding its users to reach a better quality ormore relevant content?

2. How effective is each individual signals for rank-ing resources for a given query? What are theranking correlations created by these social sig-nals?

3. How to combine these social signals in form ofsocial properties? What are the most useful ofthem to take into account in search model?

Note that we have already investigated the impactof social signals on search effectiveness using machinelearning approaches in paper [11]. It only describedsome preliminary results and it did not deeply evalu-ate and analyze the results. This paper extends signif-icantly our previous work in the following main addi-tional aspects:

1. presenting a more complete state-of-the-art onsocial signals,

2. exploiting additional signals (e.g., tag, rating andits freshness, etc),

3. conducting new and extensive experiments ontwo standard datasets, namely INEX SBS (SocialBook Search) and INEX IMDb (Internet MovieDatabase),

The remainder of this paper is organized as follows:Section 2 presents some related work and the back-ground. Section 3 details our social IR approach. Then,Section 4, reports on the results of our experimentalevaluation. Finally, Section 5 concludes this paper byannouncing perspectives.

2. Background and Related Work

In this section, we report: (i) some background in-formation about social signals and the existing searchengines using these signals (e.g., Google and Bing);(ii) related work exploiting social data to estimate doc-ument relevance.

2.1. Social Signals

Social signals represent one of the most popularUGC (User Generated Content) on the web. Indeed,according to [4] web pages include buttons of differ-ent social networks where users can express whetherthey support, recommend or dislike content (text, im-age, video, etc). These buttons which describe actions

being of social activities (e.g., like, share, +1) are re-lated to specific social networks (e.g., Facebook andGoogle+) with counters indicating the interaction ratewith the web resource (see Figure 1) .

Fig. 1. Example of a resource with social signals buttons

A social signal is a measure of the activity on so-cial media. It is a social interaction of a real personwith a resource on the web through the functionalitiesoffered by social networks. As with backlinks3, socialactions (e.g, rating, like, like, +1) can be interpreted asan approval for the resource, which can helps to im-prove the its ranking in search engines. As for politicalinformation, it is information related to government orpolitical news, for example information about the elec-tion of regional heads. Social media enable users toget and share political information in social media andalso giving opinions that can be done by pressing like,dislike buttons or commenting on information [43].

2.1.1. Types of Social SignalsIn general, each social network uses its own social

signals with different operating rules. Table 1 summa-rizes the most popular signals on social networks.

2.2. Social Signals and Search Engines

Despite the lack of clear consensus on the exact rela-tionship between social signals and famous search en-gines (e.g., Google and Bing), there are many reasonswhy the signals can not be ignored. Rather than seeing

3backlinks or inbound link is a hyperlink to a site or web page.

Ismail Badache / Learning Social Signals for Predicting Relevant Search Results 3

Table 1List of different social signals types

Type Example Social network

VoteLike Facebook, LinkedIn,+1 Google+, StumbleUpon

MessageTweet Facebook, Google+,Post LinkedIn, Twitter

ShareShare Google+, Twitter, Buffer,Re-tweet Facebook, LinkedIn

TagBookmark Delicious, Diigo, DiggPin Pinterest

CommentComment Facebook, Google+,Reply LinkedIn, Twitter

EmotionLove, Haha, Wow

FacebookSad, Angry

Event ReactionPride (Pride month)

FacebookThankful (Mother’s day)

RelationFollowers

Facebook, TwitterFriends

the signals and ranking of results by the search enginesas two distinct components. It is useful to considerthem as interconnected processes working towards theoverall goal: increasing online visibility. Since incep-tion of Facebook or other social networks, social sig-nals also became an important information for SEO(Search Engine Optimization) [39]. They provide in-formation about social interactivity, social behaviorand social relations4. The correlation between socialsignal and the ranking position of a URL is extremelyhigh. This is valid for all social networks covered bySISTRIX5 Toolbox. The works of Lewoniewski et al.[39] lead to the assumption that the results of socialsignals also correlate with the quality of Wikipedia ar-ticles. Therefore, social signals can be an indicator ofthe relevance of web resources.

Google is still mysterious about the way to ex-ploit social signals to rank its search results, but somestudies conducted annually since 2016, searchmetrics6

showed that it exists a high correlation between socialsignals and the rankings provided by search enginessuch as Google. However, the degree to which socialsignals play a role SEO is unclear. John Mueller (Web-master Trends Analyst at Google) said : "Do socialmedia signals have an impact on organic rankings inGoogle? Not directly. No. So it’s not that there’s anykind of a ranking effect there. To a large part socialnetworks also have a nofollow on the links that theyprovide when they post this content, so it’s not the case

4https://en.ryte.com/wiki/Category:Search_Engine_Optimization5https://www.sistrix.fr/6https://www.searchmetrics.com/knowledge-base/ranking-

factors-infographic-2016/

that would give you any kind of a ranking boost there.What you do sometimes see however is that the socialposts show up in the search results."7

Although Google does not have partnership withFacebook, it still has access to public data from Face-book and may use some of them to better understandthe popularity of web pages. In 2015, Google signedan agreement with Twitter to index tweets in real time,allowing them to be searchable. Free access to Twit-ter database means that all the information on Twit-ter are available for Google automatically without theuse of robots crawling. Google algorithms also focuson Twitter profiles who tweet and retweet content, buthow do it remains a black box.

In addition, it is no secret that Google gives weightto its own social network, Google+. The content andinteractions on Google+ are known to have a positiveimpact on the ranking of its results. The 2016 Search-metrics study showed that +1 is strongly correlatedwith the ranking of Google search results, compared toother well-known criteria such as Facebook signal andkeywords frequency.

Bing, the second biggest search engine after Google,is explicit in its use of signals such as tweet and Face-book like as well as other social signals as ranking fac-tors [2,37,45]. Bing has partnered with Facebook insocial search [51]. Bing algorithms focuses on the so-cial media content, links, popularity from various so-cial networks that are considered as important factorsby Bing to define the ranking of results.

Bing is a notable example of exploiting images andinformation (posts, signals) from social media to pro-vide the richest results for users [45]. The social mediaactivity is also presented on Bing results pages muchmore visible than other search engines.

Social content such as tweets, pins (coming fromPinterest) and Facebook data, containing relevant key-words, are often integrated into the Bing search results.Thus, publishing visual contents (image, video, etc.)on the social networks is an excellent way to increasethe visibility on Bing. Also, Bing is implementing aservice called Social Sidebar that exploits Facebook toenhance a search [28]. It is a third column of the resultspage that allows users connected to comment and likethe relevant results from Facebook without leaving thesearch page. This service is functional only in USA.

The popularity of UGC, particularly in the contextof social media has given birth to many new problems

7https://www.youtube.com/watch?v=WszvyRune14#t=1192


in information retrieval. Specifically, how to exploitthese social content in favor of IR is an open question,which gave birth to a new field in IR, Social Informa-tion Retrieval (SIR) [6,18].

2.3. Social Signals Approaches

Some approaches of social search rely on exploitingtags. Abel et al. [1] and Hotho et al. [31] proposed dif-ferent algorithms based on folksonomies. By analogywith PageRank, FolkRank is based on relation betweentag, resource and user, whereas with GFolkRank a re-source group is identified by a unique tag in the con-text of the resource group. In the same context Bao etal. [17] proposed two algorithms, SocialPageRank andSocialSimRank, together with Yanbe et al. [50] suggestSBRank algorithm. These algorithms are motivated bythe report that there is a strong interdependence be-tween the popularity of users, tags and resources ina folksonomy. They focus on collective social searchand doesn’t respect different engagement types or dif-ferent trust levels. SocialSimRank calculates similaritybetween two tags of folksonomy and declares that sim-ilar tags are usually assigned to similar resources. Inaddition, tags are sometimes more reliable than meta-data provided by the content producer. However, sin-gle tag can hardly cover an entire topic and is moreambiguous for the user that a contextual sentence.

Furthermore, there are several recent works that fo-cus on how to improve information retrieval (IR) ef-fectiveness by exploiting the users’ actions and theirunderlying social network. Chelaru et al. [22] stud-ied the impact of signals (like, dislike, comment) onthe effectiveness of search on YouTube. They showedthat, although the basic criteria using the similarity ofquery with video title and annotations are effective forvideo search, social criteria are also useful and im-prove the ranking of search results for 48% of queries.They used feature selection algorithms and learning torank algorithms. Karweg et al. [35] proposed an ap-proach combining topical score and social score basedon two factors: first, user engagement intensity quan-tifies the effort a user has made during an interac-tion with document, measured by the number of clicks,number of votes, number of records and recommen-dation, secondly, trust degree measured from socialgraph for each user according to his popularity, usingPageRank algorithm. They have found that social re-sults are available for most queries and usually lead tomore satisfying results. Similarly, in [36,37] Khodaeiet al. proposed a ranking approach based on several so-

cial factors including relationships between documentowners and querying user, importance of each user andusers’ actions (playcount: number of times a user lis-tens to a track on last.fm) performed on web docu-ments. They have conducted an extensive experimentsset on last.fm dataset. They showed a significant im-provement for socio-textual ranking compared to thetextual only and social only approaches. On Twitter,Hong et al. [30] used retweets as a measure of pop-ularity and apply machine learning techniques to pre-dict how often new messages will be retweeted. Theyexploited different features, the content of messages,temporal information, metadata of messages and users,and the user’s social graph. However, banal tweets(e.g., rumors, without interest) can be very popular,such as those concerning celebrities, who generallyhave a large number of followers. Chan et al. [21] pro-posed a system called PostScholar, a service that aug-ments the results returned by Google Scholar, a searchengine for academic citations. PostScholar detects theTwitter activity related to an article and displays thatinformation on the search results page returned byGoogle Scholar. An additional hyperlink appears in theresults for each article that has Twitter activity associ-ated with it (the number of tweets found for that arti-cle, the date of the most recent tweet). These tweets aresorted according to their sentiments scores. Albishreet al. [3] proposed an innovative mechanism to au-tomatically select useful feedback documents using atopic modeling technique to improve the effectivenessof pseudo-relevance feedback models. The main ideabehind their proposed model is to discover the latenttopics in the top-ranked documents that allow for theexploitation of the correlation between terms in rele-vant topics. To capture discriminating terms for queryexpansion, they incorporated topical features into arelevance model that focuses on the temporal infor-mation in the selected set of documents. Experimen-tal results on TREC 2011-2013 microblog datasets il-lustrated that the proposed model significantly outper-forms all state-of-the-art baseline models.

Finally, there are other studies initiated by MicrosoftBing researchers [41,45] that show the usefulness ofdifferent social contents generated by the network ofuser’s friends on Facebook. Pantel et al. [42] stud-ied the leverage of social annotation on the quality ofsearch results. They observed that social annotationscan benefit web search in two aspects: 1) the anno-tations are usually good summaries of correspondingweb pages, 2) the number of annotations indicates thepopularity of web pages. Hecht et al. [28] presented


a system called SearchBuddies based on any socialinformation around the user and especially what hisfriends liked and shared as web page, Facebook pages.Gou et al. [24] proposed a ranking approach takinginto account document content and similarity betweenuser and documents user owner in social network.They used a multi-level algorithm to measure the sim-ilarity between actors. Experimental results based onYouTube data show that compared with tf-idf algo-rithm, SNDocRank method returns more relevant doc-uments. According to these results, a user can enhancesearch by joining a larger social networks, having morefriends, and connecting larger communities.

In this paper, our goal aims to exploit social sig-nals for improving accuracy and relevance of conven-tion textual web search. We exploit various signals ex-tracted from different social networks. In addition, in-stead of considering social features separately as donein the previous works, we propose to combine them tomeasure specific social properties, namely the popular-ity and the reputation of a resource. We also attempt tomeasure the impact of the freshness of the signal on theperformance of IR system. Unlike our previous workon social signals [5,7,8,10,12,13,14,15,16], in this pa-per, the proposed approach is completely supervisedby exploiting social signals collected from different so-cial networks as criteria of relevance. It is evaluatedon different types of standard test data (INEX SocialBook Search and INEX Internet Movie Database). Inthe following, we present some previous work in con-nection with this work but are different in regard to thesolution proposed in this paper.

In [5] we presented our first basic preliminary studyin the form of a poster which proposes a problematicon the users’ information needs coming from the so-cial web, this problem is related to what is called so-cial search or social information retrieval. In [10,12]we have used some features such as comments andlikes for ranking web resources, and addressing the im-pact of number of shares and likes in search relevance.The use of temporal features is inspired by the use offeatures of this class (e.g., Age of the resource andthe signal) in specific domains, such as IMDb movies[13,14]; In these papers, we are particularly interestedin: first, showing the impact of signals diversity as-sociated to a resource on information retrieval perfor-mance; second, studying the influence of their socialnetworks origin on their quality. We have proposed tomodel these social features as prior that we integrateinto an unsupervised language model. In [15,16] ourobjective is to study the impact of the new emotional

social signals, called Facebook reactions (love, haha,angry, wow, sad) in the retrieval. These reactions al-low users to express more nuanced emotions comparedto classic signals (e.g. like, share). First, we have ana-lyzed these reactions and showed how users use thesesignals to interact with posts. Second, we have evalu-ated the impact of each such reaction in the retrieval,by comparing them to both the textual model withoutsocial features and the first classical signal (like-basedmodel). Similarly to the modeling signals in [13,14],these social features are modeled as document priorand are integrated into a language model. We haveconducted a series of experiments on IMDb dataset.In [8,9] our goal is to show how these social tracescan play a vital role in improving Arabic Facebooksearch. Firstly, we have identified polarities (positiveor negative) carried by the textual signals (e.g. com-ments) and non-textual ones (e.g. the reactions loveand sad) for a given Facebook post. Therefore, the po-larity of each comment expressed on a given Facebookpost, is estimated based on a neural sentiment modelin Arabic language. We note that sentiment analysisof social content is a complex task [20]. Secondly, wehave grouped signals according to their complemen-tarity using features selection algorithms. Thirdly, wehave applied learning to rank (LTR) algorithms to re-rank Facebook search results based on the selectedgroups of signals. Finally, experiments are carried outon 13,500 Facebook posts, collected from 45 topicsin Arabic language. Experiments results reveal thatRandom Forests combined with ReliefFAttributeEval(RLF) was the most effective LTR approach for thistask. In [7] we have conducted an exploratory studyin the impact of users’ traces on Arabic and EnglishFacebook search. In general, during all these years ofresearch, our findings reveal that incorporating socialfeatures is a promising approach for improving the IRranking performance in Arabic and English languages.

3. 2SRM: Social Signals Relevance Model

Our IR approach, named 2SRM, consists of exploit-ing social signals to define social properties to take intoaccount in retrieval model. We associate to each webresource a social relevance estimated based on thesesocial features (signals and properties). The social rel-evance score is then combined with a classical topicalrelevance score (see Figure 2).


Fig. 2. A modular approach for Social IR

3.1. Notation

Social information that we exploit within the frame-work of our model can be represented by 5-tuple (U,R, A, T, G) where U, R, A, T, G are finite sets of in-stances: Users, Resources, Actions, Times and Socialnetworks.

Resources. We consider a collection R={r1, r2,...rn}of n resources. Each resource r can be a web page,video or other type of web resources. A resource rcan be represented both as a set of textual keywordsrw={rw1

, rw2,...rwn } and as a set of social actions A{a1,

a2,...am} performed on this resource.Actions. There is a set A={a1, a2,...am} of m actions

that users can perform on the resources. These actionsrepresent the relation between users U={u1, u2,...ul}and resources R. For instance, on Facebook, users canperform the following actions on resources: publish,like, share or comment.

Time. It represents the history of each action, letTai ={t1,ai , t2,ai ,...tk,ai } a set of k moments (date) atwhich each action ai was produced. A moment t repre-sents the datetime for each action a of the same type.

Social networks. There is a set G={g1, g2,...gz} of zsocial networks. Each specific social network containsspecific social actions a performed on resources.

3.2. Formalization of Our Social Search Model

By analyzing various types of social actions (sig-nals) through many social networks, we define threesocial properties that are detailed below:

Popularity P. Is a social phenomenon which in-dicates which is the most known among the pub-lic. Thanks to the influence of peers, target resourcesprogress quickly in terms of its pervasive in the soci-ety. Therefore, the web resource popularity can be es-timated according to the rate of sharing this resourcebetween the users through social actions. We assume

that the popularity comes from users’ activities onsocial networks, i.e. A resource is said popular if itwas shared and commented by several users in severalsocial networks, to the point where it becomes veryknown to general public.

Reputation R. The resource popularity does not re-flect its good or bad reputation. Resource reputation isan opinion on this resource, we believe that the esti-mation of this property can be calculated based on so-cial activities that have positive meaning such as Face-book like or marking resource as favorites on Deli-cious. Indeed, resource reputation depends on the de-gree of users’ appreciation on social networks.

In a summary, we assume that some social actionsare more suitable to evaluate popularity of a resourceand others are more related to reputation. Therefore,we associate to each of these properties a score cal-culated by a simple counting (normalized using min-max) of the number of associated actions. The generalformula is the following:

fx(r,G) =

m∑i=1,ax

i ∈A

Count(axi , r,G) (1)

Formula 1 is normalized as follows:

fx(r,G)Norm =fx(r,G)− Min( fx(r,G))

Max( fx(r,G))− Min( fx(r,G))(2)

Where:

– Count(axi , r,G) represents number of occurrence

of specific action axi performed on a resource

in relation to a specific social network. x ={P,R}S ocial (with P: Popularity, R: Reputation).

– fx(r,G) arithmetic function that represents thelinear combination of m social actions that quan-tify each x social properties (rate of interactionthrough the social signals).

– Min( fx(r,G)), Max( fx(r,G)) represent the mini-mum and maximum value for fx. fx(r,G)Norm rep-resents the fx min-max normalization.

In addition to a simple counting of social actions, wepropose to consider the time associated with the sig-nal. We assume that the resource associated with fresh(recent) signals should be promoted.

Freshness F. Is an important relevance factor, ex-ploited by several search engines. The informationfreshness is often measured in relation to its publica-


tion date, but we cannot say that information is neces-sarily obsolete because it was published two years ago.Taking an example of a resource published in Septem-ber 2001, carrying an information about the attack on"World Trade Center", in 2019, the same resource wasdiscussed on social networks through different socialsignals. We assume that a resource is fresh if recent so-cial signals were associated with it. For that purpose,we define freshness as follow: "a date of each socialaction (e.g., date of comment, date of share) performedon a resource on social networks can be exploited tomeasure the recency of these social actions, hence thefreshness of information". Its formula is given as fol-lows:

fF(r,G) =1

1m

∑mi=1

(1k

∑kj=1 Time(t j,ai , r,G)

) (3)

Where:

– Time(t j,ai , r,G) calculating the time duration (re-cency) between current time and action time t j,ai

of the same type for a resource r. We notice thatfor each action, its time is initialized to: 01-01-1970 00:00:00.

– fF(r,G) freshness function that represents the in-verse average of Time(t j,ai , r,G) values for a re-source r.

– 1k

∑kj=1 Time(t j,ai , r,G) represents the time dura-

tion average related to all k action time t j,ai of thesame type.

– 1m

∑mi=1

(1k


)represents the

average value of time duration averages related toall m types of action on the resource.

Social score. Regarding the social score RelS (q, r,G), we specify that this score takes into account thesesocial properties, which are in the form of three nor-malized factors that are combined linearly by the fol-lowing formula:

RelS (q, r,G) = δ. fR(r,G) + λ. fP(r,G) + β. fF(r,G)

(4)

Where : β, λ, δ are parameters assigning relative weightsto each social properties, ∀(β, λ, δ) ∈ [0, 1]3, β + λ +δ = 1.

Example : How to calculate RelS ?Let R={r1, r2, r3} a collection of 3 resources.Let A={a1, a2, a3, a4} of 4 actions (signals) that

users can perform on the resources, where : a1 = like,a2 = bookmark, a3 = tweet and a4 = share. Wehypothesized that reputation R is quantified using a1and a2, and popularity P is quantified using a3 and a4.

Table 2 illustrates, through an example of data, thedifferent steps to calculate the final social score.

Table 2Calculation example of RelS

Functions / Resources r1 r2 r3Calculation of Reputation R

Count(aR1, r,G) 100 11 22

Count(aR2, r,G) 77 200 100

eq. 1 : fR(r,G) 177 211 122

eq. 2 : fR(r,G)Norm 0.28 1 0

Calculation of Popularity P

Count(aP3 , r,G) 111 17 35

Count(aP4 , r,G) 18 250 112

eq. 1 : fP(r,G) 129 267 147

eq. 2 : fP(r,G)Norm 0 1 0.13

Calculation of Freshness F(1k

∑kj=1 Time(t j,a1 , r,G)

)883 39 189(

1k


)128 50 111(

1k


)1519 18 372(

1k


)350 31 570

14

∑4i=1

(1k


)720 34.5 310.5

eq. 3 : fF(r,G) 0.001 0.03 0.003

Social score RelS : e.g. δ = 0.6λ = 0.2β = 0.2

eq. 4 : RelS (q, r,G) 0.224 1 0.026

Ranking of resources 2 1 3

4. Experimental Evaluation

To evaluate our approach, we conducted a seriesof experiments on two datasets, SBS (Social BookSearch) and IMDb (Internet Movie Database). We firstevaluated the impact of social signals, taken separatelyand when they are combined as properties (popularity,reputation and freshness). Secondly, we study the ef-fectiveness of each social signal using machine learn-ing with selection attributes algorithms. We comparedour approach which takes into account social signals,with the baseline formed by only a textual model. Ourmain goals in these experiments are:


1. to evaluate the impact of signals taken separatelyand grouped (properties),

2. to evaluate the effectiveness of signals using ma-chine learning techniques.

4.1. Description of INEX Datasets (IMDb and SBS)

We used the collections SBS8 and IMDb9 docu-ments provided by INEX. Each document describes abook on SBS and movie on IMDb. It is represented bya set of metadata, which has been indexed according tokeywords extracted from fields (see table 3 and 5). Foreach document, we collected specific social signals viatheir corresponding API of 6 social networks listed inTable 7. We chose 30 and 208 topics with their rele-vance judgments provided by INEX IMDb 2011 andSBS 2015, respectively. In our study, we focused onthe effectiveness of the top 1000 results.

Table 4 presents statistics on the number of sig-nals in the 1000 documents returned by each topic(30 IMDb topics). According to the averages of signalnumbers in the documents, we note that the density ofFacebook signals (in average: 85.8 like, 94.1 share and98.4 comment) is very high compared to other signals(in average: 2.5 +1, 0.9 bookmark, 17.2 tweet and 1.4share (LinkedIn)).

Table 6 presents statistics on the number of signalsin the 1000 documents returned by each topic (208SBS topics). We note that the density of Facebook sig-nals is very high compared to Amazon/LibraryThingsignals but the total number of rating and review ismuch higher compared to other signals.

4.2. Quantifying Social Properties

Table 7 presents the properties that we want to takeinto account in our retrieval model. In order to quantifythese social properties, we associate them with the cor-responding social signals. Specific social signals (ac-tions) have been associated with each property depend-ing on their nature and meaning. In Table 7, we notethat the social signals that quantify reputation carrypositive opinions, for example, bookmark a resourcelink by a user on Delicious means that this resourcehas been added to his favourites list. Concerning likeand rating, user clicks on these buttons to indicate thathe has enjoyed the resource content. So the presenceof these social signals in resource increases the degree

8http://social-book-search.humanities.uva.nl/#/data9https://inex.mmci.uni-saarland.de/tracks/dc/2011/

of resource reputation. The same applies for popular-ity, the exploited social signals to estimate it, let usknow the position of this resource on the web in termsof trend and propagation. Finally, to quantify freshnessthe dates of the different actions are not available ex-cept the dates of each rating from SBS dataset.

4.3. Result of Linear Combination Study

We conducted experiments with models based onlyon the contents of documents, as well as approachescombining content and social data. Normalized for-mula (5) is the weighted sum of social relevance andtopical relevance:

Rel(q, r,G) = α·RelT (q, r)+(1−α)·RelS (q, r,G) (5)

Where: α ∈ [0, 1] as a weighting parameter andRelT (q, r) is the normalized score of topical relevance.We used BM25 [46] and Lucene Solr scoring10 asbaselines models for our study. Lucene Solr scoringuses a combination of the Vector Space Model (VSM)and the Boolean model.

In this paper, we evaluate the contribution of eachsocial signal/social property and the effect of theircombination on relevance. We first select the best pa-rameters α (see formula 5) and β, λ, δ (see formula 4)by applying J48 learning algorithm, then we compareour approach with baselines. We note that if α = 0only the social relevance is taken into account. More-over, α = 1 corresponds to the baselines textual mod-els. The best values of parameters are the following:α ∈ [0.5, 0.6] with β ∈ [0.1, 0.2], λ ∈ [0.3, 0.5], δ ∈[0.4, 0.6] for P@10 and P@20.

Tables 8 and 9 summarize the results of precision(P@10 and P@20), nDCG and MAP for IMDb andSBS datasets, respectively. We evaluated different con-figurations, by taking into account social signals indi-vidually and their combination in the form of socialproperties. the “Freshness" configuration is only esti-mated for the SBS collection where we have the ap-pearance time of each rating on a document. In order tocheck the significance of the results, we performed theStudent test [23] and attached * (strong significanceagainst BM25) and ** (very strong significance againstBM25) to the performance number of each row in thetables 8 and 9 when the p-value < 0.05 and p-value <0.01 confidence level, respectively.

10http://lucene.apache.org/solr/


Table 3List of the different IMDb document metadata fields

Field Description Status Field Description StatusID Identifying the film (document) - Director Director of the film project IndexedTitle Film’s title Indexed Writer Writers and writers of the film IndexedYear Year of the film release Indexed Actors Main actors of the film IndexedRated Film classification by content type - Plot Text summary of the film IndexedReleased Date of making the film Indexed Poster URL of the link poster -Runtime Length of the film Indexed url URL of the web source document -Genre Film genre (Action, Drama, etc.) Indexed UGC Social signals recovered -

Table 4Statistics on the number of signals in the IMDb documents returned by 30 topics

Network Facebook Google+ Delicious Twitter LinkedInSignal Like Share Comment +1 Bookmark Tweet Share

Minimum 0 0 0 0 0 0 0Maximum 76842 43918 62281 1475 986 12223 299880

Total 2478498 2718918 2845169 73392 26143 499232 42787Average 85.8027 94.1258 98.4964 2.5407 0.9050 17.2830 1.4812

Table 5List of the different SBS document fields

Field Status Field Status Field Status Field Statusbook Index similarproducts - title Index imagecategory -

dimensions - tags Index edition Index name Indexreviews Index isbn Index dewey - role Index

editorialreviews Index ean Index creator Index blurber -images - binding Index review Index dedication Indexcreators Index label Index rating - epigraph Indexblurbers - listprice - authorid - firstwordsitem Index

dedications Index manufacturer Index totalvotes - lastwordsitem -epigraphs Index numberofpages - helpfulvotes - quotation Indexfirstwords Index publisher Index date Index seriesitem -lastwords Index height - summary Index award -quotations Index width - editorialreview Index browseNode -

series Index length - content Index character -awards - weight - source Index place Index

Table 6Statistics on the number of signals in the SBS documents returned by 208 topics

Network Facebook AmazonSignal Like Share Comment Review Rating Tag

Minimum 0 0 0 0 0 0Maximum 7213 5892 5975 3378 3378 277

Total 4760698 5222491 5465011 23856118 23856118 1811968Average 41.3082 45.3152 47.4195 206.9981 206.9981 15.7223


Table 7Exploited social signals in the quantification

Property ci Social Signal Source Dataset

Popularity

c1 Number of Comment Facebook SBS, IMDbc2 Number of Tweet Twitter IMDbc3 Number of share(LIn) LinkedIn IMDbc4 Number of share Facebook SBS, IMDbc5 Number of Review Amazon/LibraryThing SBSc6 Number of Tags Amazon/LibraryThing SBS

Reputation

c7 Number of Like Facebook SBS, IMDbc8 Number of Mention +1 Google+ IMDbc9 Number of Bookmark Delicious IMDbc10 Average of Rating Amazon/LibraryThing SBS

Freshness c11 Dates of Ratings Amazon/LibraryThing SBS

Table 8Comparing social search effectiveness to Solr and BM25 (On IMDb)

P@10 P@20 nDCG MAPBM25 0.2912 0.2276 0.3158 0.1733Lucene Solr 0.2617 0.1951 0.2744 0.1590

Like 0.3963∗∗ 0.3142∗∗ 0.4873∗∗ 0.2339∗∗

Share 0.4037∗∗ 0.3119∗∗ 0.4911∗∗ 0.2428∗∗

Comment 0.3855∗∗ 0.3004∗∗ 0.4103∗∗ 0.2141∗∗

Tweet 0.3488∗∗ 0.2613∗∗ 0.4096∗∗ 0.1882∗∗

Mention+1 0.3206∗∗ 0.2467∗∗ 0.3627∗∗ 0.1809∗∗

Share(LIn) 0.3177∗ 0.2411∗ 0.3509∗ 0.1739∗

Bookmark 0.3358∗ 0.2395∗ 0.3517∗ 0.1730∗

All Criteria 0.4225∗∗ 0.3702∗∗ 0.4974∗∗ 0.2701∗∗

Reputation 0.4513∗∗ 0.3961∗∗ 0.5478∗∗ 0.2671∗∗

Popularity 0.4167∗∗ 0.3622∗∗ 0.5510∗∗ 0.2612∗∗

All Properties 0.4984∗∗ 0.4372∗∗ 0.5971∗∗ 0.2916∗∗

Table 9Comparing social search effectiveness to Solr and BM25 (On SBS)

P@10 P@20 nDCG MAPBM25 0.0601 0.0517 0.1581 0.0540Lucene Solr 0.0528 0.0487 0.1300 0.0463

Like 0.0768∗∗ 0.0600∗∗ 0.1776∗∗ 0.0652∗∗

Share 0.0813∗∗ 0.0622∗∗ 0.1812∗∗ 0.0784∗∗

Comment 0.0701∗∗ 0.0588∗∗ 0.1719∗∗ 0.0613∗∗

Tag 0.0642∗ 0.0471∗ 0.1660∗ 0.0532∗

Rating 0.0672∗∗ 0.0442∗∗ 0.1654∗∗ 0.0622∗∗

All Criteria 0.0883∗∗ 0.0699∗∗ 0.1893∗∗ 0.0820∗∗

Freshness 0.0853∗∗ 0.0644∗∗ 0.1816∗∗ 0.0795∗∗

Reputation 0.0884∗∗ 0.0713∗∗ 0.1886∗∗ 0.0820∗∗

Popularity 0.0876∗∗ 0.0692∗∗ 0.1865∗∗ 0.0801∗∗

All Properties 0.0932∗∗ 0.0800∗∗ 0.1912∗∗ 0.0843∗∗

We observe in all cases, with taking into account so-cial features, the results are significantly better com-pared to textual models. Also, it is clear that combiningsocial signals as properties (popularity and reputation)provides better results than when they are taken indi-

vidually. The results show that reputation configura-tion provides better results than popularity. The fresh-ness in our study is related to the recency of actionsassociated with a resource. The resources that possessfresh signals are promoted in search results list. Theoverall combination of social properties provides thebest results. According to Student test, majority of theresults show a strong statistically significant improve-ment.

In general, experimental results reflect the effective-ness of social signals on search task. More specifically,the results show that the way we have combined so-cial signals to quantify different properties is effec-tive to improve precision and nDCG. Therefore, com-bination of freshness with popularity and reputation(named “All Properties" in tables 8 and 9) provides thebest improvement compared to a random combinationof “All Criteria".

We can explain these results by the positive senseof reputation property quantified through the countingof signals such as like, mention+1, bookmark and rat-ing, which means favourable and positive opinion forthe resource judgment. Social networks urge users toshare, comment, evaluate and disseminate the infor-mation on a large scale. These interactions allow us todraw conclusions about the social position and qualityof these resources in social networks across their pop-ularity, reputation and freshness. Therefore, we canalso explain our results by the high rate of user’s en-gagements on various social networks, which bringstogether more than a billion users, producing users’massive interactions "wisdom of crowds" with web re-sources through these social actions of different na-tures, often positives.


4.4. Identify the Most Effective Signals

In this section, we conducted a series of experimentsin a supervised environment, using machine learningalgorithms with the set of effective social signals iden-tified in table 7. The aim is twofold: on the one handwe wondered whether the attribute selection really im-proves the results of a search. On the other hand, weintended to measure the performance of some learningalgorithms in this type of classification.

4.4.1. MethodologyRemoving the irrelevant and redundant features

from the data helps to improve the performance oflearning models. Following are the most well-knownfeature selection algorithms:

– CfsSubsetEval: CFS, short for "Correlation-based Feature Subset Selection", evaluates theworth of a subset of attributes by considering theindividual predictive ability of each feature alongwith the degree of redundancy between them.Subsets of features that are highly correlated withthe class while having low intercorrelation arepreferred [26].

– WrapperSubsetEval: Evaluates attribute sets byusing a learning scheme. Cross validation is usedto estimate the accuracy of the learning schemefor a set of attributes [38].

– ConsistencySubsetEval: Evaluates the worth ofa subset of attributes by the level of consistencyin the class values when the training instances areprojected onto the subset of attributes. Consis-tency of any subset can never be lower than thatof the full set of attributes, hence the usual prac-tice is to use this subset evaluator in conjunctionwith a Random or Exhaustive search which looksfor the smallest subset with consistency equal tothat of the full set of attributes [33].

– FilteredSubsetEval: Class for running an arbi-trary subset evaluator on data that has been passedthrough an arbitrary filter (note: filters that alterthe order or number of attributes are not allowed).Like the evaluator, the structure of the filter isbased exclusively on the training data [27].

– ChiSquaredAttributeEval: Evaluates the worthof an attribute by computing the value of the chi-squared statistic with respect to the class [32].

– FilteredAttributeEval: Class for running an ar-bitrary attribute evaluator on data that has beenpassed through an arbitrary filter (note: filters thatalter the order or number of attributes are not al-

lowed). Like the evaluator, the structure of the fil-ter is based exclusively on the training data [27].

– GainRatioAttributeEval: Evaluates the worth ofan attribute by measuring the gain ratio with re-spect to the class [27].

GainR(Class, Attribute) =

(H(Class)− H(Class|Attribute))H(Attribute)

(6)

While: H specifies the entropy. Entropy is a mea-sure of the uncertainty associated with a randomvariable.

– InfoGainAttributeEval: Evaluates the worth ofan attribute by measuring the information gainwith respect to the class [27].

GainR(Class, Attribute) =

(H(Class)− H(Class|Attribute))(7)

– OneRAttributeEval: OneR, short for "One Rule",is a simple classifier that generates one-level de-cision tree. OneR evaluates the worth of an at-tribute by using the OneR classifier. OneR classi-fier used crossvalidation to estimate the accuracyof the learning scheme for a set of attributes [29].It combines the C4.5 decision tree and Gaussiandistribution.

– ReliefFAttributeEval: Evaluates the worth of anattribute by repeatedly sampling an instance andconsidering the value of the given attribute for thenearest instance of the same and different class.Can operate on both discrete and continuous classdata [40].

– SVMAttributeEval: Evaluates the worth of anattribute by using an SVM classifier. Attributesare ranked by the square of the weight assignedby the SVM. Attribute selection for multiclassproblems is handled by ranking attributes for eachclass seperately using a one-vs-all method andthen "dealing" from the top of each pile to give afinal ranking [25].

– SymmetricalUncertAttributeEval: Evaluates theworth of an attribute by measuring the symmetri-cal uncertainty with respect to the class [26].


S ymmU(Class, Attribute) =

2 ∗ (H(Class)− H(Class|Attribute))H(Class) + H(Attribute)

(8)

In this study, we relied on algorithms for selectingattributes to determine the best social signals to ex-ploit in the learning model. Feature selection Algo-rithms [27] aim to identify and eliminate as many ir-relevant and redundant information as possible. Weused Weka11 for this experiment. It is a powerful open-source Java-based learning tool that brings together alarge number of learning machines and algorithms forselecting attributes.

We proceeded as follows: the top 1000 documentsfor each topic from the two collections (30 IMDb top-ics and 208 SBS topics) were extracted using the de-fault Lucene Solr model. Then, the scores of all cri-teria (social signals) are calculated for each resource.We identified relevant documents and irrelevant doc-uments according to the Qrels. The resulting set con-tains 30000 documents for IMDb and 115248 docu-ments for SBS, including:

– 2765 relevant documents and 27235 irrelevantdocuments for IMDb.

– 2953 relevant documents and 112295 irrelevantdocuments for SBS.

We observed that this collection has an unbalancedrelevance classes distribution. This occurs when thereare many more elements in one class than in the otherclass of a training collection. In this case, a classi-fier usually tends to predict samples from the ma-jority class and completely ignore the minority class[48]. For this reason, we applied an approach to sub-sampling (reducing the number of samples that havethe majority class) to generate a balanced collectioncomposed of:

– 2765 relevant documents and 2765 irrelevant doc-uments for IMDb.

– 2953 relevant documents and 2953 irrelevant doc-uments for SBS.

Irrelevant documents for this study were selectedrandomly. Finally, we applied the attribute selection al-gorithms on the two sets obtained, for 5 iterations ofcross-validation.

11http://www.cs.waikato.ac.nz/ml

We note that these algorithms operate differently,some return an importance ranking of attributes (e.g.,FilteredAttributeEval), while others return the numberof times that a given attribute has been selected byan algorithm in a cross-validation (e.g., FilteredSub-setEval). In the following, we present the results ob-tained by each selection algorithm (with and withoutconsidering the time) applied on INEX SBS and IMDbdatasets. We note that we have used for each algorithmthe default setting provided by Weka.

Our goal of this study is to determine the most im-portant signals for IR task and verify if the results ob-tained previously (Table 8) are consistent. In our case,the selection algorithms are to give a score to each sig-nal based on its significance towards relevance classof the document (relevant or irrelevant). For IMDb(see Table 10), we evaluated 7 criteria where n = 7,and for SBS (see Table 11) we evaluated 6 criteria.We applied a cross-validation for 5 iterations (5 cross-validation folds). Table 10 shows the social signals se-lected through attributes selection algorithms. We usetwo types of these algorithms: a) those using rankingmethods to order the selected criteria (metric in thetables is [Rank]); and b) those using search methodsthat indicate how many times the criterion has been se-lected during the cross-validation task (metric in tablesis [Folds Number]). A social signal strongly preferredby the selection algorithm is a well-ranked signal (rank= 1) and strongly selected (folds number = 5).

According to Table 10, we remark that the Face-book signals c1: Comment, c4: Share and c7: Like arethe most selected and highly ranked compared to othersignals. Signals c2: Tweet and c8: +1 are also favoredby the attributes selection algorithms, except the algo-rithm WrapperSubsetEval that selected them only onetime among 5 iterations, and the algorithms ReliefFAt-tributeEval, SVMAttributeEval that ranked c2: Tweetin 7 and 6, respectively. Finally, the most weakest andmost disadvantaged signals are c3: Share(LIn) and c9:Bookmark. These results are correlated with the ob-tained results in Table 8.

Similarly, on SBS dataset, table 11 shows that Face-book signals c1: Comment, c4: Share and c7: Like arealways highly ranked and often validated during the5 iterations of cross-validation. In addition, the sig-nal c11: rating biased by its freshness is strongly fa-vored (average ranks = 2 and average selection = 5).This last result proves the interest of the freshness thatwe propose, i.e. resources with fresh signals should beboosted in the top list of search results. The signalsc6: Tag and c10: average of rating (without consider-


Table 10Selected signals with attribute selection algorithms (IMDb dataset)

Algorithm Metric c++1 c+2 c3 c++

4 c++7 c+8 c9

CfsSubsetEval [folds Number] 5 5 - 5 5 2 -

WrapperSubsetEval [folds Number] 1 1 1 4 5 1 3

ConsistencySubsetEval [folds Number] 5 5 5 5 5 5 4

FilteredSubsetEval [folds Number] 5 5 - 5 5 2 -

ChiSquaredAttributeEval [Rank] 3 4 6 1 2 5 7

FilteredAttributeEval [Rank] 3 4 6 1 2 5 7

GainRatioAttributeEval [Rank] 1 4 7 2 3 5 6

InfoGainAttributeEval [Rank] 3 4 6 1 2 5 7

OneRAttributeEval [Rank] 2 4 6 3 1 5 7

ReliefFAttributeEval [Rank] 3 7 5 1 2 4 6

SVMAttributeEval [Rank] 5 6 2 1 4 3 7

SymetricalUncertEval [Rank] 1 4 6 2 3 5 7

Count 12 12 10 12 12 12 10

Table 11Selected signals with attribute selection algorithms (SBS dataset)

Algorithms Metric c++1 c++

4 c+6 c++7 c+10 c++

11

CfsSubsetEval [folds number] 5 5 - 5 - 5

WrapperSubsetEval [folds number] 1 5 2 4 4 4

ConsistencySubsetEval [folds number] 4 5 3 5 4 5

FilteredSubsetEval [folds number] 4 5 - 5 - 5

ChiSquaredAttributeEval [rank] 4 1 6 2 5 3

FilteredAttributeEval [rank] 4 1 6 2 5 3

GainRatioAttributeEval [rank] 3 1 6 4 5 2

InfoGainAttributeEval [rank] 4 2 6 3 5 1

OneRAttributeEval [rank] 4 2 6 3 5 1

ReliefFAttributeEval [rank] 4 1 6 3 5 2

SVMAttributeEval [rank] 4 1 6 3 5 2

SymetricalUncertEval [rank] 4 2 6 3 5 1

Count 12 12 10 12 10 12

Fig. 3. Machine Learning Process


ing its creation dates) are weakly preferred (favored)by the selection algorithms, with average ranks of 6,5, respectively, and the average selection of 1.25, 2,respectively. By comparing these results with the re-sults listed in Table 9, we noticed that the same sig-nals highlighted by the linear combination method arehighlighted by the learning techniques.

4.5. Learning Signals for Predicting Relevance

We also conducted a series of experiments exploit-ing these signals in supervised approaches based onlearning models. We used the results returned byLucene Solr using all the queries from the two INEXcollections (IMDb and SBS), each separately, as train-ing sets. We then used three learning algorithms, thischoice being explained by the fact that they oftenshowed their effectiveness in IR by exploiting criteriaof relevance: SVM [34], J48 (C4.5 implementation)[44] and Naive Bayes [49]. The input of each algorithmis a vector of social signals, either all the signals or justthe signals selected by a precise selection algorithm.Each signal is represented by its quantity in the docu-ments. Learning algorithms predict the relevance classfor each document (relevant or irrelevant). We applieda cross validation for 5 iterations (5 cross-validationfolds). Figure 3 illustrates the learning process we usedfor evaluating social signals.

We recall that the phase of attribute selection algo-rithms has highlighted two sets of signals:

1. in the case of the CfsSubsetEval and FilteredSub-setEval algorithms, the selected signals are c1:comment, c4: share, c7: like and c11: rating (withconsidering its time) for SBS and c1: comment,c2: tweet, c4: share, c7: like, c8: +1 and c10: rat-ing (without considering its time) for IMDb.

2. in the case of the other selection algorithms, allthe social signals studied on the two collectionsare selected: c1, c4, c6, c7, c10 and c11 for theSBS collection and c1, c2, c3, c4, c7, c8 and c9for the IMDb collection.

The question at this stage relates to the specifica-tion of the input signal vector for the learning algo-rithms, either we take all the signals, or we keep onlythose selected by the attribute selection algorithms. Inthis case, with which learning algorithms these will becombined.

In order to take into account the signals chosen bythe selection algorithms in learning models, we reliedon the work of Hall and Holmes [27].

Hall and Holmes [27] have studied the effectivenessof some attribute selection techniques by confrontingthem with learning techniques. Since the performanceof the factors differs from one learning technique toanother, they have identified the best attribute selectiontechniques to find the best performing factors accord-ing to the learning techniques to be used.

Based on their study, we used the same pairs oflearning techniques and attribute selection techniques:

– signals selected by WrapperSubsetEval (WRP)are learned by Naive Bayes.

– signals selected by CfsSubsetEval (CFS) arelearned by Naive Bayes.

– signals selected by ReliefFAttributeEval (RLF)are learned by J48.

– signals selected by SVMAttributeEval (SVM) arelearned by SVM.

Table 12Machine learning results (P@20) on IMDb

Classifiers Attribute selection criteria All criteria

NaiveBayes0.4927 (CFS)

0.48020.4802 (WRP)

SVM 0.4874 (SVM) 0.4874

J48 0.5562 (RLF) 0.5562

Table 13Machine learning results (P@20) on SBS

Classifiers Attribute selection criteria All criteria

NaiveBayes0.1223 (CFS)

0.11000.1100 (WRP)

SVM 0.1114 (SVM) 0.1114

J48 0.1301 (RLF) 0.1301

Tables 12 and 13 present the results of the threelearning algorithms of signals that emerged from thestudy using attribute selection techniques. We find thatonly the CFS algorithm confirms the hypothesis putforward by Hall and Holmes. Indeed, it is the onlyone for which the results obtained with the selectionof attributes, are 0.4927 (IMDb) and 0.1223 (SBS),exceed the use of all the signals, 0.4802 (IMDb) and0.1100 (SBS). We have shown that machine learn-ing approaches have better efficiency (precision) withattribute selection approaches. We then note that alllearning models outperform textual models (LuceneSolr model and BM25) as well as our first propositionsbased on the linear combination approach. We finallyfind that the J48 decision tree is the most appropriatemodel, it takes into consideration all the social signals,


the improvement rates compared to Naive Bayes andSVM are 13% and 15% on IMDb as well as 6% and17% on SBS, respectively. In addition, the J48 givesthe best improvements over all previous approaches,the improvement rate compared to the default modelBM25 (configuration named Lucene Solr in Tables 8and 9) is 144% for the IMDb collection and 152% forthe SBS collection, while comparing to the best re-sults obtained by the model based on linear combina-tion (configuration named “All Properties" in Tables 8and 9) is 27% for the IMDb collection and 63% for theSBS collection.

Finally, all these experiments clearly show that so-cial signals allow to enhance a search. These improve-ments show the interest of social relevance, knowingthat qualitative properties (popularity and reputation)and the temporal property (freshness) provide a signif-icant improvement compared to the configuration ig-noring these properties (textual model only or signalsteken individually). We observe that the resources hav-ing more positive data (e.g., like, +1, rating) are trust-worthy than the ones don’t possess these social signals.If multiple users have found that the resource is use-ful, then it is more probable that other users will findthese resources useful too. After these experiments, weobserve that learning models are much more suitablethan linear combination on exploiting of this type ofsocial signals to enhance a search. We can say that theJ48 learning model with selection attribute algorithmimproves a precision of search results significantly.

4.6. Ranking Correlation Analysis

In order to analyze social signals and determine ifthere is a link (dependence / independence) betweenthem and the document relevance, thus that betweenthem in pairs, we conducted a correlation study. Ourgoals are as follows:

– first, determine the social signals that are corre-lated with the relevance, and facilitate the inter-pretation of the results.

– second, determine the redundant signals, andthose that have a same effect on the retrieval im-provement.

4.6.1. Correlation Between Signals and RelevanceAccording to a June 2014 study from Search-

metrics12, among 22 ranking factors identified, social

12www.searchmetrics.com/knowledge-base/ranking-factors-infographic-2014/

signals account for 5 of the 6 most highly correlatedwith Google search results. In addition, BrightEdge13

survey released in 2013, 84% of search marketers saysocial signals such as like, tweet, and mention +1 willbe either more important (53%) or much more impor-tant (31%) to their SEO (Search Engine Optimization)compared to previous years.

Social signals continue to become more and morea highly correlated factor with the results of Google.Although we did not see a lot of scientific studies onthese signals, some marketing organizations such asSearchmetrics continue to analyze them. According tothe 2015 Searchmetrics study [47], the correlations ofsocial signals rankings are practically unchanged com-pared to 2014 and remains at a high level. The first re-sults returned by Google contain more social signals,this factor increases exponentially in the first places.Figure 4 shows the 2015 results of the correlation be-tween social signals and Google results.

We analyzed the ranking correlation between sig-nals and relevance using a correlation coefficient ofSpearman’s Rho (rs), that measures the agreement be-tween each signal value and document relevance class[19]. If the value of Rho is positive, we can say thatthere is some correlation between the two variables.The value of Rho is between [−1, 1], more the Spear-man Rho is close to 1, more the relation is strong andvice versa [19].

Figure 5 shows the values of correlations betweenranges social signals (individually and grouped asproperties: popularity and reputation) with respect todocuments relevance. This study shows that Facebooklike (0.29) has the highest correlation among the otherindividual signals, followed by number of Fecebookcomment (0.28). Other high-ranking factors includeFacebook share (0.27) and tweet (0.23). Concerningthe signals grouped as properties as well as the totalof Facebook signals, are the most correlated with rel-evance compared to signals taken individually. How-ever, the popularity generates the highest correlationcompared to the reputation and total Facebook signals.

Finally, the ranking correlation analysis shows thatall social signals are positively correlated with rele-vance. This study justifies our hypothesis and the re-sults obtained above (see table 8) and confirms the in-terest of social signals exploited: Well positioned re-sources have a high number of like, share and specificresources stand out in the top search results with a very

13http://www.marketingcharts.com/uncategorized-20695


Fig. 4. Correlation between social signals and Google search results

Fig. 5. Rho Correlations between social signals and documents relevance

Table 14Spearman’s Rho correlation values for the social signals pairs

Signals Like S hare Comment S hare(LIn) Tweet Bookmark +1

Like 1S hare 0.61 1

Comment 0.31 0.26 1S hare(LIn) 0.35 0.41 0.40 1

Tweet 0.32 0.28 0.39 0.77 1Bookmark 0.34 0.48 0.51 0.31 0.76 1

Google + 1 0.34 0.61 0.40 0.32 0.30 0.71 1

high mass of social data. On the one hand, this meansthat the activity on social networks continues to in-crease, on the other hand, it means that the frequentlyliked or shared content is increasingly correlated withgood ranking of relevance.

4.6.2. Pair-wise Correlation Between Social SignalsTo examine the linear relationship for each pair of

social signals, we compute the pairwise overlap be-tween the features by averaging the similarity of theirtop-1000 rankings over all queries. Atypical methodfor measuring the similarity of two ranked lists is us-ing the Spearman’s Rho metric. The more Rho is close

to 1 (in absolute value), the more the relation is strongand vice-versa.

In table 14, we provide the Spearman’s Rho scoresthat are normalized to [0,1] range where 0 means com-pletely different rankings and 1 means equal rankings.The lower diagonal of the table presents the corre-lation of social signals based on the rankings for allqueries. We find that, the top-1000 rankings providedby the social signals pairs (tweet, share(LIn)), (book-mark, tweet) and (mention +1, bookmark) are highlycorrelated, i.e., the similarity scores of these pairs arehigher than 0.70 (see table 14). These correlations be-tween social signals imply some redundancy, at least


for the purposes of ranking. These observations justifyand confirm the results obtained by using feature selec-tion approach to filter and rank such redundant socialsignals. In this study, social signals: bookmark, share(LIn) are the less important criteria followed by +1.

Finally, this is a preliminary correlation study, weare well aware that further reflection to better addressthese issues is needed.

5. Conclusion

This paper proposes a search model exploiting so-cial signals. These signals (User Generated Content),collected from several social networks, can quantifysome social properties such as popularity, reputationand freshness. The proposed model combines linearlytwo relevance scores: (1) topical, estimated using clas-sical IR model and (2) social, estimated using somesocial features, popularity, reputation and the fresh-ness of resources. Experimental evaluation conductedon two INEX datasets IMDb and SBS shows that theintegration of social signals and their properties withina textual search model allows to improve the qualityof the search results. Our evaluations using attributesselection algorithms and three state-of-the-art learningalgorithms support our hypothesis: the rankers basedon the social signals, including both the popularity,the freshness and the reputation outperform those builtby using only basic textual features. We found thatJ48 brings the best improvement in terms of effec-tiveness compared to baseline and all our other pro-posed configurations. Analyzing ranking correlations,we note that all social signals present a positive cor-relation. Meanwhile, this correlation agreement justi-fies the significant improvement for our proposed so-cial approach.

For future research, we plan to address some limi-tations of the current study. We plan to integrate othersocial data into a proposed approach (emotions, eventreactions, etc.). Also, we plan to study the importanceof social networks and social actors of these signalsand their impact on the relevance. Further experimentson other types of collections are also needed. This re-quires tracking usersâAZ personal profiles as well asthose of their followers and those of users they share,like, rate,tweet, etc. We intend to collect these data inthe future to evaluate the user preferences, comparedto social neighbors, to solve the personalized search.This is even with these simple elements, the first re-sults encourage us to invest more this track.

References

[1] F. Abel, N. Henze, D. Krause, and M. Kriesell. On the Effectof Group Structures on Ranking Strategies in Folksonomies,pages 275–300. Springer Berlin Heidelberg, 2009.

[2] R. Agrawal, B. Golshan, and E. Papalexakis. Whither socialnetworks for web search? In Proceedings of the 21th ACMSIGKDD International Conference on Knowledge Discoveryand Data Mining, KDD ’15, pages 1661–1670, New York, NY,USA, 2015. ACM.

[3] K. Albishre, Y. Li, and Y. Xu. Query-based automatic trainingset selection for microblog retrieval. In Advances in Knowl-edge Discovery and Data Mining - 22nd Pacific-Asia Con-ference, PAKDD 2018, Melbourne, VIC, Australia, June 3-6,2018, Proceedings, Part II, pages 325–336, 2018.

[4] O. Alonso and V. Kandylas. A study on placement of socialbuttons in web pages. CoRR, abs/1410.2828, 2014.

[5] I. Badache. RI sociale: intégration de propriétés socialesdans un modèle de recherche. In Conférence francophone enRecherche d’Information et Applications - CORIA 2013, pagespp. 1–6, Neuchâtel, Switzerland, April 2013.

[6] I. Badache. Recherche d’information sociale : exploitation dessignaux sociaux pour améliorer la recherche d’information.(Social information retrieval : harnessing social signals toenhance a search). PhD thesis, Paul Sabatier University,Toulouse, France, 2016.

[7] I. Badache. Exploring differences in the impact of users’ traceson arabic and english facebook search. In IEEE/WIC/ACMInternational Conference on Web Intelligence, WI ’19, pages225–232, New York, NY, USA, 2019. ACM.

[8] I. Badache. Users’ traces for enhancing arabic facebook search.In Proceedings of the 30th ACM Conference on Hypertext andSocial Media, HT ’19, pages 241–245, NY, USA, 2019. ACM.

[9] I. Badache, A. Abu-Thaher, M. Hamdan, and L. Abu-Jaish. So-cial Information Retrieval in Arabic Language: Case of Face-book. In COnférence en Recherche d’Information et Applica-tions, Lyon, France, March 2019.

[10] I. Badache and M. Boughanem. Exploitation de signaux so-ciaux pour estimer la pertinence a priori d’une ressource. InConférence francophone en Recherche d’Information et Appli-cations (CORIA), pages pp. 163–178, Nancy, France, 2014.

[11] I. Badache and M. Boughanem. Harnessing social signals toenhance a search. In Proceedings of the 2014 IEEE/WIC/ACMInternational Joint Conferences on Web Intelligence (WI) andIntelligent Agent Technologies (IAT) - Volume 01, WI-IAT ’14,pages 303–309, Washington, DC, USA, 2014. IEEE ComputerSociety.

[12] I. Badache and M. Boughanem. Social priors to estimate rel-evance of a resource. In Proceedings of the 5th InformationInteraction in Context Symposium, IIiX ’14, pages 106–114,New York, NY, USA, 2014. ACM.

[13] I. Badache and M. Boughanem. Pertinence a Priori Basée surla Diversité et la Temporalité des Signaux Sociaux. In Con-férence francophone en Recherche d’Information et Applica-tions (CORIA), pages pp. 23–38, Paris, France, 2015.

[14] I. Badache and M. Boughanem. A priori relevance based onquality and diversity of social signals. In Proceedings of the38th International ACM SIGIR Conference on Research andDevelopment in Information Retrieval, SIGIR ’15, pages 731–734, New York, NY, USA, 2015. ACM.


[15] I. Badache and M. Boughanem. Emotional social signals forsearch ranking. In Proceedings of the 40th International ACMSIGIR Conference on Research and Development in Informa-tion Retrieval, SIGIR ’17, pages 1053–1056, New York, NY,USA, 2017. ACM.

[16] I. Badache and M. Boughanem. Les Signaux Sociaux émotion-nels : Quel impact sur la recherche d’information ? In COn-férence en Recherche d’Information et Applications (CORIA2017), Marseille, France, March 2017.

[17] S. Bao, G. Xue, X. Wu, Y. Yu, B. Fei, and Z. Su. Optimizingweb search using social annotations. In Proceedings of the 16thinternational conference on World Wide Web, pages 501–510.ACM, 2007.

[18] L. Ben Jabeur. Leveraging social relevance: Using social net-works to enhance literature access and microblog search. PhDthesis, Université de Toulouse, Université Toulouse III-PaulSabatier, 2013.

[19] S. D. Bolboaca and L. Jantschi. Pearson versus spearman,kendall’s tau correlation analysis on Structure-Activity rela-tionships of biologic active compounds. Leonardo Journal ofSciences, 5(9):179–200, 2006.

[20] F. Bravo-Marquez, E. Frank, and B. Pfahringer. Transferringsentiment knowledge between words and tweets. Web Intelli-gence, 16(4):203–220, 2018.

[21] L. Chan, S. Jhaver, and S. Soni. Postscholar: Surfacing socialsignals in google scholar search. In Proceedings of the 19thACM Conference on Computer Supported Cooperative Workand Social Computing Companion, CSCW ’16 Companion,pages 17–20, New York, NY, USA, 2016. ACM.

[22] S. Chelaru, C. Orellana-Rodriguez, and I.S. Altingovde. Howuseful is social feedback for learning to rank youtube videos?World Wide Web, 17(5):997–1025, Sep 2014.

[23] W.S. Gosset. The probable error of a mean. Biometrika,6(1):1–25, 1908.

[24] L. Gou, X. Zhang, H. Chen, J. Kim, and C.L. Giles. Socialnetwork document ranking. In Proceedings of the 10th AnnualJoint Conference on Digital Libraries, JCDL ’10, pages 313–322, New York, NY, USA, 2010. ACM.

[25] I. Guyon, J. Weston, S. Barnhill, and V. Vapnik. Gene selectionfor cancer classification using support vector machines. Ma-chine learning, 46(1-3):389–422, 2002.

[26] M.A. Hall. Correlation-based feature selection for machinelearning. PhD thesis, University of Waikato Hamilton, 1999.

[27] M.A. Hall and G. Holmes. Benchmarking attribute selectiontechniques for discrete class data mining. IEEE Trans. onKnowl. and Data Eng., 15(6):1437–1447, 2003.

[28] B. Hecht, J. Teevan, M.R. Morris, and D.J. Liebling. Search-buddies: Bringing search engines into the conversation. In Pro-ceedings of the Sixth International Conference on Weblogs andSocial Media, pages 138–145, 2012.

[29] R.C. Holte. Very simple classification rules perform well onmost commonly used datasets. Machine learning, 11(1):63–90, 1993.

[30] L. Hong, O. Dan, and B.D. Davison. Predicting popular mes-sages in twitter. In Proceedings of the 20th International Con-ference Companion on World Wide Web, WWW ’11, pages 57–58, 2011.

[31] A. Hotho, R. Jäschke, C. Schmitz, editor="Sure York Stumme,Gerd", and J. Domingue. Information retrieval in folk-sonomies: Search and ranking. In The Semantic Web: Re-search and Applications, 3rd European Semantic Web Confer-

ence, ESWC 2006, Budva, Montenegro, June 11-14, 2006, Pro-ceedings, pages 411–426, 2006.

[32] L. Huan and S. Rudy. Chi2: feature selection and discretizationof numeric attributes. In Proceedings of 7th IEEE InternationalConference on Tools with Artificial Intelligence, pages 388–391, Nov 1995.

[33] L. Huan and S. Rudy. A probabilistic approach to feature se-lection - A filter solution. In Machine Learning, Proceedingsof the Thirteenth International Conference (ICML ’96), Bari,Italy, July 3-6, 1996, pages 319–327, 1996.

[34] V. Jan, W.L. Kenneth, and N. Wilfred. Searching for qualitymicroblog posts: Filtering and ranking based on content anal-ysis and implicit links. In Database Systems for Advanced Ap-plications - 17th International Conference, DASFAA 2012, Bu-san, South Korea, April 15-19, 2012, Proceedings, Part I, pages397–413, 2012.

[35] B. Karweg, C. Huetter, and K. Böhm. Evolving social searchbased on bookmarks and status messages from social networks.In Proceedings of the 20th ACM International Conference onInformation and Knowledge Management, CIKM ’11, pages1825–1834, 2011.

[36] A. Khodaei and C. Shahabi. Social-textual search and ranking.In Proceedings of the First International Workshop on Crowd-sourcing Web Search, Lyon, France, pages 3–8, 2012.

[37] A. Khodaei, S. Sohangir, and C. Shahabi. Personalization ofWeb Search Using Social Signals, pages 139–163. SpringerInternational Publishing, Cham, 2015.

[38] R. Kohavi and G.H. John. Wrappers for feature subset selec-tion. Artificial intelligence, 97(1):273–324, 1997.

[39] W. Lewoniewski, R.C. Härting, K. Wecel, C. Reichstein, andW. Abramowicz. Application of seo metrics to determinethe quality of wikipedia articles and their sources. In Rober-tas Damaševicius and Giedre Vasiljeviene, editors, Informa-tion and Software Technologies, pages 139–152, Cham, 2018.Springer International Publishing.

[40] R.S. Marko and K. Igor. An adaptation of relief for attributeestimation in regression. In Proceedings of the Fourteenth In-ternational Conference on Machine Learning (ICML 1997),Nashville, Tennessee, USA, July 8-12, 1997, pages 296–304,1997.

[41] M.R. Morris and J. Teevan. Exploring the complementary rolesof social networks and search engines. In Microsoft Research,Human-Computer Interaction Consortium Workshop(HCIC),pages 1–10, 2012.

[42] P. Pantel, M. Gamon, O. Alonso, and K. Haas. Social annota-tions: Utility and prediction modeling. In Proceedings of the35th SIGIR Conference on Research and Development in In-formation Retrieval, pages 285–294, 2012.

[43] D.R. Pierce, D.P. Redlawsk, and W.W. Cohen. Social influ-ences on online political information search and evaluation.Political Behavior, 39(3):651–673, Sep 2017.

[44] J.R. Quinlan. C4. 5: programs for machine learning. Elsevier,2014.

[45] M. Raza. A new level of social search: Discovering the user’sopinion before he make one. Technical report, Cambridge, UK:Mi-crosoft Research Cambridge.

[46] S.E. Robertson and S. Walker. Some simple effective approx-imations to the 2-poisson model for probabilistic weighted re-trieval. In Proceedings of the 17th Annual International ACMSIGIR Conference on Research and Development in Informa-tion Retrieval, SIGIR ’94, pages 232–241, 1994.


[47] M. Tober, D. Furch, K. Londenberg, L. Massaron, andJ. Grundmann. Search ranking factors and rank correlations.SearchMetrics, 2015.

[48] S. Yen and Y. Lee. Under-sampling approaches for improv-ing prediction of the minority class in an imbalanced dataset.Intelligent Control and Automation, pages 731–740, 2006.

[49] Q. Yuan, G. Cong, and N.M. Thalmann. Enhancing naive bayeswith various smoothing methods for short text classification.In Proceedings of the 21st International Conference on WorldWide Web, WWW ’12 Companion, pages 645–646, New York,NY, USA, 2012. ACM.

[50] Y. Yusuke, A. Jatowt, N. Satoshi, and T. Katsumi. Can socialbookmarking enhance search in the web? In Proceedings ofthe 7th ACM/IEEE-CS Joint Conference on Digital Libraries,JCDL ’07, pages 107–116, New York, NY, USA, 2007. ACM.

[51] S. Zhang and N. Cabage. Does SEO matter? increasing class-room blog visibility through search engine optimization. In46th Hawaii International Conference on System Sciences,HICSS 2013, Wailea, HI, USA, January 7-10, 2013, pages1610–1619, 2013.

Learning social signals for predicting relevant search results

Documents

Transcript of Learning social signals for predicting relevant search results