Combating #Extremism: Twitter and

online terrorist content

Thesis for Bachelor of Liberal Arts and Sciences (Social Sciences Major)

Tilburg School of Humanities and Digital Sciences (TSHD)

University College Tilburg

Name: Alina Gabriela Andrei

ANR: 273897

Student no: u1273909

Supervisor: Dr. Marloes van Noorloos

Dr. Jan Hendrik Valgaeren Second reader

2

“With great power comes great responsibility”

Uncle Ben, The Spider-Man

3

Table of Contents

1. Introduction ...................................................................................................................................................... 4

1.1 Research Statement ...................................................................................................................................... 4

1.1.2 Methodology ............................................................................................................................................. 8

1.2 Relevance of the study .................................................................................................................................. 9

2. Theoretical Framework.................................................................................................................................. 10

2.1 Foucault and the idea of governmentality ................................................................................................. 10

2.2 Responsibilization Theory .......................................................................................................................... 12

3. The Role of the Internet and Social Media Companies in dealing with online illegal content ................. 18

3.1 The current state of the Internet and (some) dilemmas in the field of online crime control ...................... 18

3.2 Responsibilization of the Social Media Companies ................................................................................... 22

3.3 EU-level initiatives with regards to combating online terrorist content .................................................... 27

3.4 Developments in Twitter’s practices with regards to terrorist content ...................................................... 32

4. Conclusion ....................................................................................................................................................... 38

5. References ....................................................................................................................................................... 41

Online Sources cited: ....................................................................................................................................... 43

4

1. Introduction

1.1 Research Statement

All over the world, individuals rely on the internet to access instantaneous information. This

mainly happens because of the practicality of the internet, whereby almost effortlessly, a

broad selection of content is available just by means of a ‘click’. The wide variety of data on

the web can start from academic documents, news sources, podcasts and documentaries and

move towards more heinous content such as beheading videos, child pornography and other

types of unlawful, illegal content. As Almagor has put it in 2017 “The internet contains some

of the best products of humanity and some of the worst ones. It serves both positive and

negative elements in society” (Almagor, 2017, p.451). Therefore, it is not a surprise that

terrorists (and terrorist organizations) have found the internet to be a “godsend”, because it

offers the desired tools to undergo their recruitment and radicalization processes and

disseminate their content to the broad audience (Tsesis, 2017). The internet has become

highly favorized for sharing extremist- and terrorist content because it has the advantageous

trait of granting (almost) full anonymity of the users. More precisely, as Weimann said in

2010, “decentralized and providing perfect anonymity, it (the internet) cannot be subjected

to control or restrictions and can be accessed by anyone” (Weimann, 2010, p.45). Of course,

it is important to note that the above-mentioned traits of the internet are mainly related to

the Western world, where internet censorship is not in place as it is the case of China, for

example. Another characteristic of the internet in this respect is that the geographical

proximity between individuals does not matter, and communication can happen in real time

no matter where the users are situated. This communication can occur mainly via social media

companies, which are platforms that allow for instantaneous communication by means of

messages, videos, pictures and so on. Social media companies can be seen as the ‘gems’ of

the internet, because audiences world-wide can be reached in a matter of seconds and

information can easily become ‘viral’. These platforms convey the virtual space for debates

on topics that are very important for today’s society, such as for example democracy and

politics. Therefore, for these reasons (and of course, not limited to these), the traits of social

media companies are more special compared to traditional media and the internet.

The current study aims at explaining and comprehending the changing role of the social media

companies in today’s society, and how their responsibility with regards to the content shared

on their platforms has gradually changed over time. Social media companies have become an

important part of today’s society and their rapid growth led to increased responsibility on

their behalf, especially in detecting unlawful content (Almagor, 2017). This shift in

responsibility can be translated into the dilemma this thesis will delve into, namely that

traditionally speaking, states were considered to be in charge with combating any type of

expression that was perceived as a threat and was in violation of the state’s law, such as

incitement to terrorism and terrorist content and ultimately to set the balance between

freedom of speech and overall societal security. But with the growth in the popularity of social

5

media companies, and the fact that these platforms do play a role in disseminating terrorist

content, there is an increase call for the social media companies themselves to take

responsibility upon the content that is shared on their platforms. Therefore, what the current

study aims at, is providing a greater understanding of how the role of a social media company,

namely Twitter has developed over time, i.e. from not assuming any liability with regards to

the content shared on the platform and stressing that the users themselves are fully

responsible of what they decide to share and/or post online (Rules and Regulations Twitter,

2009) towards taking actions with regards to the content and implementing a zero tolerance

policy with respect to extremist and violent speech on their platform (Rules Regulations

Twitter, 2018). By conceptualizing on Garland’s (1996) responsibilization framework, the

present study aims at creating a bridge for further research to be conducted with respect to

the responsibility of today’s main social media companies with regards to the illegal content

that is to be found on their platforms.

While the focus of this thesis is centered around the social media platform of Twitter, a brief

description will be provided before introducing the main research question as well as the sub

questions that will be answered throughout this study. According to Kwak, H., Lee, C., Park,

H., & Moon, S. (2010) Twitter, who launched in 2006, is an instantaneous ‘microblogging’

social platform that allows for communication between users and is perceived to be a

powerful medium of information sharing. The idea of microblogging means that there is a

fixed number of 140 characters limit allowed per ‘tweet’, tweet that could evolve around any

topic and could be further read by both followers and non-followers. One other characteristic

of Twitter is that users can read, follow, re-tweet and so on messages by users that they do

not have in their ‘followers’ list, which further implies that anyone can access all profiles,

without knowing or ‘following’ that specific account per se. Moreover, Twitter is perceived as

a different social media platform compared to Facebook and YouTube for example, because

of its algorithms. Both Facebook and YouTube have algorithms that filter what each user is

seeing according to their interests, which is not the same for Twitter.

Therefore, with respect to the above-mentioned information, the research question of this

study is: “How can the developments in Twitter’s role with regards to combating online

terrorist content over time, be assessed in light of the responsibilization theory?”. The sub-

questions that will be answered in order to arrive at a pertinent answer for the main question

are:

1. What is the responsibilization theory?

2. How is it dealt with online illegal content on the internet and on social media

platforms in general?

From this second question, a following (sub) sub-question will be derived, namely what is the

role of the European Union in combating online illegal content?

6

3. What are the developments in Twitter’s practice with regards to online terrorist

content?

In order to have all the necessary information to fully be able to understand the topic of

interest and to research towards an adequate answer, it is of utmost importance to discuss

the terms of ‘terrorism’ and ‘terrorist content’. Providing a clear definition of terrorism might

seem far-fetched at this point in time, due to the sole reason that academics, heads of many

states and representatives of the main international organizations active nowadays were

unable for so many years to agree on a definition of terrorism. As accurately as it was pointed

out by Alex Schmid (2004), the main issue of terrorism in our contemporary society has to do

with the definitional problems that come along with this term. As the title of his paper,

“Terrorism - The Definitional Problem” (2004) suggests, the fact that for almost two hundred

years agreement on a definition was impossible to be met, portrays the complexity of this

term. But what can be certainly stated is that “terrorism may be the most important word in

the political vocabulary these days” (Schmid, 2004, p.376), fact that did not change for 15

years until today.

What Schmid (2004) further stresses, is that the definitional problem of terrorism and not

being able to find a consensus on what terrorism really is, is more political rather than legal.

He further gives four reasons why terrorism is difficult to define, which are:

1. “Because terrorism is a ‘contested concept’ and political, legal, social science and

popular notions of it are often diverging;

2. Because the definition question is linked to (de-)legitimization and criminalization;

3. Because there are many types of terrorism, with different forms and manifestations;

4. Because the term has undergone changes of meaning in the more than 200 years of

its existence” (Schmid, 2004, p.395).

Taking all these into consideration, a continuous debate will most likely still take place in the

future before achieving a common understanding on the definition of terrorism. Although

there are some agreements in various fields as to what terms shall be included when aiming

at defining this concept, there are disparities between the academic field, the legal one and

the (inter-)governmental definitions, but most of the common- agreed terms used in various

fields are ‘violent acts, intent to spread fear/terror, political character’ (Schmidt, 2004).

Moving towards explaining the second concept that is of great importance and crucial for the

present thesis, namely ‘online terrorist content’, the EU Directive 2017/541 that can be found

in the European Commission’s proposal, namely the regulation of the European Parliament

and of the Council in preventing the dissemination of terrorist content online will be used.

The definition of ‘online terrorist content’ should “capture material and information that

incites, encourages or advocates the commission or contribution to terrorist offences, provides

instructions for the commission of such offences or promotes the participation in activities of

7

a terrorist group. Such information includes in particular text, images, sound recordings and

videos” (EC, 2018, p.15).

8

1.1.2 Methodology

In terms of the methodology of this study, the research question and the sub-questions

derived from it will be answered by reviewing relevant academic literature from different

academic fields such as terrorism studies, criminology studies as well as sociology and

communication studies on the topic of interest as well as delving into reliable news sources

that could convey a broader perspective on the issue at hand. By using academic literature

from various fields of research, the current thesis will have an interdisciplinary background of

theories and literature to understand the current developments of the issue of interest. The

responsibilization theory, which is the core of the theoretical framework and main aspect of

the research question will be explained by means of the literature written by David Garland

(1996;2001), as he is the one who shed light on the concept of responsibilization. Garland’s

“The Limits of the Sovereign State. Strategies of crime control in contemporary society” (1996)

and “The Culture of Control: Crime and social order in contemporary society” (2001) will be

the means through which the theoretical background of this study will be conceptualized,

through a content analysis of his work and reviewing his writings.

Academic literature from terrorism- as well as communication studies will be used to

comprehend the changing role of social media companies, as well as the various standpoints

scholars have on the current dilemma that social media platforms are faced with, namely

taking upon the traditional role of the states of acting and consequently being responsible for

the content that is shared on the platforms. This dilemma will be partly answered by delving

into legal articles such as the European Commission’s proposal of 2018 and other EU-level

relevant documents (e.g. Europol documentation). After assessing the above-mentioned

aspects, the emphasis will be shifted towards the platform of Twitter specifically, in order to

assess their changing role with regards to the shared content on their platform. This will be

researched by means of reviewing academic articles on the topic of Twitter and terrorist

content online, as well as looking at Twitter’s annual reports and publicly available terms and

conditions and private policies available on the official website (Twitter.com), facts that will

be thoroughly explained in chapter 3, section 3.4. Moreover, reliable news sources will be

analyzed in order to gain a better perspective into Twitter’s practice with regards to

combating online terrorist content. The reason Twitter is the platform of choice for assessing

its developments with regards to combating the dissemination of terrorist content is its role

in the discussion about terrorist content online, and because, as research shows, Twitter is

among the preferred platforms for terrorists themselves to disseminate their content and

speech (Almagor, 2017; Jeong, 2016; Al-Rawi et al., 2018 etc.).

9

1.2 Relevance of the study

It is worthy to investigate the developments in Twitter’s role with regards to combating

terrorist content online, as nowadays social media platforms play a big role in the day-to-day

lives of millions of people. The present study is relevant from an academic point of view,

because there is little academic research done at this point in time with regards to the role of

a social media company (in this case, Twitter) in combating terrorist content online generally

speaking, let alone in relation with Garland’s (1996) Responsibilization Theory. Although a

great deal of literature nowadays deals with the pressing issue of terrorism, and consequently

terrorist speech on the online medium, the link between the developments of a social media

platforms along with the responsibilization theory with regards to combating terrorist

content has not been made yet.

This thesis therefore is relevant from an academic point of view because it presents a research

angle that has not been made yet and it provides a greater understanding of how the role of

Twitter has changed over time. As mentioned already, in 2009 Twitter was not taking any

public responsibility with regards to the content shared on the platform and was stressing

that the users are fully responsible of what they decide to make public on Twitter, towards a

different approach that is to be seen from 2018, where Twitter’s rules and regulations have

been adapted towards the implementation of a zero-tolerance policy with regards to

extremist and violent speech on their platform (Rules and Regulations Twitter, 2018).

Consequently, by means of Garland’s (1996) responsibilization theory, the present thesis aims

at setting the scene for further research to be conducted regarding the responsibility of

contemporary social media companies in their battle of combating illegal and terrorist

content online.

It is an important societal development to research into the responsibilization of the social

media companies when it comes to the content that is made publicly available. Today’s

society is gradually more and more digitally active, as ‘scrolling’ on social media platforms are

part of many individuals’ daily routines. Therefore, it is significant, at a societal level to discuss

how to deal with potentially harmful speech on these social media platforms and to assess

whether they could have a prospective negative impact on individuals. Moreover, for further

research it is important to mention that this topic will be relevant in the future as well, as

technology is in a continuous development and the trend is constantly increasing when it

comes to the use (and consequently, abuse) of the social media companies, generally

speaking.

10

2. Theoretical Framework

2.1 Foucault and the idea of governmentality

The idea of responsibility and responsibilization was theorized by many thinkers throughout

the past decades in order to fully assess its applicability and suitability to cotemporary crime

control. Of core interest to this thesis is the development of Garland’s (1996)

responsibilization theory and what he describes to be a (possible) effective strategy in crime

control. But before going in depth with Garland’s framework, Michel Foucault’s work with

respect to governmentality and responsibilization will be first brought to the fore. In his article

titled “’Governmentality’ and the problem of crime: Foucault, criminology, sociology” from

1997, Garland touches upon Foucault’s analysis with respect to the ideas of governmentality

and crime control. He ends his article with what he calls “the limits of ‘governmentality’

analysis” where he presents limitations to Foucault’s framework, whereby he calls his analysis

“to be suggestive rather than substantive” (Garland, 1997, p.195). What it is stressed in

Garland’s work about Foucault’s ideas of governmentality, is that through this whole concept

the idea of governmentality is that of shaping and ultimately controlling individuals conduct

by other means, and not those means of the state alone (Garland, 1997). This shift from the

governing people’s conduct through other means than through the state alone can be linked

to what Hache (2007) portrays as a ‘behavioral transformation’ which takes place “via a

twofold responsibilization strategy” (Hache, 2007, p12.). This transformation implies the idea

that change should occur first in the way individuals think, with regards to responsibilization

in order to have the ability to change the ways in which individuals act. The ‘twofold

responsibilization strategy’ operates both at an individual level, but it also consists of a

“transfer of the State’s traditional responsibility to protect to groups and individuals” (Hache,

2007, p.12).

The French philosopher Michel Foucault coined the term of governmentality in the 1970’s in

one of his courses about power and politics, a term which was “understood in the broad sense

of techniques and procedures for directing human behavior. Government of children,

government of souls and consciences, government of a household, of a state, or of oneself”

(Rose et al., 2009; Foucault 1997, p.82). Through his perception of governmentality, Foucault

creates a merge between the two major concepts of ‘government’ and ‘rationality’ which

have the power to lead and shape individuals’ conduct in society (Lemke, 2002). In her paper,

Hache (2007) examines the connection between Foucault’s ideas of governmentality with the

notion of responsibility perceived as a tool of neo-liberalism ideology. What the author

stresses is that, the shift from classical liberalism towards neo-liberalism represents a

reorganization of the government; in this reorganization, the emphasis is placed on the

allocation of responsibilities to various non-state actors compared to how it used to be the

case in the past, namely that only those official state bodies that had as a sole purpose to deal

with crime control, were in fact responsible for this issue alone (Hache, 2007, p.4). Important

11

to mention regarding the link between governmentality and responsibilization is that, as

mentioned above, it has been resorted to more indirect modes of governing and controlling

individuals. This has, however, another dilemma attached, namely that by having less direct

governance and control from the state agencies, it can lead to blurrier lines when it comes to

the distinction between the roles of the public and the private spheres and individual’s

liberties. Moreover, worth highlighting in this context is the idea that neo-liberal penal theory

brings to the fore the belief that the criminals are not perceived as “abnormal or monsters

but individuals like anyone else who have made a miscalculation” (Hache, 2007, p.11). This

perception implies that one should not aim at correcting offenders who behave in a deviant

way but rather aim at understanding the reasoning behind one’s actions that ultimately have

led to such behavior. This change in perception of the criminal has to do with the shift from

classical criminology to a more modernist approach, where the focus is merely set on the

assumption that one should investigate why a certain person does not behave responsibly

and ultimately, what can be done to change their ways of thinking.

12

2.2 Responsibilization Theory

In his work “The limits of the Sovereign State” David Garland (1996) illustrates an extensive

analysis of the strategies of crime control, with a main focus on the 20th century Great Britain

and other Anglo-American nations, by means of an in-depth conceptualization. The main

aspect of interest for this thesis regarding Garland’s work evolves around his

conceptualization of the responsibilization theory and its applicability to the focal research

question of this thesis. Before he (Garland) begins to introduce his perception of the

responsibilization theory and what this implies, he commences his article by stressing on the

changes in the intensity of the crime rates occurring in the western societies. Striking, as he

calls it, is the fact that the increased crime rates have gradually become “a normal social fact”

(Garland, 1996, p.446) and have become a standard in the everyday life of the individuals.

What is of interest to the author is thus not how this change occurred in the phenomena of

crime but rather how governments and (private) agencies are responding to this

normalization of crime, namely through adaptive strategies, among which is the

responsibilization one. Nonetheless, it is of utmost importance to understand that this was

the reality Garland depicted in the 1996, which was the case (in the United Kingdom) for a

decade already. Thus, it does not reflect the reality of todays’ society, because crime rates

have been dropping in many western states, although with this decrease taking place, crime

may still be regarded as a fact of life for many individuals.

In his article, Garland proceeds in introducing the idea of state sovereignty through what he

titled “The Myth of Sovereign Crime Control”, through which he implies that the basic idea of

the sovereign state having the capability to provide “security, law and order, and crime control

within its territorial boundaries” (Garland, 1996, p.448) has been coined as a myth. What

succeeded to strengthen the perception of an effective sovereign state was the emergence

of the new state forces in the late 19th century which were seen as a cooperation between

the state and the institutions of civil society (e.g. families, neighborhoods, churches, trade

unions and so on) that had ability to reduce the crime rates. Additionally, Garland touches

upon ideas that were found in Foucault’s writings as well, namely that the ‘new criminologies’

deny the previous assumptions that the committed crime steered from a pathological issue

or a faulty upbringing, but rather takes on the premise that “crime becomes a calculated risk

to be calculated (both by the offender and by the potential victim) […] rather than a moral

aberration which needs to be specially explained” (Garland, 1996, p.451). Although Garland

does not connect this directly to the responsibilization practice, he appears to be using the

example on how crime in general has changed both related to the perpetrator as well as the

victim in order to comprehend the reason for criticism of the current way of governing and

the need for adapting new (effective) strategies. This need for adapting new strategies steers

from what he calls “supply side criminology” which, Garland states that it is “aiming to modify

the everyday routines of social and economic life by limiting the supply of opportunities,

shifting risks, redistributing costs and creating disincentives” (Garland, 1996, p.451). He

13

elaborates this statement by means of examples along the lines of “ … replacing cash with

credit cards, building locks into the steering columns of cars (…) encouraging local authorities

to co-ordinate the various agencies that deal with crime and of course, encouraging citizens

to set up Neighborhood Watch schemes” (p.451). With these examples, Garland depicts the

contrast between this new approach and that of the traditional criminology, whereby in the

former one, the state, and consequently the state’s agencies are not the primary actors in the

establishment of crime control. Another contrast to traditional criminology consists in the

portrayal of the criminal subject, which is depicted by Garland as an opportunistic individual

“whose access to social goods must be barred” instead of a person who is perceived as a

societal misfit in need of assistance (p.451). From this point on, Garland reflects upon the

possibility of some new emerging strategies related to these so called new criminologies,

namely the idea that the discourse on the means behind the committed crimes has changed,

as explained above; among these needed strategies, the responsibilization theory will be

further introduced and explained as a (new) mode of governing crime (Garland, 1996, p.452).

Garland depicts the responsibilization practice as a strategy which involves “the central

government seeking to act upon crime not in a direct fashion through state agencies (police,

courts, prisons, social work etc.) but instead by acting indirectly, seeking to activate action on

the part of non-state agencies and organizations” (Garland, 1996, p.452) as a new crime-

prevention approach. Highly relevant terms associated with this strategy are “‘partnership’,

‘inter-agency cooperation’, ‘the multi-agency approach’, ‘activating communities’, creating

‘active citizens’ and ‘help for self-help’” (p.452). From these terms, one can therefore reach

the conclusion that the responsibilization theory has as a core principle the (partial)

delegation of responsibility regarding crime and the prevention of crime on to public and

private organizations and individuals which normally are not bound to the state and to

persuade them in operating accordingly (i.e. responsible). This theory basically stresses the

idea that non-state actors and private agencies are equally responsible for controlling crime

and security. The attempt to appoint private agencies and individuals into taking action

involves a set of methods and techniques to be implemented on behalf of the state, such as

either “stimulating new forms of behavior or stopping established habits” (Garland, 1996,

p.452); afterwards it can be assessed whether individuals and organizations have indeed the

necessary means available (to attempt) to effectively reduce criminal momentum and to

further evaluate whether this responsibility strategy could be prescribed.

Worth mentioning here is that the core idea of the responsibilization practice is on the one

hand to deliver the message that the state alone is highly incapable of effectively preventing

and controlling the increased crime rates and on the other hand that other actors in the

society (e.g. “property owners, residents, retailers, manufacturers, town planners, school

authorities, transport managers, employers, parents and individual citizens”) (Garland, 1996,

p.453) must become aware of their own responsibility with respect to crime prevention and

the increase of opportunities in the area of informal crime control.

14

Garland continues on the same line of reasoning by highlighting that such cooperative

measures which bring along private and public organizations in initiating preventive measures

might be, at times, perceived as forceful and compelling on behalf of the private companies.

He exemplifies this argument by the suggested idea that the government might compel retail

companies themselves to do more in order to reduce shoplifting and retail-related crimes by

warning that consequently the retailers themselves will have to bear the costs of theft-related

prosecutions. This example, he argues, is of support to the idea that such a reverse towards

a system of “private prosecution shows how the responsibilization strategy merges neatly into

strategies of privatization and public expenditure reduction which commanded such support

from conservative governments in the 1980s and 1990s”, which ultimately seems to highlight

Garland’s critical approach. (Garland, 1996, p.453)

It is crucial to be stressed in this context that the responsibilization strategies’ aim is thus not

to load off the state’s function but rather to implement a “new form of governance-at-a-

distance, which represents, in this field at least, a new mode of exercising power. It is a new

mode of governing crime, with its own forms of knowledge, its own objectives, its own

techniques and apparatus” (p.454). Accordingly, the state does not become less in terms of

power because it retains all its previous functions along with the new additional “set of

coordinating and activating roles, which, in time develop into new structures of support,

funding, information exchange and cooperation” (Garland, 1996, p.454). But, simultaneously,

the responsibilization practice slowly deteriorates the impression of the state as the “public’s

representative and primary protector” (p.454) because of the basic involvement of the

citizens and other public or private organizations. Thus, responsibilization as a strategy does

not imply that the state is ‘backing-off’ from its actual duties but rather it implies the idea

that the state acknowledges the need of taking up a new role. By means of such strategy, the

state is therefore testing the effectiveness of the various ways of “acting at a distance, of

activating the governmental powers of private agencies, of coordinating interest and setting

up chains of cooperative action” (p.454) which nonetheless can be perceived as more difficult

compared to the traditional way of issuing commands to state appointed workers.

After assessing the possible benefits of implementing the practice of responsibilization, with

the main benefit being the possibility of reducing the crime rates and dealing with the state’s

inability in solely decreasing these, Garland centers the discussion on what the main problem

related to the strategy is. The author claims that the main problem is that “the

responsibilization of non-state agencies and the routinization of crime prevention are liable to

give rise to huge disparities in the social provision and distribution of security” (Garland, 199,

p.463). In terms of the idea of security, Garland accurately points out that this problem is

likely to occur if security will not be equally provided and guaranteed to all citizens by what

he calls the ‘sovereign state’. In such situation of allocating the needed security based on

“market forces, rather than according to need” (p.463) the gap between the privileged strata

of the society and the poor will increase, thus those who are in need of security will lack the

15

necessary means to ‘purchase’ their security which will nonetheless “propel us towards a

fortified, segregated society” (p.463). By considering the main research question of the

current thesis, this idea could possibly be materialized under the premise that, maybe the

same could be applied for the case of online terrorist content on the social media platforms.

More specifically, if social media companies would undergo the practice of responsibilization,

it might be the case that some forms of illegal content would be regulated, but some forms

would not, thus, the premise is that the market forces could have a role to play in this

situation, that will be further discussed in the conclusion of this thesis.

Furthermore, Garland stressed on the issue that another aspect is decreasing the possibility

to apply the responsibilization practice, namely the fact that for the majority of the non-state

actors and private organizations, reducing crime is not the main objective to pursue. These

actors, that might have the available resources and capabilities of reducing crime rates, have

therefore as a core objective to continue to deliver their main services and increase their

profits instead of choosing to show interest in a form of cooperation for the purpose of

reducing crime rates and tighten control. Another impediment for the prospect of decreasing

crime rates is the state itself, which proved little to no willingness in recalculating costs and

delegating more funds towards initiatives for countering crime. The issue is that the state

proved to be more inclined to pursue policies that are believed to promulgate “punitive

strategies (which are easier to deliver) than to sacrifice economic or social objectives in the

services of crime control” (Garland, 1996, p.464), which only proves to undermine the possible

advancement of social change. Instead of aiming at publicizing policies to promote a vision of

social justice, the implemented penal policies shift “the burden of social control on to

individuals and organizations that are often poorly equipped to carry out this task” (p.466).

Even nowadays, it is still the case that more punitive and obligatory actions have been taken

at the state-level in order to aim at decreasing crime rates. As it will be discussed in the

following chapter, social media companies themselves lack the ability and willingness to

invest the needed funds in areas that could lead to a better control of the content that is to

be found on their platforms and decide to prioritize areas that are not (necessarily) under

‘threat’ (Almagor, 2017).

With the responsibilization practice in mind, another concept that could be of relevance to

this thesis is that of securitization. Introduced by Buzan et al. (1998) securitization is “the

process in which security measures are established in place of previous policies and routines;

it requires the establishment of an existential threat that potentially is devastating enough to

have political effects” (Rasmussen, 2015). Securitization needs thus a clear context in order

for its success to be granted, therefore the “audience” (i.e. both securitizing actors and

citizens) need to accept that the issue at hand is indeed an actual threat (Rasmussen, 2015).

With the process of responsibilization, citizens become more and more aware that it is one’s

personal duty to act and pursue a safe behavior which presents a development that Western

governance tends to adopt, namely prudentialism which implies “increasing individual effort

16

and responsibility” as opposed to welfarism which emphasizes on “collective risk solutions

and collective insurance” (Rasmussen, 2015). As the concept of interest is that of

responsibilization, specifically the responsibilization of social media companies (e.g. Twitter)

with regards to the terrorist content to be found online, the concept of securitization, can be

linked to the threat terrorism as well. Kaunert et al., (2019) pointed out that the idea of

‘collective securitization’ (Kaunert et al., 2019, p.262) is behind the developments at the

European Union’s level with regards to combating the existent terrorist threat, sparked after

the 9/11 events in the USA. What the authors stress is the idea that although the development

did not happen rapidly, there are major areas in which the EU strived in countering the issue.

Thus, an effective relationship built on the concepts of both responsibilization and

securitization, which nonetheless stress on collective efforts for its effectiveness to be

granted, could possibly lead to successful measures in combating the present terrorist threat

and decreasing the problem of crime control in contemporary societies.

After assessing Garland’s work on the responsibilization practice from 1996, the present

thesis will briefly examine his newer work on crime control and establish a possible

relationship with the aforementioned practice of responsibilization. In his book ‘Culture of

Control: Crime and Social order in contemporary society’ (2001) Garland portrays the

evolution of the crime control field and its characteristics. What he states as being a definitory

characteristic of the field of crime control are two “interlocking and mutually conditioning

patterns of action: the formal controls exercised by the state’s criminal justice agencies and

informal social controls that are embedded in everyday activities and interactions of civil

society” (Garland, 2001, p.5). He further stresses that it is important to acknowledge that the

field of crime control “involves the social ordering activities of the authorities and also the

activities of private actors and agencies (…) Too often our attention focuses on the state’s

institutions and neglects the informal social practices upon which state actions depends”

(Garland, 2001, p.6). Garland aims at putting forward the idea that this reconfiguration (of

crime control) involves manifold factors and cannot be pointed just towards the society nor

the governmental bodies, but rather seen as an overall change in responses of crime which

include, but are not limited to, “new practices of controlling behavior and doing justice,

revised conceptions of social order and social control, and altered ways of maintaining social

cohesion and managing group relations” (p.6).

Garland critically discusses this aspect which he calls ‘the commercialization of crime control’

(Garland, 2001, p.17) by pointing out that the line between the private and the public spheres

becomes thinner and thinner. What used to be the case, namely that those state institutions

that were essentially dealing with delivering the appropriate means in the field of criminal

justice were predominantly and solely active (in dealing with crime), is not necessarily in

practice anymore. On the contrary, along with those state institutions that were governing

the criminal field, an emergence of “the activity of citizens, communities, and companies, that

works with a more expansive conception of crime control and that utilizes techniques and

17

strategies that are quite different from those used by traditional criminal justice agencies”

(Garland, 2001, p.17) is to be seen more and more often, whereby the practice of

responsibilization can be seen under the emerging strategies. As his research primarily

focuses on the UK and the USA, Garland has framed that the government, especially in the

UK, aimed to constantly “reach out beyond its own criminal justice organizations and to

activate crime reduction activity on the part of individual citizens, communities, commercial

concerns and other elements of civil society” (Garland, 2001, p.17). This idea brings back to

the main theory that this thesis centers around, namely the responsibilization practice which

Garland introduced in 1996, which in his later work (i.e. Culture of Control, 2001) he applies

it more detailed to the existing policies and practices. The very premise that crime control

was a responsibility that belonged solely to those governmental organizations whose role was

in the criminal justice field had changed in contemporary (British) society. The lines between

the role of the private as well as the public sector become blurrier than ever, because the

public sector undergoes the practice of remodeling its primarily functions so that it merges

with those of the private industries (Garland, 2001, p.18).

The following chapter will analyze and apply the responsibilization practice firstly in a broader

sense, namely to generally speaking online crime relating it to a possible collaboration

between the public and the private sectors and explain what this collaboration implies.

Consequently, Garland’s practice will be applied to social media companies which will set the

scene to further discuss and analyze the main question of this thesis, namely the

developments of Twitter’s actions in combating online terrorist content over time.

18

3. The Role of the Internet and Social Media Companies in dealing

with online illegal content

3.1 The current state of the internet and (some) dilemmas in the field of online crime

control

When crime is inevitable and problematic to regulate as it is, it is even harder to regulate it

when perceived as ‘borderless’. Every nation-state abides to its own laws and regulatory

body, thus when crime occurs, depending on the state’s jurisdictions, it is dealt with

accordingly. But dealing with online crime is harder to regulate because of the trans-

nationality aspect of the issue. Important to emphasize is that under the term of online crime,

this thesis will predominantly refer to the existence of online terrorist content and

propaganda, and not the broad meaning of crime per se.

Every nation state abides to its own rules regarding the possible content that can or cannot

be shared on the internet. The internet as we know it is a place where free speech and

freedom of expression as well as freedom to access information is essential, but not

unlimited. Thus, when unlawful content appears on the internet, based on the respective

state’s legislation and jurisdiction it is dealt with and prosecuted accordingly. There is an

ongoing debate among scholars on the topic of the possibility of an international internet

governance in dealing with online. Important to acknowledge is the fact that currently, such

governance is not existent, but discussion on the topic of an international regulation,

generally speaking becomes more and more present. For example, since the Convention on

Cybercrime of 2001 which was created under the auspices of the Council of Europe, it can be

perceived that there is such type of governance when referring to the European Union

specifically. Although it is not labeled as an international internet governance per se, which is

perceived by scholars to be universally applicable, the Cybercrime Convention of 2001

stresses on an international cooperation to reduce cybercrime. Namely, the Convention

emphasizes that the “Parties shall co-operate with each other, in accordance with the

provision of this chapter, and through the application of relevant international instruments on

international co-operation in criminal matters, arrangements agreed on the basis of uniform

or reciprocal legislation, and domestic laws, to the widest extent possible for the purpose of

investigations or proceedings concerning criminal offences related to computer systems and

data, or for the collection of evidence in electronic form of a criminal offence” (Council of

Europe, Convention on Cybercrime, 2001, p.14). Under the Convention, states’ cooperation

occurs in terms of offering ‘mutual assistance’ as comprehensively as possible with regards

to the investigation of (possible) criminal offences related to data theft and computer

systems, as well as pooling collective effort in obtaining ‘evidence in electronic form of a

criminal offence’ (p.15). Relevant for this thesis is the fact that the European Council’s

Convention on Cybercrime does not include the presence of terrorist content, speech and

19

propaganda on the internet under the prosecuted crimes, but the focus is more on data theft,

child pornography, copy-right sanctions and so on.

Along the same line of reasoning, Mittal & Sharma (2017) provided a review of an

international legal framework to combat cybercrime, starting from the premise that

nowadays, in order to keep the unlawful content and the increase of cybercrime under

control, an international convention is highly necessary. The complexity of today’s cybercrime

is “(…) further complicated by the very nature of the cyber space manifested in anonymity in

space and time, rapidity of actions (…), non-attribution of actions and absence of international

borders” (Mittal & Sharma, 2017, p.1372). Furthermore, the authors stress on the idea that

the only available instrument is the above-mentioned Convention on Cybercrime

implemented by the Council of Europe (2001), which can be only applied to the region of

Europe, but non-members states such as the “the USA, Australia, Canada, Israel, Japan have

also ratified and signed” the convention which conveys even more legitimacy to this tool

available in the global fight against cybercrime (Mittal et al., 2017, p.1372). The two propose

with their research that “the Council of Europe’s Convention on Cybercrime should be the

baseline for framing an International Convention on Cybercrime” (Mittal et.al, 2017, p.1372).

The discussion about this issue of an international form of governing the internet effectively

is still very present because the presence of terrorist speech on the online medium continues

to be strikingly high. In his article titled ‘Ending the Cyber Jihad’ B.R. Davis (2006) stressed on

the issue that “Limiting the ability of terrorist networks to use the Internet as an operational

platform is one of the most significant challenges that lawmakers and national security

experts face” (Davis, 2006, p.125). He proceeds in arguing that those international

organizations that are responsible for regulating the internet in terms of decreasing the

means of terrorist communications should be in collaboration with sovereign states in

diminishing the threat. As an example of such international organization that aims at

regulating the internet, he introduces the Internet Corporation for Assigned Names and

Numbers (‘ICANN’), which is one of the main organizations that holds responsibility “for

Internet Protocol (IP) address space allocation, protocol identifier assignment, generic (gTLD)

and country code (ccTLD) Top-Level Domain name system management, and root server

system management functions” (icann.org/general) and is fully dedicated to ‘preserving the

operational stability of the internet’. Thus, several scholars, among which Davis (2006) as

well, have concluded that maybe, an international internet governance might be the answer

to being able to prosecute the unlawful content existent on the internet and on social media

platforms. Along with the regulatory incentives of ICANN, Davis (2006) stressed that the U.N

General Assembly has engaged with the pressing issue of cyber terrorism crimes, hence the

General Assembly Resolutions 57/238 from 2003 and 58/199 from 2004 aimed at creating a

universally applicable ‘culture’ of cyber security (Davis, 2006, p.164).

20

Furthermore, research on the issue of a broad, international regulatory body of the internet

stresses that a cooperation between the private and the public sectors could be an optimal

solution in decreasing online crime. This regulatory body, which some refer to as, or frame it

as an international internet governance (Callahan, 2017) could be perceived as a utopic

scenario from various points of view, but at the same time highly criticized from other ends.

What current literature suggests related to the possibility of establishing such governance,

and what exactly this governance means refers to a collaboration between the private and

the public sector in bearing responsibilities with regards to instituting regulatory policies

related to the management of the internet (Callahan, 2017). From the various academic

sources that have been analyzed throughout this study, still a debate on the definition and

possibility of achieving such governance is to be seen, due to the many variables involved and

issues regarding (data) privacy regulations and boundaries that each nation state abides to.

Moreover, this type of governance with respect to disseminating online crime (e.g. online

terrorist content) is hardly achievable due to the fact that there is no single definition on what

terrorism is, because of it being such a contested concept, as already discussed. What

academics and policy makers consider crucial in making this international internet

governance a reality is to create international unanimous definition of terrorism that is “broad

enough to receive consensus from the international community, yet specific enough to be

enforceable” (Callahan, 2017). Callahan further elaborates on this issue that such definition

should rather include concrete examples of what constitutes terrorism than for example

aiming at drafting “an overarching definition on the satisfaction of each individual nation-

state” (Callahan, 2017), fact that seems highly implausible to become a reality in the near

future. What needs to be further considered is that besides the aforementioned issue, if not

defined properly, such governance could lead to even more problems on the line of restricting

a wide variety of expressions. This could occur in the case where clear guidelines between

what is allowed and what is considered illegal content are not accordingly implemented.

Because it is such a debated topic, the aim is to present a wide range of opinions before

accurately taking a position whether such an approach to govern the internet could

effectively reduce online crime (i.e. terrorist online content). An interesting stand was

proposed by Goldsmith (2019) who critically assessed previous theories regarding the

sovereignty of the internet, whereby he concludes with the idea that internet regulations that

occur specifically in one nation can have definitory repercussion on how users from a different

nation experience the web (Goldsmith, 2019). He further states that it is hardly possible to

achieve full cooperation, whereby under cooperation he means “mutual restraint in order to

achieve larger reciprocal benefits” (p.823) between sovereign countries with respect to the

governance of the internet because “it takes painstaking negotiations about how each nation

will restrain itself, written specifications about what restraints each side assumes, penalties

for noncompliance, verification mechanisms, and the like. And even then, nations often fail to

achieve or sustain cooperation” (Goldsmith, 2019, p.823). For example, current news sources

show that Russia plans to detach itself from the global internet, which is something no other

21

country has tried before, namely “it’s going to test whether it can disconnect from the rest of

the world electronically, while keeping the internet running for its citizens” (Jee, 2019). If

Russia indeed manages to accomplish this internet detachment, it could potentially “help

Russia resist existing sanctions from the US and the EU, and any potential future measures”

as well as increase Russia’s “long tradition to control the flow of information between citizens”

(Jee, 2019). Thus, this example alone, not to mention the situation China is undergoing with

regards to internet censorship, is considerably decreasing the real possibility of an

international or global internet regulatory body/governance.

Therefore, looking through Garland’s critical lenses and simultaneously having the practice of

responsibilization in mind, it seems that the merge of the private and the public spheres in

combating online crime is still under thorough debate. As Garland has stated in his ’96 paper,

although looking at the practice and making non-state actors responsible in the field of crime

control is something that occurs in todays’ contemporary society, the actual outcomes are

not necessarily the desired ones. The interests of the private sphere will likely never be in

complete harmony with those of the public, namely having the possibility of being able to

access (unrestricted) information, and the right to freedom of expression but at the same

time to fight the threat and harm done by cybercrime, hate- and terrorist speech on the online

medium. Although some believe that the internet could be restrained under one governance

that makes all actors equally responsible in the process of disseminating online crime, strong

criticism still shadows the reality of this practice.

22

3.2 Responsibilization of the Social Media Companies

The internet is a place where unregulated content is often to be found, therefore the need to

‘responsibilize’ all the actors involved in sharing this content is perceived to be necessary.

With the worldwide increase in the use and popularity of social media platforms, there is no

surprise that terrorists use the internet and consequently social media platforms as a means

to spread their content and recruit new members. Moreover, besides terrorist content and

propaganda, social media companies also have to deal with a manifold of unlawful content,

such as (but not limited to) child pornography, extremist- and hate speech, discriminatory

and racial content that is as hard to regulate as the terrorist content online. For example,

research shows that after the crisis that arose in Syria in 2015, ISIS has been actively using

social media platforms such as Facebook, Twitter and YouTube to disseminate its ideology to

their world-wide sympathizers (Awan, 2017). In his study, Awan analyzed “100 different

Facebook pages and 50 Twitter user accounts which generated over 2050 results” which

confirmed his initial hypothesis, namely that “online hate is being used by groups such as ISIS

for a variety of reasons such as recruitment and propaganda” (Awan, p.138-139).

There is a constant debate and effort between companies and state’s organizations to

promulgate and effectively regulate the internet, but due to the state’s inability to cooperate

with private organizations that might have the available resources in helping to combat online

crime (on social media), the attempts to better police the internet and social media as a whole

have been perceived to be unsuccessful (Wu, 2015). The discussion about state’s

unwillingness, generally speaking, in better policing the internet with respect to the vast

illegal content present has been touched upon by other researchers as well, whereby Mittal

et al. (2017) state in their review of the efforts to combat online crime that “the very nature

of the internet-governance structure, titled heavily toward private players, leaves very little in

the hands of the States. The efforts for strengthening international co-operation to combat

cybercrime, including the Convention (here the discussion is brought back to the Council of

Europe’s Convention on Cybercrime, 2001), have miserably failed to tap this private element

of the governance mainly due to the conflict of private and public interest (Mittal et al., 2017,

p.1373). The emerging discussion is whether the States believe that they should take a step

back when dealing with online crime and further transfer the main responsibility to the social

media platforms themselves. This debate will be thoroughly discussed in the sections below.

With their emergence, social media platforms were perceived to be very innovative as they

allowed for instant communication between users world-wide and were basically formed of

any internet platform that allows “communication through images, videos or messages” (Wu,

2015). Important to mention is that the content on the social media platforms can be posted

by means of a username which offers the trait of (partial or full) anonymity of the user and

these images, messages and videos are not only target to a specific person per se but can be

publicly posted. Research shows that after the launch of Facebook and Twitter, the use of

23

social media grew significantly and, especially boomed after their first years of activity

because it became very practical for people to “suddenly be able to quickly post statuses or

‘tweet’ messages as soon as something happened” (Wu, 2015). Looking through available

research and statistics on the usage and growth of social media platforms, it seems that in

2018 Facebook was still the most accessed social media platform world-wide, with 2.27 billion

active users on a monthly basis, followed by YouTube (1.9 billion users), Instagram (1 billion

users) and Twitter (326 million users) (Sharma, 2018). As a result of this growth in social

media’s popularity overall, there is no surprise that terrorist use the internet with the same

ease as the average internet users especially because the internet and its social media

platforms are perceived as “cheap, pervasive, anonymous, unregulated and uncensored”

(Cilluffo, Cardesh & Whithead, 2007). This perception becomes harder and harder to discredit

with the increase in the illegal and terrorist content online, but as it will be portrayed in the

forthcoming section, social media companies are working to diminish this issue from further

evolving.

Nonetheless, current developments displayed progress in the area of what action has been

taken on social media with respect to the issue of online terrorist content as well as more

generally speaking regarding the overall illegal content that is to be found. Not only social

media companies have acknowledged their role in acting more responsible with respect to

the content that it is shared on their platforms, but also the involvement of supra-national

organizations such as the European Union had an impact in this developmental process and

will be discussed in the following sub-chapter with respect to online terrorist content. At the

same time, it can be discussed whether this development in the acknowledgment of the social

media companies of their new role (i.e. a more responsible role) was in fact a reaction towards

the outside pressure put by both the clients and users of these platforms as well as state’s

organizations and supra-national bodies. This discussion is hard to be linked with actual

(academic) facts, because both sides are to be seen as valid.

In his paper “The role of internet intermediaries in tackling terrorism online” (2017), R.C.

Almagor stresses on the role of ‘social networking sites’ (p.427) in effectively fighting

terrorism online. Although Almagor notes that his article is mainly focused on Islamic

terrorism, the reasoning behind his arguments are applicable to other categories of terrorism

as well. His main argument throughout the article is that social media companies shall engage

in a more proactive approach in dealing with the unlawful content on their networks in order

to have a real stand in the fight against modern terrorism, which “relies heavily on the

internet; both modern terrorism and the internet have common features that promote close

relations: they are global and diffusive, they do not require one center, their operation does

not require a very large budget, innovation is important to sustain both, and their operation

can be carried out through clandestine means” (Almagor, 2017, p.428). His proposal regarding

a shift from the current passive role of the social media companies to more of a proactive one

is central to his article, whereby he stressed that this all breaks down to social media

24

companies’ willingness to change and (re-)prioritize their investments (Almagor, 2017). The

main argument on behalf of the social media companies’ passivity with regards to their

assumed role is that it is simply impossible to screen all the available content shared on these

networks and that the available manpower is not enough. The example of YouTube is here

presented, whereby emphasis is put on the fact that “every second, one hour of video is

uploaded to YouTube. If liability concerns compelled YouTube to examine each video before

allowing it to be posted, YouTube could not continue to operate as an open forum for user

expression” (Almagor, 2017, p.432). One might argue that more of a proactive approach on

behalf of the social media companies themselves in dealing with online illegal content is

therefore the answer. It all breaks down to the idea of social media companies’ willingness to

adopt a proactive role with regards to combating illegal content, because generally speaking,

these platforms are indeed proactive in other respects, for example customer service, and

other branches were employees are physically working for the company. Thus, the idea

brought to the fore by Almagor is that social media companies should display more

willingness to invest in the problematic areas, such as combating online terrorist content that

is highly present on their platforms.

Shifting his focus more specifically on Facebook, where hundreds of posts and comments are

shared every hour, Almagor stresses that “Facebook has adopted a policy of continued

passivity on the whole, mitigated by activity when a request is made. This policy (according to

one of his main sources, Milner, whom he thoroughly interviewed regarding Facebook’s

policies at that time) is far more effective than monitoring vast amounts of content. There is

no need to monitor many millions of posts when only a small number of these posts are

problematic” (Almagor, 2017, p.433). Thus, as important as the issue of online terrorist

content and unlawful content in general seems to be for everyone, Facebook was hesitant in

allocating funds to the task of content monitoring specifically and maintained a passive stand.

The main issue that is to be seen here, is that it is mainly due to social media companies’

unwillingness to allocate the necessary funds in actively monitoring the content on their

platforms that leads to their reluctance to fully commit to the battle of fighting online

terrorism (Almagor, 2017). Facebook already uses sophisticated algorithms that tailors the

information that appears on each user’s timeline according to its own preferences and

interest, therefore this research suggests that Facebook engineers should develop a new

algorithm that could effectively spot any type of terrorist and extremist dangerous content

(p.432). The limitations of such algorithms are also made very clear, whereby Almagor

stressed that having people monitoring this content is far more effective than using

algorithms, because messages in general are very sensitive to the context they are put into,

but nonetheless, a new investment in this respect, namely developing even more accurate

algorithms, could be a great start before Facebook decides in investing in manpower for

inspecting the available content (Almagor, 2017). Relevant for understanding the

developments that occurred since Almagor’s research is that Facebook started to control

more in a proactive manner the content that is shared on the platform, whereby in 2018

25

“Facebook removed 14 million pieces of terrorist content” and thoroughly expanded their

algorithms to detect posts that are in violation of the company’s policies, which work across

19 different languages (Coldewey, 2018). Thus, although Almagor (and not only) stressed on

the necessity of investing in human capital in detecting unlawful content online, with the rise

of technology it is therefore possible to make use of such innovative algorithms that could

effectively detect terrorist content. However, one can still emphasize on the ‘merge’ of

human labor alongside these algorithms, because of the simple fact that some content might

be phrased in a more complex manner so that it needs the human capability of judging

whether it is indeed illegal content or not.

The concluding remarks regarding the issue of social media companies adopting a more

proactive stand in the fight against terrorist speech online breaks down to the idea that,

generally speaking, there are many methods to make the detection of illegal content online

possible, but it is a matter of the allocation of resources in the precise ‘troubled’ areas. Many

social media companies are hesitant to allocate these resources in combating the unlawful

content available only until pressured by other companies, supra-national organizations or

even the users themselves: “They relieve themselves of responsibility to combat antisocial and

violent speech to the best of their abilities; most internet intermediaries shy away from

assuming such responsibility as it is the easier and more profitable path to purse” (Almagor,

2017,p.440). Therefore, this failure constitutes more of a question of will to monitor illegal

content rather than a question of the actual ability of these companies in doing the ‘right’

thing (p.436). This brings to the fore another discussion, namely why would social media

companies themselves engage in allocating funds in the area of combating online terrorist

content if state authorities also lack the willingness to invest in this area of expertise. The

answer to such discussion is difficult to achieve and could itself be the core of an entirely

different research, but in order for effective measures to be taken both by state authorities

(e.g. international bodies) and social media companies, incentives are necessary. Research in

this area provided information that for action to be taken on behalf of the social media

companies themselves, outside pressure is most likely needed. Social media companies’ main

target is not necessarily to combat illegal content, or to fight the ‘war on terror’, but rather

to provide the necessary means for their users to have access to instant communication and

such.

As the previous section focused predominantly on the (perceived) need for cooperation

between the public and the private sectors in countering online crime, it is important to

emphasize that the need for such cooperation and collaboration has been acknowledged by

social media companies themselves. The following example will illustrate an effective

collaboration between private and public organizations with the aim of sharing expertise and

striving towards achieving the common goal of fighting terrorist presence online. In a public

blog post made in June 2017, Twitter announced the formation of the Global Internet Forum

to Counter Terrorism along with Facebook, Microsoft and YouTube (Global Public Policy

26

Twitter, 2017). The announcement acknowledges the challenges faced by these four

companies because of the widespread presence of terrorism content (e.g. propaganda) and

violent extremism speech online. Therefore, the four big internet players of the social

medium have collaborated and “developed policies and removal practices that enable (us) to

take a hard line against terrorist or violent extremist content on our hosted consumer services.

We believe that by working together, sharing the best technological and operational elements

of our individual efforts, we can have a greater impact on the threat of terrorist content

online” (Global Public Policy Twitter, 2017). The Global Internet Forum to Counter Terrorism

is therefore an example of cooperation between private companies, such as the above-

mentioned social media companies and other small tech companies, “civil society groups and

academic, governments and supranational bodies such as the European Union and the United

Nation” (Global Public Policy Twitter, 2017). Unfortunately, not much data is available on the

effectiveness of the above-mentioned cooperation. The Global Internet Forum to Counter

Terrorism appears to be involved in funding further research in the area of countering online

terrorist content and consequently developing new technology to disrupt the terrorist agenda

online (GIFCT, 2018).

Connecting the aforementioned information to Garland’s responsibilization practice, one

could state that the Global Internet Forum to Counter Terrorism could, in theory, be a positive

example of the application of Garland’s conceptualization. But due to the fact that little data

is available in terms of indicating clear changes in combating online terrorist content from the

above-mentioned initiative, one can also look at it more critically and assess that these

‘newly’ emerging enterprises to combat terrorism online might after all have their own aims

to pursue, i.e. giving the impression that these big internet players are indeed cooperating,

when in reality the cooperation does not occur as transparent as individuals would like to

think. As Garland critically assessed in his ’96 work, there are situations where the needed

security, in this situation being the effective outcome of the cooperation between the

members of the GIFCT in combating terrorism online, is based “on market forces, rather than

according to need” (Garland, 1996, p.463).

27

3.3 EU-level initiatives with regards to combating online terrorist content

To further apply the theoretical framework of Garland’s (1996) responsibilization practice to

the issue of online terrorist content on the social media platform of Twitter, the present study

will examine some of the European Union’s initiatives with regards to combating online

terrorist content. More specifically, the aim is to analyze current proposals and regulations

put forward by the European Union and EU-level agencies, such as the European Union’s law

enforcement agency Europol, with regards to combating the dissemination of unlawful

content and terrorist content on the internet. Lastly, this section will also present some

perspectives of NGO’s that are active in this sphere in order to illustrate the counter

arguments that have been brought upon the EU’s institutions in dealing with online unlawful

content.

Before going in detail with the European Union’s most recent initiative with regards to the

prevention of the dissemination of terrorist content online which came to life in 2018, it is

important to understand the previous initiatives that led to the EU Commission’s most recent

actions with regards to the issue at hand. The European Unions’ law enforcement agency,

Europol, was perceived as active in the field of countering (online) terrorism. With regards to

online terrorist activity, on the 10th and 11th of April 2017, Europol hosted the first Conference

on Online Terrorist Propaganda and released a report based on the various discussion of the

Europol’s’ European Counter Terrorism Center (ECTC) on topics pertaining to the online

terrorist hazard. During the conference, highly debated was the Islamic State’s (IS) popularity

and well-planned strategy on the online medium. The European Counter Terrorism Center

(ECTC) stressed on the necessity to tackle this challenge, tackling that can be done by means

of the responsibilization of the parties involved, or how the ECTC has put it: “… can only be

achieved through a coordinated and systematic action bringing together law enforcement,

public institutions and the private sector” (Europol.europa.eu).

Likewise, the European Union Internet Referral Unit (EU IRU) based at Europol’s ECTC has as

a main aim to detect and investigate “malicious content on the internet and in social media”

(Europol/EU IRU). Europol’s Internet Referral Unit is comprised of a team of experts with

diverse knowledge in the area of counter-terrorism, ranging from “experts in religiously

inspired terrorism, translators, information and communication technology developers and

law enforcement experts in counter terrorism”, which shows the wide availability of expertise

at hand, to aim at combating the issue of the dissemination of terrorist content online. The

core tasks of the EU IRU are the following:

“Supporting the competent EU authorities by providing strategic and operational

analysis;

Flagging terrorist and violent extremist online content and sharing it with relevant

partners;

28

Detecting and requesting removal of internet content used by smuggling networks to

attract migrants and refugees;

Swiftly carrying out and supporting the referral process, in close cooperation with the

industry”

(Europol.europa.eu/eu-internet-referral-unit-eu-iru)

In terms of the effectiveness of the IRU, it is stated that in December 2017, on average 86%

of the content that has been flagged has been removed, content that was disseminated

“across over 80 platforms in more than 10 languages” (Europol/EU IRU, figures of December

2017).

Now moving to more recent EU level initiatives, in 2018, the European Commission submitted

the ‘Proposal for a Regulation of the European Parliament and of the Council on Preventing

the Dissemination of Terrorist Content Online’ (EC, 2018) to emphasize the great need of

action on behalf of the European Union against terrorist content online. What the commission

considers illegal terrorist content corresponds with the definition drafted by the EU Directive

2017/541, namely “information which is used to incite and glorify the commission of terrorist

offences, encouraging the contribution to and providing instructions for committing terrorist

offences as well as promoting participation in terrorist groups” (EC, 2018, p.3-4). More

specifically, the European Commission’s Proposal states that:

“This Regulation seeks to provide clarity as to the responsibility of hosting service providers

in taking all appropriate, reasonable and proportionate action necessary to ensure the

safety of their services and to swiftly and effectively detect and remove terrorist content

online, taking into account the fundamental importance of the freedom of expression and

information in an open democratic society” (EC, 2018, p.2).

The European Union’s proposal corresponds with the EU Directive 2017/541 by proposing as

a main aim to harmonize the legislations of each Member State with regards to criminalizing

and prosecuting (online) terrorist offences. Furthermore, as Article 21 of the Directive on

Combating Terrorism emphasizes that Member States are responsible for taking the right

measures in guaranteeing the removal of the online terrorist content, the Commission’s

Proposal states that “(it) covers not only material inciting terrorism but also material for

recruitment or training purposes, reflecting other offences related to terrorist activities” (EC,

2018, p.3). This is as well covered by the EU Directive, whereby both state that each Member

State has the freedom in choosing what measure needs to be taken. Among these

(preventative) measures, it is proposed for Member States to introduce removal orders which

can be issued by the competent bodies of authority in the respective Member State under

the form of a judicial or administrative decision. In such cases of removal, the hosting service

providers are obliged to remove (or at least restrict the access to) the respective content in

the time span of an hour since the decision was officially taken and adopt proactive measure

29

in accordance with the level of risk linked to the removal of the specific terrorist content.

Among the proposed proactive measures, the Regulation includes “deploying automatic

detection tools” (p.4). Moreover, the Regulation at hand obliges Member States to guarantee

that their state authorities have the necessary means available to intervene when online

terrorist content is to be found. Member States are also “obliged to inform and cooperate

with each other and make use of any channels set up by Europol to ensure coordination with

regards to removal orders and referrals” (p.4). In order for the Commission to ensure that the

hosting service providers and consequently the Union’s Member States effectively implement

the proposed Regulation, penalties are to be applied in particular events. For example,

Member States should adopt “rules on penalties, including, where appropriate, fining

guidelines” (p.21) in the case where hosting service providers fail to abide repeatedly to the

obligations of removing or disabling online terrorist content within the amounted hour since

the removal order was received.

The Commission acknowledges the trans-nationality of the issue at hand and stresses that

“given the cross-border dimension of the problems addressed, the measures included in the

proposal need to be adopted at the Union level in order to achieve the objectives. The internet

is by its nature cross-border, and content hosted in one Member State can normally be

accessed from any other Member State” (EC, 2018, p.5). Terrorist content online, be it

specifically on social media or on generally speaking the online medium has the aim “to incite

individuals to carry out terrorist attacks, including by equipping them with detailed

instructions on how to inflict maximum harm. Further propaganda is commonly released after

such atrocities whereby they glorify in these acts, and encourage others to follow” (EC, 2018,

p.8). Thus, this regulation proposed by the European Commission seeks to protect the public

security with the aim of diminishing access towards such content that indorses numerous

violations and posits a real threat. The potential negative impact that this proposal could have

in relation with the fundamental human rights can affect the following areas:

a. “Rights of the content provider: right to freedom of expression; right to protection of

personal data; right to respect of private family life, the principle of non-discrimination

and the right to an effective remedy;

b. Rights of the service provider: right to freedom of conduct a business, right to an

effective remedy;

c. Rights of all citizens: and rights to freedom of expression and information” (EC, 2018,

p.8)

As acknowledged by the European Commission, the present proposal could potentially have

a negative impact on some of the basic and fundamental rights. This fact was brought to the

fore by several international non-governmental organizations, such as EDRi (European Digital

Rights), which is perceived active in the field of defending the freedom of pression and rights

on the online medium. On the 18th of March 2019, EDRi, along other NGOs drafted an open

30

letter towards the Members of the European Parliament with respect to the regulation on

terrorist content online which “endangers freedom of expression” (EDRi, 2019, Open letter).

The main points of criticism in the letter were around the issue of the definitions of terrorist

content and of hosting service providers and the “unworkable deadlines for content removal”

which is perceived as being extremely shorts for hosting service providers to be able to

comply with the removal orders. With this short and unrealistic span of one hour, EDRi

stressed that it could lead to the removal of legal content online due to the time pressure on

behalf of the enterprises obeying to the orders. In the open letter, the NGOs stress that “this

draft Regulation risks undermining current efforts and could have a strong impact on

European citizens’ fundamental rights” (EDRi.org, 2019, Open Letter).

Henceforth, the above-mentioned proposal aims at operating as a regulatory means with

regards to terrorist content online. Although some regulations are already in place, whereby

states and social media companies have various procedures and policies that are

independently implemented by nation states, the EU’s aim is to harmonize and implement all

of the above as a standard for all the companies operating in the European Union. As stressed

by the European Commission, the misuse of the internet and social media platforms have

serious consequences not only for the service providers that are accommodating such

content on their platforms, but for the average users and generally speaking for the society

as a whole. These consequences can relate to the distrust between the service providers and

the users that nonetheless lead to more severe consequences such as possible (permanent)

damages to the business as a whole. Therefore, it is of utmost importance to responsibilize

these online service providers in being aware on the shared content on their platforms in

order to “protect their services from misuse by terrorists and to help tackle terrorist content

disseminated through their services” (EC, 2018, p.13-14).

As the responsibilization practice is mainly based on the shared responsibility of the public

and private agencies in controlling crime (Garland, 1996), one can state that the European

Commission is taking stands in applying this practice with the discussed proposal. The

European Commission aims with the 2018 proposal to standardize the existing rules and

regulations regarding the online content and oblige all the Member State to obey to this, as

well as urging social media companies themselves to take a more proactive role with regards

to the issue at hand.

In concluding this section of the present thesis, Garland’s conceptions on the practice of

responsibilization (1996) might seem along the same line of reasoning with the European

Commission’s proposal. Although not implemented yet, and still at the level of proposal, this

can be seen as a way the EU is ‘passing’ the responsibility towards the hosting service

providers in obliging to undergo the presented measures in combating the dissemination of

online terrorist content. The link with what Garland has stated in his ’96 work, namely that

states are more inclined to adopt more ‘punitive measures’ that are easier to be delivered,

31

can to a certain extent be stated about the European Commission’s proposal as well. Criticism

regarding the ‘unrealistic’ one-hour deadline for content removal has been emphasized as

well as the lack of properly defining the concepts of core interest to the proposal. Critical

discussions regarding what the exact role of these companies should be is also widely present

in the public debate, therefore it is at the EU Commission’s latitude the extent to which it will

be reacted to the present criticism and whether the European Parliament will adopt the

Proposal as it was drafted in 2018 or make amendments. Important to note with respect to

the issue of responsibilization is that Garland could not have foreseen almost two decades

ago, in the late 1990’s, how social media companies would grow and develop and ultimately

become such important and big players in today’s society.

32

3.4 Developments in Twitter’s practices with regards to terrorist content

After assessing both the private and the public spheres, as well as what social media

companies and EU-level organizations are doing in combatting online crime by having the

responsibilization practice in mind, it is time to center the remaining of this thesis in the

direction of the core research theme, namely the developments in Twitter’s role with regards

to terrorist content. Furthermore, these developments will be assessed in light of the

responsibilization practice and analyze whether Garland’s critical conclusions are applicable

to the present issue, fact that will be thoroughly discussed in the conclusion section. Some of

the main developments in Twitter policies with regards to terrorist presence on the social

media platform will be explained by analyzing their official terms and conditions. Afterwards,

academic research and news articles will be evaluated with regards to Twitter’s actual

practice.

Central in understanding Twitter’s developments with regards to combatting terrorist

content on its platform, it is important to see the evolution in Twitter’s Terms of Service,

available on the Twitter website from 2009 until the most current form of these terms from

2018. Clear general developments can be seen from the first versions of these terms of service

until the most current one, firstly with respect to the length of the documents and secondly

with respect to the elaborations regarding each of the sections. The first ten versions of these

terms (starting from 2009 until January 2016) portrayed no difference in the rules and

regulations based on the country the user is accessing the social media platform from,

whereby from September 2016 up until today there are specific rules for the United States

and for users from the European continent. Interesting to see is the fact that Twitter does not

present the rules and regulations applicable beyond the European continent and the USA on

the official website. This can be perceived as a limitation with regards to the rules and

regulation of the Arabic countries and China for example, which can either be due to language

issues or a different IP address is needed to access those specific terms and conditions.

Although these rules are overlapping, it is important that Twitter acknowledged the

differences in terms of laws, rules and regulations with respect to data privacy between

Europe and the USA. Moreover, starting from the first version of the Twitter rules until the

most current ones, little is changed in terms of Twitter’s perceived responsibility with regards

to unlawful content. Under the first general conditions it is stressed that, first of all, the user

has full responsibility of all the data and information posted on the platform and second of

all, it is stated that “We may, but have no obligation to, remove Content and accounts

containing Content that we determine in our sole discretion are unlawful, offensive,

threatening, libelous, defamatory, obscene or otherwise objectionable or violates any party’s

intellectual property or these Terms of Use” (Twitter.com, Version 1, Terms of use, 2009). The

same principle is to be found under the majority of the versions, with a slight difference in

phrasing, but nonetheless it is important to stress that Twitter claims no obligation or liability

33

in taking any direct actions with regards to the content online, even though under some

circumstances the platform will remove inappropriate content.

Looking at the current public security measures on Twitter’s website (Twitter.com, 2018),

there is a high emphasize on a zero-tolerance policy against any type of discriminatory

speech, be it racially, sexually or religiously, as well as any content related to the promotion

of terrorism. A specific section related to violent extremist groups may be found under the

rules and regulations section, which states: “You may not make specific threats of violence or

wish for the serious physical harm, death, or diseases of an individual or group of people. This

includes, but is not limited to, threatening or promoting terrorism. You also may not affiliate

with organizations that, whether by their own statements or activity both on and off the

platform- use or promote violence against civilians to further their causes” (Twitter Help

Center/rules-and-policies/violent-groups). As a consequence of the above-mentioned

situation taking place, the respective account(s) will be permanently suspended. Under

violent extremist group, Twitter developed a list of criteria that serves at understanding what

the company regards as violent extremist groups, therefore when all of the following criteria

are met, actions (e.g. permanent suspension of the account(s)) will be taken. The criteria put

forward by Twitter are:

“identify through their stated purpose, publications, or actions as an extremist group”

“have engaged in, or currently engaging in, violence (and/or promotion of violence) as

a means to further their cause”

“target civilians in their acts (and/or promotion) of violence”

(Public Manual Rules and Policies Twitter Help Center/rules-and-policies/violent groups)

Besides these criteria, Twitter stresses that first of all, this policy does not apply to any military

or governmental bodies and exempted are those groups who are engaging in peaceful

resolutions as well as groups with a representative body elected through democratic

practices. Secondly, there is also certain behavior that Twitter considers when establishing

whether an account is affiliated with a violent extremist group, such as:

“stating or suggesting that an account represents or is part of a violent extremist

group”

“providing or distributing services (e.g. financial, media, propaganda) in furtherance

of progressing a violent extremist group’s stated goal”

“engaging in or promoting acts for the violent extremist group”

“recruiting for the violent extremist group”

(Public Manual Rules and Policies Twitter Help Center/rules-and-policies/violent-groups)

In conclusion, Twitter states that if the above-mentioned criteria are met and such behavior

is spotted by both first person and bystander accounts, those accounts belonging or affiliated

34

with violent extremist groups will be permanently suspended (The Twitter Rules). What will

be further discussed in the following sections is that, although Twitter is becoming more and

more active with suspending accounts, research shows that the risk (and the practice) of new

emerging accounts being created is seemingly high.

Twitter communicates the changes in the global public policies, as well as updates on current

matters the company is working on or plans to adopt to the public. These communications

take place under the form of the ‘Twitter Blog’, which can be accessed publicly from the main

page of the platform. In 2016, a blog post titled “An update on our efforts to combat violent

extremism”, Twitter offers insights and statistics on the suspended accounts, the way in which

the company responds to reported accounts and the overall progress in combating this issue

on the platform. Twitter stressed in the blog that it is an ongoing developing process in

improving its technology and finding viable ways to combat the issue in such a manner that it

will survive the ‘test of time’. The company is in collaboration with other organizations and

experts and stated that “there is no one magic algorithm for identifying terrorist content on

the Internet” but they sustain the need of utilizing and developing new forms of technology

to help in identifying such accounts (Twitter Blog, 2016). In 2017, Twitter released data in a

public statement for a well-recognized online newspaper (The Guardian, 2018) with one of

their biannual transparency reports. In this report Twitter acknowledges the intense usage of

the social platform for wrongful purposes and stated that in a span of six months (between

July and December 2017) the company banned approximatively 270.000 accounts world-wide

for promoting terrorism and various extremist content. Worth mentioning is the fact that

emphasis is put on the debate between the limits of free speech, especially whether legally

speaking there can be such limits to begin with and the “complex relationship between private

companies and the public interest” sparked from the discussion whether suspending terrorist

accounts on social media is indeed an effective approach countering the extremist activity

online (Berger & Morgen, 2015, p.2).This goes back to the previous discussion in section in

3.2 where Almagor (2017) stresses that the effectiveness of such algorithms could not be

compared with human beings checking the validity of the content, due to the importance of

the context in which these messages appear. Thus, not having individuals looking at these

messages before removing or suspending might lead to both false-positive and false-negative

results that will not be beneficial for anybody in the end. By relying solely on (new) technology

and algorithms to detect illegal messages on the platform, it could be very difficult to state

accurately whether the content is actually illegal or not because it is a very fine line even for

experts to take stand on what exactly is incitement to terrorism and what is not.

Other research was based on the effectiveness of the suspension of Twitter accounts whereby

the overall sentiment was that it is quite ineffective as more accounts emerge in the

aftermath of the suspended ones. Berger and Morgan developed a comprehensive study in

2015 with the goal of creating an overview of The Islamic State’s success in exploiting the

social media medium by sending propaganda and aiming to reach people ‘vulnerable to

35

radicalization’, whereby the focus of their study is solely concentrated on Twitter (Berger &

Morgen, 2015). The research sample was one of 20.000 ISIS related Twitter accounts and

statistical data was collected in 2014, between the months of September until December

(p.54). At the same time Berger & Morgen collected their statistical data, Twitter led “an

aggressive campaign of suspension” (p.55) whereby one important finding was that at the

beginning of the research “the primary ISIS hashtag – the group’s name in Arabic- went from

registering in 40.000 tweets per day or more around the time suspensions began (September

2014), to less than 5.000 on a typical day in February” (p.56). This finding then (partially)

supports the effectiveness of account-suspension measure taken by Twitter and is a clear

impediment for ISIS’s network ability to grow. What Berger & Morgen further discuss is the

idea that it is unlikely to examine from the available data whether the decrease in ISIS online

activity on Twitter is solely based on the suspension of the accounts, or ultimately users

simply decided to merge to another social media platform; but they concluded that

nonetheless this suspension rate had a positive effect for the company, in terms of having

less unlawful content on the platform (Berger & Morgen, 2015). Lastly, a valid point made by

the two researchers stresses that Twitter usually suspends those accounts who are visibly

affiliated with ISIS, by monitoring “the content of the tweets and user reports” (p.57) but those

accounts who have not as large of a number of followers and are perceived as smaller have a

higher change of not being suspended (p.56).

In a paper published in 2018 in the International Journal of Cyber Warfare and Terrorism by

Al-Rawi et al., the Islamic State’s social media content on Twitter was examined. This research

looked into “over 50 million tweets posted by more than 8 million unique users that referenced

the keywords ‘ISIS’ or ‘ISIL’” (Al-Rawi et al., 2018, p.1) on Twitter, by taking into account both

English and Arabic language tweets, fact that gives more accuracy and uniqueness to the

study. The researchers made use of several interviews with Islamic State’s terrorist fighters

to emphasize on the role Twitter has in disseminating ISIS propaganda, whereby a striking

statement made by one of the interviewees was “Your tweets are your weapons” (p.4) in

relation to those who tweet for ISIS with the aim to “educate its followers on the use of Twitter

(…)” (Al-Rawi et al., 2018, p.4; O’Neil, 2016). Al-Rawi (et al.) acknowledge the same as Berger

& Morgen (2015), and other researchers as well did, namely that after being largely banned

on Twitter, ISIS related accounts active on Twitter have largely decreased, although new ones

emerge. But the emergence of the newer accounts does not occur so rapidly as before the

censoring begun. For example, other data shows that “from mid-2015 to early 2016, Twitter

shut down over 125,000 ISIS-linked accounts” (Jeong, 2016). Banning the terrorist

organization from social media “had an impact on the group’s presence, and its media

production decreased considerably: specifically, from August 2016, there were 194 post from

ISIS-linked media production units which were far fewer than the 761 posts found in August

of the previous year before censoring began” (Al-Rawi et al., 2018, p.5). To better understand

how ISIS operates on the social media platform of Twitter, it is important to mention that the

terrorist organization was active with a little amount of ‘official’ ISIS accounts that ‘tweet

36

official statements and updates’ but mostly from decentralized accounts that are belonging

to jihadi sympathizers and other followers (Al-Rawi et al.,2018).

With the above-mentioned information in mind, one can conclude that the (perceived)

responsibility in Twitter’s practice was not highly emphasized under the first versions of the

official terms and conditions presented on the website from around 2009, but clear changes

have been made over time, until the most recent public policy data retrieved from 2018,

where developments can be seen with regards to Twitter’s most current practices. By linking

the official terms and conditions and public policies together with the academic research

available on the developments of Twitter with regards to combating online terrorist content,

it appears that voluntarily, Twitter did take some actions before 2018 already. In 2015, Berger

& Morgen conducted an extensive research based on terrorist content and terrorist affiliated

accounts on Twitter, where they concluded that an increase number of accounts were

suspended. Under the official terms and conditions, Twitter did not assume any liability in

2014 to act and assume responsibility with respect to what users decided to post, therefore

this can only lead to the assumption that in practice, Twitter seemed to perceive these actions

(i.e. suspending accounts in 2014) to be voluntary and not their ‘responsibility’ per se. Judging

solely from the terms and conditions and the public rules made available by the social media

company, there are disparities to be seen between Twitter’s approach and the actual image

it wants to portray to its wide audience. Regarding these disparities, it can be argued that

although Twitter highly stresses on zero-tolerance policy with respect to extremist terrorist

speech on the platform, there is still unlawful content present and maybe, Twitter could do

even more with regards to the issue at hand.

Nonetheless, with respect to Twitter’s actions towards unlawful content, and the immediate,

permanent suspension of the respective accounts, it is important to acknowledge that this

alone is already a big step towards a ‘safer’ and more responsible platform, because not all

social media companies that are faced with the problem of terrorist content, do take such

severe measure as suspending accounts. Moreover, in a (Vox) news article about Twitter’s

efforts detecting abusive and illegal content, it was stressed that Twitter’s main source of

finding illegal content was coming from users themselves, but now it has improved its

algorithms (Wagner, 2019). Moreover, what the article further stated with regards to

Twitter’s algorithms was that “Algorithms designed to improve safety are the only way Twitter

is going to keep pace with the volume of tweets people share every day. Twitter is far from

“healthy,” but it may be getting a little closer to cleaning up its act” (Wagner, 2019).

In conceptualizing his responsibilization practice (1996) Garland could not have imagined the

important role a platform like Twitter could have in the public debate. With more than 300

million monthly users (Wagner, 2019), Twitter holds a relevant role in terms of what

information is shared, under which conditions, and hence how accurate is the respective

information that reaches the public. At the same time, with having such a big role in today’s

37

society, it could be argued that Twitter, by becoming more responsible concerning the shared

content on the platform, could end up contradicting Garland’s conception with regards to the

possible advancements of social change, and how states were at that time undermining these

possibilities (of the advancement of societal change). Of course, Garland (1996) related this

issue (p.464-466) towards the state, but again he could not have foreseen that Twitter, for

example, could be perceived as an almost ‘state-like institution’, metaphorically speaking,

with regards to the influence and ‘power’ it has.

38

4. Conclusion

The present study has explored the role of Twitter’s developments with regards to combating

terrorist content over time in light of Garland’s (1996) responsibilization practice. Other issues

of interest such as whether a possible establishment of an international (or global) internet

governance in combating online crime would be a good approach and to what extent is the

European Union active in the role of countering the dissemination of online illegal content

were investigated throughout the present study. Considering all the presented information

and making use of Garland’s (1996) responsibilization practice, one can fully apply his

concluding criticism towards the case of Twitter. As Garland has accurately pointed out, one

core aspect regarding the responsibilization of the non-state actors and private organizations

for the prospect of reducing crime rates breaks down to the fact that most of the time, these

private organizations have other aims to prioritize beside that of reducing crime (Garland,

1996). Thus, by applying this theory to the issue at hand, one can see in Twitter’s

developments that it became, both in theory and in practice, more responsible towards the

terrorist content shared on the platform, but at the same time this issue is still present and

highly unlikely to be completely eradicated. The question that rises here, is what exactly has

made Twitter to change and take upon a more responsible stance towards the problem of the

unlawful content on the platform. The idea that with complex issues come along complex

answers is relatable to the case of Twitter’s developments as well, because there is not one

single factor that has led to Twitter becoming more responsible. Manifold factors had a role

in this development, such as, on the one hand, it can be argued that the public pressure (e.g.

new sources, users, other social media companies) played a role in Twitter taking more of a

proactive position in managing its content. Having a seat at the table of the big internets’

players, Twitter needs to consider the outside feedback in order to improve its platform and

continue to deliver according to user’s expectations. Various actors (i.e. online news sources)

started to put Twitter ‘on the spot’ with respect to the issue of online terrorist content on the

platform, therefor the social media company was more or less obliged to react. As it was

presented in the previous section, one reaction on behalf of Twitter was to improve its

content-detection algorithms as well as emphasizing in its official terms and conditions that

unlawful content is not acceptable on the platform. On the other hand, besides the outside

pressure it can be argued that states and authorities had an impact on this development as

well. Although the European Commission’s Proposal is still at the level of proposal, with the

possible implementation of such regulation, Twitter could face even more pressure (legally

speaking). If ratified, an instrument like the EC’s proposal would make social media

companies, consequently Twitter, to take on a bigger, and more responsible role with regards

to combating online terrorist content.

Important to keep in mind is that Twitter is a business whereby the main goal is to make profit

and gain competitive advantage over other social media platforms, thus reducing crime is not

necessarily one of the main goals of the company. Although not perceived as a main goal,

39

combating terrorist content on its platform is emphasized in the last couple of years and effort

in that respect can be seen from the high number of suspended accounts (section 3.4).

Another point of discussion here could evolve around the idea that because of the terrorist

content and propaganda on its platform, Twitter might actually risk losing legitimacy and

popularity among other platforms. This fear of losing amongst other social media platforms

might leave Twitter in a state of ‘crisis’, whereby the platform might eventually suspend an

increased number of accounts that were untruthfully associated with terrorist content just to

be ‘on the safe side’, fact which although plausible, is not the core of the present discussion.

Essential to consider here is that Garland conceptualized the responsibilization theory in

1996, therefore he could not have foreseen the immense role social media companies will

play in today’s society and ultimately how they would have a role in shaping contemporary

public debate. To quote Almagor (2017) on the important role that social media companies

have nowadays, the following illustrates it very accurately, namely “Facebook and Google

have more power than presidents and prime ministers. Power without responsibility is

dangerous, corrosive and undermines our well-being” (p.452). Consequently, because

everything centers around social media companies, they are unlikely to evade their

responsibility when it comes to illegal content on their platforms, which brings about another

discussion, namely that of whether social media companies should be the ones responsible

to draw the line between illegal content and freedom of expression, or whether this should

be solely the task of the states to legally prosecute. This can be linked to the discussion

presented in the introduction of this thesis that was centered around terrorism as a

‘contested concept’ (Schmid, 2004), because a main issue in itself is the difficulty to accurately

define terrorist speech (and terrorist content). With the increase of technology and the new

emerging detection algorithms, the question is whether social media companies can

accurately define such content and provide a reliable balance between the ‘false-positives’

and ‘false-negatives’ results that come along with using these algorithms. From the analyzed

news sources and academic literature, it can be concluded that this is a risk social media

companies are willingly assuming by making use of technological detection mechanism for

analyzing the content on their platforms. As stated, Twitter is different from other social

media companies because for example, YouTube and Facebook have algorithms that decide

based on the user’s preferences what it appears on the feed, whereby with Twitter this is not

the case. Twitter used to rely on users to spot and ‘flag’ suspicious accounts but as stated in

Wagner’s (2019) Vox’s article, Twitter is starting to adapt more proactive means of detection

(i.e. new content detection algorithms).

With the growing awareness of the existent threat of illegal content online and terrorist

propaganda, the roles of social media companies should be more than ever defined and

clarified. As Almagor (2017) stressed in his paper, “Better cooperation is required between

internet intermediaries and governments. Indeed, to have effective results in fighting highly

dangerous phenomena such as terrorism, cooperation is vital. Businesses are expected and

40

obligated to act responsibly in a way that would benefit their communities and avoid or

minimize harm to their stakeholders” (Almagor, 2017, p.452). Social media companies, and

internet platforms in general have been given the task, indirectly speaking, “to balance

between freedom of expression and social responsibility” because the internet, as innovative

as it is, is open for “use and abuse” (Almagor, 2017, p.452). This task, traditionally speaking,

did not belong to social media companies but was only entrusted to courts and authorities,

but with the immense role that social media companies have nowadays, it can be argued that

it is something necessary. Simultaneously, one has to bear in mind that entrusting social

media companies with more and more of these complex tasks could also lead to a situation

where these companies are overwhelmed with all the responsibilities that are being assigned

to them and fail to deliver according to what it is expected.

In conclusion, I believe that terrorists (and terrorist organizations) along with their supporters

and devotees will always find new means and alternative methods (or even alternative social

media platforms) to spread their desired messages and propaganda online. But because of

this sole reason, the need for new emerging strategies to limit, if not fully counter the

dissemination of such content online should be further stressed and researched on. In 1996,

Garland believed that although put in practice, society back then was not fully ready to make

use of the strategy of responsibilization, but it is possible that by now, over two decades later,

merging governmental, scholarly, public and corporate efforts in finding effective means to

counter the issue of online crime, will shed new perspectives regarding his criticism and the

effectiveness of the practice. Moreover, I argue that although social media companies are key

in shaping public and political debate nowadays, overestimating their capabilities and

ascribing too much responsibility outside of their initial competences could have possible

negative impacts. Therefore, I believe that nowadays emphasis should be centered towards

managing to concede on a general definition of terrorism and terrorist content so that general

agreement can be reached with regards to what exactly needs to be regulated and combated,

thus little room for ‘false-positives’ and ‘false-negatives’ would be available.

41

5. References

Al-Rawi, A., & Groshek, J. (2018). Jihadist Propaganda on Social Media: An Examination of ISIS Related

Content on Twitter. International Journal of Cyber Warfare and Terrorism (IJCWT), 8(4), 1-15.

Awan, I. (2017). Cyber-extremism: Isis and the power of social media. Society, 54(2), 138-149.

Berger, J. M., & Morgan, J. (2015). The ISIS Twitter Census: Defining and describing the population of ISIS

supporters on Twitter. The Brookings Project on US Relations with the Islamic World, 3(20), 4-1.

Callahan, G. (2017). Attacking ISIL on Twitter: Addressing ethical responsibility in the weaponization of social

media. Intersect: The Stanford Journal of Science, Technology, and Society, 10(2).

Cohen-Almagor, R. (2017). The Role of Internet Intermediaries in Tackling Terrorism

Online. Fordham L. Rev., 86, 425.

Coldewey, D. (2018). Facebook removed 14 million pieces of terrorist content this year, and the numbers are

rising. Retrieved from https://techcrunch.com/2018/11/08/facebook-removed- 14-million-pieces-of-

terrorist-content-this-year-and-the-numbers-are-rising/?guccounter=1

Council of Europe, Convention on Cybercrime, ETS No. 185.

Cilluffo, F. J., Cardash, S. L., & Whitehead, A. J. (2006). Radicalization: Behind bars and beyond

borders. Brown J. World Aff., 13, 113.

Davis, B. R. (2006). Ending the Cyber Jihad: Combating Terrorist Exploitation of the Internet with the Rule of Law

and Improved Tools for Cyber Governance. CommLaw Conspectus, 15, 119.

European Commission, Proposal for a Regulation on Preventing the Dissemination of Terrorist Content Online,

COM (2018) 640 final.

Garland, D. (2001). The culture of control: Crime and social order in contemporary society.

Garland, D. (1997). Governmentality' and the problem of crime: Foucault, criminology, sociology. Theoretical

criminology, 1(2), 173-214.

Garland, D. (1996). THE LIMITS OF THE SOVEREIGN STATE Strategies of Crime Control in Contemporary

Society. The British journal of criminology, 36(4), 445-471.

Goldsmith, J. (2019). Sovereign Difference and Sovereign Deference on the Internet. Past, Present.

Hache, E. (2007). Is responsibility a tool of neo-liberal governmentality? Raisons politiques, (4), 49-65.

Hill, R. (2016): Internet governance, multi-stakeholder models, and the IANA transition: shining

example or dark side?, Journal of Cyber Policy, DOI: 10.1080/23738871.2016.1227866

Jeong, S. (2016). Twitter: We’ve Blocked 125,000 ISIS Accounts Since Mid-2015. Motherboard.

Retrieved from http://motherboard.vice.com/read/twitter-we-blocked-125000-isis-

accounts-since-mid-2015

42

Jee, C. (2019, March 22). Russia wants to cut itself off from the global internet. Here's what that really

means. Retrieved from https://www.technologyreview.com/s/613138/russia-wants-to-cut-

itself-off-from-the-global-internet-heres-what-that-really-means/

Kaunert, C., & Léonard, S. (2019). The collective securitisation of terrorism in the European

Union. West European Politics, 42(2), 261-277.

Kwak, H., Lee, C., Park, H., & Moon, S. (2010, April). What is Twitter, a social network or a news media?,

In Proceedings of the 19th international conference on World wide web (pp. 591-600). AcM.

Lemke, T. (2002). Foucault, governmentality, and critique. Rethinking marxism, 14(3), 49-64.

Mittal I.P.S., S., & Sharma, P. P. (2017). A Review of International Legal Framework to Combat

Cybercrime. SSRN Electronic Journal. doi:10.2139/ssrn.2978744

O’Neill, P.H. (2016, February 2). How ISIS trains militants in online security. The Daily Dot. Retrieved

from http://www.dailydot.com/layer8/isis-aef-nsa-snowden-video/

Rasmussen, J. (2015). ‘Should each of us take over the role as watcher? ’Attitudes on Twitter towards the

2014 Norwegian terror alert. Journal of Multicultural Discourses, 10(2), 197-213.

Rose, N., O'Malley, P., & Valverde, M. (2009). Governmentality. Annu. Rev. Law Soc. Sci., 2, 83-104.

Schmid, A. (2004). Terrorism-the definitional problem. Case W. Res. J. Int'l L., 36, 375.

Sharma, R (2018). 100 Social Media Statistics 2019 Infographic for Business. Statusbrew Blog

Tsesis, A. (2017). Social media accountability for terrorist propaganda. Fordham L. Rev., 86, 605.

Wagner, K. (2019, April 16). Twitter says it's getting better at detecting abusive tweets without your

help. Retrieved from https://www.vox.com/2019/4/16/18410931/twitter-abuse-update-

health-technology-harassment

Weimann, G. (2010). Terror on facebook, twitter, and youtube. The Brown Journal of World

Affairs, 16(2), 45-54.

Wu, P. (2015). Impossible to regulate: Social media, terrorists, and the role for the UN. Chi. J. Int'l L., 16, 281.

43

Online Sources cited:

European Digital Rights Associations (EDRi)

https://edri.org/about/

https://edri.org/eu-internet-forum-document-pool/

EDRi Open Letter

https://edri.org/open-letter-regulation-on-terrorist-content-online-endangers-freedom-of-expression/

Europol Conference on online terrorist Propaganda

https://www.europol.europa.eu/newsroom/news/europol-hosts-conference-online-terrorist-propaganda

Europol Main Reports EU Terrorism situation

https://www.europol.europa.eu/activities-services/main-reports/eu-terrorism-situation-and-trend-report

EU Internet Referral Unit – EU IRU

https://www.europol.europa.eu/about-europol/eu-internet-referal-unit-eu-iru

Global Internet Forum to Counter Terrorism

https://blog.twitter.com/en_us/topics/company/2017/Global-Internet-Forum-to-Counter-Terrorism.html

https://www.gifct.org/press/global-internet-forum-counter-terrorism-update-our-efforts-use-technology-

support-smaller-companies-and-fund-research-fight-terrorism-online/

General Assembly Resolution 57/239,2003 U.N. Doc

https://www.itu.int/ITU-D/cyb/cybersecurity/docs/UN_resolution_57_239.pdf

General Assembly Resolution 58/199, 2004 U.N. Doc

https://www.itu.int/ITU-D/cyb/cybersecurity/docs/UN_resolution_58_199.pdf

Internet Corporation for Assigned Names and Numbers (ICANN)

https://icann.org/general

Social Media Statistics

https://blog.statusbrew.com/social-media-statistics-2019/

The Guardian

https://www.theguardian.com/technology/2018/apr/05/twitter-bans-270000-accounts-to-counter-terrorism-

advocacy

Twitter Blog

https://blog.twitter.com/official/en_us/a/2016/an-update-on-our-efforts-to-combat-violent-extremism.html).

Twitter Rules

https://help.twitter.com/en/rules-and-policies/violent-groups

Twitter Terms of Service (Previous)

https://twitter.com/en/tos/previous