Form Digitization in BPO: From Outsourcing to Crowdsourcing?

Jacki O’Neill1

Shourya Roy2

1 Xerox Research Centre Europe

6 Chemin de Maupertuis

Meylan, 38240, France

[name.surname]@xrce.xerox.com

Antonietta Grasso1 David Martin

1

2Xerox Research Centre India

Bangalore, India

shourya.roy@xerox.com

ABSTRACT

This paper describes an ethnographic study of an

outsourced business process – the digitization of healthcare

forms. The aim of the study was to understand how the

work is currently organized, with an eye to uncovering the

research challenges which need to be addressed if that work

is to be crowdsourced. The findings are organised under

four emergent themes: Workplace Ecology, Data Entry

Skills and Knowledge, Achieving Targets and

Collaborative Working. For each theme a description of

how the work is undertaken in the outsourcer’s Indian

office locations is given, followed by the implications for

crowdsourcing that work. This research is a first step in

understanding how crowdsourcing might be applied to

BPO activities. The paper examines features specific to

form digitization – extreme distribution and form

decomposition – and lightly touches on the crowdsourcing

of BPO work more generally.

Author Keywords

Crowdsourcing; Ethnography; Business Process;

Outsourcing.

ACM Classification Keywords

J.4 [Social and behavioral sciences] Sociology; H.5.3

[Group and Organization Interfaces]: Collaborative

computing.

General Terms

Human Factors; Design.

INTRODUCTION

When the Web moved from a publishing platform to a

collaborative one, a new set of possibilities for distributed

and collaborative working arose. Web 2.0 has made

possible a scale of collaboration that was not conceivable

before. We use the term collaboration in a loose manner:

many individuals can contribute small parts to create some

greater whole without necessarily having to work together

or coordinate overtly. Crowdsourcing – basically where

task outsourcing is delegated to a largely unknown Internet

audience – is emerging as a major example of such

collaboration.

Crowdsourcing is the act of taking a task traditionally

performed by an employee or contractor, and outsourcing

it to an undefined, generally large group of anonymous

people, in the form of an open call1. Over the last few years

crowdsourcing has been becoming increasingly popular due

to factors such as the proliferation of Internet access and

mobile devices in emerging nations like India, and

increasing numbers of people opting for alternate modes of

employment [11]. Amazon Mechanical Turk (AMT) is

probably the best known crowdsourcing micro-task

platform where a group of individuals or organizations

(requesters) post small tasks in large volumes to be taken

up by individuals (workers) for execution. After execution,

the workers post back their results for evaluation and get

paid on acceptance by the requester. Examples of such

tasks range from digitization of scanned documents,

translation of text, to transcription of audio files and so on.

AMT has thousands of such tasks of small granularity

which often can be executed in seconds and minutes, with

payments usually in the order of a few cents.

In the business world questions are being raised about

whether crowdsourcing could be a replacement for

outsourcing, and a number of small scale start-ups seem to

be quite convinced about the business potential.

CrowdEngineering, Microtask, CrowdFlower,

ClickWorker, LiveOPs, GetSatisfaction, DSTTechnologies,

are a few representative ones. Whilst some companies

employ task-specialized crowds, others make use of general

micro-task platforms such as AMT to execute tasks which

companies used to previously outsource.

Business Process Outsourcing (BPO) started gaining

critical mass nearly two decades ago when companies from

USA and Europe started migrating ‘non-core’ business

processes like administration and customer care to the new

BPO specialists primarily as a means of cost saving. This

was followed closely by ‘off-shoring’ where the work was

moved to countries with lower labour costs, such as India.

Technology innovation was crucial in this radically

different delivery model where work origin and workers

1 http://en.wikipedia.org/wiki/Crowdsourcing

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are

not made or distributed for profit or commercial advantage and that copies

bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior

specific permission and/or a fee.

CHI 2013, April 27–May 2, 2013, Paris, France.

Copyright © 2013 ACM 978-1-4503-1899-0/13/04...$15.00.

were geographically separated. For tasks like digitization of

insurance claim forms, the offshore delivery models have

been extremely successful in spite of critical data privacy

and security concerns. BPO companies send work to

thousands of workers in global service delivery centers,

proving that the security and privacy concerns (along with

geopolitical risks) can be successfully mitigated. As part of

the ongoing pursuit of cost savings, new labour markets

such as crowdsourcing come under scrutiny as possible

alternatives to the existing work organization.

Crowdsourcing offers several advantages over outsourcing:

It can be more cost effective, even when employers are

operating at the minimum wage level, because there is

a large reduction in infrastructure costs, as employers

are no longer required to host workers themselves –

thus saving on building, security and technology costs.

Crowdsourcing can bring together diverse groups of

people, from all over the world. It could tap into the

labour pool of people who, for whatever reason, are

unable or unwilling to work full-time in an office

environment, but who could put in some (extra) hours

a week. The flexibility offered by Crowdsourcing

coupled with global reach can lead to accessing better

qualified workers, leading to higher quality output.

Related to this, crowdsourcing offers interesting

possibilities for Corporate Social Responsibility

(CSR), in particular providing jobs where needed (e.g.

in rural India) to populations who currently have

limited work opportunities and could benefit from even

small amounts of extra income. This requires both

technological infrastructure and education and in India,

for example, the Government is pursuing an aggressive

program to deploy broadband to rural areas. The rise

of rural BPO’s [12] attests to the number of literate,

educated workers in these areas.

Crowdsourcing offers greater possibilities for on-

demand labour, with easier ramping up and down of

workforces, although this depends partly on task

learning and expertise requirements. There is a

possibility for easier 24/7 labour as workers self-select,

either preferring non-office hours e.g. choosing to

work 4-6am or coming from different time zones. This

is important, since the requirement to work shifts is a

major driver of attrition.

Despite these advantages, there remain a number of

challenges to be overcome before crowdsourcing can be

viewed as a viable alternative to outsourcing. Fundamental

research questions pertaining to branches of computing

such as HCI, analytics and machine learning, as well as

social science, economics and so on need to be solved.

Challenges include managing data security, quality

assurance and the extreme work distribution often involved

in crowdsourcing.

This paper contributes to the research into whether

crowdsourcing is an appropriate next step for outsourcing,

by examining one specific area of BPO work – healthcare

form digitization. We undertook an in-depth ethnographic

field study of the work involved in digitizing healthcare

forms, as done today in outsourced operations. Our goal

was to understand what challenges – social, technical and

organizational – need to be overcome and in what ways the

work might need to be reordered if it were to be completed

by a crowdsourced labour force, whilst at the same time

meeting key business constraints. This research was

undertaken as part of a larger project which aims to

understand the applicability of crowdsourcing to various

types of BPO work and to produce technology innovation

in this domain. Our approach differs from that of much of

the ongoing research in the crowdsourcing domain, which

tends to focus on experimental studies of AMT use. We

start by understanding the organization of the work as it is

currently achieved by one labor force, as a way of revealing

the challenges which might arise if distributed to an

alternate workforce. This approach was chosen in part

because the business constraints of timeliness of

completion – TurnAround Time (TAT) - and the strict

quality criteria outlined in the Service Level Agreement

(SLA) impose additional levels of complexity. We also

believe that starting with the work, as done now, brings a

fresh perspective to crowdsourcing research.

In the rest of this paper, we outline relevant related research

(Section 2). Then, in Section 3, we describe how the work

is currently organised to achieve TAT and quality at

minimum cost, and highlight some of the issues that need

to be taken into account if the work is to be undertaken

through crowdsourcing. In the following section 4, we

present our findings from the ethnographic field study and

their implications for the crowdsourcing of that work. The

findings are organised around four themes which emerged

from the data: Workplace Ecology, Data Entry Skills and

Knowledge, Achieving Targets and Collaborative Working.

In Section 5, we discuss the import of these findings for

BPO work in general and form digitization in particular,

especially around extreme distribution and form

decomposition. We sum up with a short conclusion

(Section 6). At this point we raise more questions than we

answer, but we believe this is still of value to the HCI

community given their growing interest in crowdsourcing.

If we are to design robust crowdsourcing systems for BPO

work, then we believe that an understanding of that work

can inspire innovative design to remodel that work for the

crowdsourcing environment as well as make us aware of

the challenges and risks.

RELATED WORK

Much of the research so far has consisted of experimental

studies using AMT to investigate various aspects of the

crowdsourcing of micro-tasks. Whilst they rarely include

common BPO tasks, much can be learned from these

studies. Early research focused on the wisdom of the

crowd, in particular demonstrating that the crowd could

produce results of equivalent value to those produced by

experts. For example, [10] carried out experiments on AMT

comparing crowd and expert ratings of Wikipedia articles

to understand how the crowd would perform in different

circumstances. They observed low correlation between the

two groups which was primarily owing to large number of

malicious users gaming the system. They found that it was

important to have multiple ways of detecting suspicious

responses and that honest completion of tasks should not

require significantly more effort than deliberately giving

bad answers. In Soylent [4], the authors described an AMT

based interface for shortening, proofreading and general

editing of documents. They demonstrated that good

performance could come from putting such work out to the

crowd – with good quality summaries produced. They

introduced the notion of crowd programming i.e. that by

designing tasks in certain ways the crowd could be

‘programmed’ to perform well. For example, they

described a crowd programming pattern known as Find-

Fix-Verify where complex crowd intelligence tasks were

split “into a series of generation and review stages that use

independent agreement and voting to produce reliable

results”. They also described two extremes of crowd

worker – neither of which produced the best results: the

Eager Beaver, who put in almost too much effort,

shortening the text so much that meaning was lost versus

the Lazy Turker who might only change one or two words.

Getting the best out of the crowd

Another set of papers have explored various mechanisms

for getting the best out of the crowd by manipulating

parameters such as task design, pay, difficulty and interest

of work. For example [9] manipulated a number of factors

such as effort required to complete a task, incentive and so

on. They found that higher pay increases completion rate,

time spent on the task and quality, but that qualified

workers are less affected by pay. In contrast another paper

[15] found that pay did not increase quality, although it did

increase throughput. [5] Focused on experiments used to

build theoretical models to predict how people would

respond to different price/task scenarios and also how to

tune them dynamically to account for individual biases.

These models accounted for “(1) how workers decide

whether or not to participate in a crowdsourcing project and

(2) how workers decide the amount to produce”. One

finding was that wage per job was not the sole motivator,

but that workers also focused on their ability to reach

salient targets, e.g. whether there was enough work in this

category for them to earn some target amount. [2]

Examined incentives and reputation (of the crowd worker)

in a competitive software development portal, where

challenges are set and prizes awarded to winning solutions.

Not surprisingly the prize amount was a strong determinant

of individual performance, however they also found that

reputation has significant economic value i.e. workers

would take jobs which are likely to improve their

reputation ratings. Workers also considered the specificities

of individual jobs and how they relate to their personal skill

set, rather than being swayed by task design factors such as

the length of the requirements document.

Other papers examine how quality can be determined, for

example, by using multiple workers on the same task [1].

The underlying principle being, if two or more people

independently agree on an output it can be considered

correct. This is known as “output agreement” or

“independent agreement” and works well for tasks like

image tagging, where giving a true answer exists. Where

there is no single response, e.g. translation, another

mechanism, Tournament Selection (typically used in

Genetic Algorithms to select the best individual from a

population of individuals), has been applied [21]. In this

scenario, one group of people generates a set of plausible

answers (population) which subsequently undergo multiple

rounds of pair-wise selection steps by other groups of

people where the winner moves to the next round

(crossover). It was found that after three or four rounds the

best answer is found.

A problematic issue, particularly for business process

crowdsourcing, is that for work currently crowdsourced not

only does completion time vary but some tasks never get

completed [6] and there is a tail where some tasks need an

inordinate amount of time for completion [7]. In the latter

paper it was also found that there is no stable average

completion time of tasks posted on AMT. Clearly these

findings pose potential problems for tasks where timeliness

of completion is crucial.

Other research examines crowd collaboration. [18] Looked

at translation and story writing and found that collaboration

enabled higher quality output and encouraged learning, as

users could, for example, ask for the meaning of words

while translating. They coined the term Social

Crowdsourcing to describe this. Another study

demonstrated how difficult semantic tasks could be

completed by an iterative crowdsourcing process i.e. one

where each worker builds on the output of the previous

worker [19]. Whilst this is not direct collaboration, the

output might be considered to be collaborative, and most

importantly they showed great success in addressing tasks

which were too difficult for a single person. E.g. the

transcribing of almost illegible documents (such as doctors

prescriptions) were near perfect after around five iterations.

Crowdsourcing for BPO

Some research has focused on the BPO environment. For

example, in the call centre environment, [22] described

how orchestration of resources and control of processes

with respect to time, delivery and quality are key

requirements for enterprise crowdsourcing. They proposed

a concept of customer calls being redirected (as calls or

transcribed messages) to be resolved by online

communities. This resonates well with our domain. Whilst

we are not looking at call centres with their extreme

responsiveness requirements, the ability to complete

processes rapidly to high quality standards will be crucial.

A key player exploring crowdsourcing for BPO is IBM,

who have been experimenting with different types of

crowdsourcing in their business. In one paper [23] they

used crowdsourcing for IT optimization using a tool called

PeopleCloud. PeopleCloud is a platform which a) enables

the enterprise to access an online scalable workforce, and

b) provides an interface to those services required for

carrying out the crowdsourcing tasks. It enables different

roles to be defined including Requestor (define a problem),

Business Owner (approves tasks and provides

rewards), Service Providers (execute tasks, essentially

crowd), Collaborator (team formation, providing inputs).

Using this platform they created an up-to-date data

repository for around 4,500 business applications using

user input. Previously the data repository had been very out

of date since it was maintained by overstretched systems

administrators. The crowd results considerably improved

the situation at little cost and to the benefit of all. The paper

is interesting as it presents a business crowdsourcing

application where the contributors are ‘paid’ in points not

money, but they can perform the tasks at little cost to

themselves (being as they are regular users of the business

applications and have the information to hand), with a

consequent benefit to themselves and others. This differs

from the work in the second paper from IBM [8], which is

much closer to our business domain. The authors advocate

the idea that outsourced service delivery centres will

become Virtual Service Delivery Centers (VSDC), where

currently outsourced processes will be carried out through a

mixture of automation and crowd processes. The role of

these centres then will be to pass the parts of the process

which can be automated to the relevant automation service,

and for the rest to mediate between the business and the

crowd. Overall they would be in charge of guaranteeing

availability, quality, anonymity, privacy and security. They

describe a novel way to enable an OCR correction task to

be crowdsourced and raise a number of issues of relevance

to our domain, such as: 1) Quality control and pay - by

introducing known errors into the data, worker quality can

be judged and pay scales determined; 2) Security issues are

addressed through extreme task decomposition; 3)

Workforce availability to achieve TAT - since the task is

very low skilled people can be pulled into the workforce

and trained up rapidly to maintain a large pool of

employees. Unfortunately the authors did not give any

figures or examples of how this worked in practice, given

that this was a short paper; 4) Quality assurance for SLA is

handled using the skill level determined by the worker

assessment mechanism described in 1) above to determine

whether the same data will go to 1, 2 or 3 employees and

the results matched. This research is the closest to our

application domain and is a first step at addressing some of

the major domain issues. In this paper however we take a

wider focus. Rather than considering just one aspect of the

task itself (in our case digitisation of handwritten forms not

suitable for OCR), we look at the wider context of the

work, i.e. including social and learning aspects and the

work that is done to ensure the forms are processed

according to business requirements.

Finally, other less task specific work focuses on features

essential for crowdsourcing of BPO work, such as

scheduling. [17] Describes a scheduling and monitoring

mechanism that can monitor the capabilities of the crowd,

trigger agreement violations, and deploy counteractions to

compensate service quality degradation. The idea is to

provide a means for organising the mixture of skills and

availabilities of the crowd so they will act in a way that will

fit with the customer’s business needs, in particular their

SLA’s and time and quality measures. They include the

idea of ‘distinguished’ crowd members becoming

responsible points of reference. These members mediate the

crowd, settle disagreements, organize activities, schedule

tasks, and monitor behaviour.

METHOD AND FIELD SITE

Although crowdsourcing seems to offer the potential to

revolutionise some types of outsourcing, this is not

unproblematic technically or organisationally. In an effort

to understand what it would take to crowdsource a

particular form of BPO work – relatively low-skilled data

entry work – we embarked on an in-depth field study of

that work as it is undertaken now in an outsourced

environment. The aim being to uncover the opportunities

and challenges that would be faced if this work were to be

crowdsourced and indeed at a more fundemental level to

see whether this work would be crowdsourceable.

We followed the HCI and CSCW tradition in which

ethnomethodological ethnographies have long been used to

understand the application domain and inform design

thinking [3]. By understanding the work as it is undertaken

now and given a particular technology scenario, in this case

crowdsourcing, we can begin to map out what is required to

undertake that work in this new setting, as has previously

done with, for example, mobile payments [13].

The work setting under investigation was a BPO operation

involved in the digitisation of healthcare forms for US

insurance companies. This is basically data processing

work with a heavy human element. Forms arrive in

dedicated mail rooms from all over the US. They are

scanned and where possible undergo OCR. The scanned

forms and OCR output are routed to various offshore

centres in India, Mexico, and Ghana. In these centres a

series of human and systems steps are undertaken to ensure

that the correct information is transferred from the scanned

form to the electronic database within various time limits.

This field study was carried out at two Indian locations:

Bangalore and Kochi between March and August 2011,

with approximately 5 working weeks spent in the field. The

ethnographic study primarily consisted of observation of

the entire offshore workflow handled in India for three

different clients, supported by in situ interviewing of the

people involved. The offshore human data processing

workflow consists of a number of sequential steps. The

following are undertaken for all clients 1) data entry –

both full-key (where forms are not suitable for OCR e.g.

handwritten forms, they must be keyed into the database

from scratch by hand) and partial-key (the system detects

possible errors in OCR which need to be checked by a

human); 2) verification, through double key i.e. the agents

enter the information anew, but receive prompts if their

entries are different from those of the prior agent; 3)

review, where agents check specific fields as suggested by

a rules engine – this includes potential keying errors plus

pre-defined checks e.g. if the claimed amount is greater

than $5,000. In addition, extra functions may be performed

according to client need such as look-up of healthcare

providers in a client directory; reject review, ensuring that

forms which are rejected (i.e. where one of the agents has

said they cannot be keyed because they do not meet some

pre-set criteria) have been rejected for a legitimate reason;

and splits, where a form which has been rejected for

containing e.g. two patient names is divided into two

claims, one for each patient, and sent back to the agents for

data entry.

There are a number of other activities which are carried out

outside of the workflow to ensure the smooth running of

the business, that customer requirements are met and that

the workflow activities run smoothly. This articulation [20]

and management work includes 1) quality audits and six

sigma projects to improve particular quality problems; 2)

supervision and management, including shift organisation,

reporting and floor-walking (i.e., on floor supervision,

which will be discussed in more detail later); 3) production

control – a specific function for monitoring the flow of

forms (known as claims) through the all the various

workflow steps, both onshore and offshore, and

troubleshooting when problems occur.

A fundamental difference between this type of work when

it comes to crowdsourcing and other BPO work that has

already been attempted is the ongoing nature of the work,

which is regulated by tight time and quality requirements,

of which we will discuss more later. In this paper we

concentrate on what it would mean to crowdsource the data

entry steps for non-OCR forms. This is considered low skill

data entry work, but as we will show it already poses

considerable challenges for crowdsourcing.

FIELDWORK FINDINGS AND THEIR IMPLICATIONS FOR CROWDSOURCING

In this section we will describe some of the key findings

about how the work is carried out now and discuss what

this means for crowdsourcing. The findings are organised

around four themes which emerged from the data:

Workplace Ecology, Data Entry Skills and Knowledge,

Achieving Targets and Collaborative Working. Before we

move to these themes, however, we found that there are a

number of enablers for crowdsourcing already present in

the way that the work has been organized as outsourced

labour.

Outsourcing to Crowdsourcing; Facilitators

A number of advantages, as far as crowdsourcing is

concerned, come from studying an already outsourced

process:

1) The process is already distributed: any of the sequential

steps of the workflow can be done by any capable team of

agents at any site.

2) The process is managed semi-automatically by a

powerful Workflow Tool (WT) and a group of employees

known as Production Control. Together they manage

distribution between sites and agents, given upcoming

deadlines, agent skill sets and volume.

3) There is little interdependence between each of the

sequential steps: it is not necessary to know who did the

prior step or who will do the next step, there is no need to

discuss what was done previously and what will be done

next when processing any particular claim. Any quality

implications, e.g. data entry errors uncovered in subsequent

steps, are handled outside of the processing workflow in a

separate quality process.

4) The work is carried out on specifically designed

applications, which allow agents access only to the current

form they are processing and to enter (or check) data for

that form. Other activities are not allowed and access to

client systems is not given.

In terms of technology, the agent sees the scanned image of

the form on the top of the screen and a database form on

the bottom of the screen. The database may be prefilled or

empty, depending on OCR and the stage of the workflow.

For the standard forms, the field of the form (on both the

scanned copy and the database system) to which the agent

must attend next is highlighted enabling the agent to move

quickly through the form.

In effect then one might think that the challenges of

distribution have been solved and to a large extent this is

true in the sequential steps of the workflow through which

each claim passes which is controlled by the workflow tool.

However, much of the work to make the workflow work is

carried out locally. That notwithstanding the very fact that

the process has already been outsourced and is widely

distributed across teams and countries gives us a head start

when thinking of crowdsourcing but is not sufficient.

Workplace ecology

The outsourcers’ offices in both Bangalore and Kochi are

located on several floors across a couple of multi-story

office blocks in two technical parks. Data entry operations

are configured by client. That is, employees work on data

entry for single clients; client groupings are located as

separate functional units within these floors. Either by

seating area, indicated by banners, or in separated-off

access-controlled offices. Personnel need a security pass, or

to register as a visitor, to enter the tech park itself and then

to enter the outsourcers’ general offices, which are manned

by security guards. For many clients, particularly those

with independently secured office spaces, there are lockers

outside where staff have to leave their bags and phones

during the shift. Typically no paper or pens are allowed in

the office areas. Agents usually work shifts, with each shift

having at least one team leader present. By being collocated

with their teams, team leads can monitor the activity on the

floor. They spend a large amount of their time walking

around the floor, answering queries from the agents and

keeping an eye on their activity. In addition staff occupying

other roles such as quality and management generally have

either offices or desks within the unit. They do not work

shifts however they often time their office hours to overlap

at least partially with their client’s (US) office hours. On

each floor there is a break area with free coffee and water,

and tables and chairs. In both sites the tech parks have a

number of shops, restaurants and banks. Production control

is located elsewhere: mostly in the USA, but for one group

of clients they are on another floor in the Bangalore offices.

Each production control group handles the workflows for a

number of client operations.

Since health care forms contain personal information of the

clients’ customers, including social security numbers, name

and address details, the security of the data is governed by

US laws, in particular HIPPA compliance, which strictly

controls who may access the data and protects against

unauthorized distribution and use. These laws are there to

ensure that no customer data can fall into the wrong hands

and this is the main reason why mobile phones, pens and

paper are not allowed in the office. Data security is a key

concern of the clients and therefore of the outsourcer. So

the systems and workflows have been designed to ensure

maximum security. For many of the clients, the agents

access the data through iGELs (a type of thin client) rather

than PC’s. iGELs have no USB or other data retrieval ports

and do not store any information. Data is not stored locally;

rather it is stored in the US and pulled in batches to the

agents’ computers as they are ready to process it. Any

software modifications are done through the US. In

addition to this technical enforcement of data security, data

is also secured through the physical set-up (restricted

access to offices) and social means (supervisors enforce the

rules so agents cannot copy any of the data). Agents are

also trained in HIPPA compliance when they join the

company and have refresher training regularly.

Implications for crowdsourcing

In summary, data security is currently enforced through

physical (office space), social (supervision and training)

and technical (iGels, batching, etc.) means. The main

implication for crowdsourcing comes from the distribution

of the workers from controlled office environments into

their own homes or unsecured Internet cafes and their lack

of a contractual relationship with the company. The control

the outsourcer can exercise over the people doing the work

is necessarily reduced. In effect then, security can no longer

be enforced by physical and social means and solutions to

HIPPA compliance etc. will have to be wholly technical.

Data entry skills and knowledge

Agents have a decent level of education (graduate or

undergraduate), good English language skills and good

typing speed. The data entry work is known as ‘key what

you see’ and is considered low skilled work; nonetheless

the learning curve of an average new entrant is around

seven weeks to meet speed and quality expectations. This is

because in reality data entry is not simply ‘key what you

see,’ rather the agents must interpret what they see

according to an extensive rule set. To illustrate, the most

straightforward data entry task is the HCFA form: a

standard form for claiming insurance on medical

procedures undertaken. The name field of this form has

around 13 rules for how the name should be entered i.e.

which is the first name, surname, middle name and

credentials. This is just one of 33 fields. The outsourcer

processes a number of different forms and other documents

for each client, thus agents need to learn how the various

rules apply for all the different form types. In addition, task

complexity is situational. It differs between form types e.g.

HCFAs are relatively standard, whereas Correspondence is

by its nature non-standard. Data entry for Correspondence

is only 4 fields (name, social security number, etc.)

compared to 33 fields for HCFA forms, however each item

of Correspondence can take considerably more time to

complete as the information may be anywhere or nowhere

on the form: identifying that information is not present

often takes longer than finding and entering that

information where it is present (because of the need to

double or triple check). In addition, for some types of jobs,

agents have to first check that the information matches

between different pages of the scanned image, before doing

a data entry step, e.g. Explanation of Benefits (EOB) jobs.

Whilst data entry is rapid, checking is slower. At this point

we should mention the pay structure for the agents. Agents

are paid on the basis of performance (Performance Related

Pay (PRP)) – that is, they are paid per keystroke or per

form (depending on the job type) with quality taken into

account. Thus their work speed is of immediate concern to

them.

As well as complexity differing between form types, in

predictable ways, complexity also differs within form types

i.e. across individual claims. Thus handwriting and poorly

printed forms can be difficult to read. Just like learning the

rules, handwriting deciphering is a learned skill. Even the

newest agent was better at deciphering handwriting than the

fieldworker and the team and quality leads could often

decode text that the fieldworker thought unrecognisable.

Another type of within form complexity is where forms

need to be rejected for some reason – that is they do not fit

the criteria for data entry so the agents cannot enter them

into the database. This may be because of poor scanning

(alignment, too faint, etc.) or because they do not comply to

the rules e.g. containing information about more than one

patient or health care provider. Reject decisions take time

because they also require double checking and agents are

accountable for their rejections i.e. wrongly rejected forms

will bring the supervisor’s attention. Finally, non-standard

means non-standard, so a piece of correspondence may

have a cover sheet with all the information required for data

entry on it, or the information may not be found in the

document at all. Within form complexity can only

determined on a document by document basis and would

not be easy to predict in advance.

Implications for crowdsourcing

Certainly it would be good to develop techniques to

accommodate the learning curve, starting the workers on

just one form type, or splitting the form into sections so that

agents only need to learn the rules for one particular

section. Such specialization is commonly thought to be an

essential element of crowdsourcing [8] and indeed recent

research has taken this concept a step further by

introducing the notion of Hyperspecialization [14]. We

could also imagine techniques for making the rules more

easily available for agents in situ. However, when

crowdsourcing just as within the office, it is clearly

beneficial to both employer and worker for speed and

quality to have experienced agents. Thus we may want to

think of crowd models that encourage agents to specialise

and become skilled.

Secondly, the fieldwork raises the question of how

incentives can be determined given the situational

complexity of the work. Currently different pay scales are

determined on the basis of form type, but agents have

social and organisational pressures on them to encourage

them to complete the batches they pull e.g. supervisors will

hold them accountable for rejecting batches (discussed

below). In the crowdsourcing situation, what is to stop

agents from rejecting difficult work? Assuming this will

happen we need models which spot and deal with such

instances whether these are incentive schemes - although

incentive schemes may not work on their own - or hybrid

models of in-house and crowd workers.

Achieving targets

The SLA is the contractual agreement between the

outsourcer and the client specifying what will be achieved.

In the case of healthcare data entry two components are key

- the TurnAround Time (TAT) and quality - and penalties

are often in place if either is not met to agreed levels.

Different job types for different clients have different TATs

i.e. time from receipt of form to claim being processed.

Targets include a) daily cut off times, by which all forms of

a certain type must be completed, or b) all forms must be

out of the system within some time limit based on when

they came into the system (typically 24 hours, but it can be

as low as 2-4 hours). Quality levels also vary between

clients and forms.

Performance Related Pay (PRP) is a strong motivating

factor, however pay alone is not enough to ensure the SLA

is achieved. Team and quality leads put in extra work to

make the agents accountable for their performance. For

example, during a shift the team lead will call people up

and talk to them about their performance. To illustrate, for

one client the data entry of medical records, must be

completed by 8am IST. A key concern then of the 6am shift

is completing all remaining medical records before the

deadline. The team leaders communicate the pressures of

the queue to the agents – ‘agents key fast’ ‘everyone on

medical records please’ ‘key fast but accurately’. In

addition after the deadline for finishing medical records has

past the team leader calls up 3-4 agents at a time and talks

to them (rather publically) about their performance. For

example, “Krishna, you only did 16 medical records in an

hour and a half. What was that about?” “Ok the target of 50

is not attainable but I would expect you to be doing at least

30. You need to improve.” Thus the agents are called upon

to account for their performance; similarly if they have

made particular errors. In addition, it shows how

management work to make the targets achievable for their

team – rather than telling someone only doing 16 medical

records they should be doing 50 which might seem

unachievable and therefore be demotivating, they set a

more realistic target. Thus the team lead is still showing the

agent that they need to improve, but giving them something

easier to aim for. Agents are similarly made accountable for

their performance when they reject batches of forms. Team

leads monitor which batches are being rejected and will ask

agents why they rejected a particular batch and if a batch is

rejected a couple of times will assign it to someone whom

they will tell they must complete it. In this way there a

balance is achieved between the agent’s desire to do easy

work quickly and the requirement to get all the work in the

workflow completed in a timely manner.

Currently training includes information on the whole

workflow so agents know how their work fits into the wider

process. This is done to improve quality, as agents are

made aware of which fields are checked and which are not

in later stages of the process. It is also a good general

principle, for enabling workers to make the workflow work

in the best way [16].

Implications for crowdsourcing

The current model of work is a push model, with a) the

work being assigned dynamically (by the workflow tool

and production control) according to agents skill set, i.e.

what forms and what tasks they have been trained on, and

queue; and b) various social (and financial) pressures.

In contrast the model of work in crowdsourcing is a pull

model; work is self-selected by the agents and they will not

have the same accountability for completing tasks as the

contracted workers. Any crowdsourcing system needs to be

designed to ensure that the work is completed in a timely

manner to good quality, for which coordination of the

workflow will be key. We may also want to consider

whether communication about the pressures of the queue

would be motivating for workers in the crowd or not and

whether incentive schemes linked to the urgency of the

work would be effective.

Collaborative working

Explicit collaboration has been designed out of the

workflow - claims progress through workflow steps from

agent to agent and even country to country automatically as

each prior step is completed. However, work is

collaborative at the claim level. That is, the routine troubles

encountered in data entry are solved with colleagues or

floorwalkers and it is not uncommon to see a group of two

or three people around a screen discussing an issue. Typical

issues that arise are deciphering handwriting or determining

which rule applies in this particular circumstance. Since

forms are filled in by people all sorts of phenomena may be

seen, e.g. arrows showing a name has been entered the

‘wrong way round’, crossings out and reentries,

clarifications written on the form and so on. Not all of these

are described in the rule set, which is updated as ‘typical

incidences’ arise. A large part of a team lead’s job is floor-

walking to answer such queries. However, there is typically

just one team lead and many agents so team leads are not

always immediately available when needed. Since time is

money for the agents and any time not entering data is

money not earned, they may turn to their colleagues when

the team lead is not available. Some teams encourage this,

with newcomers sat by experienced neighbours (thus

promoting collaborative learning), others discourage it. A

fairly typical rule of thumb however is that where the issue

is deciphering handwriting, agents will typically ask their

neighbour first, only turning to the team lead if needed. For

questions of rules the team lead is the first port of call, with

fellow agents only being asked when the team lead is busy

and typically it will nonetheless be followed up with the

team lead. Such troubles rarely take long to solve, in the

order of seconds rather than minutes, and it is undoubted

that this collaboration improves both speed and quality of

data entry.

Implications for crowdsourcing

We might want to think about enabling collaborative set-

ups amongst crowd members, enabling them to help one

another, or to have sub-crowds with particular skills, such

as handwriting deciphering to which data fragments might

be sent, although this implies more complex workflow

management. Although there is a general assumption that

crowdsourcing is supervision free, is there value in

considering pull models of ‘supervision’ and feedback

loops to enable on the job learning?

DISCUSSION

In the previous section we have described in some detail

how the work is organised now to achieve TAT and quality

at minimum cost and highlighted some of the issues that

need to be taken into account if the work is to be

undertaken through crowdsourcing. We hope that

understanding the detail of how such seemingly simple

work is carried out now, is already a useful contribution to

the research on crowdsourcing of business process

activities. The variety of issues it raises are not completely

new in themselves, with security issues, collaboration and

specialization, and the desire to have an experienced crowd

(as seen with the various reputation systems), being

covered elsewhere as reported in the related work section.

In contrast, some of the issues are rarely discussed, such as

how to manage situationally determined complexity (i.e.

diversity of seemingly similar tasks), support for learning

in the crowd and the impact of reduced accountability.

However, even for the issues touched on elsewhere our

study gives grounded examples from real work processes,

bringing out the subtlety of their connotations, not found

elsewhere and crucially grounded in a study of real work

processes. Before we turn to the connotations of this

research for the crowdsourcing of form digitisation, namely

extreme distribution and form decomposition, we want to

briefly discuss how the findings relate to the crowdsourcing

of BPO work more generally.

Implications for BPO work more generally

The core change for BPO work more generally is the lack

of contractual relations between the work provider and the

worker and closely related to this the reduction in

accountability of workers. This is likely to have different

impacts depending on the type of work being

crowdsourced. For example, in this case a major impact is

on data security, which can no longer be ensured through

contractual and supervisory means. However, the

implications are more generic; crowdsourcing involves a

pull model of work, where workers self-select what they

will and will not do, the employer has much less control

over them. In the employer-employee relation there are

myriad ways in which employees are made socially

accountable. In most crowdsourcing models these mixed

social and financial incentives/motivators are replaced

almost solely with monetary ones, even though it is clear

that money is not considered an adequate motivator on its

own for the in-office workers and there is no reason to

suspect that money will be the only motivator for

crowdworkers (see for e.g. [4]). Worse is that these

financial incentives tend to be at lower levels of pay per

item/hour than received by office workers! Of course, in

reality social aspects will remain part of the work; given a

fair working environment, workers will continue to do their

best and take pride in their work. Also for this work, as

with much crowdsourced work, reputation (of worker) will

be a crucial factor in determining who the work can be

selected by, at what price, etc. Along with financial

incentives, such social factors need to be understood and

fairly applied.

It is also important to remember that the seemingly

standard work of form digitization is not unique in having

‘hidden’ complexities. Currently managed by a variety of

routine techniques in house, such features become

important when the work is put out to potentially untrained

workforces, such as the crowd. To deal with this it is

important to understand in advance the detail of the work

and then we can envisage the use of hybrid workforces and

the development of methods for on-the-job learning even

for crowdworkers.

Extreme distribution and form decomposition

Turning now to form digitization specifically we need to

take into account how extreme distribution can be

managed. Although the work is already distributed between

sites, countries and agents, crowdsourcing will necessitate a

further distribution – to individuals at individual sites

(homes, internet cafes…) and quite likely of the data itself.

That is, whereas forms are currently distributed as a whole

to individual agents to work on them (for data entry, but

also for data correction, where the forms have been through

OCR) it is likely that only segments of each form would be

sent to any one agent in the crowdsourcing model. This

serves two purposes; 1) specialization e.g. reducing the

number of rules to be learned and hopefully increasing

throughput, and 2) to address issues of data security and

HIPPA compliance by removing problematic context from

presented data elements. An important question therefore

becomes, will decomposition and further distribution

disrupt the coherency of the work? The work of data entry

itself does not typically fall into a class of work where

collocation plays a major role in facilitating the work

because of, for example, the requirement for people to

troubleshoot vexing problems [19] or the strong

interrelation of different work sections [16]. For this type of

data entry there is no need to know who did the step before

you or who will do the step after you or where they will be

even if the outsourcer does train workers on how their work

fits into the larger workflow. This should make the work

easier to distribute; it is only cooperative in a limited way

and not interdependent at the same sort of level as for

example call centre and back office work – which is often

distributed to very negative effect. However, by further

decomposing and distributing the work a number of things

are potentially lost:

Learning from peers. In this case, how to apply the

right rules for this instance, plus potentially learning

the rules in the first place.

Interpreting challenging data, in this case,

handwriting. Both of these stem from the further

distribution of workers and could impact on output

quality and worker’s learning and progression.

Supervision and motivation, especially given the new

model of employment inherent in crowdsourcing

Team feeling and social interaction, although it

might be hoped that this is mitigated by the self-

selection of the crowd i.e. in an ideal world those who

would thrive in an office environment can work in that

way, whilst those who prefer to work in isolation can

choose to work in the crowd. One could also imagine

crowd working models where workers are based out of

internet centres – where social interaction and team

feeling comes not from employer-based groups but

rather from crowd working groups.

Relative ease of scheduling. Scheduling will become

even more complex, since different parts of the forms

are interdependent and some tasks (such as rejects) can

require viewing the whole form. In addition to

workflow management, ensuring there are enough

people to complete the work in a timely manner to the

required levels of quality is also a potential concern [9,

19]. Are we just to assume that the necessary people in

the crowd will always be there to do the work when

needed? This is rather big leap of faith and for the

crowdsourcing of key business process activities needs

to be proven.

The question of context. To what extent is the

knowledge that the data entry is for healthcare

insurance important? Is it important that a data element

is understood in relation to other elements in the form?

There are clearly intra-document dependencies that

relate to form validity but it is an open question as to

whether these can all be handled at a technical level.

Transparency of workflow might be negatively

impacted by form splitting, as it becomes harder to

know where your work fits in the wider workflow and

consequently what you need to take particular care

about. This might, but in this case not necessarily,

impact on quality. To understand this better one might

want to experiment with different set ups to enable

lightweight transparency and measure its effect.

CONCLUSION

Crowdsourcing is an emerging paradigm for very large

scale distribution of knowledge work. While it is being

leveraged extensively for data collection, digitization of

text and localization of content, its application in on-going

business processes has thus far been limited. In particular,

there has been little work towards attempting to answer if

crowdsourcing will be the next evolution of outsourcing. In

this paper, we presented results from an ethnographic field

study of form digitization with the objective of

understanding challenges to crowdsourcing this activity.

We identified four key issues which we described in detail,

and articulated corresponding crowdsourcing implications.

We believe that these are key issues for business process

crowdsourcing and present interesting technical and

organization challenges. In doing this we hope this helps

frame an agenda for other researchers in this area.

REFERENCES

[1] Ahn, L.V., and Dabbish, L. 2004. Labeling images

with a computer game. In Proceedings of the SIGCHI

conference on Human factors in computing systems (CHI

'04). ACM, New York, NY, USA, 319-326

[2] Archak, N. 2010. Money, glory and cheap talk:

analyzing strategic behavior of contestants in simultaneous

crowdsourcing contests on TopCoder.com. Proceedings of

the 19th international conference on World wide web

(WWW '10). ACM, New York, NY, USA, 21-30

[3] Bentley, R., Hughes, J. A., Randall, D., Rodden, T.,

Sawyer, P., Shapiro, D., and Sommerville, I. 1992.

Ethnographically informed systems design for air traffic

control. In Proc. CSCW’92. (1992), 123-129.

[4] Bernstein, M.S., Littl, G., Miller, R.C., Hartmann,B.

Ackerman, M. S., Karger, D. R., Crowell,D. and Panovich,

K. 2010. Soylent: a word processor with a crowd inside. In

Proceedings of the 23nd annual ACM symposium on User

interface software and technology (UIST '10). ACM, New

York, NY, USA, 313-322.

[5] Horton, J. J. and Chilton, L. B. 2010. The labor

economics of paid crowdsourcing. In Proceedings of the

11th ACM conference on Electronic commerce (EC '10).

ACM, New York, NY, USA, 209-218.

[6] Huang,E. Zhang,H. Parkes,D.C., Gajos,K.Z. and Chen,

Y .2010. Toward automatic task design: a progress report.

In Proceedings of the ACM SIGKDD Workshop on Human

Computation (HCOMP '10). ACM, New York, NY, USA,

77-85.

[7] Ipeirotis, P. G. 2009. Blog post Your estimated waiting

time: Infinite!. Available at http://www.behind-the-enemy-

lines.com/2009/02/your-estimated-completion-time-

infinite.html. Last Accessed on 19th Nov, 2011

[8] Karnin, E. D., Walach, E. and Drory, T. 2010.

Crowdsourcing in the document processing practice.

Proceedings of the 10th international conference on

Current trends in web engineering (ICWE'10). Springer-

Verlag, Berlin, Heidelberg, 408-411.

[9] Kazai, G. 2010. An Exploration of the Influence that

Task Parameters have on the Performance of Crowds. In

CrowdConf 2010, San Francisco, USA.

[10] Kittur,A., Chi, E. H., and Suh, B. 2008.

Crowdsourcing user studies with Mechanical Turk. In

Proceeding of the twenty-sixth annual SIGCHI conference

on Human factors in computing systems (CHI '08). ACM,

New York, NY, USA, 453-456

[11] Knowledge@Wharton. 2011. From Freelancers to

Telecommuters: Succeeding in the New World of

SolitaryWork. Published in Knowledge@Wharton. June

29, 2011. Available at

http://knowledge.wharton.upenn.edu/article.cfm?articleid=

2806. Last Accessed on 19th

Nov, 2011.

[12] Knowledge@Wharton. 2010. It Takes a Village: The

Rise of Rural BPO. Published in Knowledge@Wharton.

February 11, 2010. Available at

http://knowledge.wharton.upenn.edu/india/article.cfm?artic

leid=4450. Last Accessed on 19th

Nov, 2011.

[13] Kumar, D., Martin, D., and O’Neill, J. 2011. The times

they are a-changin': mobile payments in india. In

proceedings of CHI 2011, 7-12th

May, Vancouver, Canada.

[14] Malone, T. W., R. J. Laubacher and T. Johns. 2011.

The age of hyperspecialization. Harvard Business Rev.

89(7/8): 56-65.

[15] Mason, W. and Watts, D.J.2009. Financial Incentives

and the “Performance of Crowds”. In Proceedings of the

ACM SIGKDD Workshop on Human Computation

(HCOMP '09). ACM, Paris, France.

[16] O’Neill, J., Martin, D., Colombino, C., and Grasso, A.

2011. When a little knowledge isn’t a dangerous thing. CHI

2011.

[17] Psaier, H., Skopik, F., Schall, D. and Dustdar, S. 2011.

Resource and Agreement Management in Dynamic

Crowdcomputing Environments. In Proceedings of the

2011 IEEE 15th International Enterprise Distributed

Object Computing Conference (EDOC '11). IEEE

Computer Society, Washington, DC, USA, 193-202

[18] Rzeszotarski, J .2011. Worker Collaboration in

Crowdsourcing Markets. ACM SIGCHI 2011 Workshop on

Crowdsourcing and Human Computation.

[19] Sachs, P. 1995. Transforming Work: Collaboration,

Learning and Design. Communications of the ACM 38 ( 9).

36-44.

[20] Schmidt, K., and Bannon, L. 1992: 'Taking CSCW

Seriously: Supporting Articulation Work', Journal of

CSCW, vol. 1, 1992, no. 1-2, pp. 7-40

[21] Sun, Y., Roy, S., and Little, G. 2011. Beyond

Independent Agreement: A Tournament Selection

Approach for Quality Assurance of Human Computation

Tasks. AAAI Workshops, North America, Aug. 2011.

Available at:

http://www.aaai.org/ocs/index.php/WS/AAAIW11/paper/vi

ew/3904. Date accessed: 19 Nov. 2011.

[22] Vecchia, G.L. and Cisternino, A.. 2010. Collaborative

workforce, business process crowdsourcing as an

alternative of BPO. In Proceedings of the 10th international

conference on Current trends in web engineering

(ICWE'10), Florian Daniel and Federico Michele Facca

(Eds.). Springer-Verlag, Berlin, Heidelberg, 425-430

[23] Vukovic, M. Lopez, M., and Laredo, J. (2010).

PeopleCloud for the Globally Integrated Enterprise. In

Lecture Notes in Computer Science (6275), Service-

Oriented Computing. ICSOC/ServiceWave 2009

Workshops. 109-114.