RESOURCE AND TASK MANAGEMENT AUTOMATION FOR CROWDSOURCING

By

Delisa Simonovic

A DISSERTATION

Submitted to

The University of Liverpool

in partial fulfillment of the requirementsfor the degree of

MASTER OF SCIENCE

28/01/2015

ABSTRACT

RESOURCE AND TASK MANAGEMENT AUTOMATION FOR CROWDSOURCING

By

Delisa Simonovic

The distribution of work through online labour networks as mean of value creation and

innovation also became available to small and midsize companies specialised in web design

and development. For these companies, crowdsourcing could be a solution for on-demand

workforce scaling depending on the level in which it can support the execution of complex work.

The problem addressed by this research is the prospect of adapting typical web design and

development tasks and activities to requirements of a crowdsourcing model. For this purpose,

two typical cases in web design and development are investigated. Logo design case ad-

dresses typical design tasks while JavaScript snippet case addresses typical development

tasks. These two cases are examined in a proof of concept experiment in which the process of

logo and JavaScript snippet creation is divided into multiple crowd friendly tasks managed by a

proposed workflow. During the experiment, the feasibility of proposed workflow is tested thus

providing insights into a process of transformation in which initial specification becomes fin-

ished artefact by the means of participants’ combined effort. The output of each task together

with a response to a questionnaire provided valuable data related to factors influencing task

granularity and feasibility of proposed workflow. The research resulted with meaningful output

in form of final logos and JavaScript snippets thus validating the feasibility of the workflow and

proving that typical web design and development tasks can be adapted to crowdsourcing prin-

ciples. The combination of two methods used in this research, Case Study and Design Re-

search point to a conclusion that proposed crowdsourcing workflow can be adapted to certain

requirements by applying understandings from each finished task cycle into the next one.

ii

DECLARATION

I hereby certify that this dissertation constitutes my own product, that where the language of

others is set forth, quotation marks so indicate, and that appropriate credit is given where I

have used the language, ideas, expressions, or writings of another.

I declare that the dissertation describes original work that has not previously been presented for

the award of any other degree of any institution.

Signed,

Delisa Simonovic

Student, Supervisors and Classes:

Student name: Delisa Simonovic

Student ID number: H00018787

GDI name: Dr. Shakil Ahmed

CRMT class ID: LAUR-906-4

DA name: Dr. Ali Ahmed

CAC class ID: UKL1.CKIT.702-201424-10.H00021491

iii

ACKNOWLEDGEMENTS

First of all I would like to thank Dr. Ali Ahmed, my dissertation advisor without whom

the completion of this thesis would not be possible. He is the only person besides me, who

really had to cope with ideas and concepts presented in this thesis. I would also like to thank all

my instructors at UoL. Special thanks to Ms. Wendy Fan for her fast and informative replies to

numerous administrative inquiries from my side.

Thanks to all participants in my research especially to Mr. Igor Kvrgic and Mr. Tibor

Debus for being a new breed of professionals driven by innovation.

Many thanks to my mother and my father for being brave enough to bring my sister, my

brother, and me to this world. My father, for inspiring me to perceive the world from many differ-

ent perspectives, for showing me importance of looking behind the scenes and for cultivating

curiosity in me even about most ordinary aspects of life. Thanks to my mother, for being pro-

tective and supportive in mine most challenging times. I am thankful to my sister for inexhaust-

ible support and believe in my thoughts and ideas and for her courage which served as ex-

ample for “believing in yourself when nobody else does”. Many thanks to my brother for being

very sensitive yet critical and realistic regarding many things and situations I wouldn’t be aware

of without him.

iv

TABLE OF CONTENTS

PageLIST OF TABLES 3

LIST OF FIGURES 4

Chapter 1. Introduction 51.1 Scope 7

1.2 Problem Statement 7

1.3 Approach 8

1.4 Outcome 10

Chapter 2. Background and review of Literature 122.1 Background 12

2.2 Literature Review 142.2.1 Taxonomy...........................................................................................................142.2.2 Crowdsourcing platforms....................................................................................162.2.3 Task Automation and Coordination....................................................................182.2.4 Research methods for crowdsourcing.................................................................20

2.3 Terms 21

Chapter 3. Theory 223.1 The Crowd 23

3.2 Task Management 24

3.3 The Problem of Granularity 27

3.4 Map reduce 32

Chapter 4. Analysis and Design 344.1 Overview 34

4.2 Web design and development production process analysis 35

4.3 Proposed Workflow 38

4.4 Research Model 42

4.5 The Experiment 45

4.6 Test Data and Questionnaires 46

Chapter 5. Methods and Realization 525.1 Logo Design Case 53

5.2 JavaScript Snippet Coding Case 55

5.3 Summary 56

1

Chapter 6. Results and Evaluation 576.1 Logo Design Case 60

6.2 JavaScript Snippet Coding Case 62

Chapter 7. Conclusions 647.1 Lessons Learned 64

7.2 Academic Application and Limitations 66

7.3 Business Application and Limitations 67

7.4 Recommendations / Prospects for Future Research / Work 67

REFERENCES CITED 69

2

LIST OF TABLES

Page

Table 1. Complex Work vs Micro Task Crowdsourcing.............................................35Table 2. Production process overview.........................................................................37Table 3. Planned differences between two production cycles.....................................46Table 4. Task 1.............................................................................................................47Table 5. Task 2.............................................................................................................47Table 6. Task 3.............................................................................................................48Table 7. Task 4.............................................................................................................48Table 8. Task 5.............................................................................................................49Table 9. Task 6.............................................................................................................50Table 10. Task 1...........................................................................................................51Table 11. Task 2...........................................................................................................51Table 12. Actual design cycle differences (logo design case).....................................54Table 13. Actual design cycle differences (JavaScript snippet case).........................56Table 14. Overall research stats...................................................................................58Table 15. Logo Design Case........................................................................................72Table 16. JavaScript Coding Case...............................................................................77

3

LIST OF FIGURES

Page

Figure 1. The Three Defining Elements of Crowdsourcing (Saxton, Og & Kishore, 2013)........................................................................................................................6

Figure 2. Proposed Research Model..............................................................................9Figure 3. The relationship between the terms (van der Aalst & van Hee, 2002:39)

...............................................................................................................................25Figure 4. MapReduce word count process (www.cs.uml.edu:2014)...........................32Figure 5. Proposed Workflow......................................................................................40Figure 6. Proposed Workflow Scheme........................................................................41Figure 7. Research model............................................................................................43Figure 8. Final Artefact - First Cycle Logo Design Case...........................................60Figure 9. Final Artefact - Second Cycle Logo Design Case.......................................61Figure 10. Artefact Information Content.....................................................................62Figure 11. Growing information content of artefacts..................................................65

4

CHAPTER 1. INTRODUCTION

Contemporary crowdsourcing is a fairly new phenomenon. Modern crowd-

sourcing practice emerged as a product of synergy between several factors including new

technologies, business needs, and the internet induced change in communication and

social interaction. Jeff Howe coins the term crowdsourcing in 2005 by combining the two

words - outsourcing and crowd. It was published for the first time in 2006 in a Wired

Magazine article "The Rise of Crowdsourcing” (Safire, 2009). According to Howe (2006),

the age of the crowd has come in which "distributed labour networks are using the Inter-

net to exploit the spare processing power of millions of human brains” (Howe, 2006). New

business model is on the rise influencing companies and organisations worldwide and

changing the landscape of innovation and collaboration in a computer-mediated society.

Crucial aspect of crowdsourcing is the open call format which taps into a large network of

individuals offering them a possibility to present solutions for real life, real business prob-

lems (Howe, 2006). Social web technologies and social media infrastructure play a signi-

ficant role in providing a fertile base for the development of a new outsourcing model

which not only involves communities but also controls and manages their participation

(Saxton, Oh & Kishore, 2013).

Academia has also recognized crowdsourcing as a phenomenon which opens a

space for innovation in management science by addressing critical issues in management

situations regarding the nature of environment and changeability and uncertainty of parti-

cipating individuals (Lebraty & Lobre-Lebraty, 2013). Crowdsourcing presents a new way

of thinking for both entrepreneurs and researchers alike where the complexity of the sub-

ject blurs the distinction between the two. The identification of many new organisational,

economic and social issues that emerge in the process of maturity of crowdsourcing mod-

els serve as developmental challenges which push the boundaries and motivate innova-

tion. However, crowdsourcing is still in early stages of development and researchers

5

have yet to consolidate the taxonomy. Several attempts have been made but still need to

be accepted by researchers' community.

Interesting fact is that although crowdsourcing is a new term, the underlying prac-

tice has been used for centuries in various attempts to use individual or small group ef -

forts in accomplishing certain goals. In 1714, looking for a solution to "The Longitude

Problem" British Government offered a prize of 20.000 Ł for anyone who would come up

with the solution. Result was the invention of a marine chronometer (Hobden & Hobden,

1989). In 1936, Toyota opened a contest for their new logo design. The winning logo was

chosen from 27.000 entries and led to a brand name change (Toyota Global Site, 2014).

In 1956, a design contest was opened for Sydney Opera architectural solution (Sydney

Opera House, 2014). There are numerous initiatives throughout the history which all have

one common denominator - to use and utilize people's inspiration, ideas and innovation in

finding solutions to particular problems. Nowadays crowdsourcing is tightly related to

emerging business needs which in necessity of fast and efficient solutions mark an era of

IT mediated innovation platforms and communities.

The importance of this subject is yet to be seen in coming years in which crowd-

sourcing core goals - problem solving, generating innovation and automating individual

approach within a process of aggregation of individual resources will provoke rapid devel-

opment and will boost social and technological achievements.

Figure 1. The Three Defining Elements of Crowdsourcing (Saxton, Og & Kishore, 2013)

6

1.1 Scope

The scope of the project is an investigation of potential implementation of crowd-

sourced labour in enterprise contexts specialized for web design and development. The

primary focus of this investigation is a prospect of division and adaptation of complex and

interrelated tasks into "crowd friendly" mini tasks. "Crowd friendly" tasks should be relat-

ively simple, modular, and as much independent as possible. They should be managed

by a process workflow in order to establish integrity and co-relatedness within a produc-

tion process. Such production process should consolidate and reconcile two opposing

principles thus opening a space for investigation: The need to divide and granulate web

design and development process in order to create "crowd friendly" mini tasks is some-

what opposed to the need for integration of tasks into streamlined and managed process.

A process of automation in which principles of analysis and synthesis of the tasks play an

important role should accomplish the consolidation of the two. The research will focus on

logo design and simple JavaScript snippet coding as two particular elements of the pro-

cess of website design and development. Investigation of the entire web site development

is out of scope of this research because at this point it only focuses on essential elements

which are only fragments of the entire process. However, the investigation and compre-

hension of crowdsourcing principles applied to limited application scope represented by

these elements can contribute to eventual future application of the model on the entire

web design and development process.

1.2 Problem Statement

Many small and medium sized web design and development companies have a

problem with capacity scaling. In many cases, they are unable to allocate needed re-

sources for specific projects because they lack the flexibility of using an on-demand work-

force. Since most projects have capacity demands only within specific time frame, crowd-

sourcing as a way of aggregating resources offers a potential solution by allocating re-

sources where and when needed. In order to fully utilize crowdsourcing principles in con-

texts of web design and development, companies need a way to automate and streamline

7

the development process participated by crowd workers. This opens a problem area of

task and resource automation where work division and task assignment have a crucial

impact on the quality of deliverables. The aim of the research is to propose a solution for

managing ad-hoc virtual teams which would be coordinated by utilizing automated work-

flows for task and resource management. Automation of tasks involves project require-

ment analysis, task formulation, task division – granulation, and task assignment and

coordination.

1.3 Approach

The problems of task and resource automation, for crowdsourcing in web design

and development, require multi-layered approach for many reasons. Web design and

development are well-established practices with their methodologies and rationale. They

already exist on the premise of labour division and utilize project management principles

in a process of creating deliverables. Such context requires analysis of existing produc-

tion processes in web design and development which should be analysed from the point

where production requirements and specifications already exist and are in consent with

project stakeholder. Process analysis should result in high-level process definition needed

for adoption and implementation of production processes into a crowd friendly model. The

design of a crowd friendly model uses Design Research Methodology (DR) in order to

create constructs and methods for task division and task suitability in crowdsourcing en-

vironment. Proposed model is tested using a case study in a pilot web development pro-

ject. The process is iterative so that understandings from the case study on the pilot pro-

ject could be reintegrated into design research of a crowd friendly model. This approach

will offer deep insight into a problem domain where certain number of factors could be

followed through iterative process thus giving an overview of cause and effect relation

when introducing a change in certain factors.

8

Web Design and Development Production Pro-cess Analysis

Design Research (DR) of a crowd friendly workflow model

Case Study on a Pilot Web Design and Development Project

Figure 2. Proposed Research Model

Figure 2 depicts three major phases of research. The first phase is a production process

analysis which will gather requirements and specifications for a production of one web

design artefact and one web development artefact. These specifications include all the

information, steps and procedures needed to produce both artefacts as they would be

produced during a standard production process within a company. The second phase is

a design phase in which standard production process has to be adapted to crowdsourcing

requirements and constraints determined by major crowd features. According to Hosseini

et al. (2014) major crowd features are population diversity, unknown-ness, largeness,

undefined-ness and suitability which implies that crowdsourcing tasks also have a feature

to attract those individuals from the crowd which have specific skills and expertise, thus

being of interest to population of individuals who share similar attributes. Crowdsourcing,

as general phenomena, can be observed from the point where it attracts certain individu-

als be it for the reason of volunteering, competence, satisfaction or love of community and

sharing (Hosseini et al. 2014). For this reason, crowd friendly tasks have to be suitable

and understandable to a population as wide as possible serving as a counterbalance to

implicit filtering and narrowing of population due to specific skill requirements. Second

phase of research will yield task definition by translating production requirements into mini

tasks managed by iterative workflows. Every step of a standard production process can

be translated into an open call task which when iterated through a workflow will elevate

information content of the artefacts. The third phase is a case study in which proposed

task definition and workflow will be tested using qualitative research. During this phase,

the observation will be focused on task suitability, number of workflow iterations and

crowd worker feedback. Results from the third phase will be evaluated and returned to the

second phase in form of change to existing workflow and task definitions. The quality of

produced artefacts after every iteration between second and third phase can be used as a

validation of factors which were modified. It would be very hard to predict these factors

without doing the observation on a case study pilot project. However, most likely factors

that could influence the quality of the artefacts include and are not limited to:

task time limitation - influenced by the initial problem specification scope

9

a number of iterative tasks - determines the extent of refinement increments

where small number of iterative tasks require greater extent of each increment as

well as greater effort by the crowd worker

perceived task difficulty

perceived understanding of the problem defined by initial task and consecutive

tasks within iteration cycle

These and other factors will be observed, changed according to given feedback and put

into a second production cycle. Such procedure contributes to the interpretation of results

because it puts initial task definition and perceived task aims into relation making it pos-

sible to observe wide range of factors and their causalities.

1.4 Outcome

The research outcome showed positive results regarding the hypothesis that typ-

ical web design and development tasks can successfully be divided into manageable

mini-tasks for crowdsourcing, and that task lifecycle and task granularity can be optimized

using automation workflows. The positive outcome is evaluated only within limitations of

proposed workflow and tested cases. By using two Design Research cycles, it was pos-

sible to measure the feasibility of initial workflow model and to establish needed con-

straints which lead to more appropriate granularity regarding the task nature. For example

in logo design case, task granularity was limited only by time constraint of 30 minutes per

task within the second cycle. Since in logo design, we are dealing with non-discrete re-

finement extents, the control of granularity was only possible by using time limitations. In

JavaScript snippet case, granularity of each task is also dependent on task specification.

During the research, it became evident that whether we use time limitation or specification

information density to influence granularity, individual cognitive factor determines the out-

come. Some participants showed better performance in one type of refinement tasks

while at the same time they had trouble with other type of tasks. Factors influencing indi -

vidual differences can be many, but results point to the conclusion that individual concen-

tration span among other factors depends on motivation that is affected by the level in

10

which tasks are perceived more or less interesting. Supposed innovative contribution of

each task can be adjusted by using more or less precise specification. The more open

and abstract the specification is, the higher potential innovation can be achieved yet there

is higher probability of inadequate output.

11

CHAPTER 2. BACKGROUND AND REVIEW OF LITERATURE

2.1 Background

The project is being done for a small web design and development company in

Zagreb, Croatia. It is a company founded by my brother and me in 2005. Since the days

company started, one particular issue seemed to re-emerge from one project to another.

This issue had an impact on the growth of the company as well as on the development of

company's strategy and its positioning on the market. The issue that is shared between

companies regardless of size and industry is a problem of workforce scaling. On demand

workforce scaling is a concept of managing resources according to lean management

practices, which reduce unneeded labour costs by scaling a workforce according to actual

needs. This practice does not only save companies of unnecessary costs but also makes

project and operations planning more flexible.

Crowdsourcing as a practice and a phenomenon opens a new perspective on

workforce planning and management especially for the companies whose core busi-

nesses in major part depend on online communication and collaboration. Web design and

development companies are for this reason, perfect candidates to implement and test

crowdsourcing practices and principles in their production cycle. However, to be able to

apply crowdsourcing principles, current production activities need to be adjusted to a new

paradigm mainly because crowdsourcing represents a mediating layer between job offer-

ers - companies that have a workforce demand, and virtual crowd - globally dispersed

individuals who offer their cognitive capacities, knowledge and expertise to job offerers.

This mediating layer has a potential to provide numerous benefits yet it also sets new

rules and boundaries.

The main problem addressed by this study is whether traditional web design and

development tasks and activities can be tailored, automated and managed in order to

comply with crowdsourcing requirements. This is not a new problem and many other re-

searchers have emphasized a problem of crowdsourcing complex work, mainly because

most of existing crowdsourcing platforms either do not deal with complex tasks, they only

12

offer micro-task execution, or they do not address the need to analyse and divide com-

plex tasks because they use crowd competitions in which participants do the entire com-

plex work and compete against other participants for a single prize. One of the most fam-

ous crowdsourcing platforms is Amazon's Mechanical Turk (MTurk). According to Kittur et

al. (2011) "The types of tasks accomplished through MTurk have typically been limited to

those that are low complexity, independent, and require little time and cognitive effort to

complete." (Kittur et al., 2011:3) On the other hand when complex tasks are required,

crowdsourcing platforms usually use competition as a basis of their service. The problem

with competitions are several:

the problem of incentive - many crowd workers compete over single prize

the problem of efficiency - every individual participant completes the entire work

required

the problem of economy - the effort of only one individual is actualised while the

efforts of all the others is wasted.

The "winner takes all" model for complex tasks results in loss of effort invested by crowd

workers since only one prize is awarded to the best entry evaluated according to job

offered criteria. (Xu & Lasron, 2014)

The aim of this research is to investigate a potential use of crowdsourcing prin-

ciples in web design and development and to propose a usable workflow model which will

streamline task automation and management of resources within a scope of web design

and development. Having in mind that single task is a building block of every crowd-

sourcing effort, and that standard web development tasks have to be crowd friendly, a

fine line between excess task complexity and excess task simplicity has to be balanced.

In a case of very simple and short tasks, there is a burden of high costs for management

and coordination of tasks. The lower the task complexity, the greater number of tasks has

to be created in order to fulfil all of the requirements put upon standard task. On the other

hand, if tasks are too complex, the only way to put them into crowd labour market is by

using already discussed crowdsourcing contests.

13

2.2 Literature Review

2.2.1 Taxonomy

According to Hoseini et al. (2014) current problems and achievements in crowd-

sourcing as well as rising industrial and academic interests in the subject led to various,

sometimes divergent definitions of crowdsourcing. Primary concern here is a great di-

versity in the interpretation of the concept. Hoseini et al. (2014) proposed basic crowd-

sourcing taxonomy based on four pillars: the crowdsourcer, the crowd, the crowdsourced

task, and the platform. These should serve as a baseline for the interpretation of crowd-

sourcing and should accommodate multiple perspectives including social sciences, busi-

ness and computer science.

Agafonovas & Alonderiene (2013) discuss several important subjects related to

open innovation, value of crowdsourcing platforms and possible benefits for creative

agencies and their potential customers. Authors identify three key crowdsourcing parti-

cipants:

The Core Team

The Customer

The Crowd

Giving equal attention to perspectives of all crowdsourcing participants, Agafonovas &

Alonderiene (2013) propose integrated business model for crowdsourcing-driven organ-

isation. In such a concept, the goal of value creation cannot be established without incent-

ives for every party. Money compensation is not the only motivational driver for the parti -

cipants. They also gain experience, they communicate and they learn through feedback

and acknowledgement. The value creation within a system would be accomplished by

using collaborative innovation as one of main crowdsourcing potentials.

Simula & Vuori (2012) examined interaction of contributors and B2B companies

within crowdsourcing context which is seen as an important medium for innovation and

knowledge creation. Combined wisdom of many participants can drastically improve a

process of innovation. Authors observe the problem of crowdsourcing by presenting in-

14

volved participants as layers in value-chain structure. Layers consist of internal employ-

ees, trusted partners, pre-selected participants from the crowd, and crowd as a whole.

In an article "The Future of Crowdwork“ Kittur et al. (2013) discuss important is-

sues covering present state of crowdsourcing phenomenon as well as possible develop-

ments in a wide scope of areas related to crowdsourcing. They introduce vital points for

discussion and divide them into 12 groups:

Workflow - parallel approach used in micro-tasking lacks sufficient support

for complex work

Task Assignment - tasks have to be assigned to most appropriate individu-

als yet there is a need to address the problem of shared resources.

Hierarchy - organisational benefits of hierarchy have to be transmuted to an

environment where its useful elements can be adopted into fluid and flexible

nature of crowdsourcing.

Real-time Crowd work – on-demand workforce scaling and the problem of

crowd latency have adverse effects on work deadlines.

Synchronous Collaboration - complex tasks require cooperation yet current

platforms operate on the basis of independent work.

Quality Control - low cost of crowdsourcing and high worker turnover have

negative effect on output quality.

Crowds Guiding AIs - computational power opens an opportunity for intelli-

gent algorithms able to learn from crowd intelligence.

AIs Guiding Crowds - AI can be set to organise manage and optimize crowd

work using machine learning model.

Crowdsourcing Platforms - existing platforms offer limited usability and

flexibility. Future cloud platforms need to introduce new designs and con-

cepts.

Job Design - task design should take into account worker motivation and

fulfilment as well as organisational requirements.

15

Reputation and Credentials - reputation facilities need to provide reliable

anti-cheating mechanisms while retaining benefits of low hiring costs and

pseudonymity.

Motivation and Rewards - financial motivation is necessary but is not the

only aspect of motivation. Managerial, social and psychological aspects of

motivation should be integrated into motivational constructs.

Besides these important functional elements future crowd work should address, Kittur et

al (2013) emphasize an importance of moral and ethical aspects without which crowd-

sourcing would create a future we could not be proud of. There is a serious risk of trans-

forming crowd work into a low-cost exploitation. According to the authors, the design of

future crowdsourcing frameworks has to pass a test by positively answering the question:

“Can we foresee a future crowd workplace in which we would want our children to parti-

cipate?” (Kittur et al. 2013:1)

Asmolov (2011) explores challenges in crowdsourcing platform analysis. He

suggests CHAT - cultural-historical activity theory for crowdsourcing project analysis.

"Activity" is the primary aspect of analysis within his paper. Crowdsourcing projects are

observed as a means which contribute to the development of activity systems while

crowdsourcing platforms are seen as expression of activity systems. Activity as an aspect

of analysis involves subject - object relationships and pertaining dynamics of power con-

cepts within crowdsourcing deployments where power of networks can be seen both from

the perspective of individual empowerment and as the empowerment of traditional power-

holders.

2.2.2 Crowdsourcing platforms

Maja Vuković (2009) in her article "Crowdsourcing for Enterprises" evaluates

several crowdsourcing platforms and identifies key features as well as their shortcomings.

She identified two major crowdsourcing models: Marketplace and Competition. Each of

16

the models utilizes individual micro tasking for the benefits of requestors. Marketplace

connects requestors and the Crowd whereas in Competition, the only one winning sub-

mission is selected and awarded. She points out several challenges on the path of realiz-

ing true benefits of crowdsourcing:

Request model pooling of different providers where each of them fulfils a

single task which is just a part of a complex request

Participant matchmaking

Models for bid evaluation

Dynamic pricing according to supply and demand of participants with certain

skills

Quality Management

Platforms for virtual team formation

The design of crowdsourcing platforms needs to address certain challenges which involve

the technical design of the system, economic models for management of incentives, parti-

cipant's involvement in online communities, aspects of collaboration, multi-tenancy and

integration (Vuković, 2009). Both Marketplace and Competition models have certain

shortcomings because they represent work in progress in the process of maturity and

development of crowdsourcing as a principle. Xu & Larson (2014) addressed issues re-

lated to crowdsourcing contests, their poor social well-fare and their efficiency problem.

These issues are a by-product of "winner takes all" contests. Authors propose a mechan-

ism for filtering of high-quality contestants before they are engaged in task-related activit -

ies thus rising social welfare and crowdsourcing contest efficiency.

Hirth, Hossfeld & Tran-Gia (2011) point out problems and issues related to usabil-

ity of commercial crowdsourcing platform MicroWorkers.com. The platform uses micro-

task model, offering simple tasks to potential workers. These tasks do not require any

special skills or knowledge and can be completed in a matter of minutes. Quality control is

achieved by utilising cheat detection mechanisms. Authors address and recommend sub-

jects for future work:

workflow optimisation and automation

worker recommendation systems

17

quality management

According to the authors, workers capable of engagement on more complex tasks tend to

be more expensive; on the other hand, such investment is remunerative because of better

output quality which reduces overall task costs.

2.2.3 Task Automation and Coordination

In an interesting framework proposal article "Crowdforge: Crowdsourcing Com-

plex Work", Kittur et al. (2011) discuss prospect of facilitating complex work within the

frame of current requirement constraints of crowdsourcing. Although crowdsourcing suc-

cess can be seen globally, as companies start to envision their strategies on accomplish-

ing work via online platforms be it for crowdsourcing design (threadless.com) or crowd-

sourcing R&D (innocentive.com), crowdsourcing is still in its infancy. On one side, we

have platforms for competition based contests while on the other we have micro-task

platforms lacking the ability to support complex work. Currently, most popular micro-task

platform is Amazon Mechanical Turk (Mturk). It has gained popularity in non-specialized

crowdsourcing market where micro-tasks are used for work needing from few seconds to

a few minutes. Still, low complexity of tasks and their independence limits a potential use

of MTurk as a platform for complex work. Recognizing these limitations, authors propose

a framework that would address certain requirements in order to accomplish execution of

interdependent complex tasks. "The framework systematically breaks complex problems

down into simpler tasks by creating subtasks that define and create other subtasks and

distributes these tasks to workers" (Kittur et al. 2011:3). Being aware that management of

complex tasks would incur certain costs to the system because of complexity caused by

coordination of interdependent tasks and their subtasks as well as because such context

would require certain skills and education thus limiting workforce availability, authors pro-

pose a system that would reduce these costs by using task disaggregation and aggrega-

tion. This way all the requirements needed for successful execution of complex work such

as task definition, supervision, and coordination could be achieved in micro-task environ-

18

ments. Interestingly, authors propose utilisation of map reduce model - a commonly used

programming model for distributed computing. In order to achieve coordination of com-

plex task dependencies, they translate principles for coordination of computer systems

and use it for coordination of human activity. "Crowdsourced labor markets can be viewed

as large distributed systems in which each person, such as a worker on Mechanical Turk,

is analogous to a computer processor that can solve a task requiring human intelligence.

In this way a crowdsourcing market could be seen as a loosely coupled distributed com-

puting system." (Kittur et al. 2011:7)

Additional step towards decentralisation of task design and management can be

achieved by offloading these activities to the crowd itself. Kulkarni, Can & Hartmann

(2011) propose a model for micro task creation automation within Amazon's crowd-

sourcing marketplace - Mechanical Turk. The model involves crowd workers into a pro-

cess of workflow planning and design, and introduces a new modality into perspective.

The benefits of crowdsourcing can be used as a mean of process self-management in

which crowdsourcing resources are used for the design of control of the same resources.

Turkomatic relies on a recursive algorithm that directs crowd workers in the process of

splitting and solving tasks as well as recombining and merging task results. The process

consists of two phases. In the first phase, complex tasks are iteratively subdivided if there

is a need. If a first subdividing iteration still produces complex tasks, they are put into a

second iteration in order to divide them further. The process goes on until it outputs

simple atomic tasks that can successfully be solved as micro-tasks (HITs - Human Intelli-

gence Tasks). In the second phase, solution to every partial task is merged into a final

solution which addresses initial complex task.

Simperl, Norton & Vrandecic (2011) proposed a framework for crowdsourcing of

linked data management tasks. They explain decomposition of a main task to micro tasks

suitable for crowdsourcing. Since a large portion of linked data management relies on

human cognition and interpretation, the authors find it suitable for crowdsourcing. Such

strategy combines computational and human intelligence in an effort to automate the

process while offering certain quality. Human computation can compensate quality issues

19

in liked data management that could not be addressed by using only computational al -

gorithms.

Lopez, Vukovic & Laredo (2010) propose People-Cloud, online service for task

coordination in crowdsourcing. The service would organize virtual teams online which

would be able to integrate their efforts and knowledge in IT service delivery domain.

People, their experience and their knowledge represent a backbone of every organiza-

tion. Knowledge as a resource cannot be fully utilized without knowledge networks which

should facilitate knowledge discovery in order to support important initiatives. Knowledge

can be successfully utilized as a resource within an organization and it can also be har-

vested from external community thus providing greater value to organizational efforts and

initiatives.

2.2.4 Research methods for crowdsourcing

Marjanović & Roztocki (2013) ascertain that the majority of research papers on

Crowdsourcing are empirical and use case study as the most common method. Interest-

ingly, none of the studied researches used Design Research (DR) or Action Research

(AR) method. They argue that Crowdsourcing as emerging phenomenon should be ap-

proached using AR or DR research, thus facilitating new solutions and offering a contribu-

tion to existing body of knowledge. This way researchers would solve practical problems

and make a significant contribution for further studies.

According to Henver & Chatterjee (2010) Design Research paradigm addresses

two key issues in information system research: the importance of IT artefacts and per-

ceived insufficient relevance of IS research. Until early 1990s, Design Research has tradi-

tionally been reserved for hard sciences like Computer Science or Engineering. Later, it

became apparent that such methodology can successfully be used in design and im-

provement of IT artefacts. Its ability to solve real-world problems puts Design Research in

a position where it can largely improve IS effectiveness and help companies in fulfilling

their purpose, their economy and efficiency.

20

2.3 Terms

Artefact – An item/object of human origin observed during scientific investigation

Atomic task - Simple self-contained task which cannot or needs not to be further divided

Crowd – A large number of people temporarily gathered and engaged in some form of

collective behaviour

Crowdsourcing – The process of acquiring added value by employing distributed labour

networks

Crowd Worker – An individual who uses crowdsourcing platforms and engages into exe-

cution of different tasks available on these platforms

Crowdsourcing platform – An online application serving as a market for creation, distri-

bution and execution of tasks

Design research – Applied research method for design and evaluation of information

system artefacts

Distributed labour network – A large network of people seen as a potential processing

power and used as an online resource supported by crowdsourcing

Iterative workflow - A workflow in which actions and operations are executed by follow-

ing repeated sequence

Map-reduce – A programming model for parallel processing of large datasets. It consists

of two basic functions: Map – reads and extracts large amount of data records; Reduce -

aggregates and summarizes the data

Open call – An open invitation for participation in activities contributing to certain object-

ives

Task granularity – The level of possible task divisibility

21

CHAPTER 3. THEORY

The Theory behind crowdsourcing is fairly new. The concept was born in 2005

but it is still in the process of standardisation in terms of structure and taxonomy. How-

ever, particular theories constituting crowdsourcing are mature ones like for example Le

Bon's psychology of the crowd, Economic Theories of externalisation, cognitive psycho-

logy and communicology, business process and task management. All these aspects will

be briefly elaborated in order to offer an insight into a genealogy of the phenomenon and

pertaining problems and challenges.

Although mentioned theories are mature enough to pass the test of time, they can

only provide a starting point in a development of crowdsourcing models for several reas-

ons. First reason is that Lee Bon's theory of the crowd doesn't really thematize virtual

crowds emerged as a consequence of internet driven communication. It can provide

some insights into underlying dynamics of virtual crowds, but the theory itself needs to

develop further in order to be entirely relevant for crowdsourcing. Secondly, the medium

of communication changed the nature of the interaction between participants assigned to

certain productive activities. Internet and social network systems are connecting people

thus creating new models of communication. Since ever greater number of individuals

interact globally, it changes the aspects of communication so fast that it becomes very

difficult to investigate its principles and form functional cause and effect models. Saxton,

Oh and Kishore (2013), when discussing about web 2.0 conclude that "The technology

effectively has no pre-defined knowledge structure; it is instead is a highly flexible, inter-

active, and user-centric system that readily accommodates changing human activity and

knowledge." (Saxton, Oh and Kishore, 2013:7) The volatility of the medium most probably

has an effect on frequent changes to communication models and on the development of

new ones. Thirdly, traditional business processes are used within organisational frame-

works which allow high interdependence between members. According to Buchanan &

Huczynski (2010) high organisational interdependence requires "mutual adjustments,

unscheduled meetings, face to face discussions, physical proximity and interdepartmental

teams." (Buchanan & Huczynski, 2010:503). On the other hand, crowdsourcing has limit-

22

ations in regard to task interdependence and collaboration because it relies on a structure

based on loosely coupled members which participate in a production process only by

solving micro-tasks. Since micro-tasks have the capacity to support low interdependence,

there is a need to develop facilities which would support high interdependence needed for

accomplishing complex work. To compensate low interdependence limitations, crowd-

sourcing models need to have a way of bridging the communication and coordination gap

by centralising task coordination.

3.1 The Crowd

Wexler (2012) explicates the purpose of his paper as “to examine how the con-

temporary literature on crowdsourcing recasts the classical social scientific conception of

the crowd from social problem to problem solver."(Wexler, 2012:7) He distinguishes two

classical and one contemporary scientific view of the crowd and relates them to corres-

ponding phases in theorising. In the first phase, the crowd is seen as a factor of influence

which facilitates subjection of individual rationality to primitive unconscious instincts of the

group mind (Le Bon, 1895). In this period, the attention to crowd as a social phenomenon

has been driven by the fear of the destructive potential the crowd has, which has been

witnessed throughout the history (McClelland, 1989). Crowd challenges the social order

and is conceived by mobs, rioters and strikers. Such social aggregations require manage-

ment and control and have been perceived as a direct threat to the elite (Wexler, 2012).

In the second phase, the behaviour of the crowd is seen as showing "rationality in

context" (Wexler, 2012). Crowds were no longer seen as irrational but as a collective

behaviour which tested existing norms and bounds of establishment. From the madness

and transgression of the crowds, scientific perspective slowly moved towards comprehen-

sion of a crowd as a signal of rational protest. "The crowd and those in it are champions if

not indicators of change." (Wexler, 2012:9) However, the slight turn in understanding of

the phenomenon did not amend the overall perception of the crowd as still dangerous and

tightly bound to unpleasant uncertainty. What is most important in this phase is the emer-

23

gence of the idea of rationality of the crowd. It was not completely understood, but it was

seen as enormous potential that merely started to develop shape.

In the third phase, attitude towards crowds changed dramatically as it moved from

focusing on the destructive potential and "rationality in context" to crowd as a valuable

resource and a problem solver. Key feature of the crowd is its great diversity and number.

These two, when adequately managed, can be an inexhaustible source of new ideas,

solutions and expertise. By proponents of this phase in theorising, it is believed that the

crowd can even deliver better and more adequate results than by using traditional means

of employee engagement (Lakhani et al.,2007). Using interactive, modern technologies

like Web 2.0 as a means of communication and collaboration, as well as rewards as a

way of incentive enables crowd to deliver relatively cheap, often original and superior

results (Brabham, 2009).

3.2 Task Management

According to Van der Aalst & van Hee (2000) task is defined as "a logical unit of

work" and can be divided into three broad categories: manual, semi-automatic and auto-

matic tasks (Van der Aalst & van Hee, 2000:38). The differentiation between these three

groups is based upon the criteria which distinguishes three types of relations between

tasks and computational aid. Tasks not requiring any computer application in order to be

executed are manual tasks. For example, conduct a meeting, sign a contract, install a

physical component, etc. are all manual tasks. On the other hand, we have completely

automated tasks. These are performed without needing any human intervention and are

usually executed within a computer program or an application. They are called automatic

tasks for obvious reasons. Between these two we have semi-automatic tasks which re-

quire human intervention, but cannot be performed without help from some automated

system, computer program or an application. Crowdsourcing tasks perfectly fit into the

category of semi-automatic tasks which require human input but are coordinated using

programmed rules and conditions. In order to further contribute to understanding of tasks

we need to put them in a context. Tasks are created, transformed and executed within a

24

context of some process for which they provide particular purpose. The entire business

process depends on the management of multiple tasks which are dependent on one an-

other. Still, what is a relation between tasks and a process? According to Hollingsworth

(1995) as cited in Schulte (2012) "A workflow is the computerised facilitation or automa-

tion of a business process and may contain automated and manual activities, also re-

ferred to as human tasks." (Schulte, 2012:1). In this context, task is an indivisible piece of

work or activity within a workflow. By nature it is generic, meaning that it is not bound to

specific work item or activity. According to Van der Aalst & van Hee (2000) "A work item

is the combination of a case and a task which is just about to be carried out." (Van der

Aalst & van Hee, 2000:38)

Figure 3. The relationship between the terms (van der Aalst & van Hee, 2002:39)

Activity describes an actual work related to work item that needs to be done in order to

fulfil the task. Task atomicity (indivisibility) depends upon the context. From the point of

view of task client (task originator), task is atomic. However, for the one who has to per-

form the task (task supplier), task can be divided into set of smaller tasks which are then

indivisible from the perspective of task supplier. Theoretically, task supplier can further

subcontract some tasks which are indivisible from within his context, but can be further

divided by a subcontractor.

Let's take an example of a simple task and how it relates to work item, activity

and case. Case is a logo design according to specification. Simple task related to logo

design is to create an idea model. Work item is then the creation of idea model according

25

to given specification. Activity for proposed work item would encompass all activities

needed to fulfil the task. These can be research, decision on type of logo to be used (font

logo, picture logo, symbol logo), creation of story and message behind the logo, decision

on the combination of colours etc. We can see that these activities can be formed into

tasks since from the context of task supplier; task for creation of an idea model can be

divided into tasks which are indivisible in his context.

In order to implement existing web design and development processes into

crowdsourcing paradigm there is a need to investigate a process as an element which

combines other building blocks like tasks, cases, conditions, etc. into a functional whole.

According to van der Aalst & van Hee (2000) "A process is a collection of tasks, condi-

tions, sub processes and their relationships with one another." (van der Aalst & van Hee,

2000:65) Every particular process refers to a certain number of cases which envelop

tasks, work items and activities. Each case is controlled by a process under which tasks

are executed. Tasks are not specific to a certain case because they are generic. When

tasks and cases are combined, they constitute a work item. According to Workflow Man-

agement Coalition Specification (1999) the process is a "representation of a business

process in a form which supports automated manipulation, such as modelling, or enact-

ment by a workflow management system." (WFMC, 1999:11) Business Process is a set

of related activities and procedures which are performed in order to achieve certain busi-

ness objective within existing organisational context. When the context is changed to a

global internet context where everyone is invited to participate in the production process,

tasks and their relationships have to be approached differently.

Since a workflow is an intermediary between business process and tasks, it has a

role of integration and consolidation of human intelligence tasks with business process

needs. In crowdsourcing, workflows also have a potential to utilise internet communica-

tion and computing power in order to successfully bridge distributed human resources

and business goals. According to Joosten & Brinkkemper (1995) "There are different

types of research related to workflow systems: technical, conceptual, organisational and

methodological." (Joosten & Brinkkemper, 1995:2) The research of crowdsourcing work-

flows focuses mainly on organisational and methodological aspects. Methodological be-

26

cause the implementation of workflows composed by micro-tasks in loosely coupled en-

vironments requires research of new models and methods. Organisational because busi-

ness processes are applied to particular environments in which organisation of tasks

needs to coordinate large and diverse population of potential workers. Coordination prob-

lem has partially been addressed in theory of formal models for decentralised decision

making and multi-agent planning. (Suken & Zilberstein:2008) Such models have many

possible applications like multiple robot coordination, load balancing and overall coordina-

tion of various agents with partitioned reality. These models also have potential applica-

tion in crowdsourcing environments where multiple agents cannot directly communicate

their observation to other agents because it would cause communication overhead. Addi-

tionally, every agent doesn't have a detailed knowledge of the entire production process

but has some information on immediate task context. Seuken & Zilberstein (2008) call this

the problem of “decentralised control of partially observable stochastic system." (Seuken

& Zilberstein, 2008:2). However, multiple perspectives and views on the same problem

can be a source of innovation in crowdsourcing, yet these need to be coordinated to a

certain extent in order to provide meaningfully combined output.

3.3 The Problem of Granularity

One of concerns of this research is the problem of task granularity. This important

aspect determines the extent of task division into ever more simple tasks. Since crowd-

sourcing evolved from outsourcing, both strategies share the aim towards externalisation.

Major difference between these two strategies has a basis in granularity of work. Out -

sourcing typically encompasses externalisation of entire projects executed by traditional

division of labour whereas crowdsourcing tends to outsource small chunks of work to

virtual crowd. According to Hossfeld, Hirth & Tran-Gia (2011) "Crowdsourcing platforms

can be viewed as a further development of the outsourcing principle, by reducing the ad-

ministrative overhead and the size of the outsourced tasks." (Hossfeld, Hirth & Tran-Gia,

2011:143) In both crowdsourcing and outsourcing, administrative overheads can be re-

duced because all the expenses otherwise needed to cover the costs of an in-house em-

27

ployment are cut. However, crowdsourcing implies higher management and coordination

overhead because many simple tasks are assigned to many individuals. In outsourcing,

one single project is delegated to a subcontractor so that management, coordination and

other activities which direct and combine all the work needed are accomplished by a sub-

contractor. The subcontractor actually approaches the project in a similar manner as it

would be approached by the company doing it in-house. The rationale behind outsourcing

lies in the specialisation of subcontracting company for the work that is outsourced which

ends in better optimisation of resources and cost reduction. On the other hand, in crowd-

sourcing, the focus is not on the single project but on individual tasks constituting the

project. Not that project elements like planning, resources and scope are deemed irrelev-

ant, but tasks represent the only aspect through which immediate collaboration is

achieved. The costs of all single tasks combined can be lower than the costs of compared

tasks within an in house project, still these tasks need more coordination, direction and

management because they are divided and assigned to workers which are loosely

coupled. In such environments, task specification needs to contain additional metadata to

be able to provide the description of the context in which task execution is expected.

Without appropriate metadata, crowdsourced tasks are limited to basic simple tasks

which cannot be utilised for complex work. Such tasks are predominantly populating task

listings on amazon's mturk.com and on several other popular crowdsourcing services

where crowd worker is asked to validate search result, input scanned data, add picture

description or find company or product information online. According to Kittur et al.

(2011) "This approach works well when the tasks involve what Van de Ven et al. call

“pooled interdependence”, in which the group product is a simple aggregation of the indi-

vidually produced products." (Kittur et al., 2011:5) Pooled interdependence is represented

by many independent contributions from individuals aggregated into one synergetic

whole. Although tasks are independent of one another they still require some basic type

of control in form of rules, procedures and common forms for data input. Cost of coordina-

tion in case of pooled interdependence is low because tasks do not require adjustments

and coordination with other tasks of the same type. For instance, writing a simple Wikipe-

dia article can be divided into simple tasks where each member writes only one subsec-

28

tion of an article in a parallel manner. Later, all subsections are combined into a single

article. Naturally, the content of subsections has to be connected in some way but this

requires only basic interrelation between subsections. Obviously the complexity of the

task has greater impact on granularity than sheer size of the task, however, complexity

and size are in most cases proportional.

Complex work in crowdsourcing environments has certain limits in regard to gran-

ularity. Dependent task have to be processed sequentially versus parallel in pooled inter-

dependence. Some tasks have to be finished before other tasks can be started. When

approaching the granularity problem, most evident aspect is the aspect of management

overhead because on one side we have a tendency towards small, simple and cheap

tasks, yet very simple tasks require high cost of coordination if they are part of a complex

work. For instance "create a textbox with certain size and functions" seems like a simple

task, and it actually would be a simple task if people would find the way to make use of

textboxes themselves. However, textboxes or any other elements are never used in void.

The usage of textbox requires a context which describes where on the page is textbox

positioned, which data will be entered, where will the data go, what other functions de-

pend on data input and validation, etc. These aspects complicate seemingly simple task

and create a need for additional coordination. It seems that optimal granularity for the task

depends on task information content and on metadata information content. If the task is

truly very simple, how can it be integrated as a part of complex work? It would need very

large metadata description so that crowd worker can have all needed requirements to

complete a simple task. But can such task be called simple?

Let's make an overview on how tasks are decomposed based on employed

externalisation strategy. According to Hirth, Hossfeld & Tran-Gia (2011) there are several

levels of task granularity which are correspondent to each of externalisation strategy. The

level of granularity is based on externalised units of work referred to as "work packaging".

The lowest granularity and largest package of work is a project. When project is assigned

to a subcontracting company all terms and conditions as well as required work is negoti-

ated with that particular company. This type of externalisation is called outsourcing. Since

project consists of multiple tasks, companies saw an opportunity to outsource single

29

tasks. The granularity of the task is higher than the granularity of a project in terms of

"work packaging". Further granulation of tasks brings about subtasks. Subtasks are ex-

ternalized in form of out-tasking which is commonly used as a measure of work for freel-

ance jobs. To accomplish this, companies use out-tasking platforms which then take all

correspondence and coordination with freelancers on themselves. Further stage of work

granulation is represented by micro-tasks which have the finest level of granularity. Micro

tasks are therefore building blocks of crowdsourcing because they don't require any long

term obligations and can be accomplished in a short time. According to Hirth, Hossfeld &

Tran-Gia (2011) "A micro task can be accomplished within a few minutes to a few hours

and thus does not need a long term employment." (Hirth, Hossfeld & Tran-Gia, 2011:143)

These micro-task features allow higher flexibility for workload balancing but also provide

greater opportunity for innovation, generation of ideas, design of solutions and wisdom

harvesting from the crowds. Still, there are no explicit definitions for optimal task granular -

ity in certain environments or for factors that would open some insights on dynamics of

task granularity in crowdsourcing. In most cases time as a factor is yet the only measur-

able constraint in defining needed granularity of tasks.

Interesting solution proposed by Kulkarni, Can & Hartmann (2011) involves crowd

as a deciding agent on needed division and granularity of tasks. Since micro tasks are

basic units of work in most crowdsourcing platforms, there is a demanding need to effi -

ciently manage task size and complexity in order to fit as wide possible range of work

activities into a crowdsourcing paradigm. According to Kulkarni, Can & Hartmann (2011)

"Effective task and workflow design remain something of a black art among crowd-

sourcing users, involving substantial planning, software development, and testing."

(Kulkarni, Can & Hartman, 2011:1). Micro-tasks appear in the context of workflows

through which tasks are divided and assigned to each online worker. The complexity of

these workflows is a limiting factor because only experts are able enough to cope with the

challenge of planning and design of such workflows. The authors devise a model which

would pass this challenge to the crowd and effectively give an opportunity to standard

crowd workers to have their part in design and planning of recursive workflows. Proposed

system would create pre-structured HIT's (Human Intelligence Tasks) which would be

30

further decomposed by Amazon's Mechanical Turk users (Turkers). Decomposed tasks

are then assigned to other Turkers and solved in parallel or sequentially. Solved tasks are

then combined into the whole solution. Decomposition of tasks doesn't take place only in

one step of the process but can be conducted multiple times across different steps. It is a

recursive process which involves both standard Turkers and experts in workflow manage-

ment. The contribution of the authors certainly provided new direction for further research

in crowdsourcing workflow management including task definition and granularity, still au-

thors are aware that this field is still uncharted territory and needs more investigation.

"Crowd-assisted workflow generation also makes no guarantees for optimality in pricing

or execution time. The question of optimizing crowd-produced workflows is fascinating

and merits future investigation." (Kulkarni, Can & Hartman, 2011:6)

Task granularity can also be approached from the perspective of cognitive organ-

isation of information. According to Perry (1998), "Distributed cognition provides a means

of describing the co-ordination of collaborative activity."(Perry, 1998:1) In crowdsourcing,

tasks are distributed to crowd workers and are processed within a distributed cognition

system represented by crowd. Each crowd worker acts as a single cognitive system

which takes input, processes information, and takes action and outputs results. Informa-

tion input is provisioned by using tasks which are divided into a problem spaces. Informa-

tion processing is done by using artefacts which represent the information as a mean of

work performance and as a resource for coordination of a collaborative effort. Granular-

ity, from the perspective of distributed cognition, is observed as a level of individual work

activity. As each performing agent processes information (artefacts) within a mental do-

main, such representation of information is invisible and cannot be directly observed.

Hence, granularity can only be analysed by observing information input, output, and its

transformative process. However, individual cognitive abilities should be taken as an im-

portant factor when task granularity is concerned. Attention span of crowd workers can be

seen as a deciding factor in task design. The criteria for the division of tasks should there-

fore conform to two time constraints. The first constraint is task time limitation implied by

crowdsourcing itself. Tasks need to be short enough to support economic benefits and

independence indicated by smaller tasks. Second constraint is a concentration span

31

boundary within which full uninterrupted cognitive potential can be possible. Distributed

cognition and collaborative function of work are concepts which occurred more than a

decade before crowdsourcing concept was even defined, yet they can serve as a valu-

able resource for research on cognitive aspects of labour division in crowdsourcing.

3.4 Map reduce

Map reduce was introduced by Google in 2004 in a response to a rising need for

processing large datasets. With growing internet usage and daily creation of large

amounts of data, Google needed a solution for managing large data clusters. (Dean &

Ghemawat, 2004) It is a programming model with two core functions. Map function reads

large amount of data records and extracts specific data from each record. It then stores

data in a key-value pairs which are suitable for further processing. Reduce function sorts,

aggregates and filters out data according to a specific need and writes down the results.

Figure 4. MapReduce word count process (www.cs.uml.edu:2014)

From architectural and functional viewpoint map-reduce model implements hier-

archical structure between working nodes. In Map phase, Master node processes the

input and divides it into small problems. These are then distributed to worker nodes, pro-

cessed, and the result of each node is passed back to a Master node. In Reduce phase,

Master Node receives processed results from Worker nodes and combines them accord-

ing to initial problem specification. All tasks from a single map series are processed in

32

parallel. Interestingly, map reduce combines both parallel and sequential principle. Typ-

ical map reduce cycle consists of several phases: input, splitting, mapping, shuffling and

reducing. These phases are conducted in sequential order whereas each phase consists

of multiple parallel jobs which are executed each on different working node. The logic

behind map-reduce lays in parallel independent processing of large datasets which are

structurally divided in order to balance workload and to provide time efficiency. Such

design is very adaptable to crowdsourcing, especially in providing requirements for com-

plex work.

33

CHAPTER 4. ANALYSIS AND DESIGN

4.1 Overview

Current research proposes particular approach for investigation of crowdsourcing

constructs used for web design and development. These constructs encompass such

task and workflow design which have to support complex work within constraints of

crowdsourcing. Main constraints within crowdsourcing are time limitation and independ-

ence of a single task. Each task has to provide sufficient information on the requirements

needed in order a worker can complete it successfully. A worker has to be able to com-

plete a single task instantly without the need to stick to other tasks in the project. On the

other hand, if there are tasks he finds interesting he can choose to fulfil them all. How-

ever, the choice of completing a single or multiple tasks within a project should not be

determined by interdependence between individual tasks but by worker’s personal motiv-

ation and interests. Time limitation is a requirement but not the only one because al-

though small tasks require short working time they also need to be independent. They

have to be self-sufficient without requiring additional knowledge of context within which

task is being accomplished. In other words, all requirements for successful accomplish-

ment of the task have to be present within a task definition and need to be concise and

understandable within the time frame of a single task. It is very hard to accomplish com-

plex work by implementing micro-task strategy for several reasons. Typical micro tasks in

crowdsourcing markets are limited to activities such as data entry, comment posts, site

ratings, image descriptions, etc. These tasks are atomic by themselves and have little or

no interdependence. The projects behind these tasks have some particular purpose, still

individuals working on these tasks do not have to understand what the creator of the

tasks wants to accomplish. Some individuals will see the logic behind the task and some

will not, yet the understanding of real goals and objectives behind the task are not man-

datory for successful micro task accomplishment. Complex work on the other hand,

needs these understandings because it requires a higher level of coordination and task

interdependence.

34

Complex work in traditional environ-ments

Micro Task Crowdsourcing

Tasks are interdependent Tasks are independent

Participants are tightly coupled Participants are loosely coupledNo strict time limitation for certain task Time limitation exists due to micro task

strategyContext of work is needed and implied Context of work is not needed

Table 1. Complex Work vs Micro Task Crowdsourcing

To address the discrepancy between complex work and crowdsourcing in context

of web design and development a workflow is proposed which combines three principles:

divide et impera iterative workflows, map-reduce and layered approach. The workflow is

tested in an initial proof of concept experiment which involves known web designers and

developers with at least two years of experience. During the second cycle, workflow is

changed and tested in the second experiment which involves unknown professionals from

freelancer.com. Observations and insights from the first experiment are used to refine the

workflow and to gather additional insights regarding workflow feasibility, control of output

quality and production process of individual workers on freelancer.com. In second logo

design production cycle, voting is conducted by creating voting tasks on microworkers.-

com. Tasks for voting on JavaScript snippets coded by freelancers in the second produc-

tion cycle were given to professionals who participated in the first cycle. Voting tasks on

JavaScript snippets in the second production cycle were not given to individuals on micro-

workers.com because there was no guarantee in the required competencies and skill

level needed to evaluate JavaScript artefacts.

4.2 Web design and development production process analysis

Website development is a structured process which involves different stages.

These stages determine general strategy for website development planning. For the pur-

pose of this research, there is the need to provide an overview of this process because it

represents a starting point in the design of crowd-friendly workflows. According to Lynch

& Horton (2009), there are six major stages in the process of website development:

35

# Website devel-opment stage

Typical tasks Deliverables

1 Site definition and planning

Analyse site requirements (site content, potential users and their interaction with content)

Define scope, goals, and object-ives

Decide on technology used (op-erating system, supported browsers, database, network connectivity capacity)

Define site features (simple info site, CMS, Ecommerce, com-munity site, etc.)

Plan for execution (budgets, staffing, timelines)

Site requirement specification documentation

Management and execution plan

2 Information architecture

Define site organisation and content

detailed design specification

content descrip-tion

design sketches and wireframes

3 Site Design Edit and proofread text

Design visual elements

Design and organise pages

Design database

Code logical components

text content ready

finished HTML page templates

finished visual elements (logos, backgrounds, buttons)

finished server side code

finished JavaS-cript code snip-pets

4 Site construc-tion

Assemble site components

Fill in the content

Test site functionality

Finished html for entire site

Link navigation structure in place

Visual elements in place

Textual content in place

5 Site marketing Plan site advertisement strategy Advertising ma-

36

Create site advertisements and promotional material

Create viral content

terial

paid online cam-paigns

Mail campaigns

Social network viral marketing campaigns

6 Tracking, eval-uation, and maintenance

Analyse server logs

Track website usage

Track page errors

Generate traffic reports

Maintain site content and links

Backup site files and database

Error reports

Usage reports

Up to date con-tent and links

Site file and database archives

Table 2. Production process overview

The scope of the research is focused on tasks within “site design stage” and “site

construction stage” which are represented by two observed cases within the research:

logo design and JavaScript snippet coding. However proposed workflow model could

potentially be used in any of the mentioned stages by adjusting initial task specification,

number of iterations, scope of iterations, target task population, etc. The reason for focus-

ing only on site design and site construction stage in this research is because these two

stages pronounce, materialise and actualize the concepts and plans conveyed from previ-

ous phases. They contain concrete workload which needs to be accomplished according

to specifications and plans delivered. The creation of specifications and requirement doc-

umentation usually involves customer input and correspondence and are for this reason

hard to formalise and execute through discrete steps within a crowdsourcing workflow.

These steps could also be formalised, but it would extend the scope of experiments to an

unmanageable proportion. Another reason is that if we want to observe the possibility of

crowdsourcing of the entire web development process, we should divide the process into

functional phases and observe each phase independently, partly because of controllable

and manageable scope, and consequentially because of needed detail level. Site design

and site construction phase can be divided into discrete tasks because deliverables have

37

to conform to certain structural rules. These rules come in form of technology and stand-

ards used, in our case, JavaScript and HTML5. In regard to logo design, the rules are not

so strict but logo design follows a certain predictable path, from concept idea to finished

logo, and can therefore be observed as a structural element of the entire design process.

Other visual elements could be designed by reusing a procedure used for logo design.

The logic behind proposed workflow follows top-down design strategy in which

high level specification is piped through workflow iterations. The specification is seen as a

blueprint which contains potential artefacts. Entire web site design specification can be

divided into small definitions which become initial artefacts. The specification is divided

according to preferred granularity of an artefact which is then translated into specific task.

Task granularity depends on the granularity of specification (first generation artefact).

Proposed workflow model is used to observe the transformation of information built into

the specification (first generation artefact) through a process where it becomes a deliver-

able, a real concrete information artefact (logo and JavaScript snippet). Crowdsourcing

workflow adds value to the first generation artefacts by giving them additional information

content which originates from each individual process of artefact transformation. Every

crowd worker transforms information given as a specification, adds value to it and re-

leases it to a next iteration in which additional information content is added by another

crowd worker. As site development is based on components and integration. Initial spe-

cification is divided into components which are translated to atomic tasks and iterated

through proposed workflow where each component is delivered. This research is focused

on production of components by utilising proposed workflow.

4.3 Proposed Workflow

The workflow design is based on analysis of web design and development pro-

cess which is adapted to conform to constraints induced by crowdsourcing. The problems

regarding granularity of web design/development tasks and feasibility of proposed crowd-

sourcing model will be approached by leveraging map-reduce model (Kittur et al. 2011)

38

combined with iterative approach commonly used in micro-task crowdsourcing. This way,

complex tasks can be divided into a simple tasks and the output of every series of simple

tasks can be combined using layered approach. Since every series of simple tasks pro-

duce some output, the output has to be reduced in order to eliminate low quality submis-

sions. The process of elimination is done by creating a new series of tasks for voting on

output created by previous series of tasks. For example, there is a need to design a logo

for the customer. A logo design process can be crowd sourced by splitting the tasks

needed in traditional production cycle into more suitable mini tasks. Let's take an over-

view of simplified proposed workflow for crowd sourced logo design for the customer.

39

Figure 5. Proposed Workflow

40

Figure 6. Proposed Workflow Scheme

Figure 6. shows generalised workflow scheme used in both cases (logo design

and JavaScript snippet coding) and in both experimental cycles. It depicts only two itera-

41

tions which are enough to provide understanding of workflow logic in which single spe-

cification is delivered as a task to multiple workers. As it passes from one iteration to an-

other, specification is transformed by workers and gains information as it becomes more

concrete. Low information content high-level (abstract) specification gains information

content and loses abstraction. Quality control is achieved by voting in which only one

artefact with the highest grade goes to the next iteration.

4.4 Research Model

The research combines case study with the design research as to provide an

insight into crowdsourcing phenomenon within a context of web design and development.

Research aims are to offer a perspective in regard to a hypothesis that typical web design

and development tasks can successfully be divided into manageable mini-tasks for

crowdsourcing, and that task lifecycle and task granularity can be optimized using auto-

mation workflows. The research is designed to cover two separate cases. One case will

be used to observe the effectiveness of proposed workflow model in a process of logo

design. Another case is used to observe coding of a simple JavaScript snippet using sim-

ilar workflow model. This covers both, web design and web development production

cycles and represents basic web design and development tasks. Research participants in

the first cycle are four designers and four web developers with at least two years of exper-

ience in the relevant field. These participants are known to the researcher. They are in-

formed about the details of the research and are not offered any monetary incentive. In

the second cycle, the participants are professionals from freelancer.com who will be

offered a monetary incentive as to provide real life circumstances. Test data is represen-

ted by two short specifications for each cycle, one for logo design and another for JavaS-

cript snippet coding. These specifications should mimic typical task specifications for

crowdsourcing within a proposed workflow framework.

42

Figure 7. Research model

Case study was chosen as a preferred method in this research for several reasons:

to provide additional insights regarding feasibility of proposed workflow

to provide an in-depth examination of the process

to determine various factors which are influencing the transformation of artefacts

within the workflow

43

to observe how the participants react to limitations caused by mini-task environ-

ment such as time and monetary incentive

to determine factors influencing task granulation and atomicity of tasks

to observe how the participants perceive the work on other people's artefacts

to observe voting on results as a mean of quality control

to provide a basis for eventual further quantitative research

Although a case study is a qualitative method usually used for hypothesis development

and as a precursor to quantitative research, it can serve as a valuable scientific approach

to problems covered by this research and by proposed hypothesis. According to Flyvbjerg

(2006), case studies can be used as a valid method for hypothesis testing. He points out

that one of misunderstandings about the case study is that it is most valuable for devising

hypothesis because other methods should be used for hypothesis testing. This idea is a

consequence of another misunderstanding which assumes that generalisation cannot be

based on individual case. The problem of case study as a basis for generalisation is over-

ruled by a simple test called "falsification" and introduced by Karl Popper (1959). Falsific-

ation is represented by an idea in which one particular observation, if shown unfitting with

the proposition, becomes a challenge to general validity of the proposition. A famous ex-

ample used by Popper also known as "all swans are white" shows that if only one swan is

found to be black, general proposition that "all swans are white" loses significance and

requires revision and/or further investigation. According to Flyvbjerg (2006) "One can

often generalize on the basis of a single case, and the case study may be central to sci-

entific development via generalization as supplement or alternative to other methods"

(Flyvbjerg, 2006:228) Furthermore, "The case study is useful for both generating and

testing of hypotheses but is not limited to these research activities alone." (Flyvbjerg,

2006:229) The main advantage of case study according to Flyvberg (2006) is the ability to

"“close in” on real-life situations and test views directly in relation to phenomena as they

unfold in practice." (Flyvberg, 2006:235)

44

4.5 The Experiment

The experiment follows structured timeline and steps in which test data is trans-

formed. In the first cycle of the experiment brief specification for the task is given to the

participants. They are informed about the objectives and aims of the research and are

asked to provide some information about their professional background and experience.

Brief outline of the experiment is given to participants in order to make them aware of the

process and its requirements. In this cycle, the overall feasibility of the workflow is tested

as no time limitation is given, and participants are asked to express their price for the time

spent on each task. In the second cycle, time limitation as well as the monetary incentive

is given to participants according to observations from the first cycle. The two cycles of

the experiment are needed because of the iterative nature of design research in which

workflow model is tested in the first cycle, then evaluated using participants feedback as

well as by observing the transformation of the artefacts, all in order to refine the model

workflow and evaluate it in the second cycle. According to Hevner & Chatterjee (2010)

"The internal design cycle is the heart of any design science research project. This cycle

of research activities iterates more rapidly between the construction of an artefact, its

evaluation, and subsequent feedback to refine the design further." (Henver & Chaterjee,

2010:18). In both cycles, participants are asked to answer few open questions about their

experience after every task, as well as few measurement questions like difficulty of the

task, required time for finishing the task, participants’ monetary valuation of the task, etc.

After all, there are some planned differences between the two production cycles. These

differences were planned before the experiment was conducted in order to observe the

impacts on participant's ability to complete the task under different conditions. Other vari-

ations will be introduced into second production cycle after the first cycle is completed

and will be based on the observation of the first production cycle.

First production cycle Second production cycle

45

Participants are professionals known to the researcher

Participants are professionals on freelan-cer.com

Time for the task is not limited Time for the task is limited according to the results from the first production cycle

There is no monetary incentive for the task (participants express perceived monetary value of their effort)

Tasks are paid according to experiment results from the first production cycle

Table 3. Planned differences between two production cycles

The experiment begins with initial specification for logo and JavaScript case which

is followed throughout the production cycle by observing its transformation. These spe-

cifications serve as test data into which each participant has to embed his perspective,

knowledge, experience and effort in the production process managed by proposed work-

flow. The specification for first production cycle workflow used in each case is presented

in Figure 7.

4.6 Test Data and Questionnaires

The task specification and questionnaire for participants following the steps of the first

production cycle in logo design case:

Task 1 - initial task

The task is to write a description of initial idea for a company logo design:

The purpose behind this task is a research in crowdsourcing work ergonomics within a micro task environments.

Only one logo concept idea is to be delivered You will also be asked to answer few simple questions in order to provide additional data for the

research. The questions are completely anonymous and will be used solely for the purpose of current academic research.

Please answer these questions before you start working on a task:

1. How many years of experience do you have in your profession?2. Are you familiar with crowdsourcing?3. What is your specialty?4. What grade would you give yourself as a professional? (1-5 scale)

Main Task:

Write down a concept, an idea for a company logo in few sentences. The company specializes in production and distribution of chocolate with messages for certain occa-

sions on their package. For example: Thanksgiving chocolate, Chocolate for the doctor, Chocolate for

46

the Boss, Chocolate for new family member, Love chocolate, etc. Target population is North American continent

After you finish the task please answer these simple questions:

1. How much time did you spend for the main task?2. How much money would you ask for the time spent on this task?3. Add your opinion regarding this task. (anything you think should be mentioned)

Thank you

Table 4. Task 1

Gathered results from each participant are put to voting in order to provide quality control

by eliminating all except one artefact which is forwarded to the next workflow step.

Task 2 - first voting task

Give a unique grade in a scale from 1 to 5 for every proposed solution:

Initial task was:

Write down a concept, an idea for a company logo in few sentences. The company specializes in production and distribution of chocolate with messages for certain occa-

sions on their package. For example: Thanksgiving chocolate, Chocolate for the doctor, Chocolate for the Boss, Chocolate for new family member, Love chocolate, etc.

Target population is North American continent

Proposed solutions are (Task 1 outputs):

1. Solution 12. Solution 23. Solution 3

Thank you

Table 5. Task 2

Voted solution comes in as a specification for the next production step. This time the spe-

cification for the first step and voting results were given to participants in order to provide

understanding of the task continuity. Since all four participants vote, every participant

votes only for three presented results from other participants. Such voting should address

probable bias which could occur if participants are given their own artefacts for voting.

Task 3 - production taskInitial task was:

Write down a concept, an idea for a company logo in few sentences. The company specializes in production and distribution of chocolate with messages for certain occa-

sions on their package. For example: Thanksgiving chocolate, Chocolate for the doctor, Chocolate for the Boss, Chocolate for the new family member, Love chocolate, etc.

Target population is North American continent

47

You sent your solutions and have voted for this solution:

(Task 2 output)

Main Task:

The task is to create initial logo draft (sketch) based on proposed specification.

After you finish the task please answer these simple questions:

1. How much time did you spend for the main task?2. How much money would you ask for the time spent on this task?3. Which tool did you use for accomplishing this task?4. Which application functionalities you find necessary for this type of task?

Thank you

Table 6. Task 3

The output of step 3 should be initial logo draft based upon voted specification from step

2.

Task 4 - second voting task

Initial task was:

Write down a concept, an idea for a company logo in few sentences. The company specializes in production and distribution of chocolate with messages for certain occa-

sions on their package. For example: Thanksgiving chocolate, Chocolate for the doctor, Chocolate for the Boss, Chocolate for new family member, Love chocolate, etc.

Target population is North American continent

Chosen solution is:

(Task 2 output)

Main task was:

Create initial logo draft (sketch) based on proposed specification.

Proposed solutions are (Task 3 outputs):

1. Solution 12. Solution 23. Solution 3

Give a unique grade in a scale from 1 to 5 for every proposed solution:

Thank you

Table 7. Task 4

Task specifications for logo design case in the first production cycle is different from

JavaScript development case in the way in which specification for every following task is

a cumulation of all previous specifications in order to provide substantial metadata for

every successing task. Both metadata and artefacts are gaining information content after

48

every production task. Such methodology is needed in order to provide enough informa-

tion to participants to adhere to initial logo specification. Without it, participants could lose

orientation in regard to initial specification goals thus losing the context and direction for

their work.

Task 5 - production task

Initial task was:

Write down a concept, an idea for a company logo in few sentences. The company specializes in production and distribution of chocolate with messages for certain occa-

sions on their package. For example: Thanksgiving chocolate, Chocolate for the doctor, Chocolate for the Boss, Chocolate for new family member, Love chocolate, etc.

Target population is North American continent

Chosen solution is:

(Task 2 output)

Based on this solution logo drafts were created.

You voted for:

(Task 4 output)

Main task:

There is initial draft logo provided (Task 4 output).

Your task is to refine the logo by adding or changing some design elements.

After you finish the task please answer these simple questions:

1. How much time did you spend for the main task?2. How much money would you ask for the time spent on this task?

Thank you

Table 8. Task 5

Task 6 - third voting task

Initial task was:

Write down a concept, an idea for a company logo in few sentences. The company specializes in production and distribution of chocolate with messages for certain occa-

sions on their package. For example: Thanksgiving chocolate, Chocolate for the doctor, Chocolate for the Boss, Chocolate for new family member, Love chocolate, etc.

Target population is North American continent

Chosen solution is:

(Task 2 output)

Based on this solution logo drafts were created.

49

You voted for:

(Task 4 output)

Main task was:

There is initial draft logo provided (Task 4 output) Your task is to refine the logo by adding or changing some design elements.

Proposed solutions are (Task 5 outputs):

1. Solution 12. Solution 23. Solution 3

Give a unique grade in a scale from 1 to 5 for every proposed solution:

Thank you

Table 9. Task 6

In the first cycle of JavaScript snippet case, the design logic is identical to logo

design case. Still, there were few differences related to different types of tasks represen-

ted in test data specification under the "main task" section. Additionally there was no need

to cumulate metadata in form of previous specifications as in logo design case because

initial specification is contained in the artefact itself and can be exactly reproduced just by

reading and testing the code snippet. Following is the first task and voting task in the first

cycle of JavaScript snippet case:

Task 1 - initial task

The task is to code initial JavaScript snippet according to specification:

The purpose behind this task is a research in crowdsourcing work ergonomics within micro task envir-onments.

Only one JavaScript snippet is to be delivered You will also be asked to answer few simple questions in order to provide additional data for the

research. The questions are completely anonymous and will be used solely for the purpose of current academic research.

Please answer these questions before you start working on a task:

1. How many years of experience do you have in your profession?2. Are you familiar with crowdsourcing?3. What is your specialty?4. What grade would you give yourself as a professional? (1-5 scale)

Main Task:

Create a popup window 200*200px at the centre of the screen. The window opens on page load after 500 milliseconds. Place a 50*50px picture in the upper right corner of the window.

50

Place a textbox of 20px width in upper left corner. Place a close button in the lower centre

After you finish the task please answer these simple questions:

1. How much time did you spend for the main task?2. How much money would you ask for the time spent on this task?3. Add your opinion regarding this task. (anything you think should be mentioned)

Thank you

Table 10. Task 1

Task 2 - first voting task

Initial task was:

Create a popup window 200*200px at the centre of the screen. The window opens on page load after 500 milliseconds. Place a 50*50px picture in the upper right corner of the window. Place a textbox of 20px width in upper left corner. Place a close button in the lower centre.

Proposed solutions are (Step1 outputs):

1. Link to Solution 12. Link to Solution 23. Link to Solution 3

Give a unique grade in a scale from 1 to 5 for every proposed solution:

Thank you

Table 11. Task 2

51

CHAPTER 5. METHODS AND REALIZATION

The experiment covers two separate cases - logo design and JavaScript snippet

coding. Each of the cases consists of two phases - cycles, as to facilitate the benefits of

applying insights from the first to the second production cycle. Insights were provided by

observation of artefacts within a workflow defined transforming process taking place in a

working environment both unknown and new to the participants. Since cases are inde-

pendent from one another, it was possible to minimize expected time for the first produc-

tion cycle by executing it for both cases simultaneously. On the basis of task difficulty, it

was expected that the first production cycle for both cases would be completed within a

similar timeframe. First production cycle for both cases started at the same time, and

each followed its own propagation through iterative steps. Participants in the first cycle

were individuals known to the researcher either through professional or private circles.

They were picked according to two criteria:

They needed to have at least 2 years of related professional experience

They expressed interest to participate in the research

There was no monetary incentive for participants in the first production cycle, but they

were informed about typical limitations regarding the time and the scope for the tasks

involved. These limitations are corresponding to logic and nature of crowdsourcing. In

ideal circumstances, participants would be professionals in the related field with the ex-

perience in micro-task crowdsourcing. Although such participants undoubtedly exist, they

were not within the reach of the researcher. Additionally, participants in the first cycle had

to be known to the researcher because of assurance of their skill set and professional

experience. These are important factors as potential transformation of artefacts should

not be limited by a lack of professional experience or by the inability to provide articulate

output even under normal circumstances.

52

5.1 Logo Design Case

The entire communication with participants in the first production cycle was con-

ducted via e-mail. Email with initial task specification was sent to every participant. After

the task had been completed, each participant sent back their results. Task results were

combined and sent back to participants for voting. Participants sent their votes back,

votes are calculated, and an email with highest graded artefact together with specification

for the next task is sent back to participants. Since participants had to vote on the quality

of artefacts by using unique grades in range from 1 to 5 on every series of artefacts, the

winning artefact was selected by comparing the sum of grades from voters on every par-

ticular artefact.

The second production cycle involved participants from freelancer.com. Tasks

were created as mini projects with a single milestone. Every production/refinement task

was assigned to four freelancers which had no obligation to continue participation in suc-

ceeding tasks. For each task, first four freelancers who made a bid on the task were ap-

pointed. According to the results from the first production cycle, task duration was limited

to 30 minutes and the compensation for each task was determined to be 10$. The spe-

cification for the initial task was identical to a specification for the third task in the first

production cycle, meaning that in second production cycle there were no tasks asking for

conceptual specification of the artefact. Such structural change was needed in order to

put more focus on the refinement tasks which proved to be more challenging than all

other tasks in the first cycle. The change was implemented by redesigning the workflow

structure in which initial production task was followed by two refinement tasks. After the

completion of each task, results were gathered and sent for voting on microvorkers.com.

For the purpose of voting, each voting task was implemented as a short survey on sur-

veymonkey.com containing the choice of four artefacts (logo designs). The link for a sur-

vey was posted as a survey task in microworkers.com. Since microvorkers.com defines a

minimum of 30 participants for each task, each voting had to conform to this limitation.

The example of the voting survey can be found in the appendix.

1st production cycle 2nd production cycleParticipants Four professionals known to

the researcherFour freelancers on freelancer.-com

53

Task time limita-tion

No time limitation Task limited to 30 minutes

Incentive No monetary incentive 10$ per taskStructure of the tasks

specification

production

refinement

production

refinement

refinement

Voting Voters are participants Voters are crowd workers from microworker.com

Questionnaire 1. How much time did you spend for the main task?

2. How much money would you ask for the time spent on this task?

3. Which tool did you use for accomplishing this task?

4. Which application functionalities you find necessary for this type of task?

5. Add your opinion re-garding this task. (anything you think should be mentioned)

1. How much time did you spend for logo draft cre-ation?

2. Is given specification for logo draft good enough to provide you with all needed information for the task? Your opinion.

3. What is your opinion on task time limitation?

4. Do you rather prefer short tasks or more elab-orate ones? Why?

5. Did you complete the task at once or with inter-ruptions?

6. Describe your creative process in 2 sentences

7. Were you motivated for the task? Why?

8. Task complexity (1-5):

9. If you would like to add your opinion, a problem regarding the task or anything that you find important please provide it here:

Table 12. Actual design cycle differences (logo design case)

Table 12. represents actual differences in design implementation between two

production cycles. Participants, Task Time Limitation & Incentive differences were

planned ahead. Structure of the tasks and questionnaire were changed according to

gathered insights from the first cycle. The difference in voting emerged out of the need to

test artefact validation process in the wild.

54

5.2 JavaScript Snippet Coding Case

The implementation of the first production cycle in JavaScript snippet case fol-

lowed logic similar to logo design case. The participants were four web developers with at

least two years of experience in creating websites and web based solutions. They were

known to the researcher and were offered no monetary incentive. However, based on

experience from the logo design case and following the design research cycle it became

apparent that JavaScript case was perceived as much more convenient to participants

and was therefore significantly easier to implement. Additionally, data gathered from the

first two tasks in JavaScript case together with experience from logo design case were

valuable enough to support early changes to a planned workflow before the end of the

first cycle. With this in mind, it was possible to continue third task of the first cycle on

freelancer.com

1st production cycle 2nd production cycleParticipants Four professionals known to the

researcherFour freelancers on freelancer.com

Task time limitation No time limitation Task limited to 30 minutesIncentive No monetary incentive 10$ per taskStructure of the tasks production

refinement

refinement

refinement

refinement

refinement

Voting Voters are participants Voters are participants from the first cycleQuestionnaire 1. How much time did you

spend for the main task?

2. How much money would you ask for the time spent on this task?

3. Add your opinion regarding this task. (anything you think should be mentioned)

1. How much time did you spend for logo draft creation?

2. Is given specification for logo draft good enough to provide you with all needed information for the task? Your opinion.

3. What is your opinion on task time limitation?

4. Do you rather prefer short tasks or more elaborate ones? Why?

5. Did you complete the task at once or with interruptions?

6. Describe your creative process in 2 sentences

7. Were you motivated for the task? Why?

8. Task complexity (1-5):

9. If you would like to add your opinion, a problem regarding the task or anything you find important please provide it

55

here:

Table 13. Actual design cycle differences (JavaScript snippet case)

5.3 Summary Gathered data came in the form of artefacts and their transformation during each

series of tasks as well as in the form of participant's answers to a questionnaire after each

complete task. Since workflow was implemented in a proof of concept setup and all work-

flow steps were executed without using any programmed workflow engine, it was possible

to change the workflow during design research cycles and to observe multiple aspects

which had influence on the usability of workflow. Data revealed by participant's answers

to questionnaire are interpreted within the framework of qualitative research having in

mind original purpose of the research which is to evaluate the proposition that typical web

design and development tasks can successfully be divided into manageable mini-tasks

for crowdsourcing. In the chapter "Results and Evaluation" data will be analysed using

direct interpretation and by establishing relational patterns between multiple task execu-

tion occurrences within two cases (logo design and JavaScript snippet coding).

56

CHAPTER 6. RESULTS AND EVALUATION

During the case study experiment, a large amount of data has been gathered. To

be able to provide meaningful description and interpretation of the results, a tabular case

study database is created and can be found under the Appendices section. It consists of

all artefacts, questions, and answers from participants. The database also provides an

overview of the tasks and the transformation of artefacts created during the tasks by each

participant in exact order as it happened throughout the experiment on each case. At

least two factors had a negative impact on the quality of data gathered during second

cycles for both cases. These are:

Limitations arising out of using freelancer.com platform for research - paid freel-

ancers tend to bias their questionnaire answers towards perceived researcher's

needs. They are sometimes overly complaisant in their aim to leave good impres-

sion.

Some participants were less informative than the others due to a language bar-

rier.

Overall research results contribute to a positive evaluation of proposed workflow model

and indicate that it is possible to adapt typical web design and development tasks to a

crowdsourcing model. However there are certain limitations to the research which will be

covered in the next chapter. To provide an overview on research results some statistics

regarding the experiment are presented:

Logo artefacts created 20

JavaScript snippets created 20

Number of participants involved in production tasks 16

Number of voters via microvorker.com 102

Number of individual tasks (production + voting) 171

Actual research budget (freelancer.com, surveymonkey.net, micro-

workers.com)

398$

57

Table 14. Overall research stats

The major difference between logo and JavaScript artefacts regarding the quality

evaluation lies in the possibility of establishing minimum requirements criteria. Such cri -

teria can easily be established for JavaScript snippets because it is represented by spe-

cified functionality which was tested on every delivered artefact. Discrete functional ele-

ments in JavaScript artefacts were easy to evaluate and even track throughout every

new generation of artefacts. In logo design case however, functional elements are not so

discrete, and evaluation could be conducted only subjectively by voting and eliminating

lower quality artefacts. The problem in such setup is that in case of relatively weak logos

given for voting, one logo would always win, whereas in case when four JavaScript arte-

facts would be given for voting, and if they all were supposedly below minimum function-

ality requirements, they would all be disqualified. With this in mind, throughout the entire

experiment only one JavaScript artefact was disqualified for not providing specified func-

tionality.

During the process, some new questions were introduced as to give additional

insights into participants' perspective of the workflow. One of these questions made in-

quiry into emotional state of the participants at the onset of task by asking "How do you

feel?" Possible answers were: Very poor, Poor, Average, Good, and Excellent. Higher

emotional state can be linked to confidence level and self-value which can lead to

greater working performance and increased ability of adaptation to working environment

(Febrilia, Warokka, & Abdullah, 2011). However, the analysis of results has shown that

other questions which implicitly gave information about emotional state of participants

have proven to be more accurate. For example in several statements participant 4 stated

her perceived professional level which did not correspond to her perceived emotional

state at time of task execution. At the onset of the first task in second JavaScript produc-

tion cycle, she stated her professional level to be 4.5 out of 5. Later at the beginning of

fifth task of the same cycle, she raised perceived professional level to 5 of 5 which she

later demoted to 4 out of 5 at the onset of Task 7 (See Appendices). On the other hand

she stated to feel excellent every time. This was an indicator that emotional influence

surely exists, and that it may have effects on other measured factors but it cannot be

58

measured only by using explicit inquiry. Additionally, if emotional state could be facilit -

ated or induced via task ergonomics by adapting task specification content to particular

participants' needs, it could have positive impact on task output quality. Another question

introduced at the end of second JavaScript cycle investigated the possibility of finishing

tasks within a timeframe of 15 minutes and corresponding circumstances that would be

beneficial in such setup by asking "Can this task be accomplished in 15 minutes? Under

which circumstances?". This factor was investigated during execution of comparatively

more difficult tasks for which participants needed more time than specified 30 minutes.

The aim was to gather participants' opinion and valuation regarding requirements for

such prospect and to determine whether additional details in the task specification could

be refined in order to support time-efficient task execution. Only one participant in logo

design case answered negatively to the question “Can this task be accomplished in less

than 15 minutes?” asked during last logo refinement task. Other participants think that it

is possible. Only one of them stated that it would be possible if existing logo draft con -

tained all needed visual element layers of existing logo draft and if he had the authority

to change all of them. The answer points to other problem encountered during logo

design cycle which is the problem of refinement definition. Contrary to JavaScript case

refinements, logo design refinement cannot be discretely defined because it depends on

subjectively perceived design elements for which it is very hard to create formal categor-

isation. Such categorisation is easily applicable to program code creation because it

presumes strong relation between specification elements and functional elements within

the actual code.

59

6.1 Logo Design Case

Figure 8. Final Artefact - First Cycle Logo Design Case

Figure 8 represents the final result of the first logo design production cycle. It

contains several versions of a logo adapted to different tastes of chocolate. The final res-

ult is a compound of series of tasks defined by a workflow model. It started as a require-

ment specification which was transformed into textual logo concept specification during

the first task. The concept specification was transformed into first logo draft during the

third task, and it was further refined during the fifth task. At this stage, it became evident

that task specification can be used for guidance and direction of participants in regard to

needed outcome. For innovation, tasks need to be more open and abstract and with less

information content. If there is a need for a certain quality or feature the task need to be

more focused and with higher information content.

60

Figure 9. Final Artefact - Second Cycle Logo Design Case

Figure 9 represents the final result of the second logo design production cycle. It was

created as a combined product of multiple consecutive individual tasks each having a

time limitation of 30 minutes thus showing that it is possible to create crowdsourcing tasks

which coordinate many individual cognitive efforts while producing non discrete but mean-

ingful outcome.

61

6.2 JavaScript Snippet Coding Case

Figure 10. Artefact Information Content

62

JavaScript case was much more deterministic in the task specification as well as

in results evaluation. Functional elements could be more easily mapped to workflow tasks

than in logo design case. It was also possible to exactly follow artefact information content

growth. Figure 10 shows enclosed coloured areas representing the output of different

tasks. Areas of code without any colour represent the output of the first production cycle

(Participant 2). Blue area contains the code produced during the first task of the second

production cycle (Participant 4, Task 1). The code produced as output of the third task of

the second production cycle is contained within the red area (Participant 1, Task 3). As

the artefact propagates through refinement cycles and gets additional information content,

every new refinement task becomes more complex than the previous one not because of

the complexity of the refinement specification but because it is more difficult to interface

refinement elements into existing code without breaking existing features. Higher number

of iterations also induces greater information content within metadata in form of existing

code. In terms of usability of the workflow and task specification, one freelancer ex-

pressed his satisfaction with working on tasks and regarding positive educative effects of

the tasks. He stated that he learned a lot about programming concepts and that he is

looking forward to a new series of tasks because they are interesting and instructive.

63

CHAPTER 7. CONCLUSIONS

7.1 Lessons Learned

The pursuit of this project was very awarding, inspirational and insightful because crowd-

sourcing is a young field having a potential for significant future innovation. Researchers

are already tapping into this field both because of its multidisciplinarity as well as because

of its broad application areas. One of these areas is general task ergonomics and its ties

to satisfaction of work. There is also a core aspect of crowdsourcing which supposes

innovation through distributed collaboration which was also one of the aspects investig-

ated in this research. In a very limited environment, it was proven that production process

can be divided into atomic tasks whose creation is based on logic of continual refinement

of low complexity initial artefacts. Such refinement also enables integration of multiple

features into one existing artefact which was clearly observable in JavaScript coding

case. Proposed concept is different from existing crowdsourcing concept for complex

work used by Kittur et al. (2011) which also uses map reduce for structural division of

project scope. While existing model maps structure in parallel (horizontal), proposed

model can map structure by mapping production process vertically (by using consecutive

tasks). There is no doubt that structural division of tasks is needed in high level definition

of big projects. However, crowdsourcing can be more efficient for complex tasks if they

are divided into intermediary task specifications which would then further be divided into

atomic tasks such as used in proposed model. In the example of this research, it became

clear that the main benefit of dividing intermediary tasks into atomic tasks similar to those

used in research experiment, relates to inter-task coordination where every atomic task

produces atomic functional element. When tasks are divided into refinement layers, co-

ordination and integration into one existing artefact becomes successive, meaning that

there is no need to coordinate parallel executions at this level. Later as every specific

artefact grows in complexity, it can be integrated into greater context using the same prin-

ciple. This led to understanding that atomic tasks within a complex work environment can

be defined as tasks which can be accomplished within a single attention cycle. It also

64

became apparent that time limitation within this research was only a starting point which

directed focus to individual cognitive cycles. Time limitation was not the goal, but a way of

investigating individual cognitive cycle boundaries which are relevant to applicability of

proposed workflow model. The aspect of parallelization within a proposed model was

used solely for the purpose of innovation where multiple participants (workers) embed

their individual perspective by working in parallel on elements defined by the same spe-

cification. In high-to-low level abstraction model, participants within the same task series

are always working in parallel. In such case, coordination of tasks and integration of arte-

fact functional elements is vertical as opposed to horizontal in the case of integration of

artefacts of higher abstraction. Figure 11 represents production cycle which consists of

four production iterations. Red line represents transformation of artefacts. In every trans-

formation, artefacts gather additional information content while integrating acquired in-

formation vertically. Due to elimination of low-quality artefacts, the result of such process

is one single artefact. Every horizontal integration of finished artefacts of the same level of

abstraction would exist as new vertical integration of artefacts within a new production

cycle. Such integration was out of scope of this research.

Figure 11. Growing information content of artefacts

65

Proposed workflow model parallelises and coordinates individual cognitive cycles thus

allowing higher added value in artefact production because it promotes best out of many

possible solutions. These solutions resulted by approaching the problem from many indi-

vidual perspectives.

7.2 Academic Application and Limitations

At some point during the research, it became evident that specific task ergonomics tied to

crowdsourcing contexts can be applied to general work ergonomics. As proposed re-

search model combines design research with a case study method, it allows researchers

to conduct similar research on task ergonomics within traditional working environments. It

also allows usage of precise research instruments (surveys and questionnaires) in a relat-

ively cheap setup. Additional application of the research is for educational purposes. By

participating in a workflow defined production process individuals can learn about the

relation between functional elements, their level of abstraction and their representation as

implemented units of content within functional artefacts (code snippets and visual ele-

ments). Such concept can be more easily implemented in code programming but can also

be used in other areas of interest (design, technical and business innovation, creation of

various content).

Limitations

The usability of proposed workflow model has been tested only on atomic tasks defined

for purposes of development of JavaScript code snippets and the design of logos based

on simple requirements. These represent only two typical artefacts within web design and

development practice which operates by utilising many collections of possible artefacts.

Further limitation arises out of the implementation of the research model which lacked

interpersonal communication during the phases of data gathering. This limitation is in-

duced by using internet based communication during experiment. One additional limita-

tion of the research is insufficient examination of economic aspects and its impacts on

66

task output quality, however it is assumed that multiple perspectives in task solving pro-

cess can be more valuable than single extensive perspective.

7.3 Business Application and Limitations

According to the results of logo design case, it seems that combined professional

and regular opinions contribute to artefact usability. This is very useful insight especially

for companies which target their audience. Best combination of both worlds can be

achieved if professional skills and experience are combined with the taste of target audi-

ence especially when every step of the process is overseen by members of target popula-

tion. In addition, the need for defining a new model of communication between various

services supporting crowdsourcing resulted in a definition proposal for the “open call

framework”, a theoretical model for integration of different collaborative and crowd-

sourcing platforms. It should be used to connect different services like microworker.com

and freelancer.com with a workflow management services thus allowing information ex-

change between them in order to optimize their control of crowdsourcing process as well

as to gather additional insights through iterative research. Open call framework is a theor-

etical model used for information and resource exchange. A typical communication packet

should consist of management and control part, a research part, and payload in form of

actual task description. It should be able to connect to existing services’ API and to use

their databases with the aim of mutual benefits in design of new service features in devel-

opment of crowdsourcing principle.

7.4 Recommendations / Prospects for Future Research / Work

Future research should point towards further investigation of following areas:

The prospect of the creation of tasks for integration of finished elementary arte-

facts (such artefacts were created in this research). These tasks should be a

product of decomposition of the work needed for building an entire functional

67

website. Current research investigated only size and scope of atomic tasks which

produce elementary (atomic) functional elements.

Investigation of factors which influence and promote positive emotional response

within a crowdsourcing platform user interface

How functional artefact elements can be mapped to cognitive cycles.

In what areas can task ergonomics based on cognitive cycles influence and

change the logic behind web design and development project structure definition

Investigate the usability limit of proposed model in relation to refinement-induced

rising complexity of produced code

Investigate other areas of complex work which could implement proposed model

elements:

Creation of complex specifications

The creation of Law

Product design

Design of technical solutions

Direct governance and participatory democracy

Quantitative research which would be focused on the examination of individual

factors within proposed model setup:

Cultural, gender, age and education oriented differences on the level of

adaptation of proposed model

Relation of differences in the task specification formulation to the level of

participant’s motivational resistance during task execution

68

REFERENCES CITED

Agafonovas, A. & Alonderiene, R. (2013) Value Creation in Innovations Crowdsourcing. Example of Creative Agencies . Organisations and Markets in Emerging Economies, Vol. 4, No. 1(7), 72-103.

Asmolov, G. (2011) Crowdsourcing as an Activity System: Online Platforms as Mediating Artifacts. In Proceedings of the Sintelnet WG5 Workshop on Crowd Intelligence: Foundations, Methods, and Practices. Barcelona, Catalonia, 8-9 January, 2014. on-line: CEUR-WS.org. [ONLINE] Available at: http://ceur-ws.org/Vol-1148/paper3.pdf [Accessed 17 May 14]

Brabham, D.C. (2009) Crowdsourcing the public participation process for planning pro-jects, Planning Theory, Vol. 8 No. 3, pp. 242-62

Dean, J. & Ghemawat, S. (2004) MapReduce: Simplified Data Processing on Large Clusters. In 6th Symposium on Operating Systems Design & Implementation . San Francisco, CA, December 5. San Francisco: OSDI. 1-13.

Febrilia, I., Warokka, A. & Abdullah, H. (2011) University Students’ Emotional State and Academic Performance: New Insights of Managing Complex Cognitive . Journal of e-Learning and Higher Education, Vol. 2011, Article ID 879553, 15p

Flyvbjerg, B. (2006) Five Misunderstandings About Case-Study Research. Qualitative Inquiry, vol. 12 no. 2, 219-245

Hevner, A. & Chatterjee, S., 2010. Design Research in Information Systems. , 22, pp.9–23. Available at: http://link.springer.com/10.1007/978-1-4419-5653-8 [Accessed July 9, 2014].

Hirth, M.,Hossfeld, T. & Tran-Gia, P. (2011) Modeling of crowdsourcing platforms and granularity of work organization in Future Internet. In Proceedings of the 2011 23rd International Teletraffic Congress. San Francisco, USA, 6-8 September. San Fran-cisco: IEEE. 142-149.

Hirth, M., Hossfeld, T. & Tran-Gia, P. (2012) Analyzing a Commercial Microtask Crowd-sourcing Platform from the Involved Actors’ Viewpoints. informatik.uni-wuerzburg.de. Available at: http://www3.informatik.uni-wuerzburg.de/events/summerschool2012/pro-ceedings/Hirth.pdf [Accessed August 4, 2014].

Hobden, H. & Hobden, M. (1989) John Harrison and the problem of longitude. … , Eng-land: Cosmic Elk, 1989. 2nd ed. Available at: http://scholar.google.com/scholar?hl=en&btnG=Search&q=intitle:No+Title#0 [Accessed August 16, 2014].

Hoseini, M et al. (2014) The Four Pillars of Crowdsourcing: a Reference Model. In IEEE 8th International Conference on Research Challenges in Information Science . Mar-rakesh, Morocco, May 2014. Morocco: IEEE. 12.

Joosten, S. & Brinkkemper, S.(1995) Fundamental Concepts for Workflow Automation in Practice . In ICIS'95. Amsterdam, December. Amsterdam: ICIS. 1-12.

Kittur, A., Nickerson, J. & Bernstein, M. (2013) The future of crowd work. 2013 ACM 978-1-4503-1331-5/13/02, p.1301. Available at: http://dl.acm.org/citation.cfm?doid=2441776.2441923 [Accessed August 4, 2014].

Kittur, A. et al. (2011) CrowdForge: Crowdsourcing Complex Work. In: UIST, 2011, Palm Springs. pp. n.a. [ONLINE] Available at: http://smus.com/crowdforge/crowdforge-uist-11.pdf [Accessed 14 May 14]

Kulkarni, A., Can, M. & Hartmann, B. (2011) Turkomatic: Automatic, Recursive Task and Workflow Design for Mechanical Turk. Papers from the 2011 AAAI Workshop, 11/11, 91-96. [ONLINE] Available at: http://www.aaai.org/ocs/index.php/WS/AAAIW11/pa-per/viewFile/3884/4252 [Accessed 17 May 14]

69

Lakhani, K., Jeppesen, L., Lohse, P., & Panetta, J. (2007). The value of openness in sci-entific problem solving. Working paper, accessed September. 14, 2014: hbswk.hb-s.edu/item/5612.html

Le Bon, G. (1895), The Crowd: A Study of the Popular Mind, Ernest Benn, London (trans-lated 1947)

Lebraty, J.F. & Lobre-Lebraty, K. (2013) Crowdsourcing, One Step Beyond. 1st ed. Lon-don: John Wiley & Sons, Inc.

Lopez, M., Vukovic, M. & Laredo, J. (2010) PeopleCloud service for enterprise crowd-sourcing.Proceedings - 2010 IEEE 7th International Conference on Services Comput-ing, SCC 2010, 538-545. [ONLINE] Available at: http://libguides.liv.ac.uk.ez-proxy.liv.ac.uk/onlineprogrammes [Accessed 15 May 14]

Lynch, P.J. & Horton, S. (2011) The Site Development Process. [ONLINE] Available at: http://webstyleguide.com/wsg3/1-process/7-development-process.html. [Accessed 12 October 14].

Marjanovic, O. & Roztocki, H. (2013) The Current Research Landscape of Crowd-sourcing: Implications for Emerging Economies. In Proceedings of the Second Inter-national Conference on ICT Management for Global Competitiveness and Economic Growth in Emerging Economies. Wroclaw, Poland, September 16-17. Wroclaw: AIS Special Interest Group on ICT and Global Development . 289-311.

McClelland, J.S. (1989), The Crowd and the Mob: From Plato to Conetti, Unwin Hyman, Boston, MA

Perry, M. (1998) Process, representation and taskworld. Distributed cognition and the organisation of information.. In ISIC: Exploring the contexts of information behaviour. Scheffield, England, 17 September. Sheffield: ISIC. 1-11.

Popper, K. (1959). The logic of scientific discovery. New York: Basic Books

Safire W.(2009) Fat Tail. [ONLINE] Available at: http://www.nytimes.com/2009/02/08/magazine/08wwln-safire-t.html?_r=4&ref=magazine&. [Accessed 02 July 14].

Saxton, G., Oh, O. & Kishore, R. (2013) Rules of crowdsourcing: Models, issues, and systems of control. Information Systems Management. Available at: http://www.tand-fonline.com/doi/abs/10.1080/10580530.2013.739883 [Accessed August 18, 2014].

Schulte, D. (2012) Towards a Human Task Management Reference Model. In Proceed-ings of the 4th Central-European Workshop on Services and their Composition. Bam-berg, Germany, 23-24 February. online: ceur-ws.org 104-111.

Seuken, S. & Zilberstein, S.(2008) Formal models and algorithms for decentralized de-cision making under uncertainty . Autonomous Agents and Multi-Agent Systems, vol. 17 issue 2 , 190-250.

Simperl, E., Norton, B. & Vrandecic, D. (2011) Crowdsourcing tasks in Linked Data man-agement. In 10th International Semantic Web Conference 2011. Bonn, Germany, October 23, 2011. online: CEUR-WS.org. [ONLINE] Available at: http://ceur-ws.org/Vol-782/SimperlEtAl_COLD2011.pdf [Accessed 17 May 14]

Simula, H. & Vuori, M., (2012) Benefits and Barriers of Crowdsourcing in B2B Firms: Gen-erating Ideas With Internal and External Crowds. International Journal of Innovation Management, 16(06), p.1240011. Available at: http://www.worldscientific.com/doi/abs/10.1142/S1363919612400117 [Accessed July 18, 2014].

Sydney Opera House (2014) Sydney Opera House History 1954 - 1958 . [ONLINE] Available at:http://www.sydneyoperahouse.com/about/house_history/1954_1958.aspx. [Accessed 17 September 2014].

Toyota Global Site | History. 2014. Toyota Global Site | History. [ONLINE] Available at: http://www.toyota-global.com/showroom/emblem/history/. [Accessed 17 September 2014].

70

Van der Aalst, W. & van Hee, K. (2000) Workflow Management: Models, methods and systems. 1st ed. Eidnhoven: University of Technology.

Vukovic, M. (2009). Crowdsourcing for Enterprises . In 2009 IEEE Congress on Services. L.A. Califronia, 610 July 2009. Los Angeles: IEEE Computer Society. 686-692.

Wexler, M., 2011. Reconfiguring the sociology of the crowd: exploring crowdsourcing. International Journal of Sociology and Social Policy. [Online] Available at: http://www.emeraldinsight.com/journals.htm?articleid=1907199&show=abstract [Accessed September 19, 2014].

WFMC (1999) Workflow Management Coalition Terminology & Glossary. 3rd ed. Winchester: WFMC.

www.cs.uml.edu (N.D) MapReduce word count process [Online Image] Available at: http://www.cs.uml.edu/~jlu1/doc/source/report/MapReduce.html. [Accessed 17 September 14].

Xu, H. & Larson, K. (2014) Improving the efficiency of crowdsourcing contests. Proceed-ings of the 2014 international conference on …. Available at: http://dl.acm.org/cita-tion.cfm?id=2615807 [Accessed August 3, 2014]

71

APPENDICES

APPENDIX A. CASE STUDY DATABASE

Table 15. Logo Design Case

First production cycle - prototype testing

Professionals known to the researcher No time limit No monetary incentive Participants are familiarized with expectations for crowdsourcing tasks All participants are engaged on every task Task specification contains cumulative metadata

Tasks & questionnaires

Task 1 Participant 1 Participant 2 Participant 3 Participant 4

1. How many years of experience do you have in your profession?

10 6 2 2

2. Are you familiar with crowd-sourcing?

yes no No Yes

3. What is your specialty? logo design, web design graphic design illustrator web design

4. What grade would you give yourself as a professional? (1-5 scale)

3 3 4 4

Main task:

Write down a concept, an idea for a company logo in few sentences. The company specializes in production and distribution of chocolate with messages for certain occasions on their package. For example: Thanksgiving chocol-ate, Chocolate for the doctor, Chocolate for the Boss, Chocolate for new family member, Love chocolate, etc. Target population is North American continent.

Chocolate emerges out of half peeled cacao fruit. Care must be taken in order not to misinterpret the cacao fruit with banana or some other fruit. Illustration should be as simple as possible containing two main colours, the colour of cacao fruitpeel and the colour of the chocolate, and variations of these colours for the details. Possible problem of the concept is the fact that cacao fruit is rarely pealed but cut in half horizontally.

Chocolate, USA, money. Bribe chocolate comes first to my mind as well as chocolate coin envel-oped into golden foil which we used to eat when we were kids, it was like real money to us.Logo in shape of coin with half pealed golden foil resembling the chocolate beneath.Statue of liberty with big chocol-ate in place of the flame!

Anthropomorphic peace of chocolate with hands, legs and eyes, styled as clear black and white graphic (can be coloured if needed). It changes attributes depending on personified message: Chocolate for doctor has round head mirror, Reconciliation chocolate has "peace sign" around the neck.

Something resembling a combina-tion mark logo based on the name "Say Choco!" placed within a comic dialog cloud.

1. How much time did you spend for the main task?. How much time did you spend for the main task?

20-25 min 5 min 5 min. 5 min

2. How much money would you ask for the time spent on this task? ask for the time spent on this task?

24$ 5$ 10$ 10$

3. Add your opinion regarding this task. (anything you think should be mentioned)n

Without the name for the product or the company which is a structural part of the logo, it is hard to give a good solution. Words can provide additional inspiration. The solution can be based solely on a word or a slogan which can be styled in a certain fashion. My proposed solution contains only a symbol.

The task was clear and articulate. In order to come to the solution I had to draw a sketch using plain pencil and paper :) My process always starts that way. Regarding the task, the name for the company is missing. Thanksgiving chocolate or chocolate for the doctor both sound like a brand for the product within the company but we are talking about the logo for company brand (like Nestle, Milka etc.) This part is not specified precisely enough.

For a logo design a brand name is needed. Since it was missing I invented a slogan which could also be a brand name for the company.

Task 2 - voting task

Give a unique grade in a scale from 1 to 5 for every proposed solution (every participant votes for 3 solutions from other participants): to 5 for every proposed solution (every participant votes for 3 solutions from other participants):

Participant 2 solution = 2Participant 3 solution = 4Participant 4 solution = 5

Participant 1 solution = 3Participant 3 solution = 4Participant 4 solution = 5

Participant 1 solution = 3Participant 2 solution = 2Participant 4 solution = 5

Participant 1 solution = 4Participant 2 solution = 3Participant 3 solution = 5

Sum of grades for participant's solution

3+3+4=10 2+2+3=7 4+4+5=13 5+5+5=15

(This is a winning solution and it becomes a specification for the next task.)

72

Task 3

(In original questionnaire main task specification was preceded by specification for Task 1 in order to provide a context - metadata)

Main task:

The task is to create initial logo draft (sketch) based on proposed specification:

Something resembling a combina-tion mark logo based on the name "Say Choco!" placed within a comic dialog cloud.

I had to constantly repeat to myself that I have to stop creating different versions of logo and have to decide quickly what to send. Stressful but nothing unusual for me. This way I resolved the versioning dilemma but I still think that better solu-tions require more effort and would cost more.

1. How much time did you spend for the main task?

37 min 30 min 15 min 20 min

2. How much money would you ask for the time spent on this task?

50$This is a very hard question. Under normal circumstances and most probably, I would not work using a method being used in this research. Never before have I worked this way. Therefore I can only express the amount for the time needed under circum-stances of this research.

320$ 10$ 10$

3.Which tool did you use for accomplishing this task?

Paper, pencil, adobe illustrator Adobe illustrator Gimp Photoshop

4.Which application functionalities you find necessary for this type of task?

Paper, pencil, font viewer, vector graphics editor

Layers, vectors, transformation tools

Vector graphics supported editor, layers

Any tool supporting vectors and paths

Task 4 - voting task

Give a unique grade in a scale from 1 to 5 for every proposed solution (every participant votes for 3 solutions from other participants):

Participant 2 solution = 5Participant 3 solution = 2Participant 4 solution = 1

Participant 1 solution = 5Participant 3 solution = 4Participant 4 solution = 3

Participant 1 solution = 4Participant 2 solution = 5Participant 4 solution = 2

Participant 1 solution = 4Participant 2 solution = 5Participant 3 solution = 3

Sum of grades for participant's solution

5+4+4=13 5+5+5=15

(This is a winning solution and it becomes a specification for the next task.)

2+4+3=9 1+3+2=6

Task 5

(In original questionnaire main task specification was preceded by specification for all preceding tasks in order to provide a context - metadata)

Main taskMain task:There is initial draft logo provided (Task 4 output).Your task is to refine the logo by adding or changing some design elements.There is initial draft logo provided. Your task is to refine the logo by adding or changing some design elements.

1. How much time did you spend for the main task?me did you spend for the main task?

5 min 30 min 20 min 25 min

2. How much money would you ask for the time spent on this task?

3.5 $ 80 $ 8 $ 10 $

Task 6 - voting task

Give a unique grade in a scale from 1 to 5 for every proposed solution (every participant votes for 3 solutions from other participants):

Participant 2 solution = 5Participant 3 solution = 2Participant 4 solution = 1

Participant 1 solution = 5Participant 3 solution = 4Participant 4 solution = 3

Participant 1 solution = 3Participant 2 solution = 4Participant 4 solution = 2

Participant 1 solution = 3Participant 2 solution = 5Participant 3 solution = 4

Sum of grades for participant's solution

5+3+3=11 5+4+5=14(This is a winning solution)

2+4+4=10 1+3+2=6

73

Second production cycle - freelancer.com

Professionals are from freelancer.com Time limit for task is 30 minutes Monetary incentive for the task is 10$ Brief explanation of the project aims given to participants No cumulative metadata in task specification

Tasks & questionnaires

Task 1 Participant 1 Participant 2 Participant 3 Participant 4

1. How do you feel? (Possible answers: Very poor, Poor, Average, Good, Excellent )

5, Feeling very confident 5, I feel great and just tired little bit

I feel different about this project, especially about the questionnaire.

5

How many years of experience do you have in your profession?

I am in my third year of this profession. Currently a student studying digital media

2 years of experience I have 2 years of working experi-ence in my profession.

I have 5+years of experience in graphics design field.

Are you familiar with crowd-sourcing?

Yes Yes I am. Gathering of ideas, knowledge or content from the large group of people.

Yes, a little bit.

What is your specialty? I am in my third year of this profession. Currently a student studying digital media

I am easy and calm person and I try to do my work as soon as possible. I have great skills for web.

My specialty in web design and development.

I can create an “Eye Catching Design“.

What grade would you give yourself as a professional? (1-5 scale)

3, I rate myself this as I am aware that I am new to the profession and have a lot of experience to gain.

4 By profession I’ll give myself 3 as professional.

4.5

Main task

Create Combination Mark initial logo draft for the company. The company specializes in production and distribution of chocolate with messages for certain occasions on their package.For example: Thanksgiving chocol-ate, Chocolate for the doctor, Chocolate for the Boss, Chocolate for new family member, Love chocolate, etc. You only have to provide initial logo draft for the company with de-scribed background Target population is North Amer-ican continentType of logo: Combination Mark.Company name: Say ChocoProvide your result in two files: native format and exported jpeg

1. How much time did you spend for logo draft creation?

I spent a total of 28 minutes on this draft logo

20 min Its takes about 25 to 30 minutes to complete the draft.

About 20 minutes.

2. Is given specification for logo draft good enough to provide you with all needed information for the task? Your opinion.

Yes, as soon as I read the specification I had a few ideas that sprung to mind.

No.

There is no specified colour and size of logo and even not there about the company slogan. Etc.

Well provided information is enough to complete the rough task, but for complete logo brand-ing there will be some more point to be clear to make 100% unique and awesome logo.

It’s excellent . Very easy to under-stand.

3. What is your opinion on task time limitation?

The limitation on time restrained me to fully fulfil the logo to its full potential however it’s enough time to get basic ideas and a draft sketch made.

It’s challenging. For creativity, time limit is a huge hurdle, time limit will lower the thinking capability.For branding there should be free time for the designer to produce something great.

It’s normal time for this type of task.

4. Do you rather prefer short tasks or more elaborate ones? Why?

I don’t mind however the elabor-ated tasks I feel you are able to learn a lot more from.

More elaborate ones so that I can provide a quality output for the client.

Yeah I prefer short tasks, because they finish quickly and provide good profit.

Obviously elaborated one’s because I can easily understand the task.

5. Did you complete the task at once or with interruptions?

This task was completed without any interruptions apart from having to go get a coffee

There were no interruptions. It depends on complexity of project and requirement understanding. Mostly my projects completed without interruption.

I completed the task at once but it depends on description.

6. Describe your creative process in 2 sentences

Once I read the brief I instantly had some ideas come to mind

I thought about a design, draw a concept and decided about the

There is no Limit for creativity. Firstly I focused on targeted audience.Used the creativity on given descrip-

74

where I put down in a simple sketch. I then opened illustrator and roughly made up this sketch. The colours and fonts wouldn’t be the same and the end product would look a little more 3D just help make it a little more eye catching.

colour. Then I wrote a company name and chose font colour. Then I used a blur tool.

tion to complete the process.

7. Were you motivated for the task? Why?

I found this task great fun and I think it leaves a lot of opportunity to create a great logo. I also think what the company does is a good business idea and with the right company design and marketing company would be very success-ful

Yes. Because it’s simple. Truly speaking I Bid for this project for some bucks and a review. So these are my motivations for the task.

I was self-motivated to complete the task.

8. Task complexity (1-5) 2 – This was not a complicated task, it was fun and easy to do

1 3.5 1

9. If you would like to add your opinion, a problem regarding the task or anything that you think should be mentioned please provide it here:

- The problem is in deciding about the design and the colour needed for the perfect match.

I don’t think there is any problem regarding task, Experience something different.

-

Task 2 - voting task(voters are 31 unknown workers on microworkers.com who had to choose the best logo)

3 9 8 11 (This is a winning solution and it becomes a specification for the next task.)

Task 3 Participant 5 Participant 6 Participant 2 Participant 4

1. How do you feel? (Possible answers: Very poor, Poor, Average, Good, Excellent )

Good Average Good Excellent

2. How many years of experience do you have in your profession?

2-3 4 2 5

3. Are you familiar with crowd-sourcing?

yes, a little No yes Yes

4. What is your specialty? Frontend design and develop-ment

Webmaster I have experience in graphics design.

I’m multi talented

5. What grade would you give yourself as a professional? (1-5 scale)

2-3 3 4 5

Main Task:

There is initial draft logo provided (attachment). Your task is to refine the logo by adding or changing some design elements.

1. How much time did you spend for logo refinement?

20 min 20 min 30 min 25 min

2. What is your opinion on task time limitation?

I do not like it in general Time limit is fair Time is limited plus all the layers are merged. If there was more time the result would be better.

Perfect time for refinement.

3. Do you rather prefer short tasks or more elaborate ones? Why?

Short tasks, cause it's often fast and easy.

Depends on the time feasibility and the money I’m getting for it.

I prefer more elaborate ones because I can invest more time to achieve adequate quality.

I prefer short tasks because they are easily done in couple of mins.

4. Did you complete the task at once or with interruptions?

Without interruptions. At once. At once.At once due to clear description.

5. Describe your creative process in 2 sentences.

EnthusiasticImaginative

I did blending. I just provided some blending moment because all the layers were merged.

Understand the specification and start working.

6. Were you motivated for the task? Why?

I liked it cause it was simple Kind of as it was comparatively a simpler task than I have done before.

Yes, because the task was challenging and timely taken.

Task complexity

7. Task complexity (1-5 scale) 2 3 3 1

8. What is your opinion about this task?

It was fairly easy and did not require much work

Tasks involving graphics and design are always interesting.

With more time and good budget the quality of refinement would be better.

Simple task.

Task 4 - voting task(Voters are 42 unknown workers on microworkers.com who had to choose the best logo)

9 9 9 15(This is a winning solution)

Task 5 Participant 7 Participant 3 Participant 2 Participant 4

75

1. How do you feel? (Possible answers: Very poor, Poor, Average, Good, Excellent )

Excellent ... Very confident. Good. Excellent

2. How many years of experience do you have in your profession?

2 & half years experience .. Almost 3 years. 2 years. 5 years

3. Are you familiar with crowd-sourcing?

Yeah , but not much. Only familiar.

Yeah, I’m familiar. Yes. Yes

4. What is your specialty? My Specialty is Client Satisfaction through my quality work ... Design ...

Graphics and web development. I am hard working and I always try to make my clients happy.

I’m multi talented

5. What grade would you give yourself as a professional? (1-5 scale)

4 3 4 5

Main Task

There is a logo provided (attach-ment).Your task is to refine the logo by adding or changing some design elements. You can change as many elements you want just keep in mind that this is refinement of existing logo so do not create a completely different logo.

1. How much time did you spend for logo refinement?

35 min 20 min 30 min 10 min

2. What is your opinion on task time limitation?

My Opinion is all that all logos should be designed in 30 to 40 minutes time.

I’m comfortable with time limits. It’s okey but as the given psd was image only in 1 layer.

About refinement ... 30 minutes is perfect time.

3. Do you rather prefer short tasks or more elaborate ones? Why?

Yes Prefer short tasks because of time efficiency..

I prefer short tasks because they finish quickly and take less effort.

I prefer more elaborate tasks. I prefer short task because its take less time.

4. Did you complete the task at once or with interruptions?

I have completed the task at once.

I completed the task at once. I completed it at once. The task was completed without interruptions.

5. Describe your creative process in 2 sentences.

Always Think Positively when approaching any logo and web design.

Think, refine, create. I just changed the font and colour and background.

Try to understand the logic behind the description and start the work accord-ing to it.

6. Were you motivated for the task? Why?

I was motivated for the main task and was not so eager to answer-ing the questions.

I am always motivated to do different tasks, and the client is a good person

Yes but the time limitation bind to do creativity.

Because I love to work with graphics.

7. Task complexity (1-5 scale) 3 1 4 2

8. Can this task be accomplished in less than15 minutes?

No, I don't think this task can be accomplished in less than 15 minutes.

Yeah. Yes Yes

9. Under what circumstances? None. If I do the same task again. If there would be all layer present and the authority to change almost things.

Since original design was created by me I could easily refine the logo.

10. What is your opinion about this task?

My opinion about this task is that it is very unique and powerful especially to me ..

Task is easy, described clearly, everything is just fine. The task was easy but without

layers it is hard to work in PSD. Additionally in logo design we also need a time to think... Since that was flat image, not so many changes can be made and even if it wasn't, there was no need to make bigger changes to original design.

The task very simple. I just refined the meaning of existing logo and applied the change.

Task 6 - voting task(Voters are 30 unknown workers on microworkers.com who had to choose the best logo)

11( winning solution)

82

9

Table 16. JavaScript Coding Case

First production cycle - prototype testing

76

Professionals known to the researcher No time limit No monetary incentive Participants are familiarized with expectations for crowdsourcing tasks All participants are engaged on every task No explicit cumulative metadata

Tasks & questionnaires

Task 1 Participant 1 Participant 2 Participant 3 Participant 4

1. How many years of experience do you have in your profession?

7 8 20 2

2. What is your specialty? Ruby/Java .NET C#, web development Web frontend PHP, HTML, web

3. Are you familiar with crowd-sourcing?

Yes :) Yes No Yes

4. Do you have experience working on tasks within any crowdsourcing platforms?

No No No No

5. What grade would you give yourself as a professional? (1-5 scale)

4, on a motivating job 5. 4 4 4.5

6. Do you use facebook and to what extent?

No Occasionally Extensively Moderately

Main Task:

Create a popup window 200*200px at the centre of the screen.The window opens on page load after 500 milliseconds.Place a 50*50px picture in the upper right corner of the window.Place a textbox of 20px width in upper left corner.Place a close button in the lower centre.

Participant 1 JavaScript code Participant 2 JavaScript code Participant 3 JavaScript code Participant 4 JavaScript code

1. Is given specification good enough to provide you with all needed information for the task? Your opinion.

Everything is clearly specified. Questions that could be asked are probably even not important for the task itself.

Very concise, clear and exact. The Specification vas completely clear. There is one thing to mention though and that is that the pop up windows are not used as often as before because almost all browsers are blocking them automatically and exactly that happened to me when I tried to test my solution.

Seems sufficient.

2. How much money would you ask for the time spent on this task?

Around 10$ with a fun modifier depending on how interesting the task is. Programmers will usually ask a lower price for a piece of work which they like and want to do. If they are good at communic-ating this it could give a good idea about their work and to whom it might be cheaper than it would normally be. Discount season :)

9$ 16$ 10$

3. How much time did you spend for coding?

60 min 60 min 70 min 35 min

Task 2 - voting task

Give a unique grade in a scale from 1 to 5 for every proposed solution (every participant votes for 3 solutions from other participants):

Participant 2 solution = 4Participant 3 solution = 2Participant 4 solution = 1

Participant 1 solution = 5Participant 3 solution = 3Participant 4 solution = 2

Participant 1 solution = 3Participant 2 solution = 4Participant 4 solution = 2

Participant 1 solution = 4Participant 2 solution = 5Participant 3 solution = 3

Sum of grades for participant's solution

5+3+4=12 4+4+5=13(This is a winning solution - becomes initial artefact for refinement in second cycle)

2+3+3=8 1+2+2=5

Second production cycle (JavaScript snippet coding) - freelancer.com

Professionals are from freelancer.com Time limit for task is 30 minutes Monetary incentive for the task is 10$ Brief explanation of the project aims given to participants No cumulative metadata

77

Tasks & questionnaires

Task 1 Participant 1 Participant 2 Participant 3 Participant 4

1. How do you feel? (Possible answers: Very poor, Poor, Aver-age, Good, Excellent )

Good Average Good Excellent

2. How many years of experience do you have in your profession?

2-3 4 2 3

3. Are you familiar with crowd-sourcing?

Yes, a little No Yes A bit.

4. What is your specialty? Java Apps development Web sites and services I’m designer as well as developer

5. What grade would you give yourself as a professional? (1-5 scale)

2-3 3 4 4.5

Main taskThere is a JavaScript snippet provided.Your task is to add this features:The upper left textbox should accepts only numbers.Popup window cannot be closed until a number is entered in the textbox.When clicking on Close without entered number, open an alert window with "You must enter a number" and focus on the textbox.

Participant1 JavaScript code Participant 2 JavaScript code Participant 3 JavaScript code Participant 4 JavaScript code

1. How much time did you spend for coding?

32 min 10 min 20 min 10 min

2. Is given specification good enough to provide you with all needed information for the task? Your opinion.

The information you gave was perfectly informative.

Yes, good enough. Yes. Well Formed.

3. What is your opinion on task time limitation?

I don't really like it in general. As it was a very simple task, 30 mins was more than sufficient.

Yes it was challenging and the task for experts in not more then 20 minutes.

30 minutes is adequate time for this task.

4. Do you rather prefer short tasks or more elaborate ones? Why?

Short tasks, cause it's often fast and easy.

Depends on the time feasibility and the money I’m getting for it.

I will prefer elaborated one. Because there we can do a quality work.

I prefer short task that task not take too much time.

5. Did you complete the task at once or with interruptions?

With interruptions. At once At once The task was completed at once because the description of this project is in very informative and easy to understand.

6. Describe your creative process in 2 sentences

Devise a concept based on specification. Implement a concept. Test. Refine a concept.

Used the textbox key press event and assigned a function to it that limits the input to only numbers. On the close buttons click event checked if the textbox is empty, or textbox still contains letters, if not then the dialog close event is invoked.

Implementing the field validation within existing code.

Understand the description and start developing.

7. Were you motivated for the task? Why?

I liked it cause it was simple. Not much as it was a fairly simple task.

Is was interesting but there were some difficulties.

Because I like programming.

8. Task complexity (1-5) 2 3 2 2

9. What is your opinion about this task?

The task was easy. It’s a simple task, those experi-enced in JavaScript programming should be able to complete this fast.

Actually the task was short but interesting. Few more things could be done on the code to make it perfect.

It is a simple task.

Task 2 - voting task 3 4 2 5(winning solution)

Task 3 Participant 1 Participant 2 Participant 3 Participant 5

1. How do you feel? (Possible answers: Very poor, Poor, Aver-age, Good, Excellent )

Good Average Excellent Excellent

2. How many years of experience do you have in your profession?

2-3 4 2 2

3. Are you familiar with crowd-sourcing?

Yes, a little No Yes Yes

4. What is your specialty? Java Apps development Web sites and services I am very hardworking code lover.

5. What grade would you give yourself as a professional? (1-5 scale)

3 3 4 5

Main task

There is a JavaScript snippet

Participant 1 JavaScript code Participant 2 JavaScript code Participant 3 JavaScript code Participant 5 JavaScript code

78

provided.Your task is to add this features:Close is possible by clicking outside the popup window (also initiates the alert function ).Create an arbitrarily effect (fade in, slide in...choose one) on the picture 1000ms after page load.Do not break existing JavaScript features.

1. How much time did you spend for coding?

15 min 37 minutes 45 min 50 min

2. Is given specification good enough to provide you with all needed information for the task? Your opinion.

Yes, I understood what I had to do .

Good enough. Yes specification was good. yes it was good

3. What is your opinion on task time limitation?

Not that bad. If the task is pretty easy it's ok.

30 minutes was a bit challenging. Yes it’s challenging but as fresher it’s something hard.

One hour should be enough.

4. Do you rather prefer short tasks or more elaborate ones? Why?

Short cause you can easily see the progress. With big tasks it feels more rewarding to complete.

Depends on the time feasibility and the money I’m getting for it.

In the case of java script now I prefer short.

I like short tasks because they are more time efficient.

5. Did you complete the task at once or with interruptions?

At once. At once. At once but very hardly. I completed this task without interruptions.

6. Describe your creative process in 2 sentences

Full of energy, full of good ideas. Defined click event for the outer-most div for close. Used the settimeout method to get the effect.

As a requirement I did an image fade and popup close by clicking outside of the popup.

I tried to find and implement solution for given specification.

7. Were you motivated for the task? Why?

Yes, because it was easy and fun. Yes, this was a bit challenging. To add new feature without touching already available code was very interesting.

Yes I was very motivated for the task. I learned something new.

8. Task complexity (1-5) 2 4 4 1

9. Can this task be completed in 15 minutes?

It can for sure. It’s possible but difficult. Yes, more experienced coder can do it in 10 minutes.

yes

10. What is your opinion about this task?

IT was easy and fast but fun to do. Only those experienced in JavaS-cript programming should be able to complete this fast, rest will need to do some thinking.

It was interesting and had some logic behind it.

It was very interesting.

Task 4 - voting task 5(winning solution)

3 4 2

Task 5 Participant 1 Participant 2 Participant 3 Participant 4

1. How do you feel? (Possible answers: Very poor, Poor, Aver-age, Good, Excellent )

Average Average Good Excellent

2. How many years of experience do you have in your profession?

2-3 4 2 3

3. Are you familiar with crowd-sourcing?

Yes No Yes Yes

4. What is your specialty? Java Apps development Web sites and services I’m designer as well as developer.

5. What grade would you give yourself as a professional? (1-5 scale)

2 3 4 5

Main Task:

There is a JavaScript snippet provided. (attachment)Check JavaScript snippet featuresYour task is to add this features:Create a link on a page so that a user can open a popup without the need to refresh the page.Change the text on the popup button from "close" to "submit".After a number is entered in a popup textbox and submit button clicked - write entered value within the page body together with date and time of submittance.If values are added multiple times, every new value-date pair is added within the page body as a next row.Do not break existing features.

Participant 1 JavaScript code Participant 2 JavaScript code Participant 3 JavaScript code Participant 4 JavaScript code

1. How much time did you spend for coding?

30 min 30 min 45 min 25 min

2. Is given specification good enough to provide you with all needed information for the task? Your opinion.

I got the all the information I needed.

Yes. Good enough. Yes specification was good and logical.

Specification is Well maintained.

3. What is your opinion on task time limitation?

If the task is easy it's ok, but if the task is hard it can be stressing.

30 mins was a bit challenging. Yes it’s challenging. As fresher, when something is hard I just try to find helpful information from the

Enough time.

79

internet.

4. Do you rather prefer short tasks or more elaborate ones? Why?

Short , because you can see the progress fast.

Depends on the time feasibility and the money I’m getting for it.

In the case of JavaScript I think small tasks are ok.

I prefer short task that not take too much time.

5. Did you complete the task at once or with interruptions?

At once. At once. At once. At once.

6. Describe your creative process in 2 sentences

Imaginative and new thinking. Used the same code as docu-ment.ready on click event of the link for displaying the popup. Used JavaScript date functions to get date and time of submission.

As a requirement I created a link and then tried to show date and time. Lastly, I tried not to change existing functional parts of the script.

Understand the description and start working.

7. Were you motivated for the task? Why?

Yes. Because I like to help. Yes, this was a bit challenging. Yes the task was interesting and very logical, and I succeeded in solving it.

Because it was simple.

8. Task complexity (1-5) 3-4 4 4 0.5

9. Can this task be accomplished in 15 minutes?

No. At least I would not be able to. It’s possible. Maybe I am not sure.

10. Under which circumstances? - If you understand the JavaScript code which was already there and are familiar with date functions.

Experienced programmer can do it in 20ish minutes.

-

11. What is your opinion about this task?

It was hard and I got stuck. I got stuck when listing new lines to the history, when a new line was added, all the lines before will change to the text in the newest one.

Only those experienced in JavaS-cript programming should be able to complete this fast, rest will need to do some thinking.

The task was interesting and logical.

The task is so simple just refine and existing style change and present in different style.

Task 6 - voting task 1(disqualified)

4 5(winning solution)

2

Task 7 Participant 2 Participant 3 Participant 4 Participant 5

How do you feel? (Possible answers: Very poor, Poor, Aver-age, Good, Excellent )

Average Good Excellent Excellent

How many years of experience do you have in your profession?

4 2 3 2

Are you familiar with crowd-sourcing?

No Yes Yes Yes

What is your specialty? Apps development Web sites and services I’m designer as well as developer I am very passionate and code loving person.

What grade would you give yourself as a professional? (1-5 scale)

3 4 4 5

Main Task:

There is a JavaScript snippet provided. (attachment)Check JavaScript snippet features.Your task is to add this feature:Calculate the sum and average for all entered numbers and write them under the value date list.Do not break other existing fea-tures.

Participant 2 JavaScript code Participant 3 JavaScript code Participant 4 JavaScript code Participant 5 JavaScript code

1. How much time did you spend for coding?

15 min 50 min 25 min 50 min

2. Is given specification good enough to provide you with all needed information for the task? Your opinion.

Yes good enough. Yes it’s enough. Yes. Yes it was good.

3. What is your opinion on task time limitation?

Quite doable in 30 mins. It’s interesting because task was logical for me.

Perfect time. One hour should be long enough for this task.

4. Do you rather prefer short tasks or more elaborate ones? Why?

Depends on the time feasibility and the money I’m getting for it.

Yes, I prefer short task because it takes less time and you get output asap.

I prefer short tasks. I like short tasks cause they save the time and effort .

5. Did you complete the task at once or with interruptions?

At once. Yes, at once. I completed the task at once. I completed this task without interruption.

6. Describe your creative process in 2 sentences

Used 3 global variables to store the sum, count and average.

I provided the functionality to add the calculation and then provided the actual space to give the output.

Understand and start working. I have taken simple coding for summing and average.

7. Were you motivated for the task? Why?

Fairly. Yes I was motivated because task was logical and I wanted to do it asap. And it’s looking easy.

Yes. The task was simple. Yes I was very motivated for the task cause I love core program-ming.

8. Task complexity (1-5) 2 2 1 1

80

9. Can this task be accomplished in 15 minutes?

It’s possible. yes I don't know yes

10. Under which circumstances? If you’re quite experienced in JavaScript.

If you know all about basic functions like adding and average and only need to implement the knowledge then it could be done in 15 minutes.

- Task is simple so there are no special circumstances.

11. What is your opinion about this task?

Fairly simple task. This is very simple 15 minutes task and it shouldn’t take more than 30 minutes. It was very easy but logical.

It's a simple task. It was very interesting.

Task 8 - voting task 4 3 2 5(wining solution)

APPENDIX B. JAVASCRIPT CODE SNIPPETS

First production cycle

Participant 1

Index.html<html>

<head><title> Proba </title>

</head>

<body></body><script>

function popupCenteredWindow(url, title, w, h) { var left = (screen.width/2)-(w/2); var top = (screen.height/2)-(h/2); return window.open(url, title, 'toolbar=no, location=no, directories=no, status=no, menubar=no, scrollbars=no,

resizable=no, copyhistory=no, width='+w+', height='+h+', top='+top+', left='+left);}

window.onload = function () {

setTimeout(function(){popupCenteredWindow("popup.html", "Popup prozor", 200, 200);

},500);

}

</script></html>

Popup.html<html>

<head><style type="text/css">

#close_btn {position: absolute;left: 40%;top: 80%;

}</style>

</head>

<body><input type="text" style="width:20px"><img src="a50.png" style="float: right"><button id="close_btn" onClick="window.close()">Close</button>

</body>

</html>

81

Participant 2

Index.html<html lang="en"><head><meta charset="utf-8"><title>Popup</title><link rel="stylesheet" href="http://code.jquery.com/ui/1.11.1/themes/smoothness/jquery-ui.css"><script src="http://code.jquery.com/jquery-1.10.2.js"></script><script src="http://code.jquery.com/ui/1.11.1/jquery-ui.js"></script><link rel="stylesheet" href="/resources/demos/style.css"><script>

setTimeout(function () { $(function () { $("#dialog").dialog({ width: 200, height: 200 }); }); }, 500);

</script></head><body>

<div id="dialog" style="display:none"> <table width="100%" height="100%"> <tr> <td align="left" valign="top"><input type="text" style="width:20px" /></td> <td align="right" valign="top"><img src="50x50.jpg" /></td> </tr> <tr> <td colspan="2" align="center" valign="bottom"><input type="button" value="Close" onclick="$('#dialog').dialog('close');" /></td> </tr> </table></div>

</body></html>

Participant 3

Index.html<html lang="en"><body>

<script>var popup;

var div = document.createElement( "div" );div.id = "div";div.style.width = div.style.height = "200px";div.style.position = "relative";div.style.display = "block";div.style.top = div.style.left = "50%";div.style.marginLeft = div.style.marginTop = "-100px";div.style.border = "1px solid #f00;";div.style.backgroundColor = "#f00";div.style.textAlign = "center";

var img = document.createElement( "img" );img.style.width = img.style.height = "50px";img.style.position = "absolute";img.style.right = img.style.top = "0";img.src = "mysti_04_by_gpphotos-d32osqu.jpg";div.appendChild( img );

var txt = document.createElement( "textarea" );txt.style.width = txt.style.height = "20px";txt.style.position = "absolute";txt.style.left = txt.style.top = "0";div.appendChild( txt );

var butt = document.createElement( "input" );butt.type = "button";butt.onclick = function() { popup.close(); }butt.style.position = "relative";butt.style.margin = "0";butt.style.left = "-50%";butt.value = "close";

var buttDiv = document.createElement( "div" );buttDiv.style.position = "absolute";buttDiv.style.bottom = "0";buttDiv.style.left = "50%";buttDiv.appendChild( butt );

82

div.appendChild( buttDiv );

window.onload = function() {

setTimeout(function() {

popup = window.open( '', '', 'width=200,height=200,top=' + (screen.height/2 - 100) + ',left=' + (screen.width/2 - 100) );

popup.document.body.style.overflow = "hidden";popup.document.body.style.margin = "0";popup.document.body.style.padding = "0";popup.document.body.appendChild( div );

},500

);};

</script></body></html>

Participant 4Index.html<!DOCTYPE html><html><head></head>

<script type="text/javascript">function open_on_entrance(url,name){ window.open("a.html",'google', ' menubar,resizable,dependent,status,width=300,height=200,left=500,top=300')}</script><body onload="open_on_entrance()"></body></body></html>

a.html<html><head><style>body { background-color: linen;}.main{height:100%;width:100%;margin:0px;}.image{height:150px;width:5%;float:right;

}.textbox{height:150px;float:left;width:90%;

}.submit{margin-left:50%;width:20px;height:150px;margin-bottom:20px;}</style></head><body><div class="main"><div class="textbox">

<input type="text" style="width:20px;"/></div><div class="image"><img src="IMG_24102014_214428.png"alt="Smiley face" style="width:50px; height:50px;"/>

</div>

</div><div class="submit">

<input type="submit" value="Submit"></div></body></html>

83

Second Production Cycle

Participant 1 Task 1

Index.html<html lang="en"><head><meta charset="utf-8"><title>Popup</title><link rel="stylesheet" href="http://code.jquery.com/ui/1.11.1/themes/smoothness/jquery-ui.css"><script src="http://code.jquery.com/jquery-1.10.2.js"></script><script src="http://code.jquery.com/ui/1.11.1/jquery-ui.js"></script><link rel="stylesheet" href="/resources/demos/style.css"><script>

setTimeout(function () { function isNumeric(){ var numexp = /^[0-9]+$/; if(document.getElementById('text').value.match(numexp) ){ return true; }else{ return false; } }; document.getElementById('button').onclick = function() { if(isNumeric() === false){ alert("You must enter a number"); }else{ $('#dialog').dialog('close'); } } $(function () { $("#dialog").dialog({ width: 200, height: 200 }); }); }, 500);</script></head><body>

<div id="dialog" style="display:none"> <table width="100%" height="100%"> <tr> <td align="left" valign="top"><input type="text" style="width:20px" id="text" /></td> <td align="right" valign="top"><img src="50x50.jpg" /></td> </tr> <tr> <td colspan="2" align="center" valign="bottom"><input type="button" value="Close" id="button" /></td> </tr> </table></div>

</body></html>

Participant 2 Task 1

Index.html<html lang="en"><head><meta charset="utf-8"><title>Popup</title><link rel="stylesheet" href="http://code.jquery.com/ui/1.11.1/themes/smoothness/jquery-ui.css"><script src="http://code.jquery.com/jquery-1.10.2.js"></script><script src="http://code.jquery.com/ui/1.11.1/jquery-ui.js"></script><link rel="stylesheet" href="/resources/demos/style.css"><script type="text/javascript"> setTimeout(function () { $(function () { $("#dialog").dialog({ width: 200, height: 200 }); }); }, 500);

function isNumberKey(evt) { var charCode = (evt.which) ? evt.which : event.keyCode if (charCode > 31 && (charCode < 48 || charCode > 57)) return false;

return true; }

function checkclose() { if (document.getElementById("txtInput").value == "") { document.getElementById("txtInput").value = "enter a number"; document.getElementById("txtInput").focus(); } else if (isNaN(document.getElementById("txtInput").value)) {

84

document.getElementById("txtInput").value = "enter a valid number"; document.getElementById("txtInput").focus(); } else $('#dialog').dialog('close');

}</script></head><body>

<div id="dialog" style="display:none"> <table width="100%" height="100%"> <tr> <td align="left" valign="top"><input id="txtInput" type="text" style="width:20px" onkeypress="return isNumberKey(event)" /></td> <td align="right" valign="top"><img src="50x50.jpg" /></td> </tr> <tr> <td colspan="2" align="center" valign="bottom"><input type="button" value="Close" onclick="return checkclose();" /></td> </tr> </table></div>

</body></html>

Participant 3 Task 1

Index.html<html lang="en"><head><meta charset="utf-8"><title>Popup</title><link rel="stylesheet" href="http://code.jquery.com/ui/1.11.1/themes/smoothness/jquery-ui.css"><script src="http://code.jquery.com/jquery-1.10.2.js"></script><script src="http://code.jquery.com/ui/1.11.1/jquery-ui.js"></script><link rel="stylesheet" href="/resources/demos/style.css"><script>function isNumberKey(evt) { var charCode = (evt.which) ? evt.which : event.keyCode if (charCode > 31 && (charCode < 48 || charCode > 57)) return false;

return true; }

setTimeout(function () { $(function () { $("#dialog").dialog({ width: 200, height: 200 }); }); }, 500);

function Validate() { var txt = document.getElementById("txt"); if (txt.value == "") { txt.value = "Please enter something"; txt.focus(); } else if (isNaN(txt.value)) { txt.value = "Please enter a number"; txt.focus(); } else $('#dialog').dialog('close');

}

</script></head><body>

<div id="dialog" style="display:none"> <table width="100%" height="100%"> <tr> <td align="left" valign="top"><input id="txt" type="text" onkeypress="return isNumberKey(event)" style="width:100px" /></td> <td align="right" valign="top"><img src="50x50.jpg" /></td> </tr> <tr> <td colspan="2" align="center" valign="bottom"><input type="button" value="Close" onclick="return Validate();" /></td> </tr> </table></div>

</body></html>

85

Participant 4 Task 1

Index.html<html lang="en"><head><meta charset="utf-8"><title>Popup</title><link rel="stylesheet" href="http://code.jquery.com/ui/1.11.1/themes/smoothness/jquery-ui.css"><script src="http://code.jquery.com/jquery-1.10.2.js"></script><script src="http://code.jquery.com/ui/1.11.1/jquery-ui.js"></script><script>

</script><link rel="stylesheet" href="/resources/demos/style.css"></head><body>

<div id="dialog" style="display:none"> <table width="100%" height="100%"> <tr> <td align="left" valign="top"><input type="text" style="width:40px" class="onlynum" id="num"/></td> <td align="right" valign="top"><img src="50x50.jpg" /></td> </tr> <tr> <td colspan="4" align="center" valign="bottom"><input type="button" id="close" value="close" onClick=""></td> </tr> </table></div><script>$(document).ready(function(){

setTimeout(function(){ $(function(){ $("#dialog").dialog({ width: 200, height: 200 }); }); }, 500);

$("#dialog").dialog({open: function(event, ui) { $(".ui-dialog-titlebar-close").hide(); },modal: true,width:800

});$(".onlynum").keydown(function (e)

{ // Allow: backspace, delete, tab, escape, enter and . if ($.inArray(e.keyCode, [46, 8, 9, 27, 13, 110, 190]) !== -1 || // Allow: Ctrl+A (e.keyCode == 65 && e.ctrlKey === true) || // Allow: home, end, left, right (e.keyCode >= 35 && e.keyCode <= 39))

{ // let it happen, don't do anything return; } // Ensure that it is a number and stop the keypress if ((e.shiftKey || (e.keyCode < 48 || e.keyCode > 57)) && (e.keyCode < 96 || e.keyCode > 105)) { e.preventDefault(); } });

$("#close").click(function () { var textBox = $.trim( $('#num').val() )

if (textBox == ""){

alert('Please Enter Value');}else{

$('#dialog').dialog('close');}

});

});

</script></body></html>

86

Participant 1 Task 3

Index.html<html lang="en"><head><meta charset="utf-8"><title>Popup</title><link rel="stylesheet" href="http://code.jquery.com/ui/1.11.1/themes/smoothness/jquery-ui.css"><script src="http://code.jquery.com/jquery-1.10.2.js"></script><script src="http://code.jquery.com/ui/1.11.1/jquery-ui.js"></script><script>

</script><link rel="stylesheet" href="/resources/demos/style.css"></head><body>

<div id="dialog" style="display:none"> <table width="100%" height="100%"> <tr> <td align="left" valign="top"><input type="text" style="width:40px" class="onlynum" id="num"/></td> <td align="right" valign="top"><img src="50x50.jpg" /></td> </tr> <tr> <td colspan="4" align="center" valign="bottom"><input type="button" id="close" value="close" onClick=""></td> </tr> </table></div><script>$(document).ready(function(){ $("#dialog").fadeIn(1000);

setTimeout(function(){ $(function(){ $("#dialog").dialog({ width: 200, height: 200 }); }); }, 500);

$("#dialog").dialog({open: function(event, ui) { $(".ui-dialog-titlebar-close").hide(); },modal: true,width:800

});$(".onlynum").keydown(function (e)

{ // Allow: backspace, delete, tab, escape, enter and . if ($.inArray(e.keyCode, [46, 8, 9, 27, 13, 110, 190]) !== -1 || // Allow: Ctrl+A (e.keyCode == 65 && e.ctrlKey === true) || // Allow: home, end, left, right (e.keyCode >= 35 && e.keyCode <= 39))

{ // let it happen, don't do anything return; } // Ensure that it is a number and stop the keypress if ((e.shiftKey || (e.keyCode < 48 || e.keyCode > 57)) && (e.keyCode < 96 || e.keyCode > 105)) { e.preventDefault(); } });

$("#close").click(function () { var textBox = $.trim( $('#num').val() )

if (textBox == ""){

alert('Please Enter Value');}else{

$('#dialog').dialog('close');}

}); $("html").click(function () {

var textBox = $.trim( $('#num').val() )if (textBox == ""){

alert('Please Enter Value');}else{

$('#dialog').dialog('close');}

}); $('#dialog').click(function(event){ event.stopPropagation();});});

87

</script></body></html>

Participant 2 Task 3

Index.html<html lang="en"><head><meta charset="utf-8"><title>Popup</title><link rel="stylesheet" href="http://code.jquery.com/ui/1.11.1/themes/smoothness/jquery-ui.css"><script src="http://code.jquery.com/jquery-1.10.2.js"></script><script src="http://code.jquery.com/ui/1.11.1/jquery-ui.js"></script><script>

</script><link rel="stylesheet" href="/resources/demos/style.css"></head><body>

<div id="dialog" style="display:none"> <table width="100%" height="100%"> <tr> <td align="left" valign="top"><input type="text" style="width:40px" class="onlynum" id="num"/></td> <td align="right" valign="top"><img src="50x50.jpg" style="display:none" /></td> </tr> <tr> <td colspan="4" align="center" valign="bottom"><input type="button" id="close" value="close" onClick=""></td> </tr> </table></div><script> $(document).ready(function () { setTimeout(function () { $(function () { $("#dialog").dialog({ width: 200, height: 200 }); }); }, 500); $("#dialog").dialog({ open: function (event, ui) { $(".ui-dialog-titlebar-close").hide(); }, modal: true, width: 800 }); $(".onlynum").keydown(function (e) { // Allow: backspace, delete, tab, escape, enter and . if ($.inArray(e.keyCode, [46, 8, 9, 27, 13, 110, 190]) !== -1 || // Allow: Ctrl+A (e.keyCode == 65 && e.ctrlKey === true) || // Allow: home, end, left, right (e.keyCode >= 35 && e.keyCode <= 39)) { // let it happen, don't do anything return; } // Ensure that it is a number and stop the keypress if ((e.shiftKey || (e.keyCode < 48 || e.keyCode > 57)) && (e.keyCode < 96 || e.keyCode > 105)) { e.preventDefault(); } });

$("#close").click(function () { var textBox = $.trim($('#num').val()) if (textBox == "") { alert('Please Enter Value'); } else { $('#dialog').dialog('close'); } });

$('.ui-widget-overlay').click(function (e) { var textBox = $.trim($('#num').val()) if (textBox == "") { alert('Please Enter Value'); } else { $('#dialog').dialog('close'); }

});

setTimeout("$('img').fadeIn(2000)", 1000); });</script></body></html>

88

Participant 3 Task 3

Index.html<html lang="en"><head><meta charset="utf-8"><title>Popup</title><link rel="stylesheet" href="http://code.jquery.com/ui/1.11.1/themes/smoothness/jquery-ui.css"><script src="http://code.jquery.com/jquery-1.10.2.js"></script><script src="http://code.jquery.com/ui/1.11.1/jquery-ui.js"></script><script>

</script><link rel="stylesheet" href="/resources/demos/style.css"></head><body>

<div id="dialog" style="display:none"> <table width="100%" height="100%"> <tr> <td align="left" valign="top"><input type="text" style="width:40px" class="onlynum" id="num"/></td> <td align="right" valign="top"><img src="50x50.jpg" style="display:none" /></td> </tr> <tr> <td colspan="4" align="center" valign="bottom"><input type="button" id="close" value="close" onClick=""></td> </tr> </table></div><script> $(document).ready(function () { setTimeout(function () { $(function () { $("#dialog").dialog({ width: 200, height: 200 }); }); }, 500); $("#dialog").dialog({ open: function (event, ui) { $(".ui-dialog-titlebar-close").hide(); }, modal: true, width: 800 }); $(".onlynum").keydown(function (e) { // Allow: backspace, delete, tab, escape, enter and . if ($.inArray(e.keyCode, [46, 8, 9, 27, 13, 110, 190]) !== -1 || // Allow: Ctrl+A (e.keyCode == 65 && e.ctrlKey === true) || // Allow: home, end, left, right (e.keyCode >= 35 && e.keyCode <= 39)) { // let it happen, don't do anything return; } // Ensure that it is a number and stop the keypress if ((e.shiftKey || (e.keyCode < 48 || e.keyCode > 57)) && (e.keyCode < 96 || e.keyCode > 105)) { e.preventDefault(); } }); $("#close").click(function () { var textBox = $.trim($('#num').val()) if (textBox == "") { alert('Please Enter Value'); } else { $('#dialog').dialog('close'); } });

$('.ui-widget-overlay').click(function (e) { // var position = 'X: ' + (e.layerX || e.offsetX) + ' Y: ' + (e.layerY || e.offsetY);

// var x = e.layerX || e.offsetX; // var y = e.layerY || e.offsetY;

// var width = $(".ui-dialog").width; // var height = $(".ui-dialog").height;

// if ((x >= $(".ui-dialog").position().left && x <= $(".ui-dialog").position().left + $(".ui-dialog").width()) && (y >= $(".ui-dialog").position().top && y <= $(".ui-dialog").position().top + $(".ui-dialog").height())) {

// } // else { // alert('Click position within image: ' + position); // }

var textBox = $.trim($('#num').val()) if (textBox == "") { alert('Please Enter Value'); } else { $('#dialog').dialog('close'); }

89

});

setTimeout("show()", 1000);

});

function show() { $('img').fadeIn('slow'); }

</script>

</body></html>

Participant 5 Task 3

Index.html<html lang="en"><head><meta charset="utf-8"><title>Popup</title><link rel="stylesheet" href="http://code.jquery.com/ui/1.11.1/themes/smoothness/jquery-ui.css"><script src="http://code.jquery.com/jquery-1.10.2.js"></script><script src="http://code.jquery.com/ui/1.11.1/jquery-ui.js"></script><script>

</script><link rel="stylesheet" href="/resources/demos/style.css"></head><body>

<div id="dialog" style="display:none"> <table width="100%" height="100%"> <tr> <td align="left" valign="top"><input type="text" style="width:40px" class="onlynum" id="num"/></td> <td align="right" valign="top"><img id="image" src="50x50.jpg" style="display:none" /></td> </tr> <tr> <td colspan="4" align="center" valign="bottom"><input type="button" id="close" value="close" onClick=""></td> </tr> </table></div><script> $(document).ready(function () {

setTimeout("$('#image').slideDown('slow')", 1000);

setTimeout(function () { $(function () { $("#dialog").dialog({ width: 200, height: 200 }); }); }, 500); $("#dialog").dialog({ open: function (event, ui) { $(".ui-dialog-titlebar-close").hide(); }, modal: true, width: 800 }); $(".onlynum").keydown(function (e) { // Allow: backspace, delete, tab, escape, enter and . if ($.inArray(e.keyCode, [46, 8, 9, 27, 13, 110, 190]) !== -1 || // Allow: Ctrl+A (e.keyCode == 65 && e.ctrlKey === true) || // Allow: home, end, left, right (e.keyCode >= 35 && e.keyCode <= 39)) { // let it happen, don't do anything return; } // Ensure that it is a number and stop the keypress if ((e.shiftKey || (e.keyCode < 48 || e.keyCode > 57)) && (e.keyCode < 96 || e.keyCode > 105)) { e.preventDefault(); } });

$("#close").click(function () { var textBox = $.trim($('#num').val()) if (textBox == "") { alert('Please Enter Value'); } else { $('#dialog').dialog('close'); } });

$('.ui-widget-overlay').click(function() { var textBox = $.trim($('#num').val()) if (textBox == "") { alert('Please Enter Value');

90

} else { $('#dialog').dialog('close'); }

}); });</script></body></html>

Participant 1 Task 5

Index.html<html lang="en"><head><meta charset="utf-8"><title>Popup</title><link rel="stylesheet" href="http://code.jquery.com/ui/1.11.1/themes/smoothness/jquery-ui.css"><script src="http://code.jquery.com/jquery-1.10.2.js"></script><script src="http://code.jquery.com/ui/1.11.1/jquery-ui.js"></script><link rel="stylesheet" href="/resources/demos/style.css"></head><body> <div> <a href="#" id="popupLink">add</p> </div>

<ul id= "history">

</ul><div id="dialog" style="display:none"> <table width="100%" height="100%"> <tr> <td align="left" valign="top"><input type="text" style="width:40px" class="onlynum" id="num"/> <td align="right" valign="top"><img src="50x50.jpg" /></td> </tr> <tr> <td colspan="4" align="center" valign="bottom"><input type="button" id="close" value="Submit" onClick=""></td> </tr> </table></div><script>$(document).ready(function(){

var d = new Date(); var day= d.getDate(); var month = d.getMonth(); var year = d.getYear(); var sec = d.getSeconds(); var min = d.getMinutes(); var hour = d.getHours(); var test = true; $("#popupLink").click(function(){ $("#dialog").fadeIn(1000);

setTimeout(function(){ $(function(){ $("#dialog").dialog({ width: 200, height: 200 }); }); }, 500);

$("#dialog").dialog({open: function(event, ui) { $(".ui-dialog-titlebar-close").hide(); },modal: true,width:800

});$(".onlynum").keydown(function (e)

{ // Allow: backspace, delete, tab, escape, enter and . if ($.inArray(e.keyCode, [46, 8, 9, 27, 13, 110, 190]) !== -1 || // Allow: Ctrl+A (e.keyCode == 65 && e.ctrlKey === true) || // Allow: home, end, left, right (e.keyCode >= 35 && e.keyCode <= 39))

{ // let it happen, don't do anything return; } // Ensure that it is a number and stop the keypress if ((e.shiftKey || (e.keyCode < 48 || e.keyCode > 57)) && (e.keyCode < 96 || e.keyCode > 105)) { e.preventDefault(); } });

var para = document.createElement("p"); var oldNode = node; var node = document.createTextNode("error");

$("#close").click(function () { var textBox = $.trim( $('#num').val() )

91

if (textBox == ""){

alert('Please Enter Value');}else{

node.nodeValue=textBox + ". " +"date: " + day +"/"+month+"/"+year+" time:" + hour + ":" + min + ":" + sec; para.appendChild(node);

$('#dialog').dialog('close');}

}); $("html").click(function () {

var textBox = $.trim( $('#num').val() )if (textBox == ""){

alert('Please Enter Value');}else{

$('#dialog').dialog('close');}

}); $('#dialog').click(function(event){ event.stopPropagation(); }); var element = document.getElementById("history");

element.appendChild(para); });});

</script></body></html>

Participant 2 Task 5

Index.html<html lang="en"><head><meta charset="utf-8"><title>Popup</title><link rel="stylesheet" href="http://code.jquery.com/ui/1.11.1/themes/smoothness/jquery-ui.css"><script src="http://code.jquery.com/jquery-1.10.2.js"></script><script src="http://code.jquery.com/ui/1.11.1/jquery-ui.js"></script><script>

</script><link rel="stylesheet" href="/resources/demos/style.css"></head><body><div id="link" style="display:none"><a href="#" onclick="ShowDialog();">Open</a><div id="values">

</div></div>

<div id="dialog" style="display:none"> <table width="100%" height="100%"> <tr> <td align="left" valign="top"><input type="text" style="width:40px" class="onlynum" id="num"/></td> <td align="right" valign="top"><img src="50x50.jpg" /></td> </tr> <tr> <td colspan="4" align="center" valign="bottom"><input type="button" id="close" value="submit" onClick=""></td> </tr> </table></div><script> $(document).ready(function () { $("#dialog").fadeIn(1000); setTimeout(function () { $(function () { $("#dialog").dialog({ width: 200, height: 200 }); }); }, 500); $("#dialog").dialog({ open: function (event, ui) { $(".ui-dialog-titlebar-close").hide(); },

92

modal: true, width: 800 }); $(".onlynum").keydown(function (e) { // Allow: backspace, delete, tab, escape, enter and . if ($.inArray(e.keyCode, [46, 8, 9, 27, 13, 110, 190]) !== -1 || // Allow: Ctrl+A (e.keyCode == 65 && e.ctrlKey === true) || // Allow: home, end, left, right (e.keyCode >= 35 && e.keyCode <= 39)) { // let it happen, don't do anything return; } // Ensure that it is a number and stop the keypress if ((e.shiftKey || (e.keyCode < 48 || e.keyCode > 57)) && (e.keyCode < 96 || e.keyCode > 105)) { e.preventDefault(); } });

$("#close").click(function () { var textBox = $.trim($('#num').val()) if (textBox == "") { alert('Please Enter Value'); } else { $('#dialog').dialog('close'); SetValues(); } }); $("html").click(function () {

var textBox = $.trim($('#num').val()) if (textBox == "") { alert('Please Enter Value'); } else { $('#dialog').dialog('close'); } }); $('#dialog').click(function (event) { event.stopPropagation();

}); });

function ShowDialog() { setTimeout(function () { $(function () { $("#dialog").dialog({ width: 200, height: 200 }); $('#link').toggle(); }); }, 500);}

function SetValues() { var date = new Date();

$('#values').html($('#values').html() + "<br/>" + $('#num').val() + " " + date.getMonth() + "/" + date.getDate() + "/" + date.getFullYear() + " " + date.-getHours() + ":" + date.getMinutes() + ":" + date.getSeconds()); $('#link').toggle(); }

</script></body></html>

Participant 3 Task 5

Index.html<html lang="en"><head><meta charset="utf-8"><title>Popup</title><link rel="stylesheet" href="http://code.jquery.com/ui/1.11.1/themes/smoothness/jquery-ui.css"><script src="http://code.jquery.com/jquery-1.10.2.js"></script><script src="http://code.jquery.com/ui/1.11.1/jquery-ui.js"></script><script>

</script><link rel="stylesheet" href="/resources/demos/style.css"><style>#open{display:none}</style></head><body><div id="open"><a href="#" onclick="OpenPopUp();">Open</a><div id="display"></div></div>

93

<div id="dialog" style="display:none"> <table width="100%" height="100%"> <tr> <td align="left" valign="top"><input type="text" style="width:40px" class="onlynum" id="num"/></td> <td align="right" valign="top"><img src="50x50.jpg" /></td> </tr> <tr> <td colspan="4" align="center" valign="bottom"><input type="button" id="close" value="submit" onClick=""></td> </tr> </table></div><script> $(document).ready(function () { $("#dialog").fadeIn(1000); setTimeout(function () { $(function () { $("#dialog").dialog({ width: 200, height: 200 }); }); }, 500); $("#dialog").dialog({ open: function (event, ui) { $(".ui-dialog-titlebar-close").hide(); }, modal: true, width: 800 }); $(".onlynum").keydown(function (e) { // Allow: backspace, delete, tab, escape, enter and . if ($.inArray(e.keyCode, [46, 8, 9, 27, 13, 110, 190]) !== -1 || // Allow: Ctrl+A (e.keyCode == 65 && e.ctrlKey === true) || // Allow: home, end, left, right (e.keyCode >= 35 && e.keyCode <= 39)) { // let it happen, don't do anything return; } // Ensure that it is a number and stop the keypress if ((e.shiftKey || (e.keyCode < 48 || e.keyCode > 57)) && (e.keyCode < 96 || e.keyCode > 105)) { e.preventDefault(); } });

$("#close").click(function () { var textBox = $.trim($('#num').val()) if (textBox == "") { alert('Please Enter Value'); } else { $('#dialog').dialog('close'); ShowValue(); $('#open').toggle(); } }); $("html").click(function () {

var textBox = $.trim($('#num').val()) if (textBox == "") { alert('Please Enter Value'); } else { $('#dialog').dialog('close'); } }); $('#dialog').click(function (event) { event.stopPropagation();

}); });

function makeArray() { for (i = 0; i < makeArray.arguments.length; i++) this[i + 1] = makeArray.arguments[i]; }

function ShowValue() { var months = new makeArray('January', 'February', 'March', 'April', 'May','June', 'July', 'August', 'September', 'October', 'November', 'December'); var date = new Date(); var day = date.getDate(); var month = date.getMonth() + 1; var yy = date.getYear(); var year = (yy < 1000) ? yy + 1900 : yy; var display = $('#num').val() + " " + month + "/" + day + "/" + year + " " + date.getHours() + ":" + date.getMinutes() + ":" + date.getSeconds(); $('#display').html($('#display').html() + "<br/>" + display); }

function OpenPopUp() { setTimeout(function () { $(function () { $("#dialog").dialog({ width: 200, height: 200 }); $('#open').toggle(); $('#num').val(""); }); }, 500);}

94

</script></body></html>

Participant 4 Task 5

Index.html<html lang="en"><head><meta charset="utf-8"><title>Popup</title><link rel="stylesheet" href="http://code.jquery.com/ui/1.11.1/themes/smoothness/jquery-ui.css"><script src="http://code.jquery.com/jquery-1.10.2.js"></script><script src="http://code.jquery.com/ui/1.11.1/jquery-ui.js"></script><script>

</script><link rel="stylesheet" href="/resources/demos/style.css"></head><body>

<a href="#" id="showModal"> Show Modal</a>

<div id="text"></div>

<div id="dialog" style="display:none"> <table width="100%" height="100%"> <tr>

<td align="left" valign="top"><input type="text" style="width:40px" class="onlynum" id="num"/></td> <td align="right" valign="top"><img src="50x50.jpg" /></td> </tr> <tr> <td colspan="4" align="center" valign="bottom"><input type="button" id="close" value="Submit" onClick=""></td> </tr> </table></div><script>$(document).ready(function(){

$('div#dialog').dialog({ autoOpen: false })$('#showModal').click(function(){

$("#dialog").dialog({open: function(event, ui) { $(".ui-dialog-titlebar-close").hide(); },modal: true,

});$('div#dialog').dialog('open');

$("#dialog").fadeIn(1000);setTimeout(function(){

$(function(){$("#dialog").dialog({

width: 200,height: 200

});});

}, 500);});

$(".onlynum").keydown(function (e){

if ($.inArray(e.keyCode, [46, 8, 9, 27, 13, 110, 190]) !== -1 ||(e.keyCode == 65 && e.ctrlKey === true) ||(e.keyCode >= 35 && e.keyCode <= 39)){

return;}

if ((e.shiftKey || (e.keyCode < 48 || e.keyCode > 57)) && (e.keyCode < 96 || e.keyCode > 105)) {e.preventDefault();

}});$("#close").click(function () {

var textBox = $.trim( $('#num').val() ) var textVal = $('#num').val(); var dNow = new Date(); var localdate= (dNow.getMonth()+1) + '/' + dNow.getDate() + '/' + dNow.getFullYear() + ' ' +

dNow.getHours() + ':' + dNow.getMinutes();if (textBox == ""){

alert('Please Enter Value');}else{

$('#text').append('<p> '+ textVal +' '+ localdate +'</p>');$('#dialog').dialog('close');

} });

/* $("html").click(function () {

var textBox = $.trim( $('#num').val() )if (textBox == ""){

95

alert('Please Enter Value');}else{

$('#dialog').dialog('close');}

});*/

$('#dialog').click(function(event){event.stopPropagation();

});});

</script></body></html>

Participant 2 Task 7

Index.html<html lang="en"><head><meta charset="utf-8"><title>Popup</title><link rel="stylesheet" href="http://code.jquery.com/ui/1.11.1/themes/smoothness/jquery-ui.css"><script src="http://code.jquery.com/jquery-1.10.2.js"></script><script src="http://code.jquery.com/ui/1.11.1/jquery-ui.js"></script><script> var sum = 0; var avg = 0; var count = 0;</script><link rel="stylesheet" href="/resources/demos/style.css"><style>#open{display:none}</style></head><body><div id="open"><a href="#" onclick="OpenPopUp();">Open</a><div id="display"></div><div id="total"></div></div>

<div id="dialog" style="display:none"> <table width="100%" height="100%"> <tr> <td align="left" valign="top"><input type="text" style="width:40px" class="onlynum" id="num"/></td> <td align="right" valign="top"><img src="50x50.jpg" /></td> </tr> <tr> <td colspan="4" align="center" valign="bottom"><input type="button" id="close" value="submit" onClick=""></td> </tr> </table></div><script> $(document).ready(function () { $("#dialog").fadeIn(1000); setTimeout(function () { $(function () { $("#dialog").dialog({ width: 200, height: 200 }); }); }, 500); $("#dialog").dialog({ open: function (event, ui) { $(".ui-dialog-titlebar-close").hide(); }, modal: true, width: 800 }); $(".onlynum").keydown(function (e) { // Allow: backspace, delete, tab, escape, enter and . if ($.inArray(e.keyCode, [46, 8, 9, 27, 13, 110, 190]) !== -1 || // Allow: Ctrl+A (e.keyCode == 65 && e.ctrlKey === true) || // Allow: home, end, left, right (e.keyCode >= 35 && e.keyCode <= 39)) { // let it happen, don't do anything return; } // Ensure that it is a number and stop the keypress if ((e.shiftKey || (e.keyCode < 48 || e.keyCode > 57)) && (e.keyCode < 96 || e.keyCode > 105)) { e.preventDefault(); }

96

});

$("#close").click(function () { var textBox = $.trim($('#num').val()) if (textBox == "") { alert('Please Enter Value'); } else { $('#dialog').dialog('close'); ShowValue(); $('#open').toggle(); } }); $("html").click(function () {

var textBox = $.trim($('#num').val()) if (textBox == "") { alert('Please Enter Value'); } else { $('#dialog').dialog('close'); } }); $('#dialog').click(function (event) { event.stopPropagation();

}); });

function makeArray() { for (i = 0; i < makeArray.arguments.length; i++) this[i + 1] = makeArray.arguments[i]; }

function ShowValue() { var months = new makeArray('January', 'February', 'March', 'April', 'May','June', 'July', 'August', 'September', 'October', 'November', 'December'); var date = new Date(); var day = date.getDate(); var month = date.getMonth() + 1; var yy = date.getYear(); var year = (yy < 1000) ? yy + 1900 : yy; var display = $('#num').val() + " " + month + "/" + day + "/" + year + " " + date.getHours() + ":" + date.getMinutes() + ":" + date.getSeconds(); sum += Number($('#num').val()); count += 1; avg = sum / count; $('#display').html($('#display').html() + "<br/>" + display); $('#total').html("<hr/>sum="+ sum + " avg=" + avg); }

function OpenPopUp() { setTimeout(function () { $(function () { $("#dialog").dialog({ width: 200, height: 200 }); $('#open').toggle(); $('#num').val(""); }); }, 500);}</script></body></html>

Participant 3 Task 7

Index.html<html lang="en"><head><meta charset="utf-8"><title>Popup</title><link rel="stylesheet" href="http://code.jquery.com/ui/1.11.1/themes/smoothness/jquery-ui.css"><script src="http://code.jquery.com/jquery-1.10.2.js"></script><script src="http://code.jquery.com/ui/1.11.1/jquery-ui.js"></script><script> var sum = 0; var count = 0; var average = 0;</script><link rel="stylesheet" href="/resources/demos/style.css"><style>#open{display:none}</style></head><body><div id="open"><a href="#" onclick="OpenPopUp();">Open</a><div id="display"></div><div id="summary">sum=<span id='sum'></span>average=<span id='average'></span></div></div>

<div id="dialog" style="display:none">

97

<table width="100%" height="100%"> <tr> <td align="left" valign="top"><input type="text" style="width:40px" class="onlynum" id="num"/></td> <td align="right" valign="top"><img src="50x50.jpg" /></td> </tr> <tr> <td colspan="4" align="center" valign="bottom"><input type="button" id="close" value="submit" onClick=""></td> </tr> </table></div><script> $(document).ready(function () { $("#dialog").fadeIn(1000); setTimeout(function () { $(function () { $("#dialog").dialog({ width: 200, height: 200 }); }); }, 500); $("#dialog").dialog({ open: function (event, ui) { $(".ui-dialog-titlebar-close").hide(); }, modal: true, width: 800 }); $(".onlynum").keydown(function (e) { // Allow: backspace, delete, tab, escape, enter and . if ($.inArray(e.keyCode, [46, 8, 9, 27, 13, 110, 190]) !== -1 || // Allow: Ctrl+A (e.keyCode == 65 && e.ctrlKey === true) || // Allow: home, end, left, right (e.keyCode >= 35 && e.keyCode <= 39)) { // let it happen, don't do anything return; } // Ensure that it is a number and stop the keypress if ((e.shiftKey || (e.keyCode < 48 || e.keyCode > 57)) && (e.keyCode < 96 || e.keyCode > 105)) { e.preventDefault(); } });

$("#close").click(function () { var textBox = $.trim($('#num').val()) if (textBox == "") { alert('Please Enter Value'); } else { $('#dialog').dialog('close'); ShowValue(); $('#open').toggle(); } }); $("html").click(function () {

var textBox = $.trim($('#num').val()) if (textBox == "") { alert('Please Enter Value'); } else { $('#dialog').dialog('close'); } }); $('#dialog').click(function (event) { event.stopPropagation();

}); });

function makeArray() { for (i = 0; i < makeArray.arguments.length; i++) this[i + 1] = makeArray.arguments[i]; }

function ShowValue() { var months = new makeArray('January', 'February', 'March', 'April', 'May','June', 'July', 'August', 'September', 'October', 'November', 'December'); var date = new Date(); var day = date.getDate(); var month = date.getMonth() + 1; var yy = date.getYear(); var year = (yy < 1000) ? yy + 1900 : yy; var display = $('#num').val() + " " + month + "/" + day + "/" + year + " " + date.getHours() + ":" + date.getMinutes() + ":" + date.getSeconds(); sum += Number($('#num').val()); count += 1; average = sum / count; $('#display').html($('#display').html() + "<br/>" + display); $('#sum').html(sum); $('#average').html(average); }

function OpenPopUp() { setTimeout(function () { $(function () { $("#dialog").dialog({ width: 200, height: 200 }); $('#open').toggle(); $('#num').val("");

98

}); }, 500);}</script></body></html>

Participant 4 Task 7

Index.html<html lang="en"><head><meta charset="utf-8"><title>Popup</title><link rel="stylesheet" href="http://code.jquery.com/ui/1.11.1/themes/smoothness/jquery-ui.css"><script src="http://code.jquery.com/jquery-1.10.2.js"></script><script src="http://code.jquery.com/ui/1.11.1/jquery-ui.js"></script><script>

</script><link rel="stylesheet" href="/resources/demos/style.css"><style>#open{display:none}</style></head><body><div id="open"><a href="#" onClick="OpenPopUp();">Open</a><div id="display"></div></div>

<div id="dialog" style="display:none"> <table width="100%" height="100%"> <tr> <td align="left" valign="top"><input type="text" style="width:40px" class="onlynum" id="num"/></td> <td align="right" valign="top"><img src="50x50.jpg" /></td> </tr> <tr> <td colspan="4" align="center" valign="bottom"><input type="button" id="close" value="submit" onClick=""></td> </tr> </table></div><script> $(document).ready(function () { $("#dialog").fadeIn(1000); setTimeout(function () { $(function () { $("#dialog").dialog({ width: 200, height: 200 }); }); }, 500); $("#dialog").dialog({ open: function (event, ui) { $(".ui-dialog-titlebar-close").hide(); }, modal: true, width: 800 }); $(".onlynum").keydown(function (e) { // Allow: backspace, delete, tab, escape, enter and . if ($.inArray(e.keyCode, [46, 8, 9, 27, 13, 110, 190]) !== -1 || // Allow: Ctrl+A (e.keyCode == 65 && e.ctrlKey === true) || // Allow: home, end, left, right (e.keyCode >= 35 && e.keyCode <= 39)) { // let it happen, don't do anything return; } // Ensure that it is a number and stop the keypress if ((e.shiftKey || (e.keyCode < 48 || e.keyCode > 57)) && (e.keyCode < 96 || e.keyCode > 105)) { e.preventDefault(); } });

$("#close").click(function () { var textBox = $.trim($('#num').val()) if (textBox == "") { alert('Please Enter Value'); } else { $('#dialog').dialog('close'); ShowValue(); $('#open').toggle(); } }); $("html").click(function () {

var textBox = $.trim($('#num').val()) if (textBox == "") { alert('Please Enter Value'); } else { $('#dialog').dialog('close'); } }); $('#dialog').click(function (event) {

99

event.stopPropagation(); }); });

function makeArray() { for (i = 0; i < makeArray.arguments.length; i++) this[i + 1] = makeArray.arguments[i];

} function ShowValue() { var months = new makeArray('January', 'February', 'March', 'April', 'May','June', 'July', 'August', 'September', 'October', 'November', 'December'); var date = new Date(); var day = date.getDate(); var month = date.getMonth() + 1; var yy = date.getYear(); var year = (yy < 1000) ? yy + 1900 : yy; var display = "<div class='sum'>"+$('#num').val()+"</div>" + " " + month + "/" + day + "/" + year + " " + date.getHours() + ":" + date.getMinutes() + ":" + date.getSeconds() + "<div class='totalsum'></div>" + "<div class='totalavg'></div>"; $('#display').html($('#display').html() + "<br/>" + display);

var sum = 0;var num = 0;$('.sum').each(function(){

sum += parseFloat($(this).text()); //Or this.innerHTML, this.innerTextnum += 1;

});$('.totalsum').last().html(" Sum :" + sum);$('.totalavg').last().html(" Avg :" + sum/num);

}function OpenPopUp() { setTimeout(function () { $(function () { $("#dialog").dialog({ width: 200, height: 200 }); $('#open').toggle(); $('#num').val(""); }); }, 500);}</script></body></html>

Participant 5 Task 7

Index.html<html lang="en"><head><meta charset="utf-8"><title>Popup</title><link rel="stylesheet" href="http://code.jquery.com/ui/1.11.1/themes/smoothness/jquery-ui.css"><script src="http://code.jquery.com/jquery-1.10.2.js"></script><script src="http://code.jquery.com/ui/1.11.1/jquery-ui.js"></script><script type="text/javascript"> var sum = 0; var cnt = 0; var avg = 0;</script><link rel="stylesheet" href="/resources/demos/style.css"><style>#open{display:none}</style></head><body><div id="open"><a href="#" onclick="OpenPopUp();">Open</a><div id="display"></div><div></div></div>

<div id="dialog" style="display:none"> <table width="100%" height="100%"> <tr> <td align="left" valign="top"><input type="text" style="width:40px" class="onlynum" id="num"/></td> <td align="right" valign="top"><img src="50x50.jpg" /></td> </tr> <tr> <td colspan="4" align="center" valign="bottom"><input type="button" id="close" value="submit" onClick=""></td> </tr> </table></div><script> $(document).ready(function () { $("#dialog").fadeIn(1000); setTimeout(function () { $(function () { $("#dialog").dialog({ width: 200, height: 200 }); }); }, 500); $("#dialog").dialog({ open: function (event, ui) { $(".ui-dialog-titlebar-close").hide(); }, modal: true, width: 800 });

100

$(".onlynum").keydown(function (e) { // Allow: backspace, delete, tab, escape, enter and . if ($.inArray(e.keyCode, [46, 8, 9, 27, 13, 110, 190]) !== -1 || // Allow: Ctrl+A (e.keyCode == 65 && e.ctrlKey === true) || // Allow: home, end, left, right (e.keyCode >= 35 && e.keyCode <= 39)) { // let it happen, don't do anything return; } // Ensure that it is a number and stop the keypress if ((e.shiftKey || (e.keyCode < 48 || e.keyCode > 57)) && (e.keyCode < 96 || e.keyCode > 105)) { e.preventDefault(); } });

$("#close").click(function () { var textBox = $.trim($('#num').val()) if (textBox == "") { alert('Please Enter Value'); } else { $('#dialog').dialog('close'); ShowValue(); $('#open').toggle(); } }); $("html").click(function () {

var textBox = $.trim($('#num').val()) if (textBox == "") { alert('Please Enter Value'); } else { $('#dialog').dialog('close'); } }); $('#dialog').click(function (event) { event.stopPropagation();

}); });

function makeArray() { for (i = 0; i < makeArray.arguments.length; i++) this[i + 1] = makeArray.arguments[i]; }

function ShowValue() { var months = new makeArray('January', 'February', 'March', 'April', 'May','June', 'July', 'August', 'September', 'October', 'November', 'December'); var date = new Date(); var day = date.getDate(); var month = date.getMonth() + 1; var yy = date.getYear(); var year = (yy < 1000) ? yy + 1900 : yy; var display = $('#num').val() + " " + month + "/" + day + "/" + year + " " + date.getHours() + ":" + date.getMinutes() + ":" + date.getSeconds(); $('#display').html($('#display').html() + "<br/>" + display); Calculate(); }

function OpenPopUp() { setTimeout(function () { $(function () { $("#dialog").dialog({ width: 200, height: 200 }); $('#open').toggle(); $('#num').val(""); }); }, 500);}

function Calculate() { cnt += 1; sum += Number($('#num').val()); avg = sum / cnt; $('#open div:last-child').html("<b>Sum</b> - " + sum + "<br/> <b>Average</b> - " + avg);}</script></body></html>

101

APPENDIX C. VOTING QUESTIONNAIRE EXAMPLE

Choose the best logo for the company!

A start-up company requires a new logo. The company specializes in production and distribution of chocolate with messages for certain occasions on their package. For ex-ample: Thanksgiving chocolate, Chocolate for the doctor, Chocolate for the Boss, Chocol-ate for the new family member, Love chocolate, etc.

Please choose the best logo for the company.

Logo1

Logo2

Logo3

Logo4

Which logo do you like the best?

102

Justify your choice in two sentences?

Your industry?

Years of working experience?

103

104