Copyright 2004 Psychonomic Society Inc 304

Behavior Research Methods Instruments amp Computers2004 36 (2) 304-311

Scientific LogAnalyzerrsquos task is to mine data and analyzelog filesmdashfor example those produced by Web serversScientific LogAnalyzer filters out the information containedin the files and organizes this information for easy importinto standard software such as SPSS Excel and so forth(via tab-delimited file)

Scientific LogAnalyzer was created to meet the needs ofthose who collect data on the Internet Although ScientificLogAnalyzer is useful for the analysis of any type of logfile it currently is the only tool of its kind with featuresgeared to behavioral and social scientists such as handlingof factorial designs response time analysis and drop-outanalysis More specifically Scientific LogAnalyzer wasdeveloped to include options that have turned out be of im-

portance in Internet-based experimenting (eg computationof response times session tracking by identifier in searcharguments choosing either first or last response from sameIP flagging of potential multiple submissions and of pre-defined IP addresses andor domain names and free def-inition of session timeout)

Internet-based experimenting like any type of scientificinvestigation relies on the principle of raw data collectionand storage (Reips 2001 2002b) Raw data need to be re-tained for scrutiny by other researchers from the community(American Psychological Association 2001) In Internet-based research server log files are the raw data There arethree reasons why nothing less than the complete log filesfrom a Web server used in an Internet-based investigationfulfill the requirement set by this principle of raw data(1) Log files contain information about the number ofthose who visit the first Web page with the announcementfor the experiment and then decide not to participate (2) logfiles contain information about technical conditions dur-ing the investigation (ie the general Web traffic condi-tions at the server and the particular conditions of each re-quest by a participant) and (3) log files contain incompletedata sets (ie drop-outs and other partial nonresponses)that may reveal information about potential confoundings(Reips 2002b)

This article was presented as part of the Advanced Techniques in In-ternet Research symposium (chaired by Michael Birnbaum) at the So-ciety for Computers in Psychology (SCiP) conference in VancouverCanada November 6 2003 We thank Michael Birnbaum for helpful com-ments on an earlier version of this article and Alexandre de Spindler andAndreas Umbach for their help in the development of early versions ofScientific LogAnalyzer The authors retain a financial interest in someof the Web-based analysis services described in this article (for condi-tions of free use see note 4) Correspondence concerning this articleshould be addressed to U-D Reips Sozial- und WirtschaftspsychologieUniversitaumlt Zuumlrich Raumlmistr 62 CH-8001 Zurich Switzerland (e-mailureipspsychologieunizhch)

Scientific LogAnalyzerA Web-based tool for analyses

of server log files in psychological research

ULF-DIETRICH REIPSUniversity of Zurich Zurich Switzerland

and

STEFAN STIEGERMedical University of Vienna Vienna Austria

Scientific LogAnalyzer is a platform-independent interactive Web service for the analysis of log filesScientific LogAnalyzer offers several features not available in other log file analysis toolsmdashfor exampleorganizational criteria and computational algorithms suited to aid behavioral and social scientists Sci-entific LogAnalyzer is highly flexible on the input side (unlimited types of log file formats) while strictlykeeping a scientific output format Features include (1) free definition of log file format (2) searchingand marking dependent on any combination of strings (necessary for identifying conditions in experi-ment data) (3) computation of response times (4) detection of multiple sessions (5) speedy analysisof large log files (6) output in HTML andor tab-delimited form suitable for import into statistics soft-ware and (7) a module for analyzing and visualizing drop-out Several methodological features specif-ically needed in the analysis of data collected in Internet-based experiments have been implementedin the Web-based tool and are described in this article A regression analysis with data from 44 log fileanalyses shows that the size of the log file and the domain name lookup are the two main factors deter-mining the duration of an analysis It is less than a minute for a standard experimental study with a 2 2 design a dozen Web pages and 48 participants (ca 800 lines including data from drop-outs) The current version of Scientific LogAnalyzer is freely available for small log files Its Web address ishttpgenpsylab-logcrunshunizhch

SCIENTIFIC LOGANALYZER 305

Not being able to report the information listed abovemdashfor example because a CGI recorded data sets only fromcompleted participationsmdashwill likely become a disadvan-tage in the review process of publications as standards forInternet-based research evolve (Reips 2002b)

Scientific LogAnalyzer is the last piece in a chain of toolsour group has developed for all the steps involved in con-ducting an Internet-based experiment The starting pointin this chain is our experiment generator named WEXTOR(Reips amp Neuhaus 2002) In a guided 10-step procedureWEXTOR can be used to create HTML and JavaScriptcode for experiments that can be run in the lab or on theWeb1 Recruitment of a large and diverse participant sam-ple can be achieved through the Web experiment list2 andthe Web Experimental Psychology Lab3 (Reips 2001)

With Scientific LogAnalyzer we hope to improve thesituation for potential Web experimenters without majorprogramming andor Internet knowledge Important in-formation from the HTTP protocol is kept from gettinglost or unused behavioral data hidden in visitorrsquos paths be-come available to analysis and entries by individual visi-tors become visible with the transformation to the ldquooneuser per rowrdquo format

Scientific LogAnalyzer is publicly available at the fol-lowing Web address httpgenpsylab-logcrunshunizhch4

Other ApplicationsThere are very few scientific applications for Web log

analysis The tool STRATDYN (Berendt 2002 Berendt ampBrenstein 2001) provides classification and visualizationof movement sequences in Web navigation and tests dif-ferences between navigation patterns It was developed asan application to optimize Web site design and for hyper-text studies As such it can be used to find relevant differ-ences between usersrsquo navigation behaviors The tool is di-rected at analyzing navigation patterns in hypertexts and isnot geared toward analyzing data provided on forms Con-sequently it does not create output suitable for analysis ofdata from most types of Internet-based experimenting

LOGPAT (Richter Naumann amp Noller 2003) has itsstrength in analyzing sequential measuresmdashthat is count-ing the frequency of specific paths or path types in a logfile Just like Scientific LogAnalyzer LOGPAT was devel-oped as a platform-independent Web-based tool LikeSTRATDYN the program is limited to analyzing naviga-tion behavior and therefore it cannot be used to select andorganize form input (ie search args) The latter taskhowever is the predominant type of analysis needed in Internet-based experimenting and Web surveying whereasanalyses of navigation paths are more predominant in non-reactive research (for a categorization of Internet-basedresearch along these lines see Reips 2002c 2003)

Commercial and free log file analysis software is focusedalmost entirely on helping the user maintain a Web site interms of identifying access errors points of entry and userpaths through the site Many of these applications are easyto use and create presentation-ready graphical output Ex-ample programs are Analog (httpwwwanalogcx)

TrafficReport (httpwwwseacloakcom) Summary(httpwwwsummarynet) and FunnelWeb (httpwwwquestcomfunnel_webanalyzer) If an academic institu-tion can afford the license fees it is quite advisable to useone or several of these programs as an adjunct to scientificlog file analysis

Features and LimitationsScientific LogAnalyzer offers a number of features to

those who would like to extract information from log filesThese features are listed below and will be explained inthe following sections Web services offer two general ad-vantages For the user there is no need to install anythingThrough the upload of the log file and the download of theanalyzed output file Scientific LogAnalyzer can be usedfrom any location as long as it is connected to the InternetAlso there is no need to update anything Scientific Log-Analyzer is continuously updated server-side and newfeatures will be added to further extend Scientific Log-Analyzerrsquos functionalities in the future

Currently (in Version 5) Scientific LogAnalyzer has thefollowing features (1) a free definition of log file format(2) an easy definition of log file type via a ldquoseparatorseekerrdquo submodule (3) a choice of preset log file formatsand omission of superfluous lines for speedier handling(4) searching and marking dependent on any combinationof strings (necessary for identifying conditions in experi-ment data) (5) a free definition of user input (search ar-guments) to be included in the analysis (6) a computationof time differences (response time measurement) (7) ses-sion management and detection of multiple submissionsin multiple sessions via time IP operating system Webbrowser search argument and combinations thereof (8) detection of multiple submissions within sessions (9) automatic marking of IP addresses and domain namesto be excluded for methodological reasons (10) output inHTML andor tab-delimited form suited for import intostatistics software (11) a module for analyzing and visu-alizing drop-out (12) an information page with summaryof analysis (helps in repeated analyses) (13) speed (a typ-ical log file analysis will take only a few minutes) and(14) extensive help in both English and German

In principle the size of the log file to be analyzed is lim-ited only by the available processing time on the computerhosting Scientific LogAnalyzer and the memory allocatedto the userrsquos Web browser if displaying the resulting tablein the browser is desired In our own tests analyses witheven the largest log files (10 MB each line a HTML-page) took only a few minutes if both domain name (DN)lookup and HTML display were turned off

The TaskWeb server log files come in a format that cannot directly

be analyzed by common statistical applications One per-sonrsquos visit to the Web site creates several lines of varyinglength in the log file one for each accessed item (egHTML files or image files) All lines created by one per-sonrsquos requests show the same IP address or one from a

306 REIPS AND STIEGER

cluster of dynamically assigned IP addresses Figure 1shows a portion of a log file

Scientific LogAnalyzer was built to extract informationfrom such log files in accordance with the following cri-teria (1) implementation of organizational criteria suitedto aid behavioral and social scientists (eg organize thedata according to a factorial design) (2) inclusion of op-tions that will turn out be of value in Internet-based exper-imenting (eg collection andor computation of responsetimes choosing either first or last response from the sameIP) and (3) high flexibility on the input side (unlimitedtypes of log file formats) while strictly keeping a scien-tifically useful output format

Two desired output formats for applications that analyzelog files are Format 1 a tab-delimited file suitable for im-port into statistical applications (this file should be in theldquoone participantndashone rowrdquo format) and Format 2 a drop-out analysis (for instance in tree format) that shows attri-tion rates by path taken through the Web site

The output should include information about how longit took before the next Web page was accessed Thesetimes can be used as approximations of response times5

ProcedurePreparing and uploading the log file Naturally the

first step in analyzing a log file is possessing a log file Ac-cess to the log files created in Internet-technologyndash basedresearch is most easily achieved by running onersquos ownserver Many modern operating systems (OSs) have a built-

in Web server For example on Mac OS X it is a matter ofdropping the materials to be served in a designated folderand hitting two buttons to start serving Web pages to theworld on a computer connected to the Internet The log fileis then created on onersquos own computer Log files may alsobe obtained from the institutionrsquos server administrator

Currently the line feeds contained in the log file to beanalyzed by Scientific LogAnalyzer need to be in DOSformat Changing the type of line feedmdashfor examplefrom Unix or Mac formatmdashcan easily be done in a texteditor If sensitive participant data are involved the logfile should be stored in a location suitable for secure trans-mission (HTTPS) Scientific LogAnalyzer can then ac-cess the file via its URL The log file can also be uploadedfrom the userrsquos hard disk

Identifying the log file format Once the log file hasbeen uploaded to Scientific LogAnalyzer several subrou-tines can be used to identify its format If the format iscompletely unknown to the user the first step in this pro-cedure is to evoke the ldquoSeparator Seekerrdquo This subroutineanalyzes a portion of the log file and searches for charac-ters that separate the columns in the log file The Separa-tor Seeker makes a recommendation based on the numberof characters found Once the separator is known it can beselected from a list of frequent choices or specified in atextbox if it is an unusual separator

After establishing the separators between the columns ofthe log file the positions of the relevant columns have to bespecified All columns found in the log file are presented

Figure 1 A portion of a log file (IP addresses anonymized Referer information shortened) In case of dynamic IP addressing andsimultaneous user accesses (A) browser and operating system information (B) can be used to identify lines from the same user ses-sion Form entries are saved as search arguments (C)

SCIENTIFIC LOGANALYZER 307

to the user and for each the first 50 entries are listed indrop-down menus These drop-down menus are intendedto help deciding which column is being evaluated Posi-tions of the following columns have to be specified date(eg in format mmddyy) time (eg 121058) IP address(eg 1306023996) URL (eg experimentconditionxpage17html) and search argumentsmdashthat is data that weretransferred from one Web page to another by use of the GETmethod6 (shown by the following format age 20ampre-search 0ampexper 0) These five columns are the min-imally required data for Scientific LogAnalyzer to carryout an analysis Browsersystem information will providea better separation of multiple sessions

In many cases the log file format will follow a com-mon predefined formatmdashfor example the Webstar logfile format that is used in the Web Experimental Psychol-ogy Lab (The Webstar log file format is in the followingorder CONNECTION_ID DATE TIME RESULT HOSTNAME URL

BYTES_SENT AGENT REFERER TRANSFER_TIME SEARCH_ARGS see Figure 1) Scientific LogAnalyzer provides anoption for selecting such a predefined format sparing theuser from going through the separator-seeking and relatedprocedures To match a predefined format the user mayalso rearrange columns in the log file before uploading itto Scientific LogAnalyzer which can easily be done intext editors and spreadsheet programs

Reducing the size of the log file The cleaning submodule Before beginning with the analysis the useris asked to consider ldquocleaningrdquo the log file If the log filecontains a large number of lines with data irrelevant to thepresent analysis (retrieval of image files calls from CGIsor aCGIs hits by ldquorobotsrdquo and ldquospidersrdquo [search enginequeries] HTML files that are not part of the study mater-ial etc) using Scientific LogAnalyzerrsquos cleaning sub-module is likely to substantially reduce the duration of theanalysis for large log files A large log file of 50000 lines(approximately 10 MB) takes between 10 and 25 min de-pending on the number of factors (maximum assumed 3)and whether a DN lookup is performed or not For a smalllog file of 800 lines the range is 1 sec to about a minute(see the empirical analysis later in this article) The mostcommon extensions of potentially irrelevant image filesand server-side Web techniques are preset and may besubstituted or extended by the user

Following the deletions a short descriptive statistic ispresented and shows how many lines were deleted fromthe server log file and its respective reduction as a per-centage An option allows the user to view the records ofthe script in order to understand which lines were deleted

Defining study factor and level characteristics Sci-entific LogAnalyzer follows a logical sequence in askingthe user to fill in characteristic strings that identify a studyand if applicable experiment factors and levels

Because all accesses to pages residing on a Web serverusually enter the server log file a number or another iden-tifier that shows the visits to the Web pages making up thestudy or body of data material at interest must be speci-fied The identifier must be present in the path names con-

tained in the URLs listed in the log file If no identifier isgiven all the lines of the log file will be analyzed

If applicable the number of experimental factors in thedesign to be analyzed can be filled in and Scientific Log-Analyzer dynamically creates the appropriate number ofform fields where the factors can then be named and asearch string can be specified Specifying names andsearch strings for the factors is not obligatory factors canalso be defined implicitly via the search strings for levels(to be entered in the next step)

A necessary specification is the number of levels mak-ing up each factor7 As with the factors Scientific Log-Analyzer dynamically creates the appropriate number ofform fields where the levels can then be named and a searchstring or a combination of search strings can be specified

Defining IP addresses and domain names as markedfor exclusion Scientific LogAnalyzer provides optionsthat allow definition of IP addresses and domain names tobe marked in the output filemdashfor example test connec-tions from onersquos own computer that would compromise astudyrsquos data quality if they were not excluded from analy-sis This option may also be used for other reasonsmdashforinstance to mark the domain name of the userrsquos universitywith the purpose of identifying a local sample

The user may also use the so-called ldquomaster listsrdquo thatcontain IP addresses known to be unsuitable for analysisand domain names with dynamic IP addressing8 (egAOL t-online) The master lists are updated and managedby the authors

Output format options Scientific LogAnalyzer con-tains a submodule to select and exclude nonrelevant searcharguments from the analysis (such as ldquoSubmit Clickrdquo)Using this option ensures that an uncluttered table will re-sult from the analysis making it easier to focus on the es-sentials and facilitating import into other programs

Scientific LogAnalyzer provides three choices for thedata output format HTML limited only tabulator-separatedtext and HTML plus tabulator-separated text HTML lim-ited is valuable in testing settings for analyses The usercan opt to analyze only a small portion of the log file bysetting a limited number of lines Because the resultingtable is small the user may view the output within abrowser window If the whole log file is to be analyzedand a very large resulting table is expected only tabulator-separated text is the only option of choice because the at-tempted display of HTML would likely cause the browserto crash because of limited display memory This optionalso has the advantage of being the speediest calculationprocess since only a tabulator-separated file is createdThe third option HTML plus tabulator-separated text isa hybrid version of the two other ones and is good foranalyses of small log files The resulting table is smallenough to be viewed in a browser window immediatelyafter analysis

Handling of multiple submissions Although truemultiple submissions are rare in most Internet-based stud-ies (Birnbaum 2004 Reips 2000a 2000b 2002b VoracekStieger amp Gindl 2001) reloads of pages happen fre-

308 REIPS AND STIEGER

quently This is due to technical reasons in some casesCertain versions of Netscape Navigator for instance usedto reload a Web page every time the user resized thebrowser window Often reloads result from accidental orcurious user behavior In most cases reloads happen immediatelymdashthat is immediately after the first access-ing of a pagemdashor within a few seconds or minutes Thelength of what is considered the time between two sessionstechnically influences the number of multiple submis-sions because any break a visitor to the Web site takes maybegin a new session Scientific LogAnalyzer allows freedefinition of session timeout with 15 min set as the default

In addition to proximity in time Scientific LogAnalyzeruses information about the usersrsquo OS and Web browserscontained in the log file to identify multiple submissionsAn example for such a piece of information is ldquoMozilla40(compatible MSIE 55 Windows 98 H010818)rdquo It ishighly unlikely that 2 participants who happen to log inwithin the same time frame via the same Internet providerand are assigned the same cluster of dynamically andrapidly changing IP addresses (leading to the same IP beingtemporarily assigned to both participants) also use the sameOSbrowser combination (If both were accessing the pro-gram from the same academic computer laboratory or class-room with uniform computer systems they would usuallybe assigned two stable and different IPs As a safeguardagainst the unlikely event of completely identical IPbrowser and OS information Scientific LogAnalyzer willmark repeated access of the same Web page from the sameIP address in the output file allowing the user to selec-tively recheck the log file for anomalies)

Multiple submissions can also be identified via searcharguments For instance in experiments created withWEXTOR (Reips amp Neuhaus 2002) an identifier (ldquoses-sion keyrdquo) is created immediately as soon as a participantenters the first Web page This identifier is then passed onfrom page to page and is written to the log file on everyline created for this participant Scientific LogAnalyzercontains an option to use one of the search arguments thatwere found as the identifier

A researcher using Scientific LogAnalyzer can be flex-ible in choosing either the first or the last of occasionalmultiple submissions of differing values in a single field(eg two lines with the entries ldquo25rdquo and ldquo35rdquo for ldquoagerdquoapparently submitted by the same participant) that aresubmitted from the same IP address Cases with this typeof multiple submission are marked a ldquo1rdquo is entered in thecolumn ldquodanger-argsrdquo in the output file These markedcases should later be excluded or inspected by reviewingthe raw data

Time-variable output Scientific LogAnalyzer includesan option that will automatically calculate response timesif checked Response times are recorded in log files astime differences between downloads of HTML pages Ifcombined with client-side time measurementmdashfor exam-ple by using JavaScript code created by WEXTOR (Reipsamp Neuhaus 2002) or the Java procedure developed by

Eichstaedt (2001)mdashresponse time measurements can becross-checked9

Information page At the end of the analysis a Webpage with three links is presented to the user One willshow the resulting table if HTML was selected as an out-put format The second link points to the tab-delimitedtext file containing the output the third link leads to an in-formation page that contains a summary of the conditionsand speed of the analysis Saving a screen shot or keepinga print-out of this information page is of valuable help ifthe analysis is repeated at a later time

Factors That Influence Duration of Analysis An Empirical Test

From more than 200 analyses that were conducted usingScientific LogAnalyzer up to now a sample of 44 analyseswas selected according to the following criteria (1) Onlyfull analyses of log files were counted (none with limitedline numbers) (2) if the same log file was analyzed re-peatedly within one session the analysis with the highestdegree of informativeness was chosen (in most cases thelast analysis) and (3) the number of factors three or less

A regression analysis was performed on the data setwith number of factors number of levels number of linesuse of DN lookup and use of time variable output as fac-tors and duration of analysis as the dependent variableTwo factors explained most of the variance in duration oflog file analysis [R2 826 F(243) 97008 p 001]These factors were number of lines analyzed [ 842t(43) 12082 p 001] and whether a DN lookup wasrequested [ 157 t(43) 2251 p 030] Theconstant did not reach significance

A systematic test In order to determine a worst casescenario or at least a conservative estimate for the dura-tion of analyses in Scientific LogAnalyzer in a more sys-tematic fashion we conducted the following test We chosethe log file used in the analysis that produced the outlierwith the longest duration (58 min see the upper right circlein the top half of Figure 2) This log file of 76366 lineswas exceptional in the sense that it contained only lines ofrequests for HTML files (no images etc) For the test 4(number of lines) 2 (DN lookup) 4 (number of fac-tors) 32 analyses were performed with the log file orportions of it Even partitioning resulted in sizes of 7736657274 38183 and 19091 lines Within Scientific Log-Analyzer session length was always kept at 900 sec ldquoOurIPrdquo and DN lists were not used time variable output wassuppressed and number and type of search args was heldconstant Search strings defining factors were also heldconstant

Figure 2 shows the duration of the analyses dependenton number of lines in the log file number of experimentalfactors and whether a DN lookup was performed or notDuration ranges from 1 sec to 58 min with medians of000817 (19091 lines) 001516 (38183 lines) 002559(57274 lines) and 003313 (76366 lines) respectivelyDN lookup can significantly increase the duration of

SCIENTIFIC LOGANALYZER 309

analyses because domain name servers are very busy dur-ing certain times of day Consequently only 9 of 16 real-life analyses took less than 4 min in the DN lookup con-dition (6 took less than a minute) whereas there were 26of 28 in the no DN lookup condition (22 took less than aminute) Calculated from our conservative test results ananalysis takes about 50 sec with DN lookup for a standardexperimental study with 2 2 design a dozen Web pagesand 48 participants (800 lines including data from drop-outs) The analysis takes about 5 sec without DN lookup

Drop-Out AnalysisIn addition to the main analysis Scientific LogAnalyzer

contains a submodule for analysis of drop-out (attritionmortality) that is an important dependent andor controlvariable in Internet-based experimenting (Frick Baumlchtigeramp Reips 2001 Reips 2000b 2002a 2002b Schwarz ampReips 2001) If the corresponding button is pressed in itsmain analysis window Scientific LogAnalyzer generatesa visual display of the drop-out tree Figure 3 shows a sam-ple portion of a drop-out tree Branches can be expandedor collapsed in order to conveniently display only those

drop-out paths currently of interest Each branch alsoshows absolute and relative numbers of paths chosen fromthe earlier Web page

The complexity of analyzing dozens of paths made it nec-essary to add a MySQL database to Scientific LogAnalyzer

Version HistoryAn early precursor of Scientific LogAnalyzer was de-

veloped in 1997 as a stand-alone program for the Macin-tosh Since September 2001 a Web-based version hasbeen available that is written in Perl (Release 580)

As of this writing Scientific LogAnalyzer is in its fifthmajor development stagemdashnamely Version 5

Version 1 included the option of defining factors andlevels and the time-based session management with con-figurable duration of session breaks From that versionScientific LogAnalyzer was available on the WWW

In Version 2 a number of features were implementedthat improved speed and methodological usefulness in-cluding cleaning of the log file and defining IP and DNaddresses to be marked for exclusion Several differencesfrom previous versions enhanced Scientific LogAnalyz-

DN LOOKUP

no DN LOOKUP

test - no factortest - 1 factortest - 2 factorstest - 3 factorsreal - no factorreal - 1 factorreal - 2 factorsreal - 3 factors

test - no factor

test - 1 factor

test - 2 factors

test - 3 factors

real - no factor

real - 1 factor

real - 2 factors

real - 3 factors

05736

05150

04605

04019

03434

02848

02302

01717

01131

00546

00000

05736

05150

04605

04019

03434

02848

02302

01717

01131

00546

00000

0 10000 20000 30000 40000 50000 60000 70000 80000

0 10000 20000 30000 40000 50000 60000 70000 80000

Number of lines in the log file

Number of lines in the log file

Co

mp

uta

tio

n t

ime

Co

mp

uta

tio

n t

ime

Figure 2 Results from 44 real analyses and from the empirical test of duration of analyses in Scientific LogAnalyzer de-pending on DN lookup number of lines and number of factors Of the real analyses 35 were carried out in less than 5 min 28in less than 1 min

310 REIPS AND STIEGER

errsquos usability Changes of programming routines in Ver-sion 2 also resulted in speedier analyses

In Version 3 user administration and guest accountwere added Prefixes and suffixes allowed for almost un-limited possibilities in defining factors and levels even incomplicated log file analyses

Version 4 saw the various options in defining types oflog files Furthermore the first version of the submodulefor drop-out analysis was added Due to the complexity ofthe task to be accomplished by this submodule a MySQLdatabase (Version 32355-nt) was added

Finally Version 5 is the first version with comprehen-sive help files all available on line and in both German andEnglish Session management was refined using informa-tion about OS and browser type and identification bysearch argument was added

OutlookScientific LogAnalyzer will be further integrated with

the materials developed in the Swiss Virtual Campus pro-ject entitled ldquoExperimental Design and Web-based Ex-perimentationrdquo in order to enable learning by doing inInternet-based experimenting

Other improvements will include additional types ofanalyses and additional options for the current analysesFor example we are hoping to add more predefined log for-mats and include more options for visualizing results infuture versions of Scientific LogAnalyzer

We hope that Scientific LogAnalyzer will serve as auseful tool for Web and off-line experimenters as well asfor learning and teaching methodological concepts of Internet-based data collection and analysis in psychologyand neighboring disciplines Beyond these core applica-tions Scientific LogAnalyzer may be used universallywherever log files are produced

REFERENCES

American Psychological Association (2001) Retaining raw data(sect710) In Publication manual of the American Psychological Asso-ciation (5th ed p 342) Washington DC Author

Berendt B (2002) Using site semantics to analyze visualize and sup-port navigation Data Mining amp Knowledge Discovery 6 37-59

Berendt B amp Brenstein E (2001) Visualizing individual differ-ences in Web navigation STRATDYN a tool for analyzing navigationpatterns Behavior Research Methods Instruments amp Computers 33243-257

Birnbaum M H (2004) Human research and data collection via the In-ternet Annual Review of Psychology 55 803-832

Eichstaedt J (2001) An inaccurate-timing filter for reaction time mea-surement by JAVA applets implementing Internet-based experimentsBehavior Research Methods Instruments amp Computers 33 179-186

Frick A Baumlchtiger M T amp Reips U-D (2001) Financial incentivespersonal information and drop out in online studies In U-D Reips ampM Bosnjak (Eds) Dimensions of Internet science (pp 209-219)Lengerich Pabst

Reips U-D (2000a) Das psychologische Experimentieren im Internet(2 uumlberarbeitete Auflage) [Psychological experimenting on the Inter-net (Rev ed)] In B Batinic (Ed) Internet fuumlr Psychologen (pp 319-343) Goumlttingen Hogrefe

Reips U-D (2000b) The Web experiment method Advantages disad-

Figure 3 An example display from the drop-out submodule Absolute numbers of participantstaking a path are shown at the arrows percentages denote relative numbers having made the deci-sion to continue along this particular path from the previous Web page The further the indenta-tion the further the steps through the Web site Paths can be collapsed via mouse click at the ldquordquosigns and expanded at the ldquordquo signs

SCIENTIFIC LOGANALYZER 307

to the user and for each the first 50 entries are listed indrop-down menus These drop-down menus are intendedto help deciding which column is being evaluated Posi-tions of the following columns have to be specified date(eg in format mmddyy) time (eg 121058) IP address(eg 1306023996) URL (eg experimentconditionxpage17html) and search argumentsmdashthat is data that weretransferred from one Web page to another by use of the GETmethod6 (shown by the following format age 20ampre-search 0ampexper 0) These five columns are the min-imally required data for Scientific LogAnalyzer to carryout an analysis Browsersystem information will providea better separation of multiple sessions

In many cases the log file format will follow a com-mon predefined formatmdashfor example the Webstar logfile format that is used in the Web Experimental Psychol-ogy Lab (The Webstar log file format is in the followingorder CONNECTION_ID DATE TIME RESULT HOSTNAME URL

BYTES_SENT AGENT REFERER TRANSFER_TIME SEARCH_ARGS see Figure 1) Scientific LogAnalyzer provides anoption for selecting such a predefined format sparing theuser from going through the separator-seeking and relatedprocedures To match a predefined format the user mayalso rearrange columns in the log file before uploading itto Scientific LogAnalyzer which can easily be done intext editors and spreadsheet programs

Reducing the size of the log file The cleaning submodule Before beginning with the analysis the useris asked to consider ldquocleaningrdquo the log file If the log filecontains a large number of lines with data irrelevant to thepresent analysis (retrieval of image files calls from CGIsor aCGIs hits by ldquorobotsrdquo and ldquospidersrdquo [search enginequeries] HTML files that are not part of the study mater-ial etc) using Scientific LogAnalyzerrsquos cleaning sub-module is likely to substantially reduce the duration of theanalysis for large log files A large log file of 50000 lines(approximately 10 MB) takes between 10 and 25 min de-pending on the number of factors (maximum assumed 3)and whether a DN lookup is performed or not For a smalllog file of 800 lines the range is 1 sec to about a minute(see the empirical analysis later in this article) The mostcommon extensions of potentially irrelevant image filesand server-side Web techniques are preset and may besubstituted or extended by the user

Following the deletions a short descriptive statistic ispresented and shows how many lines were deleted fromthe server log file and its respective reduction as a per-centage An option allows the user to view the records ofthe script in order to understand which lines were deleted

Defining study factor and level characteristics Sci-entific LogAnalyzer follows a logical sequence in askingthe user to fill in characteristic strings that identify a studyand if applicable experiment factors and levels

Because all accesses to pages residing on a Web serverusually enter the server log file a number or another iden-tifier that shows the visits to the Web pages making up thestudy or body of data material at interest must be speci-fied The identifier must be present in the path names con-

tained in the URLs listed in the log file If no identifier isgiven all the lines of the log file will be analyzed

If applicable the number of experimental factors in thedesign to be analyzed can be filled in and Scientific Log-Analyzer dynamically creates the appropriate number ofform fields where the factors can then be named and asearch string can be specified Specifying names andsearch strings for the factors is not obligatory factors canalso be defined implicitly via the search strings for levels(to be entered in the next step)

A necessary specification is the number of levels mak-ing up each factor7 As with the factors Scientific Log-Analyzer dynamically creates the appropriate number ofform fields where the levels can then be named and a searchstring or a combination of search strings can be specified

Defining IP addresses and domain names as markedfor exclusion Scientific LogAnalyzer provides optionsthat allow definition of IP addresses and domain names tobe marked in the output filemdashfor example test connec-tions from onersquos own computer that would compromise astudyrsquos data quality if they were not excluded from analy-sis This option may also be used for other reasonsmdashforinstance to mark the domain name of the userrsquos universitywith the purpose of identifying a local sample

The user may also use the so-called ldquomaster listsrdquo thatcontain IP addresses known to be unsuitable for analysisand domain names with dynamic IP addressing8 (egAOL t-online) The master lists are updated and managedby the authors

Output format options Scientific LogAnalyzer con-tains a submodule to select and exclude nonrelevant searcharguments from the analysis (such as ldquoSubmit Clickrdquo)Using this option ensures that an uncluttered table will re-sult from the analysis making it easier to focus on the es-sentials and facilitating import into other programs

Scientific LogAnalyzer provides three choices for thedata output format HTML limited only tabulator-separatedtext and HTML plus tabulator-separated text HTML lim-ited is valuable in testing settings for analyses The usercan opt to analyze only a small portion of the log file bysetting a limited number of lines Because the resultingtable is small the user may view the output within abrowser window If the whole log file is to be analyzedand a very large resulting table is expected only tabulator-separated text is the only option of choice because the at-tempted display of HTML would likely cause the browserto crash because of limited display memory This optionalso has the advantage of being the speediest calculationprocess since only a tabulator-separated file is createdThe third option HTML plus tabulator-separated text isa hybrid version of the two other ones and is good foranalyses of small log files The resulting table is smallenough to be viewed in a browser window immediatelyafter analysis

Handling of multiple submissions Although truemultiple submissions are rare in most Internet-based stud-ies (Birnbaum 2004 Reips 2000a 2000b 2002b VoracekStieger amp Gindl 2001) reloads of pages happen fre-

308 REIPS AND STIEGER

quently This is due to technical reasons in some casesCertain versions of Netscape Navigator for instance usedto reload a Web page every time the user resized thebrowser window Often reloads result from accidental orcurious user behavior In most cases reloads happen immediatelymdashthat is immediately after the first access-ing of a pagemdashor within a few seconds or minutes Thelength of what is considered the time between two sessionstechnically influences the number of multiple submis-sions because any break a visitor to the Web site takes maybegin a new session Scientific LogAnalyzer allows freedefinition of session timeout with 15 min set as the default

In addition to proximity in time Scientific LogAnalyzeruses information about the usersrsquo OS and Web browserscontained in the log file to identify multiple submissionsAn example for such a piece of information is ldquoMozilla40(compatible MSIE 55 Windows 98 H010818)rdquo It ishighly unlikely that 2 participants who happen to log inwithin the same time frame via the same Internet providerand are assigned the same cluster of dynamically andrapidly changing IP addresses (leading to the same IP beingtemporarily assigned to both participants) also use the sameOSbrowser combination (If both were accessing the pro-gram from the same academic computer laboratory or class-room with uniform computer systems they would usuallybe assigned two stable and different IPs As a safeguardagainst the unlikely event of completely identical IPbrowser and OS information Scientific LogAnalyzer willmark repeated access of the same Web page from the sameIP address in the output file allowing the user to selec-tively recheck the log file for anomalies)

Multiple submissions can also be identified via searcharguments For instance in experiments created withWEXTOR (Reips amp Neuhaus 2002) an identifier (ldquoses-sion keyrdquo) is created immediately as soon as a participantenters the first Web page This identifier is then passed onfrom page to page and is written to the log file on everyline created for this participant Scientific LogAnalyzercontains an option to use one of the search arguments thatwere found as the identifier

A researcher using Scientific LogAnalyzer can be flex-ible in choosing either the first or the last of occasionalmultiple submissions of differing values in a single field(eg two lines with the entries ldquo25rdquo and ldquo35rdquo for ldquoagerdquoapparently submitted by the same participant) that aresubmitted from the same IP address Cases with this typeof multiple submission are marked a ldquo1rdquo is entered in thecolumn ldquodanger-argsrdquo in the output file These markedcases should later be excluded or inspected by reviewingthe raw data

Time-variable output Scientific LogAnalyzer includesan option that will automatically calculate response timesif checked Response times are recorded in log files astime differences between downloads of HTML pages Ifcombined with client-side time measurementmdashfor exam-ple by using JavaScript code created by WEXTOR (Reipsamp Neuhaus 2002) or the Java procedure developed by

Eichstaedt (2001)mdashresponse time measurements can becross-checked9

Information page At the end of the analysis a Webpage with three links is presented to the user One willshow the resulting table if HTML was selected as an out-put format The second link points to the tab-delimitedtext file containing the output the third link leads to an in-formation page that contains a summary of the conditionsand speed of the analysis Saving a screen shot or keepinga print-out of this information page is of valuable help ifthe analysis is repeated at a later time

Factors That Influence Duration of Analysis An Empirical Test

From more than 200 analyses that were conducted usingScientific LogAnalyzer up to now a sample of 44 analyseswas selected according to the following criteria (1) Onlyfull analyses of log files were counted (none with limitedline numbers) (2) if the same log file was analyzed re-peatedly within one session the analysis with the highestdegree of informativeness was chosen (in most cases thelast analysis) and (3) the number of factors three or less

A regression analysis was performed on the data setwith number of factors number of levels number of linesuse of DN lookup and use of time variable output as fac-tors and duration of analysis as the dependent variableTwo factors explained most of the variance in duration oflog file analysis [R2 826 F(243) 97008 p 001]These factors were number of lines analyzed [ 842t(43) 12082 p 001] and whether a DN lookup wasrequested [ 157 t(43) 2251 p 030] Theconstant did not reach significance

A systematic test In order to determine a worst casescenario or at least a conservative estimate for the dura-tion of analyses in Scientific LogAnalyzer in a more sys-tematic fashion we conducted the following test We chosethe log file used in the analysis that produced the outlierwith the longest duration (58 min see the upper right circlein the top half of Figure 2) This log file of 76366 lineswas exceptional in the sense that it contained only lines ofrequests for HTML files (no images etc) For the test 4(number of lines) 2 (DN lookup) 4 (number of fac-tors) 32 analyses were performed with the log file orportions of it Even partitioning resulted in sizes of 7736657274 38183 and 19091 lines Within Scientific Log-Analyzer session length was always kept at 900 sec ldquoOurIPrdquo and DN lists were not used time variable output wassuppressed and number and type of search args was heldconstant Search strings defining factors were also heldconstant

Figure 2 shows the duration of the analyses dependenton number of lines in the log file number of experimentalfactors and whether a DN lookup was performed or notDuration ranges from 1 sec to 58 min with medians of000817 (19091 lines) 001516 (38183 lines) 002559(57274 lines) and 003313 (76366 lines) respectivelyDN lookup can significantly increase the duration of

SCIENTIFIC LOGANALYZER 309

analyses because domain name servers are very busy dur-ing certain times of day Consequently only 9 of 16 real-life analyses took less than 4 min in the DN lookup con-dition (6 took less than a minute) whereas there were 26of 28 in the no DN lookup condition (22 took less than aminute) Calculated from our conservative test results ananalysis takes about 50 sec with DN lookup for a standardexperimental study with 2 2 design a dozen Web pagesand 48 participants (800 lines including data from drop-outs) The analysis takes about 5 sec without DN lookup

Drop-Out AnalysisIn addition to the main analysis Scientific LogAnalyzer

contains a submodule for analysis of drop-out (attritionmortality) that is an important dependent andor controlvariable in Internet-based experimenting (Frick Baumlchtigeramp Reips 2001 Reips 2000b 2002a 2002b Schwarz ampReips 2001) If the corresponding button is pressed in itsmain analysis window Scientific LogAnalyzer generatesa visual display of the drop-out tree Figure 3 shows a sam-ple portion of a drop-out tree Branches can be expandedor collapsed in order to conveniently display only those

drop-out paths currently of interest Each branch alsoshows absolute and relative numbers of paths chosen fromthe earlier Web page

The complexity of analyzing dozens of paths made it nec-essary to add a MySQL database to Scientific LogAnalyzer

Version HistoryAn early precursor of Scientific LogAnalyzer was de-

veloped in 1997 as a stand-alone program for the Macin-tosh Since September 2001 a Web-based version hasbeen available that is written in Perl (Release 580)

As of this writing Scientific LogAnalyzer is in its fifthmajor development stagemdashnamely Version 5

Version 1 included the option of defining factors andlevels and the time-based session management with con-figurable duration of session breaks From that versionScientific LogAnalyzer was available on the WWW

In Version 2 a number of features were implementedthat improved speed and methodological usefulness in-cluding cleaning of the log file and defining IP and DNaddresses to be marked for exclusion Several differencesfrom previous versions enhanced Scientific LogAnalyz-

DN LOOKUP

no DN LOOKUP

test - no factortest - 1 factortest - 2 factorstest - 3 factorsreal - no factorreal - 1 factorreal - 2 factorsreal - 3 factors

test - no factor

test - 1 factor

test - 2 factors

test - 3 factors

real - no factor

real - 1 factor

real - 2 factors

real - 3 factors

05736

05150

04605

04019

03434

02848

02302

01717

01131

00546

00000

05736

05150

04605

04019

03434

02848

02302

01717

01131

00546

00000

0 10000 20000 30000 40000 50000 60000 70000 80000

0 10000 20000 30000 40000 50000 60000 70000 80000

Number of lines in the log file

Number of lines in the log file

Co

mp

uta

tio

n t

ime

Co

mp

uta

tio

n t

ime

Figure 2 Results from 44 real analyses and from the empirical test of duration of analyses in Scientific LogAnalyzer de-pending on DN lookup number of lines and number of factors Of the real analyses 35 were carried out in less than 5 min 28in less than 1 min

310 REIPS AND STIEGER

errsquos usability Changes of programming routines in Ver-sion 2 also resulted in speedier analyses

In Version 3 user administration and guest accountwere added Prefixes and suffixes allowed for almost un-limited possibilities in defining factors and levels even incomplicated log file analyses

Version 4 saw the various options in defining types oflog files Furthermore the first version of the submodulefor drop-out analysis was added Due to the complexity ofthe task to be accomplished by this submodule a MySQLdatabase (Version 32355-nt) was added

Finally Version 5 is the first version with comprehen-sive help files all available on line and in both German andEnglish Session management was refined using informa-tion about OS and browser type and identification bysearch argument was added

OutlookScientific LogAnalyzer will be further integrated with

the materials developed in the Swiss Virtual Campus pro-ject entitled ldquoExperimental Design and Web-based Ex-perimentationrdquo in order to enable learning by doing inInternet-based experimenting

Other improvements will include additional types ofanalyses and additional options for the current analysesFor example we are hoping to add more predefined log for-mats and include more options for visualizing results infuture versions of Scientific LogAnalyzer

We hope that Scientific LogAnalyzer will serve as auseful tool for Web and off-line experimenters as well asfor learning and teaching methodological concepts of Internet-based data collection and analysis in psychologyand neighboring disciplines Beyond these core applica-tions Scientific LogAnalyzer may be used universallywherever log files are produced

REFERENCES

American Psychological Association (2001) Retaining raw data(sect710) In Publication manual of the American Psychological Asso-ciation (5th ed p 342) Washington DC Author

Berendt B (2002) Using site semantics to analyze visualize and sup-port navigation Data Mining amp Knowledge Discovery 6 37-59

Berendt B amp Brenstein E (2001) Visualizing individual differ-ences in Web navigation STRATDYN a tool for analyzing navigationpatterns Behavior Research Methods Instruments amp Computers 33243-257

Birnbaum M H (2004) Human research and data collection via the In-ternet Annual Review of Psychology 55 803-832

Eichstaedt J (2001) An inaccurate-timing filter for reaction time mea-surement by JAVA applets implementing Internet-based experimentsBehavior Research Methods Instruments amp Computers 33 179-186

Frick A Baumlchtiger M T amp Reips U-D (2001) Financial incentivespersonal information and drop out in online studies In U-D Reips ampM Bosnjak (Eds) Dimensions of Internet science (pp 209-219)Lengerich Pabst

Reips U-D (2000a) Das psychologische Experimentieren im Internet(2 uumlberarbeitete Auflage) [Psychological experimenting on the Inter-net (Rev ed)] In B Batinic (Ed) Internet fuumlr Psychologen (pp 319-343) Goumlttingen Hogrefe

Reips U-D (2000b) The Web experiment method Advantages disad-

Figure 3 An example display from the drop-out submodule Absolute numbers of participantstaking a path are shown at the arrows percentages denote relative numbers having made the deci-sion to continue along this particular path from the previous Web page The further the indenta-tion the further the steps through the Web site Paths can be collapsed via mouse click at the ldquordquosigns and expanded at the ldquordquo signs

308 REIPS AND STIEGER

quently This is due to technical reasons in some casesCertain versions of Netscape Navigator for instance usedto reload a Web page every time the user resized thebrowser window Often reloads result from accidental orcurious user behavior In most cases reloads happen immediatelymdashthat is immediately after the first access-ing of a pagemdashor within a few seconds or minutes Thelength of what is considered the time between two sessionstechnically influences the number of multiple submis-sions because any break a visitor to the Web site takes maybegin a new session Scientific LogAnalyzer allows freedefinition of session timeout with 15 min set as the default

In addition to proximity in time Scientific LogAnalyzeruses information about the usersrsquo OS and Web browserscontained in the log file to identify multiple submissionsAn example for such a piece of information is ldquoMozilla40(compatible MSIE 55 Windows 98 H010818)rdquo It ishighly unlikely that 2 participants who happen to log inwithin the same time frame via the same Internet providerand are assigned the same cluster of dynamically andrapidly changing IP addresses (leading to the same IP beingtemporarily assigned to both participants) also use the sameOSbrowser combination (If both were accessing the pro-gram from the same academic computer laboratory or class-room with uniform computer systems they would usuallybe assigned two stable and different IPs As a safeguardagainst the unlikely event of completely identical IPbrowser and OS information Scientific LogAnalyzer willmark repeated access of the same Web page from the sameIP address in the output file allowing the user to selec-tively recheck the log file for anomalies)

Multiple submissions can also be identified via searcharguments For instance in experiments created withWEXTOR (Reips amp Neuhaus 2002) an identifier (ldquoses-sion keyrdquo) is created immediately as soon as a participantenters the first Web page This identifier is then passed onfrom page to page and is written to the log file on everyline created for this participant Scientific LogAnalyzercontains an option to use one of the search arguments thatwere found as the identifier

A researcher using Scientific LogAnalyzer can be flex-ible in choosing either the first or the last of occasionalmultiple submissions of differing values in a single field(eg two lines with the entries ldquo25rdquo and ldquo35rdquo for ldquoagerdquoapparently submitted by the same participant) that aresubmitted from the same IP address Cases with this typeof multiple submission are marked a ldquo1rdquo is entered in thecolumn ldquodanger-argsrdquo in the output file These markedcases should later be excluded or inspected by reviewingthe raw data

Time-variable output Scientific LogAnalyzer includesan option that will automatically calculate response timesif checked Response times are recorded in log files astime differences between downloads of HTML pages Ifcombined with client-side time measurementmdashfor exam-ple by using JavaScript code created by WEXTOR (Reipsamp Neuhaus 2002) or the Java procedure developed by

Eichstaedt (2001)mdashresponse time measurements can becross-checked9

Information page At the end of the analysis a Webpage with three links is presented to the user One willshow the resulting table if HTML was selected as an out-put format The second link points to the tab-delimitedtext file containing the output the third link leads to an in-formation page that contains a summary of the conditionsand speed of the analysis Saving a screen shot or keepinga print-out of this information page is of valuable help ifthe analysis is repeated at a later time

Factors That Influence Duration of Analysis An Empirical Test

From more than 200 analyses that were conducted usingScientific LogAnalyzer up to now a sample of 44 analyseswas selected according to the following criteria (1) Onlyfull analyses of log files were counted (none with limitedline numbers) (2) if the same log file was analyzed re-peatedly within one session the analysis with the highestdegree of informativeness was chosen (in most cases thelast analysis) and (3) the number of factors three or less

A regression analysis was performed on the data setwith number of factors number of levels number of linesuse of DN lookup and use of time variable output as fac-tors and duration of analysis as the dependent variableTwo factors explained most of the variance in duration oflog file analysis [R2 826 F(243) 97008 p 001]These factors were number of lines analyzed [ 842t(43) 12082 p 001] and whether a DN lookup wasrequested [ 157 t(43) 2251 p 030] Theconstant did not reach significance

A systematic test In order to determine a worst casescenario or at least a conservative estimate for the dura-tion of analyses in Scientific LogAnalyzer in a more sys-tematic fashion we conducted the following test We chosethe log file used in the analysis that produced the outlierwith the longest duration (58 min see the upper right circlein the top half of Figure 2) This log file of 76366 lineswas exceptional in the sense that it contained only lines ofrequests for HTML files (no images etc) For the test 4(number of lines) 2 (DN lookup) 4 (number of fac-tors) 32 analyses were performed with the log file orportions of it Even partitioning resulted in sizes of 7736657274 38183 and 19091 lines Within Scientific Log-Analyzer session length was always kept at 900 sec ldquoOurIPrdquo and DN lists were not used time variable output wassuppressed and number and type of search args was heldconstant Search strings defining factors were also heldconstant

Figure 2 shows the duration of the analyses dependenton number of lines in the log file number of experimentalfactors and whether a DN lookup was performed or notDuration ranges from 1 sec to 58 min with medians of000817 (19091 lines) 001516 (38183 lines) 002559(57274 lines) and 003313 (76366 lines) respectivelyDN lookup can significantly increase the duration of

SCIENTIFIC LOGANALYZER 309

analyses because domain name servers are very busy dur-ing certain times of day Consequently only 9 of 16 real-life analyses took less than 4 min in the DN lookup con-dition (6 took less than a minute) whereas there were 26of 28 in the no DN lookup condition (22 took less than aminute) Calculated from our conservative test results ananalysis takes about 50 sec with DN lookup for a standardexperimental study with 2 2 design a dozen Web pagesand 48 participants (800 lines including data from drop-outs) The analysis takes about 5 sec without DN lookup

Drop-Out AnalysisIn addition to the main analysis Scientific LogAnalyzer

contains a submodule for analysis of drop-out (attritionmortality) that is an important dependent andor controlvariable in Internet-based experimenting (Frick Baumlchtigeramp Reips 2001 Reips 2000b 2002a 2002b Schwarz ampReips 2001) If the corresponding button is pressed in itsmain analysis window Scientific LogAnalyzer generatesa visual display of the drop-out tree Figure 3 shows a sam-ple portion of a drop-out tree Branches can be expandedor collapsed in order to conveniently display only those

drop-out paths currently of interest Each branch alsoshows absolute and relative numbers of paths chosen fromthe earlier Web page

The complexity of analyzing dozens of paths made it nec-essary to add a MySQL database to Scientific LogAnalyzer

Version HistoryAn early precursor of Scientific LogAnalyzer was de-

veloped in 1997 as a stand-alone program for the Macin-tosh Since September 2001 a Web-based version hasbeen available that is written in Perl (Release 580)

As of this writing Scientific LogAnalyzer is in its fifthmajor development stagemdashnamely Version 5

Version 1 included the option of defining factors andlevels and the time-based session management with con-figurable duration of session breaks From that versionScientific LogAnalyzer was available on the WWW

In Version 2 a number of features were implementedthat improved speed and methodological usefulness in-cluding cleaning of the log file and defining IP and DNaddresses to be marked for exclusion Several differencesfrom previous versions enhanced Scientific LogAnalyz-

DN LOOKUP

no DN LOOKUP

test - no factortest - 1 factortest - 2 factorstest - 3 factorsreal - no factorreal - 1 factorreal - 2 factorsreal - 3 factors

test - no factor

test - 1 factor

test - 2 factors

test - 3 factors

real - no factor

real - 1 factor

real - 2 factors

real - 3 factors

05736

05150

04605

04019

03434

02848

02302

01717

01131

00546

00000

05736

05150

04605

04019

03434

02848

02302

01717

01131

00546

00000

0 10000 20000 30000 40000 50000 60000 70000 80000

0 10000 20000 30000 40000 50000 60000 70000 80000

Number of lines in the log file

Number of lines in the log file

Co

mp

uta

tio

n t

ime

Co

mp

uta

tio

n t

ime

Figure 2 Results from 44 real analyses and from the empirical test of duration of analyses in Scientific LogAnalyzer de-pending on DN lookup number of lines and number of factors Of the real analyses 35 were carried out in less than 5 min 28in less than 1 min

310 REIPS AND STIEGER

errsquos usability Changes of programming routines in Ver-sion 2 also resulted in speedier analyses

In Version 3 user administration and guest accountwere added Prefixes and suffixes allowed for almost un-limited possibilities in defining factors and levels even incomplicated log file analyses

Version 4 saw the various options in defining types oflog files Furthermore the first version of the submodulefor drop-out analysis was added Due to the complexity ofthe task to be accomplished by this submodule a MySQLdatabase (Version 32355-nt) was added

Finally Version 5 is the first version with comprehen-sive help files all available on line and in both German andEnglish Session management was refined using informa-tion about OS and browser type and identification bysearch argument was added

OutlookScientific LogAnalyzer will be further integrated with

the materials developed in the Swiss Virtual Campus pro-ject entitled ldquoExperimental Design and Web-based Ex-perimentationrdquo in order to enable learning by doing inInternet-based experimenting

Other improvements will include additional types ofanalyses and additional options for the current analysesFor example we are hoping to add more predefined log for-mats and include more options for visualizing results infuture versions of Scientific LogAnalyzer

We hope that Scientific LogAnalyzer will serve as auseful tool for Web and off-line experimenters as well asfor learning and teaching methodological concepts of Internet-based data collection and analysis in psychologyand neighboring disciplines Beyond these core applica-tions Scientific LogAnalyzer may be used universallywherever log files are produced

REFERENCES

American Psychological Association (2001) Retaining raw data(sect710) In Publication manual of the American Psychological Asso-ciation (5th ed p 342) Washington DC Author

Berendt B (2002) Using site semantics to analyze visualize and sup-port navigation Data Mining amp Knowledge Discovery 6 37-59

Berendt B amp Brenstein E (2001) Visualizing individual differ-ences in Web navigation STRATDYN a tool for analyzing navigationpatterns Behavior Research Methods Instruments amp Computers 33243-257

Birnbaum M H (2004) Human research and data collection via the In-ternet Annual Review of Psychology 55 803-832

Eichstaedt J (2001) An inaccurate-timing filter for reaction time mea-surement by JAVA applets implementing Internet-based experimentsBehavior Research Methods Instruments amp Computers 33 179-186

Frick A Baumlchtiger M T amp Reips U-D (2001) Financial incentivespersonal information and drop out in online studies In U-D Reips ampM Bosnjak (Eds) Dimensions of Internet science (pp 209-219)Lengerich Pabst

Reips U-D (2000a) Das psychologische Experimentieren im Internet(2 uumlberarbeitete Auflage) [Psychological experimenting on the Inter-net (Rev ed)] In B Batinic (Ed) Internet fuumlr Psychologen (pp 319-343) Goumlttingen Hogrefe

Reips U-D (2000b) The Web experiment method Advantages disad-

Figure 3 An example display from the drop-out submodule Absolute numbers of participantstaking a path are shown at the arrows percentages denote relative numbers having made the deci-sion to continue along this particular path from the previous Web page The further the indenta-tion the further the steps through the Web site Paths can be collapsed via mouse click at the ldquordquosigns and expanded at the ldquordquo signs

SCIENTIFIC LOGANALYZER 309

analyses because domain name servers are very busy dur-ing certain times of day Consequently only 9 of 16 real-life analyses took less than 4 min in the DN lookup con-dition (6 took less than a minute) whereas there were 26of 28 in the no DN lookup condition (22 took less than aminute) Calculated from our conservative test results ananalysis takes about 50 sec with DN lookup for a standardexperimental study with 2 2 design a dozen Web pagesand 48 participants (800 lines including data from drop-outs) The analysis takes about 5 sec without DN lookup

Drop-Out AnalysisIn addition to the main analysis Scientific LogAnalyzer

contains a submodule for analysis of drop-out (attritionmortality) that is an important dependent andor controlvariable in Internet-based experimenting (Frick Baumlchtigeramp Reips 2001 Reips 2000b 2002a 2002b Schwarz ampReips 2001) If the corresponding button is pressed in itsmain analysis window Scientific LogAnalyzer generatesa visual display of the drop-out tree Figure 3 shows a sam-ple portion of a drop-out tree Branches can be expandedor collapsed in order to conveniently display only those

drop-out paths currently of interest Each branch alsoshows absolute and relative numbers of paths chosen fromthe earlier Web page

The complexity of analyzing dozens of paths made it nec-essary to add a MySQL database to Scientific LogAnalyzer

Version HistoryAn early precursor of Scientific LogAnalyzer was de-

veloped in 1997 as a stand-alone program for the Macin-tosh Since September 2001 a Web-based version hasbeen available that is written in Perl (Release 580)

As of this writing Scientific LogAnalyzer is in its fifthmajor development stagemdashnamely Version 5

Version 1 included the option of defining factors andlevels and the time-based session management with con-figurable duration of session breaks From that versionScientific LogAnalyzer was available on the WWW

In Version 2 a number of features were implementedthat improved speed and methodological usefulness in-cluding cleaning of the log file and defining IP and DNaddresses to be marked for exclusion Several differencesfrom previous versions enhanced Scientific LogAnalyz-

DN LOOKUP

no DN LOOKUP

test - no factortest - 1 factortest - 2 factorstest - 3 factorsreal - no factorreal - 1 factorreal - 2 factorsreal - 3 factors

test - no factor

test - 1 factor

test - 2 factors

test - 3 factors

real - no factor

real - 1 factor

real - 2 factors

real - 3 factors

05736

05150

04605

04019

03434

02848

02302

01717

01131

00546

00000

05736

05150

04605

04019

03434

02848

02302

01717

01131

00546

00000

0 10000 20000 30000 40000 50000 60000 70000 80000

0 10000 20000 30000 40000 50000 60000 70000 80000

Number of lines in the log file

Number of lines in the log file

Co

mp

uta

tio

n t

ime

Co

mp

uta

tio

n t

ime

Figure 2 Results from 44 real analyses and from the empirical test of duration of analyses in Scientific LogAnalyzer de-pending on DN lookup number of lines and number of factors Of the real analyses 35 were carried out in less than 5 min 28in less than 1 min

310 REIPS AND STIEGER

errsquos usability Changes of programming routines in Ver-sion 2 also resulted in speedier analyses

In Version 3 user administration and guest accountwere added Prefixes and suffixes allowed for almost un-limited possibilities in defining factors and levels even incomplicated log file analyses

Version 4 saw the various options in defining types oflog files Furthermore the first version of the submodulefor drop-out analysis was added Due to the complexity ofthe task to be accomplished by this submodule a MySQLdatabase (Version 32355-nt) was added

Finally Version 5 is the first version with comprehen-sive help files all available on line and in both German andEnglish Session management was refined using informa-tion about OS and browser type and identification bysearch argument was added

OutlookScientific LogAnalyzer will be further integrated with

the materials developed in the Swiss Virtual Campus pro-ject entitled ldquoExperimental Design and Web-based Ex-perimentationrdquo in order to enable learning by doing inInternet-based experimenting

Other improvements will include additional types ofanalyses and additional options for the current analysesFor example we are hoping to add more predefined log for-mats and include more options for visualizing results infuture versions of Scientific LogAnalyzer

We hope that Scientific LogAnalyzer will serve as auseful tool for Web and off-line experimenters as well asfor learning and teaching methodological concepts of Internet-based data collection and analysis in psychologyand neighboring disciplines Beyond these core applica-tions Scientific LogAnalyzer may be used universallywherever log files are produced

REFERENCES

American Psychological Association (2001) Retaining raw data(sect710) In Publication manual of the American Psychological Asso-ciation (5th ed p 342) Washington DC Author

Berendt B (2002) Using site semantics to analyze visualize and sup-port navigation Data Mining amp Knowledge Discovery 6 37-59

Berendt B amp Brenstein E (2001) Visualizing individual differ-ences in Web navigation STRATDYN a tool for analyzing navigationpatterns Behavior Research Methods Instruments amp Computers 33243-257

Birnbaum M H (2004) Human research and data collection via the In-ternet Annual Review of Psychology 55 803-832

Eichstaedt J (2001) An inaccurate-timing filter for reaction time mea-surement by JAVA applets implementing Internet-based experimentsBehavior Research Methods Instruments amp Computers 33 179-186

Frick A Baumlchtiger M T amp Reips U-D (2001) Financial incentivespersonal information and drop out in online studies In U-D Reips ampM Bosnjak (Eds) Dimensions of Internet science (pp 209-219)Lengerich Pabst

Reips U-D (2000a) Das psychologische Experimentieren im Internet(2 uumlberarbeitete Auflage) [Psychological experimenting on the Inter-net (Rev ed)] In B Batinic (Ed) Internet fuumlr Psychologen (pp 319-343) Goumlttingen Hogrefe

Reips U-D (2000b) The Web experiment method Advantages disad-

Figure 3 An example display from the drop-out submodule Absolute numbers of participantstaking a path are shown at the arrows percentages denote relative numbers having made the deci-sion to continue along this particular path from the previous Web page The further the indenta-tion the further the steps through the Web site Paths can be collapsed via mouse click at the ldquordquosigns and expanded at the ldquordquo signs

310 REIPS AND STIEGER

errsquos usability Changes of programming routines in Ver-sion 2 also resulted in speedier analyses

In Version 3 user administration and guest accountwere added Prefixes and suffixes allowed for almost un-limited possibilities in defining factors and levels even incomplicated log file analyses

Version 4 saw the various options in defining types oflog files Furthermore the first version of the submodulefor drop-out analysis was added Due to the complexity ofthe task to be accomplished by this submodule a MySQLdatabase (Version 32355-nt) was added

Finally Version 5 is the first version with comprehen-sive help files all available on line and in both German andEnglish Session management was refined using informa-tion about OS and browser type and identification bysearch argument was added

OutlookScientific LogAnalyzer will be further integrated with

the materials developed in the Swiss Virtual Campus pro-ject entitled ldquoExperimental Design and Web-based Ex-perimentationrdquo in order to enable learning by doing inInternet-based experimenting

Other improvements will include additional types ofanalyses and additional options for the current analysesFor example we are hoping to add more predefined log for-mats and include more options for visualizing results infuture versions of Scientific LogAnalyzer

We hope that Scientific LogAnalyzer will serve as auseful tool for Web and off-line experimenters as well asfor learning and teaching methodological concepts of Internet-based data collection and analysis in psychologyand neighboring disciplines Beyond these core applica-tions Scientific LogAnalyzer may be used universallywherever log files are produced

REFERENCES

American Psychological Association (2001) Retaining raw data(sect710) In Publication manual of the American Psychological Asso-ciation (5th ed p 342) Washington DC Author

Berendt B (2002) Using site semantics to analyze visualize and sup-port navigation Data Mining amp Knowledge Discovery 6 37-59

Berendt B amp Brenstein E (2001) Visualizing individual differ-ences in Web navigation STRATDYN a tool for analyzing navigationpatterns Behavior Research Methods Instruments amp Computers 33243-257

Birnbaum M H (2004) Human research and data collection via the In-ternet Annual Review of Psychology 55 803-832

Eichstaedt J (2001) An inaccurate-timing filter for reaction time mea-surement by JAVA applets implementing Internet-based experimentsBehavior Research Methods Instruments amp Computers 33 179-186

Frick A Baumlchtiger M T amp Reips U-D (2001) Financial incentivespersonal information and drop out in online studies In U-D Reips ampM Bosnjak (Eds) Dimensions of Internet science (pp 209-219)Lengerich Pabst

Reips U-D (2000a) Das psychologische Experimentieren im Internet(2 uumlberarbeitete Auflage) [Psychological experimenting on the Inter-net (Rev ed)] In B Batinic (Ed) Internet fuumlr Psychologen (pp 319-343) Goumlttingen Hogrefe

Reips U-D (2000b) The Web experiment method Advantages disad-

Figure 3 An example display from the drop-out submodule Absolute numbers of participantstaking a path are shown at the arrows percentages denote relative numbers having made the deci-sion to continue along this particular path from the previous Web page The further the indenta-tion the further the steps through the Web site Paths can be collapsed via mouse click at the ldquordquosigns and expanded at the ldquordquo signs

SCIENTIFIC LOGANALYZER 311

vantages and solutions In M H Birnbaum (Ed) Psychological ex-periments on the Internet (pp 89-114) San Diego Academic Press

Reips U-D (2001) The Web Experimental Psychology Lab Five yearsof data collection on the Internet Behavior Research Methods In-struments amp Computers 33 201-211

Reips U-D (2002a) Internet-based psychological experimenting Fivedos and five donrsquots Social Science Computer Review 20 241-249

Reips U-D (2002b) Standards for Internet-based experimenting Ex-perimental Psychology 49 243-256

Reips U-D (2002c) Theory and techniques of Web experimenting InB Batinic U-D Reips amp M Bosnjak (Eds) Online social sciences(pp 229-259) Seattle Hogrefe amp Huber

Reips U-D (2003) Psychologische Forschung zum und im Internet[Psychological research on and in the Internet] Psychologie in Oumlster-reich 22 19-25

Reips U-D amp Neuhaus C (2002) WEXTOR A Web-based tool forgenerating and visualizing experimental designs and procedures Be-havior Research Methods Instruments amp Computers 34 234-240

Richter T Naumann J amp Noller S (2003) LOGPAT A semi-automatic way to analyze hypertext navigation behavior Swiss Jour-nal of Psychology 62 113-120

Schmidt W C (2001) Presentation accuracy of Web animation methodsBehavior Research Methods Instruments amp Computers 33 187-200

Schwarz S amp Reips U-D (2001) CGI versus JavaScript A Web ex-periment on the reversed hindsight bias In U-D Reips amp M Bosnjak(Eds) Dimensions of Internet science (pp 75-90) Lengerich Pabst

Voracek M Stieger S amp Gindl A (2001) Online replication ofevolutionary psychology evidence Sex differences in sexual jealousyin imagined scenarios of matersquos sexual versus emotional infidelity InU-D Reips amp M Bosnjak (Eds) Dimensions of Internet science(pp 91-112) Lengerich Pabst

NOTES

1 As Reips (2000a 2000b 2002b) points out any materials pro-grammed for Internet-based experiments can always also be used in thelaboratory Therefore it seems wise to conceptualize experiments asWeb experiments whenever possible

2 Available at httpgenpsylab-wexlistunizhch3 Available at httpwwwpsychologieunizhchsowiUlfLab

WebExpPsyLabhtml4 Readers of this article may freely use Scientific LogAnalyzer for

noncommercial purposes in analyses of log files with up to 2000 linesusing ldquobrmicrdquo as login and password (the average log file from a Web ex-periment created with WEXTOR contains fewer than 1000 lines) Pricelist for noncommercial licenses with log files of 30 MB or less by num-ber of analyses (logins) 5 analyses 15 Euro 20 analyses 50 Euro 100analyses 200 Euro 1000 analyses 1500 Euro by time period 1 week50 Euro 4 weeks 150 Euro 3 months 400 Euro 1 year 1500 Euro Com-mercial and site licenses upon request

5 More accurate response times can be measured via JavaScript (egautomatically created in WEXTOR) or if highest accuracy is desiredvia a Java-based technique described in Eichstaedt (2001)

6 The GET method is a request method in httpmdashthe WWW trans-mission protocol The two most often used methods are GET and POSTThe GET method is used to ask for a specific documentmdashwhen you clickon a hyperlink GET is being used Information from a form using theGET method is appended onto the end of the action address being re-quested for example in httpwwwgenpsylabunizhchresponsethisthe answer ldquothisrdquo in an item ldquoresponserdquo was appended to the URL of theWeb page that a userrsquos action (pressing a submit button etc) leads to

7 If the analysis at hand is not an analysis of a factorial experimentldquo1rdquo should be filled in for both number of factors and number of levelsHowever it also works without any entry

8 Large Internet providers use dynamic IP addressing to balance theload of traffic on their servers A user is repeatedly assigned different IPaddresses Although it is possible to analyze data from such users theprocess is more tedious and therefore often less desirable than simplyrestricting an analysis to data from sites without dynamic addressing

9 Although it concerns timing of Web animation the article by Schmidt(2001) gives a good sense of issues with timing accuracy of absolutemeasurements depending on the programming technique used

(Manuscript received January 31 2004 revision accepted for publication April 15 2004)