University of Pennsylvania University of Pennsylvania

ScholarlyCommons ScholarlyCommons

Departmental Papers (CIS) Department of Computer & Information Science

November 2003

The Penn-Lehman Automated Trading Project The Penn-Lehman Automated Trading Project

Michael J. Kearns University of Pennsylvania, mkearns@cis.upenn.edu

Luis Ortiz University of Pennsylvania

Follow this and additional works at: https://repository.upenn.edu/cis_papers

Recommended Citation Recommended Citation Michael J. Kearns and Luis Ortiz, "The Penn-Lehman Automated Trading Project", . November 2003.

Copyright 2003 IEEE. Reprinted from IEEE Intelligent Systems, Volume 18, Issue 6, November-December 2003, pages 22-31. Publisher URL: http://ieeexplore.ieee.org/xpl/tocresult.jsp?isNumber=27968&puNumber=5254

This material is posted here with permission of the IEEE. Such permission of the IEEE does not in any way imply IEEE endorsement of any of the University of Pennsylvania's products or services. Internal or personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution must be obtained from the IEEE by writing to pubs-permissions@ieee.org. By choosing to view this document, you agree to all provisions of the copyright laws protecting it.

This paper is posted at ScholarlyCommons. https://repository.upenn.edu/cis_papers/35 For more information, please contact repository@pobox.upenn.edu.

The Penn-Lehman Automated Trading Project The Penn-Lehman Automated Trading Project

Abstract Abstract The Penn-Lehman Automated Trading Project is a broad investigation of algorithms and strategies for automated trading in financial markets. The PLAT Project’s centerpiece is the Penn Exchange Simulator (PXS), a software simulator for automated stock trading that merges automated client orders for shares with real-world, real-time order data. PXS automatically computes client profits and losses, volumes traded, simulator and external prices, and other quantities of interest. To test the effectiveness of PXS and of various trading strategies, we’ve held three formal competitions between automated clients.

Comments Comments Copyright 2003 IEEE. Reprinted from IEEE Intelligent Systems, Volume 18, Issue 6, November-December 2003, pages 22-31. Publisher URL: http://ieeexplore.ieee.org/xpl/tocresult.jsp?isNumber=27968&puNumber=5254

This material is posted here with permission of the IEEE. Such permission of the IEEE does not in any way imply IEEE endorsement of any of the University of Pennsylvania's products or services. Internal or personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution must be obtained from the IEEE by writing to pubs-permissions@ieee.org. By choosing to view this document, you agree to all provisions of the copyright laws protecting it.

This journal article is available at ScholarlyCommons: https://repository.upenn.edu/cis_papers/35

22 1094-7167/03/$17.00 © 2003 IEEE IEEE INTELLIGENT SYSTEMSPublished by the IEEE Computer Society

A g e n t s a n d M a r k e t s

The Penn-LehmanAutomated TradingProjectMichael Kearns and Luis Ortiz, University of Pennsylvania

The Penn-Lehman Automated Trading Project is a broad investigation of algo-

rithms and strategies for automated trading in financial markets. The PLAT Proj-

ect’s centerpiece is the Penn Exchange Simulator (PXS), a software simulator for auto-

mated stock trading that merges automated client orders for shares with real-world,

real-time order data. PXS automatically computesclient profits and losses, volumes traded, simulatorand external prices, and other quantities of interest.To test the effectiveness of PXS and of various trad-ing strategies, we’ve held three formal competitionsbetween automated clients.

PLAT backgroundThe PLAT Project (www.cis.upenn.edu/~mkearns/

projects/plat.html) has several underlying motiva-tions. From a research perspective, we’re among thegrowing number of AI and computer scienceresearchers with an interest in all forms of e-com-merce, computational markets, algorithmic mecha-nism design and electronic auctions, and related top-ics. In addition to a burgeoning theoretical literature,1

this line of research has a growing platform and sys-tems component. The best example of such a systemis perhaps the popular, successful Trading AgentCompetition (TAC)2–4 (see also http://auction2.eecs.umich.edu/researchreport.html), which has fo-cused primarily on multicommodity auction simu-lations. So, one primary motivation for the PLATProject is to contribute to this line of systems andcompetition work in automated markets. In thisregard, a distinguishing characteristic of the projectis its investigation of a real and widely studied classof automated markets and strategies. Indeed, WallStreet has many quantitative traders who do for a liv-ing what PLAT Project participants do in the safetyof the PXS environment. In the same vein, we’re alsointerested in designing challenging, realistic com-petitions in automated trading in financial markets,

using PXS as the testbed.We also actively use PXS as a platform for devel-

oping novel, principled automated trading strategies(clients). The real-data, real-time nature of PXSlets us examine computationally intensive, high-frequency, high-volume trading strategies (althoughthis last property always presents the challenges ofestimating the market impact—the effect on prices).We’re particularly interested in developing clientsthat make predictive use of limit order book data,including those using statistical modeling andmachine learning. We hope that, over time, the proj-ect will generate a library of clients with varying fea-tures (trading strategy, volume, frequency, and so on)that can serve to create realistic simulations withknown properties.

In addition, the project has a major educational com-ponent. Aside from the project staff team of five (whooversee PXS maintenance and development, managethe Island data, and run the competitions), over 30 stu-dents are developing automated trading strategies forPXS, and they regularly participate in the competitions.Many of the students are in joint programs with theUniversity of Pennsylvania’s Computer and Informa-tion Science Department and the Wharton School (ofbusiness). These students must undertake a year-longsenior research project; many have chosen to do so asparticipants on the PLAT Project developing novelautomated-trading strategies. Several students are fromother universities.

Finally, the PLAT Project is an educational andinstitutional partnership between the University ofPennsylvania and Lehman Brothers’ Proprietary

The PLAT Project has

developed a trading

simulation that merges

automated clients with

real-time, real-world

stock market data.

This simulation has

been used for three

competitions.

Trading Group in New York City, a group ofWall Street professionals who actively designand implement a rich, sophisticated collec-tion of automated-trading strategies. OurLehman colleagues have provided invaluablescientific guidance on the technical designof PXS and the competitions and have actedas mentors to the participating students.

Market microstructureTo understand the PLAT Project, as well

as PXS, you need to understand some detailsof the underlying mechanics of financialmarkets and exchanges. These computationaland transactional details are sometimesbroadly called market microstructure. Wefocus on the market microstructure of NAS-DAQ exchanges and stocks, both for speci-ficity’s sake and because it’s most directlyrelated to PXS. However, many of the sameelements appear in other exchanges, includ-ing the New York Stock Exchange.

Unlike the NYSE, NASDAQ is entirely elec-tronic. All orders, whether an algorithm or aperson generates them, are sent to NASDAQ viaan electronic interface and order-routing sys-tem. Also, a computer executes all matchesbetween buyers and sellers. Another distinc-tion between the NYSE and NASDAQ is thedegree of distribution. While all transactionsin an NYSE stock must eventually be clearedthrough a single firm or individual (known asthe stock’s specialist), NASDAQ lets many firmsand individuals provide markets in NASDAQ

stocks. This environment led to the advent ofElectronic Crossing Networks (also calledElectronic Communication Networks), whichare firms providing essentially independent,competing markets for NASDAQ stocks. A typ-ical NASDAQ stock trader will track prices andactivity simultaneously in multiple ECNs. Heor she might either break large orders up overseveral of them or prefer certain ECNs for cer-tain types of transactions, depending on dif-fering fee structures. Recently, some mergersand institutional efforts toward consolidationhave occurred, but several large, independentECNs still exist, including Island (www.island.com), Instinet (www.instinet.com), andArchipelago (www.archipelago.com).

A fundamental distinction in stock trad-ing is that between a limit order and a mar-ket order. Suppose we wish to purchase1,000 shares of Microsoft (whose NASDAQ

ticker symbol is MSFT) stock. In a limitorder, we specify not only the desired vol-ume but also the desired price. Suppose thatMSFT is trading at roughly $24.07 a share

(see Figure 1), but we only want the 1,000shares at $24.04 a share or lower. We cansubmit a limit order with this specification,and our order will be automatically placed ina queue called the buy order book, which isordered by price, with the highest offeredunexecuted buy price at the top (often calledthe bid). If multiple limit orders have thesame price, they are ordered by arrival time(with older orders higher in the book). In theexample in Figure 1, our order would beplaced immediately after the extant order for5,503 shares at $24.04. Although we offerthe same price, this order arrived before ours.Similarly, a sell order book for sell limitorders (for instance, we might want to sell500 shares of MSFT at $24.10 or higher) ismaintained, this time with the lowest sellprice offered (often called the ask).

So, the order books are automaticallysorted from the most competitive limit ordersat the top (high buy prices and low sellprices) down to less competitive limit orders.The bid and ask together are sometimescalled the inside market, and the differencebetween them is the spread. By definition,the order books always consist exclusivelyof unexecuted orders—they’re queues oforders waiting for the price to move in theirdirection.

How then do orders get executed? By def-inition, when a market order arrives, it’s im-mediately matched with the most competi-tive limit orders on the opposing book. So, amarket order to buy 2,000 shares will bematched with enough volume on the sellorder book to fill the 2,000 shares. Forinstance, for the example in Figure 1, suchan order would be filled by the two limit sellorders for 500 shares at $24.069, the 500shares at $24.07, the 200 shares at $24.08,and then 300 of the 1,981 shares at $24.09.The remaining 1,681 shares of this last limitorder would remain as the new top of the selllimit order book. A limit buy order with aprice much higher than the current ask iseffectively a market order. Likewise, a limitsell order with a price much lower than thecurrent bid is effectively a market order. Forthis reason, some ECNs (including Island)don’t offer a separate market order mecha-nism. But conceptually, the two types havean important difference: a limit order is guar-anteed price (if executed) but not execution,whereas a market order is guaranteed execu-tion but not price (because the books mightchange before the order arrives at theexchange and executes).

In this setting, every market or limit orderarrives atomically and instantaneously—orders arrive in a strict temporal sequence,and two orders can never arrive simultane-ously. This gives rise to the definition of theexchange’s last price, which is simply thelast price at which the exchange executed anorder. This quantity is what people usuallymean when they casually refer to a stock’s(ticker) price. The last price might changemore slowly than the order books, especiallyin less liquid stocks.

The market microstructure we’ve describedhas existed since the dawn of financial mar-kets, with some variations in the details (forinstance, NYSE specialists have some flex-ibility in how and when they executematches between buyers and sellers). What’smore recent is the automation of this processin markets such as NASDAQ. The fundamen-tal role of ECNs (and PXS) is the comput-erized maintenance of buy and sell orderbooks in the offered stocks, automated orderexecution, and various other related func-tionalities (such as withdrawing or changingunexecuted orders or checking a previouslyplaced order’s status).

Even more recent than the automation ofmarket microstructure is the publication ofreal-time order book data. The publication ofsuch data presents intriguing opportunities (it

NOVEMBER/DECEMBER 2003 www.computer.org/intelligent 23

MSFT

BUY ORDERS SELL ORDERS

SHARES PRICE500

6,0005,000

1001,100

1005,000

2003,2941,0003,000

1005,5032,1002,800

24.062024.061024.060024.060024.055024.050024.050024.050024.050024.050024.043024.040024.040024.030024.0300

SHARES

LAST MATCH TODAY'S ACTIVITY

Price 24.0700Time 14:57:07.72

Orders 52,983Volume 10,243,212

PRICE500500500200

1,981412

3,000500100

2,8005,0001,0005,000

4001,000

24.069024.069024.070024.080024.090024.090024.098024.100024.120024.140024.140024.140024.150024.160024.1700

Figure 1. A typical order book for an electronic crossing network.

has also spawned a relatively new literatureattempting to come to both a theoretical andempirical understanding of order book behav-ior5,6). This is because the limit order bookscan be viewed as an expression of market sen-timent and, more prosaically, might providestrategic guidance for order placement. Indeed,Wall Street traders commonly examine thelimit order books carefully and place theirorders accordingly (for instance, by “steppingin front” of an existing limit order in the bookby just a fractional amount).

The availability of order book data makesPXS possible. More specifically, Island is amajor ECN for NASDAQ stocks, accountingfor approximately one of every seven NAS-DAQ trades. It’s thus among the most impor-tant providers of liquidity in NASDAQ, andlike many ECNs, it’s a technologicallysophisticated company. Besides acceptingorders arriving through a variety of standardWall Street brokerage order-routing systems,it provides an API for automated orderplacement and management.

The Penn Exchange SimulatorPXS is basically an experimental ECN that

merges limit order data from two sources:actively connected trading clients and limitorders from Island. We obtain the Island datain real time from Island’s live Web-basedBookViewer, which shows the top 15 limitorders (price and volume) in the buy and sellorder books. (Because BookViewer showsonly the top 15 orders rather than the entirebooks, accurate maintenance of the PXSbooks involves some technical challenges. Wediscuss these challenges in an expanded ver-sion of this article, available at www.cis.upenn.edu/~mkearns/papers/plat.pdf.)

PXS simulations run in one of two modes.Live mode updates the Island data in realtime from BookViewer approximately everythree seconds. Historical mode requires thatthe requested day of Island data has alreadybeen archived. Live simulations provide themost realistic merging of the Island and PXSmarkets because they’re faithful to the timingdetails of activity on Island. Historical sim-ulations are considerably faster because PXScan process each update of Island data at itsown internal processing rate.

Every major Wall Street brokerage hasmultiple platforms and data sources on whichto test new trading strategies; these are oftencalled backtesting environments. To date,most or possibly all backtesting environmentsemploy either price information alone or, in

some cases, price and inside-market data. Thisis almost certainly because order book data hasbeen widely available for NASDAQ only veryrecently (and is still not yet available in manymajor markets).

One major advantage of an order-basedsimulation platform such as PXS is that itobviates the need for a fill or executionmodel. When only price information is avail-able for a stock, and a proposed strategywishes to place a limit order away from thecurrent price, any simulation must make adecision about whether such an order willexecute at a future time based on only thatinformation. Obviously, if the price neverreaches the limit order price, the order willnever get filled. But, if the price does crossthe limit order price, it might or might nothave executed in the real market, dependingon the depth of demand in the books at thatprice. A typical fill model might probabilis-tically execute the limit order on the basis ofthe historical data and the order’s volume.

Just as in real exchanges, PXS fills a limitorder only if an opposing order matches it atsome point in the book-based simulation.Rather than modeling the depth at differentprice levels in the books, we have the booksthemselves. To our knowledge, PXS is thefirst simulation tool employing real-worldorder book data in this manner.

We implemented PXS in C on Unix andLinux platforms. When it’s invoked, it takesthese arguments:

• The four-letter ticker symbol of the Island-traded NASDAQ stock for which to run asimulation (such as MSFT).

• The port number over which PXS willcommunicate with trading clients.

• For historical simulations, the date for whichthe simulation should run. If this argumentis omitted, PXS executes a live simulationusing the current day’s Island data.

• For the trading day being simulated(whether live or historical), the simula-tion’s start and stop times.

For example, the command

pxs -p 9800 -n MSFT -h 04292003093000 -e 160000

starts an execution of PXS on Island MSFTorder book data from 29 April 2003 at 9:30am and terminates with an Island update closeto 4 pm on that day. This execution wouldthen accept connections from trading clientsover port 9800. The same command without

the -h argument immediately starts the simu-lation, using the current Island MSFT data.

The client APIOnce a PXS simulation is under way, any

number of automated trading clients mayjoin at any time by connecting via the desig-nated port. To each connecting client, PXSassigns a client identifier.

The PXS client API contains a rich set ofdata structures and functions that permit theplacement and withdrawal of orders in thePXS market, and the computation of certainmarket information. Technically speaking,the functionality we discuss here is splitbetween PXS and a client shell process insidewhich participants implement their tradingstrategies. Generally, PXS computes quanti-ties of global interest to the simulation, whilethe client shell computes more client-specificquantities. For simplicity, we’ll blur this dis-tinction and simply refer to PXS.

The most basic client API functions are fororder placement and management:

• buyOrder(p, v) places a limit buy order atprice p for v shares. It returns an identify-ing number for the order.

• sellOrder(p, v) places a limit sell order atprice p for v shares. It returns an identify-ing number for the order.

• withdraw(o) withdraws the order with theidentifier o from the PXS order books. Thiswill fail if the order already has executed.

Some functions compute these agent-specific quantities of interest:

• The clients’current cash and share holdings• The clients’ current profit and loss, under

either the PXS or Island last-price valua-tion for share holdings

• The volume of orders outstanding in thePXS buy and sell books for the client

Some functions provide information aboutthe overall PXS market state:

• The last price of the PXS or Island markets.• The current time according to PXS. For live

simulations, this will always be close to theactual (wall clock) time. For historical sim-ulations, it will be the time stamp of themost recently processed Island update.

• The total volume of shares and number oforders in the PXS buy and sell order books.

• The total volume and number of orders thatPXS has already matched in the simulation.

24 www.computer.org/intelligent IEEE INTELLIGENT SYSTEMS

A g e n t s a n d M a r k e t s

• Various statistics of the PXS order books,including volume-weighted average prices.

More generally, clients can receive a copyof the entire PXS order book data structure.This copy shows the price and volume of everylimit order in the book, with indications ofwhich orders came from Island and which fromPXS clients. So, PXS provides clients with alevel of internal market visibility that matchesthat offered on real ECNs such as Island. Thislets PXS clients exploit market microstructuredata at different levels of sophistication.

The execution engineAt the heart of PXS is the execution en-

gine, which

• Maintains the PXS order books• Integrates Island client buy and sell limit

orders into these books• Executes matching orders of PXS clients• Computes PXS clients’ share position,

cash holdings, and profit and loss

A detailed description of the PXS execu-tion engine is beyond this article’s scope; formore information, see the expanded articleat www.cis.upenn.edu/~mkearns/papers/plat.pdf. Here we simply give an overview andexplain some of the engine’s subtle aspects.

At a high level, the execution engine con-sists of five steps that execute repeatedlythroughout a simulation:

1. Update the PXS buy and sell order bookswith any new orders detected on Island.

2. “Clean” the PXS books after theupdates, to address our partial observ-ability of the Island books.

3. Execute the PXS orders that arematched by executions on Island be-tween updates.

4. Execute matches between buy and sellorders in the PXS books; update the PXSlast price; and update client share posi-tions, cash holdings, and profit and losses.

5. Insert the newly arrived client ordersinto the PXS books.

The valuation of holdingsIn any PXS simulation, all PXS clients

begin with no cash and no shares of the stockbeing traded. At any time, a PXS client canbuy or sell shares, regardless of its currentshare and cash holdings. So, clients can sellmore shares than they hold (selling short) orbuy shares without cash. Share and cash hold-ings might thus be either positive or negative.PXS maintains these holdings for each client.

The valuation (profit and loss) of a client’sholdings at any time is the sum of its cashbalance (positive or negative) and its shareposition (positive or negative) times thestock’s current (last) price. Thus, PSX makesan important “infinite liquidity” assumption:at any moment, any client could return itsshare position to zero instantaneously byplacing all held shares on the market andreceive the current price for all of them. Forlarge share positions, this assumption isclearly unrealistic, because the immediateattempt to buy or sell large numbers of shareswill move the price unfavorably. Further-

NOVEMBER/DECEMBER 2003 www.computer.org/intelligent 25

Call for Paperswww.computer.org/intelligent/author.htm

Submission Guidelines: Submissions should be 3,000 to 7,500 words (counting a standard fig-ure or table as 250 words) and should follow the magazine’s style and presentation guidelines.References should be limited to 10 citations.

To Submit a Manuscript: To submit a manuscript for peer-reviewed consideration, please access the IEEE Computer Society Web-based system, Manuscript Central, at http://cs-ieee.manuscriptcentral.com/index.html.

Dependability characterizes the set of properties of a system thatlets us trust that the system will behave according to its re-quirements. Dependable systems are deployed in critical appli-

cations such as defense, transportation, energy, process control, andfinance. When systems are designed to withstand malicious behav-ior, we sometimes refer to them as survivable, to emphasize that theymust be dependable even when under attack. To assure these sys-tems’ dependability, they’re often built using very conservative prin-ciples that allow extensive validation throughout the life cycle. Forexample, you’d expect such systems to use well-tried and reliable(albeit slow) hardware or conservative programming practices such asno memory allocation from the heap to ensure that the computationcan be predictably within the bounds of what the hardware can sup-port. Other methods include building systems with enough security,diversity, and redundancy to be able to withstand random or mali-cious stresses; building systems using self-checking and self-repairingtechnology; or combinations of approaches.

Is it then ever possible to build a dependable system using agent technol-ogy? Can the (by its nature) nondeterministic and evolving agent com-putation paradigm be predictable enough to deploy in time-critical or

other critical applications? Is the concept of emergent properties funda-mentally at odds with the requirements for dependable software? Even ifwe can build a dependable agent system, would we ever be able to pro-duce a convincing argument to that effect? Paradoxically, the very natureof agent systems might supply enough diversity and redundancy to satis-fy dependability constraints. This special issue will explore the interplaybetween the need for guarantees and the need for flexible, agent-basedcomputations.

Topics of interest include but are not limited to

■ Specifying agent system requirements■ Formal modeling of agent systems■ Model-based approaches to building dependable agent systems■ Complexity analysis of agent computations■ Testing of agent systems■ Verification of agent systems■ Assurance of agent systems through risk

analysis and decision management■ Survivability arguments and dependability

cases for agent systems

S p e c i a l I s s u e o n t h e D e p e n d a b i l i t y o f A g e n t S y s t e m s

S u b m i s s i o n s d u e 9 A p r. 2 0 0 4

Guest EditorsMark Greaves, DARPA

Bob Laddaga, MITVictoria Stavridou, SRI

more, there’s the choice of whether to use theIsland or PXS last price to compute the val-uation of share positions. PXS computes thevaluations both ways. We further discuss val-uation in the next section.

The competitionsAlthough PXS can perform simulations

for any stock traded on Island (whichincludes all of NASDAQ), all three competi-tions involved Microsoft stock. Besides thestudents from the University of Pennsylva-nia, two teams from the University of Texasat Austin and one from Carnegie Mellon Uni-versity have participated.

An issue common to all the competitionswas risk management. Just as on Wall Street,it’s important to encourage teams to developstrategies that intelligently balance their riskand return. In the project’s setting, risk gener-ally comes in the form of large share positions(long or short), which, as we mentionedbefore, are extremely vulnerable to unfavor-able changes in the share price. So far, theproject has focused exclusively on intradaytrading; the competitions “cash out” eachclient at each day’s end, with no positions heldovernight. From this perspective, an idealstrategy would end each day with large cashholdings and a zero or small share position.

We wanted to prevent the results frombeing dominated by strategies that simplyplace large bets in the form of excessive sharepositions. So, all the competitions had thefirm rule that, during any trading day, aclient’s share position must always remainwithin a window of ±100,000 shares. Viola-tion of this was grounds for disqualification.We forgave minor infractions, but such a dis-missal occurred in one of the early competi-tions. This limit is a crude and easily verifi-able way of ensuring that no client succeedssimply by taking much larger positions thanall the others. Because (at the time of thecompetitions) MSFT traded at roughly $25a share, the share position limit effectivelymeant that clients would never have morethan approximately $2.5 million of virtualcapital at risk.

Another issue common to all the compe-titions was how to value a client’s share posi-tion at the end of each trading day. When thePXS market closes, each client will gener-ally have a positive or negative cash position(the balance of the cash it spent on purchas-ing shares throughout the day and the cash itreceived on sales). It will also have a long orshort balance of MSFT shares. To convert

this portfolio to a cash valuation, we mustassign a value to the share position.

For large share positions, simply takingthe share position and multiplying it by thestock’s last price is unrealistically optimistic.This is because the placement of a marketorder for a large number of shares might eatdeep into the limit order books, resulting inprogressively less favorable prices. This isthe well-known, difficult problem of assess-ing the market impact of large orders. It’salso why accurate measurement of strategyperformance from historical data is difficultand why brokerages almost always breaklarge orders into small increments over time.

The presence of other trading clients in thePXS market can act as an important simula-tor of market impact (a topic we’ll return to).However, for the purposes of day-end valua-tion, we do indeed make a somewhat unreal-istic “infinite liquidity” assumption and valueevery client’s share position at the last price.The assumption is somewhat justified by theaforementioned share position limit on clientsand because we’re trading a stock (MSFT)with high Island liquidity. Other valuationmethods are possible, however, and we mightexplore them in future competitions.

The three competitions took place duringNovember to December 2002, February toMarch 2003, and April to May 2003. In eachcompetition, we divided the client strategiesinto pools, both for the sake of populationdiversity and to reduce each simulation’scomputational load. The first two competi-tions placed no restrictions on clients otherthan the share position limit, and we deter-mined the winners strictly by their cumula-tive profitability. We won’t discuss theseearly competitions’ results in detail here, butone chief lesson we learned was that addi-tional rules or scoring criteria were desirableto encourage increased realism among clients.This led to the third competition, dubbed thePlatinum Platter Competition.

PPC 2003 divided the 14 entrant strategiesinto a Blue and a Red pool, each with sevenclients. The division was somewhat arbitrarybut took into account coarse preliminaryexperiments to ensure that each pool had areasonable amount of client liquidity.

The competition occurred on each of the10 trading days of the weeks of 28 April and5 May. Each day’s PXS simulation ran from9:30 am to 4 pm, the normal NASDAQ tradinghours. (Although Island conducts after-hourstrading, the liquidity tends to be considerablylower than during normal exchange hours.)

While daily and overall profitability naturallyremained important components of clientevaluation, we used an interesting and con-siderably richer set of scoring criteria (seeFigure 2) to encourage client realism andgood trading practices.

The strategiesTable 1 briefly describes each PPC 2003

client and indicates its pool and its overallfinal pool ranking according to the scoringcriteria in Figure 2. Many participants inves-tigated variants of the Static Order BookImbalance (SOBI) client that we providedfor them. This client attempts to predict pricemovement on the basis of volume and priceimbalances between the buy and sell orderbooks.

One theme that emerged was how tradingclients exploited order book information. Notonly does PXS itself use Island and internalorder books to conduct its simulations, it alsomakes its books available to trading clientsin real time, thus permitting strategies thatattempt to derive predictive or other valuefrom this information. The interest in, andchallenge of, consistently and profitablyexploiting order book data in our project mir-rors similar interest on Wall Street. Theclients CBR-SOBI and CReaTiv heavilyused order book behavior and statistics. Twoclients applied machine learning techniquesto feature vectors derived from order books;CBR-SOBI used case-based reasoning, andCIA used the classification learning algo-rithm called boosting. Overall, nine of the 14entrants exploited order book data; however,the centrality of this data to the strategy var-ied considerably.

Several clients, such as MoneyFlow,OBCrossover, and OBBreakout, imple-mented variations on more traditional tech-nical trading strategies. OBMM, DAMM-STAT, and RapidMM implemented someform of market-making strategy, which seeksto profit from price volatility rather thanoverall movement.

Overall, the entrants formed two diversepools of interesting strategies, varying fromthe extremely simple (such as Contrarian) tothe rather complex. They exhibited a rangeof trading styles that includes both thosecommonly found on Wall Street, such asmarket-making and certain technical tradingmethods, and rather new (and untested)methods, such as those relying heavily onorder book data. We have deliberatelyencouraged this diversity throughout the proj-

26 www.computer.org/intelligent IEEE INTELLIGENT SYSTEMS

A g e n t s a n d M a r k e t s

ect, believing it increases the simulations’interest and realism. Of course, PPC 2003’sresults depend strongly on the particular setof clients and even on the specific divisioninto pools, as you’ll see.

The resultsTable 2 summarizes each client’s overall

performance. The table sorts clients by theirpool and their final ranking according to thescoring criteria of Figure 2. In addition tolisting the overall point totals that determinedthe final standings, the table lists subtotalsfor each point category.

The PPC 2003 winners were CBR-SOBI(Blue pool) and DAMM-STAT (Red pool).It’s striking how differently each strategymanaged to emerge as the victor in its respec-tive pool. CBR-SOBI was also among the topperformers in terms of raw profitability (dis-cussed shortly) and earned most of its pointsin categories directly related to positive earn-ings. DAMM-STAT was barely profitableoverall but succeeded by consistent adher-ence to good trading practices. It managedto earn the maximum of 20 points for risk

NOVEMBER/DECEMBER 2003 www.computer.org/intelligent 27

Table 1. 2003 Platinum Platter Competition clients.

Client Description Performance

CBR-SOBI Case-based reasoning applied to the Static Order Book First in the Blue pool; statistically significant profitabilityImbalance strategy’s parameters

MoneyFlow A predictive strategy using money flow (price movement times Second in the Blue poolvolume traded) as a trend indicator

OBMM A market-maker that positions orders in front of the nth orders Third in the Blue poolon both books

CReaTiv Capitalization on Real Time Volatility—SOBI modified by recent Fourth in the Blue poolvolatility

OBCrossover Exponential Moving Average crossover strategy moderated by Tied for fifth in the Blue poolconfirmation of order book quartile volume-weighted average prices

OBBreakout A breakout strategy applied to trend lines on the volume-weighted Tied for fifth in the Blue poolaverage prices of buy and sell books

RaSTa Resistance and Support Trading Agent—computes support and Seventh in the Blue poolresistance levels on the basis of peaks in the order book volume

DAMM-STAT A Mixture of a Dynamically Adjusted Market-Maker that calibrates First in the Red pool; stellar position managementby recent volatility and a trend-based predictive strategy

Contrarian Sells on rising prices, buys on falling prices Second in the Red pool

OBSigma Trades on the basis of relative spreads in the buy and sell books, Third in the Red poolinterpreting a small standard deviation as a sign of confidence

OBVol A simple predictive strategy using total volumes in buy and Fourth in the Red pool; highest Sharpe ratio and statistically sell books significant profitability

RapidMM A market-maker with rapid revision of quotes based on the Fifth in the Red poolcurrent inside market

CIA Central Intelligent Agent—a predictive strategy applying boosting Sixth in the Red poolto order book snapshots

SimpleTrend A simple trend prediction strategy Seventh in the Red pool; statistically significant negative earnings

Figure 2. The client scoring criteria for the 2003 Platinum Platter Competition.

Criteria emphasizing profitability• Daily profit and loss. On a daily basis, award 3 points to each client whose end-of-day P&L

is highest in its seven-client pool; 2 points to the second highest; 1 point to the thirdhighest. Maximum possible award: 30 (3 × 10) points.

• Overall consistency of P&L. A one-time award of 15 points goes to any client that has apositive cumulative P&L over the competition’s 10 trading days and that ends with anegative daily P&L on three trading days at most. Maximum possible award: 15 points.

Criteria emphasizing robustness, with weak profitability prerequisites• Daily intraday position reversals. On a daily basis, award 2 points to any client that finishes

with a positive P&L for the trading day and that held share positions in excess of 10,000shares in both the long and short direction at some point during the day. Maximumpossible award: 20 (2 × 10) points.

• Robustness to market variation. Award 5 points to each client that has a positive P&L onany pair of trading days in which the share price rose overall (open to close) on one day ofthe pair and fell on the other. For each additional such pair, award an additional 5 points.Maximum possible award: 25 (5 × 5) points, if exactly 5 up days and 5 down days for thestock occur during the competition.

Criteria emphasizing good trading practices, with no profitability prerequisites• Daily risk saturation. On a daily basis, award 2 points to each client that achieves a share

position in excess of 50,000 shares (long or short) at some point during the trading day,without exceeding the maximum-allowed share position of 100,000 shares. Alternatively,award 2 points to clients whose total matched volume of shares for the trading day exceeds1/14 (one-half of the per-client average of 1/7) of the total matched volume of all clients.Maximum possible award: 20 (2 × 10) points.

• Daily position unwinding. On a daily basis, any client that’s awarded that day’s risk saturationpoints can earn an additional 2 points by ending the trading day with a share position offewer than 5,000 shares (long or short). Maximum possible award: 20 (2 × 10) points.

saturation and earned 14 points for unwind-ing (that is, returning to zero) its position onseven of the 10 days. The next best perfor-mance in unwinding in either pool earnedonly six points. Overall, we were quitepleased with the balance between profitabil-ity, consistency, and good trading practicesthat the scoring criteria brought out amongthe better performers. Although many of thelower-ranked clients had positive earnings,they all consistently failed in one or more ofthe basic practices or behaviors we encour-aged.

In terms of profitability, 11 of the 14clients ended with overall positive cumula-tive earnings for the 10-day competition. Ofcourse, just as on Wall Street, we must con-sider the question of both the statistical sig-nificance of earnings and the trade-offbetween risk and return. A common measureof the latter is known as the Sharpe ratio,which is the empirical daily average ofreturns divided by the standard deviation.The ideal is to have a large Sharpe ratio—consistently high earnings with a very smallspread in the returns. Among the 14 clients,two (CBR-SOBI and OBVol) achieved note-

worthy Sharpe ratios, and one (SimpleTrend)actually displayed a strongly negative Sharperatio. Despite OBVol’s strong monetary per-formance, it fared worse by the scoring cri-teria. Its poor score was due primarily to aconsistent failure to saturate the allowed riskcompared to the higher-ranked clients, whichall received all 20 points in this category. (Forsimplicity, the Sharpe ratios given simplytook the 10-day profit and loss figures dividedby their standard deviation, as opposed to themore standard annualized values.)

Although the Sharpe ratio accounts for therisk-return trade-off, it’s insensitive to theamount of data available, so it’s generally anunreliable indicator of statistical significance.We thus also provide 95 percent confidenceintervals around each client’s average. Bythis measure, two clients (CBR-SOBI andOBVol again) had confidence intervals lyingexclusively in the region of positive earnings.They thus pass this standard test for statisti-cal significance at the 0.05 level on just 10days of data. By the same token, we canassert with high confidence that SimpleTrendis a money-losing strategy.

Overall, the PPC 2003 scoring criteria

seemed to effectively balance profitabilityconsiderations with our other interests. Moreprecisely, the correlation coefficient betweenthe clients’ point totals and their profit andloss totals was 0.41; so, profitability was con-siderably important but not dominant. Thisisn’t surprising, considering the motivationbehind the criteria’s design.

AnalysisRecall that a main motivation behind the

design of PXS and the competition is to cre-ate a hybrid market in which an incomingstream of real market data can influence or“correct” a diverse market of virtual clients.We’re thus naturally interested in examiningthe extent to which the internal PXS marketcorrelated with or deviated from the Islandmarket. Overall, the competition seems tohave quite successfully balanced the virtualand external markets’ influences. A typicalplot of the PXS and Island last prices gener-ally shows that over the entire trading day,the two prices were extremely close, withoccasional small short-term deviations (wediscuss some notable exceptions to this later).

More quantitatively, Figure 3a illustrates

28 www.computer.org/intelligent IEEE INTELLIGENT SYSTEMS

A g e n t s a n d M a r k e t s

Table 2. PPC 2003 results.

Points for

95% 10-dayAvg. confidence Shape

Strategy Pool Rank P&L ($) interval ratio

CBR-SOBI Blue 1 74 9 18 2 10 15 20 4,187 ± 3,733 0.70

MoneyFlow Blue 2 69 15 20 0 4 15 15 2,007 ± 15,692 0.08

OBMM Blue 3 46 8 20 0 8 0 10 258 ± 7,909 0.02

CReaTiv Blue 4 42 7 20 0 10 0 5 (2,410) ± 6,770 (0.22)

OBCrossover Blue 5 33 6 6 0 6 0 15 3,242 ± 4,220 0.45

OBBreakout Blue 5 33 10 18 0 0 0 5 3,680 ± 7,963 0.29

RaSTa Blue 7 21 5 2 0 4 0 10 1,182 ± 2,441 0.30

DAMM-STAT Red 1 65 6 20 14 10 0 15 685 ± 5,195 0.08

Contrarian Red 2 55 6 20 2 12 0 15 2,022 ± 3,658 0.34

OBSigma Red 3 54 8 20 6 10 0 10 1,649 ± 2,382 0.43

OBVol Red 4 53 14 0 0 4 15 20 4,037 ± 1,900 1.32

RapidMM Red 5 50 10 20 0 10 0 10 3,649 ± 9,121 0.25

CIA Red 6 30 13 12 0 0 0 5 (1,451) ± 9,822 (0.09)

SimpleTrend Red 7 27 3 20 2 2 0 0 (24,467) ± 17,974 (0.84)

Tota

l poi

nts

Daily

prof

it & lo

ssDa

ily ri

sk sa

tura

tion

Daily

posit

ion u

nwin

ding

Daily

intra

day p

ositi

on re

vers

al

Cons

isten

cy of

P&L

Robu

stnes

s to m

arke

t var

iatio

n

the differences between the Island and PXSlast prices for the Blue pool (results for theRed pool are similar), aggregated over all 10competition days. The greatest mass in thesehistograms lies close to 0 cents, and virtuallyall the mass is contained in a margin of ±3cents. So, the Island data’s external influenceseems to have indeed caused a close, but notperfect, correlation between the two markets.The histograms’ symmetry around 0 indi-cates that PXS deviations from the Islandprice are unbiased, as you might expect.

Figures 3b and 3c illustrate the differencesbetween the PXS and Island bids and asks(the inside market). Here we again see a closecorrespondence, with the mass entirely con-tained in a margin of ±3 cents. Now, how-ever, the histograms are asymmetric: the PXSbid was much more frequently above theIsland bid, and the PXS ask much more fre-quently below the Island ask. In other words,the PXS inside market was generally tighterthan Island’s, a sign of greater liquidity andcompetition in the PXS market.

We now examine more client-specificbehaviors. Perhaps the easiest way to visu-alize individual clients’ overall trading be-havior is to examine their share positionsthroughout the trading day. Figure 4 containsa graph for each client in the Blue pool, witheach graph containing 10 curves showing thenumber of shares (long or short) the clientheld as a function of time over the 10 com-petition days. The curves’ crowded naturemakes tracking a client’s precise position onany single day difficult. However, we caninfer from these curves a great deal of macro-scopic information about the client’s fre-quency and volume of trading, the biastoward long or short positions, and manyother properties.

For instance, in the Blue pool (we omitdetailed Red pool analysis), we see that thevictorious client CBR-SOBI tended to exe-

cute relatively large transactions at relativelylong intervals. This contrasts with clientssuch as OBMM, whose positions increase ordecrease in much smaller increments butmuch more frequently. We can also see CBR-SOBI’s frequent reversal of position within atrading day, and an overall balance betweenlong and short selling. OBBreakout, on theother hand, seems incapable of anything butoverall short positions. The dark mass ofheavy trading activity by CReaTiv betweenapproximately 10 and 11 am each day isprobably at least partially, if not primarily,responsible for the heavy increases in over-all PXS volume that occurred about that timein the Blue pool on several trading days.

The Red pool’s SimpleTrend deservesspecial mention because its behavior nicelydemonstrates that an internal pool of diverse,aggressive virtual clients can act as a proxyfor the market impact that excessively largeorders typically have in the real world.Unlike the other clients in either pool, Sim-pleTrend often engaged in sudden transac-tions (both buying and selling) for close to100,000 shares, as Figure 5 shows. Theselarge deals proved disastrous for Simple-Trend and profitable for its trading partners,because the huge orders ate deep into theopposing book and left SimpleTrend withprogressively worse prices. (Recall that Sim-pleTrend had the worst P&L performanceand actually passes a statistical significancetest for unprofitability.) Despite the previ-ously discussed statistical closeness of theIsland and PXS markets, SimpleTrend’sbehavior caused the Red pool last price toinstantaneously deviate dramatically fromIsland’s several times. For example, Figure5 shows where large SimpleTrend ordersdirectly cause sudden changes in the PXSprice at three distinct moments during theday. Figure 5 illustrates a typical day forSimpleTrend, where several precipitous

tumbles deeper into the red are directlyaligned in time with the large positionchanges.

Our small pool of virtual clients probablydeviates from typical Wall Street traders(even automated ones) in the large volumeof trading they engage in. Nonetheless, wewere pleased overall with the PXS market’srealism, liquidity, and tightness; the externalIsland data’s healthy influence; the correc-tive effects of the other virtual clients in thecase of SimpleTrend; and the client popula-tion’s diversity.

The Penn-Lehman Automated TradingProject is a work in progress; we’re

actively planning extensions to all aspects ofthe project. On the systems and platformside, we’re enhancing the client API, improv-ing the speed and robustness of the executionengine’s internal algorithms, and designing aWeb-based GUI that will permit remote par-ticipants to use PXS and join our competi-tions. On the strategy side, we’re using PXSto investigate a range of order-book-basedtrading algorithms.

We actively solicit external participationin the project from researchers in both aca-demic and industrial settings. If you’re inter-ested, contact Michael Kearns at mkearns@cis.upenn.edu.

AcknowledgmentsBerk Kapicioglu and Byoungjoon Kim initially

developed and tested the Penn Exchange Simula-tor, and Narayanan Mahesh, Nick Montfort, andRashid Tuweiq made numerous contributions tothe system. The Proprietary Trading Group atLehman Brothers in New York City gave generousscientific, professional, and financial support tothe project. Michael Kearns especially thanks hisLehman colleagues Michael Bleich, Michael Bos,

(a) (c)(b)

00.10.20.30.40.5

-0.03 -0.02 -0.01 0 0.01 0.02 0.03

Price difference

00.10.20.30.40.5

-0.03 -0.02 -0.01 0 0.01 0.02 0.03

Price difference

00.10.20.30.40.5

-0.03 -0.02 -0.01 0 0.01 0.02 0.03

Price difference

Figure 3. The differences between the PXS and Island markets for the Blue pool, aggregated over the 10 competition days: (a) lastprices; (b) bids; (c) asks.

NOVEMBER/DECEMBER 2003 www.computer.org/intelligent 29

–100,000–80,000–60,000–40,000–20,000

020,00040,00060,00080,000

100,000

–100,000–80,000–60,000–40,000–20,000

020,00040,00060,00080,000

100,000

9 10 11 12 13 14 15 16

CBR-SOBI

9 10 11 12 13 14 15 16

MoneyFlow

–100,000–80,000–60,000–40,000–20,000

020,00040,00060,00080,000

100,000

9 10 11 12 13 14 15 16

OBMM

–100,000

–50,000

0

50,000

100,000

150,000

9 10 11 12 13 14 15 16

CReaTiv

–150,000

–100,000

–50,000

0

50,000

100,000

150,000

9 10 11 12 13 14 15 16

OBCrossover

–100,000–90,000–80,000–70,000–60,000–50,000–40,000–30,000–20,000–10,000

010,000

9 10 11 12 13 14 15 16

OBBreakout

–60,000–50,000–40,000–30,000–20,000–10,000

010,00020,00030,00040,00050,000

9 10 11 12 13 14 15 16

RaSTa

Time

No. o

f sha

res

No. o

f sha

res

No. o

f sha

res

No. o

f sha

res

No. o

f sha

res

No. o

f sha

res

No. o

f sha

res

Time

Time

Time Time

Time

Time

Figure 4. Blue pool client share positions over time (9:30 am to 4 pm). Each graph shows the number of shares a client owns or owesas a function of time, with each curve corresponding to a day of the competition.

A g e n t s a n d M a r k e t s

30 www.computer.org/intelligent IEEE INTELLIGENT SYSTEMS

Andy Ellner, Amy Papandreou, Colin Rust, andMark Sanborn for their time and knowledge. Wealso thank all the participants in the Penn-LehmanAutomated Trading Project who developed PXSclients for their hard work and patience. We’re par-ticularly grateful for the suggestions and com-ments of Yuriy Nevmyvaka of Carnegie MellonUniversity and Peter Stone of the University ofTexas at Austin.

References

1. Proc. 4th ACM Conf. Electronic Commerce,ACM Press, 2003.

2. P. Stone et al., “ATTac-2000: An AdaptiveAutonomous Bidding Agent,” J. ArtificialIntelligence Research, vol. 15, Sept. 2001, pp.189–206.

3. M. Wellman et al., “The 2001 Trading AgentCompetition,” Electronic Markets, vol. 13,no. 1, 2003, pp. 4–12.

4. M. Wellman et al., “Designing the MarketGame for a Trading Agent Competition,”IEEE Internet Computing, vol. 5, no. 2,Mar./Apr. 2001, pp. 43–51.

5. J. Bouchaud, M. Mezard, and M. Potters, “Sta-tistical Properties of Stock Order Books:Empirical Results and Models,” QuantitativeFinance, vol. 2, no. 4,Aug. 2002, pp. 251–256.

6. J. Hasbrouck and G. Saar, Limit Orders andVolatility in a Hybrid Market: The IslandECN, working paper, Dept. of Finance, NewYork Univ., 2002.

NOVEMBER/DECEMBER 2003 www.computer.org/intelligent 31

–40,000

–35,000

–30,000

–25,000

–20,000

–15,000

–10,000

–5,000

0

5,000

10,000

9 10 11 12 13 14 15 16Do

llars

Time

Island present valueSimulator present value

–100,000

–80,000

–60,000

–40,000

–20,000

0

20,000

40,000

60,000

80,000

100,000

9 10 11 12 13 14 15 16

Num

ber o

f sha

res

Time

Share position

25.725.825.926.026.126.226.326.426.5

9 10 11 12 13 14 15 16

Dolla

rs

Time

Island last priceSimulator last price

(a)

(b)

(c)

Figure 5. The client SimpleTrends’ behavior for 9 May: (a) profit and loss; (b) shareposition; (c) share price. Three trades exceeding 60,000 shares each (a sell shortly after 10 am, a sell around 1 pm, and a buy around 3 pm) are each accompanied by sharp lossesbecause the large orders eat deep into the PXS order books. Although the last prices areindistinguishably close for almost the entire day, the first and last of SimpleTrend’s largetrades cause instantaneous, large deviations of the PXS price in the corresponding direction. At the large trade around 1 pm, the PXS books apparently had enough depthnear the inside market to absorb the trade and prevent a deviation between the Islandand PXS prices.

T h e A u t h o r sMichael Kearns is a professor of computer andinformation science at the University of Penn-sylvania, where he has a joint appointment in theWharton School and is codirector of the Insti-tute for Research in Cognitive Science. Hisresearch interests include AI and machine learn-ing, computational aspects of game theory andeconomics, and computational finance. Contacthim at the Dept. of Computer and InformationScience, Univ. of Pennsylvania, 507 Levine Hall,3330 Walnut St., Philadelphia, PA 19104;mkearns@cis.upenn.edu; www.cis. upenn.edu/~mkearns.

Luis Ortiz is a postdoctoral researcher at the Uni-versity of Pennsylvania’s Department of Com-puter and Information Science. His researchfocuses on AI, computational game theory andeconomics, and machine learning. He previouslywas a postdoctoral fellow and researcher at thesame university’s Institute for Research in Cog-nitive Science. He received his PhD from BrownUniversity’s Department of Computer Science.Contact him at the Dept. of Computer & Infor-mation Science, Univ. of Pennsylvania, LevineHall, 3330 Walnut St., Philadelphia, PA 19104-6389; leortiz@linc.cis.upenn.edu.