Supplement5EffectiveTestingonPortfolio123Testing

(backtest,rankperformancetest,simulation)

Portfolio123VirtualStrategyDesignClassByMarcGerstein

ACommonButPotentiallyDangerousUse

• EmpiricalStudy– testingtodetermine,fromscratch,whetheranideaissound– IfIcreatethisratio,(x+y)/zanduseitasabasistorankstocks,willitwork?• Ifso,Inextaskwhethermytest“robust”accordingtostatistical“bestpractices”• Ifyes– Great!Icanandwilluseitinamodel.

2

TheChallengeinFinancialTesting

• Wedon’t“interpolate”– Interpolationinvolvesmodelingrelationshipsobservedwithininapopulation(orwithinaproperlyconstructedsample),inordertoputthoserelationshipstousewithinthesamepopulation

• We“extrapolate”– Thisinvolvesmodelingrelationshipsobservedwithinapopulationorsampleinordertoputthoserelationshipstouseinacompletelydifferentpopulation

3

Example:Interpolatevs.Extrapolate• Interpolation– Medicalresearch– Ifsuch-as-suchchemicalcompoundisingestedbyhumanshavingsuch-and-suchdisease,thediseasewilldisappearandwithoutadversesideeffects.

– Ifarelationshipisdemonstrated,andifitisrobust(whichiswherecompaniesfightitoutwiththeFDA),thenweknowthecompoundcanbesafelyandeffectivelyadministeredamonghumans

• Extrapolation– AWholeDifferentBallgame– Thatstudywouldnotsupportuseofthesamecompoundtocuredogs

– Dogsandhumansaresimilarinsomewaysbutdifferentinotherways(example:chocolateisapopulartreatamonghumans,butitcankilldogs)

4

ApproachingExtrapolation

• Onewaytoextrapolateistodoadifferentstudyoneachpossiblepopulation– E.g.,studythesamechemicalcompoundagainondogs

• Anotherapproachistofocustheinitialstudyoncharacteristicsofthefirstpopulationthatwecanreasonablyassumewillcarryoverintootherpopulations

5

TheTwoPopulationsonPortfolio123

• Population1– Thepast,thepointintimedatabase(“insample”)– Wecanwith100%certaintyidentifyandmodelrelationshipsthatarevalidwithinthispopulation

– Everybodydoesit• Population2- Thefuture,livemoney(outofsample)– Becausewedon’t(can’t)havedatafromthefuture,we’reforcedtousethesecondapproachtoextrapolation– workingwithconceptswecanreasonablyexpecttocarryoverfromPopulation1(thepast)toPopulation2(thefuture)

6

PracticalReality• Whathappensifweaimourtestingentirelyat

interpolation?• Itmightwork

– Thiswillhappenifthefutureresemblesthepast– Thiscanhappenforashorttime,itcanhappenoverthecourse

ofmanyyears,oritcancomeandgoatwhattheuserperceivestoberandomintervalsorbasedonhot-and-coldfads

• Itmightnotwork– Thiswillhappenifthefuturediffersfromthepastinamanner

thatisrelevanttothemodel• Thebettertheinterpolation– themoretightlytunedthemodelistothepast– themoreopportunitiesthereareforthefuturetodiffer

• Thisiswhy5-stock90%Alphasims aremorelikelytodisappointthan20-stock15%alphamodels

• Robustnessisirrelevant;allrobustnessmeansisthatyoudidasuperiorjobinterpolating

– Theuserwillexperiencethisasdisappointingout-of-sampleperformance

7

HowtomodelforExtrapolation• TestideasyouReasonablyexpecttobecapableofoperatinginboththepastandthefuture

• WilliamO’Shaughnessy,inWhatWorksonWallStreet, echoesthiswithawarningthatactuallyprovidesahintastohowwemightaccomplishit:– “Ifthereisnosoundtheoretical,economic,orintuitive,commonsensereasonfortherelationship,it'smostlikelyachanceoccurrence.”• Tortoriello followsthissameprincipalandworksthesameway

• Thekeyisn’tinwhattheytest– it’swhattheydon’tbothertotest(thingstheyknowaheadoftimelacktheeconomic,intuitiveorcommonsensebasis)!

8

TheSolution• Workwithandtestideasyoureasonablyexpecttoimpactstockprices– Don’tfocusontrendsorwhat’shappeningnoworwhathappenedinthepast

– Focusonideasyouknow,aheadoftimeandwithouttesting,shouldwork

• Thisrequiresthattherebeafoundationalideaabouthowstocksarepricedandwhytheymove– Goodnews:Suchafoundationexistsandiseasytounderstand

– Badnews:Theideaisbrutallydifficulttoapplywhichiswhywedon’tallbecomebigwinnersoneverytrade

– Testingisusedtoprovidefeedbackontheefficacyofourimplementationideas

9

TheSingle-BiggestError• Peoplewhotestimproperly,whointerpolate,arguethatstocksbehaveirrationallyandthatthereisnosuchthingas“afoundationalideaabouthowstocksarepricedandwhytheymove”

• Theyarewrong,verybadlywrong• Suchanideadoesexistandiswellknown• Thekeyforpropertesting,testingthatcouldbeusedtoextrapolate(whichiswhatwemustdo)isunderstandingthefoundationalidea,howwecanapplyittotherealworldandhowwecanusetheadaptationsasthebasisforourmodels

10

TheFoundation:theDividendDiscountModel(DDM)

• Ittellsusthatthefairpriceofastockisequaltothepresentvalueofallthemoneyweexpecttoreceiveasaresultofowningit:– Dividends– Proceedsfromaneventualsale

• THISABSOLUTELYPOSITIVELYMUSTBETRUE!– Toargueagainstitwouldmeanadvocatinginfavorofsomething

likeanimmediateexchangeof$100for$5;nobodywoulddothat

• ImplementationProblem:Howdoweknowwhenwe’llsellandwhatwe’llget– Answer:Mathematiciansrestatetheideaassumingwe’llholdto

infinity,thusallowingustofocusondividendsonly

11

TheDDMFormula

• P=D/(R– G)• P=fairprice• D=dividend• R=requiredrateofreturn• G=expected(throughinfinity)dividendgrowthrate

– WecanalsouseaformulaforR(theCapitalAssetPricingModelor“CAPM”)• R=RF+(B*(RM– RF))

– RF=RiskFree(treasury)rate– RM=expectedreturnoftheequitymarketasawhole– B=Beta,theindicatorofstock’svolatilityrelativetothemarket

12

PracticalChallenges

• Wecan’tliterallyuseDDM– Manystocksdon’tpaydividends

• TheideaherewouldbetodoaDDMvaluationasofsomeforecastedfuturedatewhendividendsareexpectedtobeintroducedandthenapplytheanswertotodayusinga“presentvalue”computation

– Themainproblemisthatit’stoodifficultgeneratecredibleinputs• TheworstproblemiswithG,mustbeaninfinitegrowthratethatpresumesaverymaturecompany(i.e.wehavetomakesureGislessthanRlestwewindupwithanegativefairprice)

13

CopingMechanism• “Itisbettertobevaguelyrightthanpreciselywrong.”– Craveth Read(Britishlogician)

• Thatworksforussincewe’redealingwiththefutureand,thus,havenohopeofbeingpreciselyright...Sowemightaswellaimforvaguelyright(orasProf.Asawath Damodaran says,lesswrongtheneverybodyelse)– Datamined“interpolated”sims,ontheotherhand,unwittinglyaimtobeandusuallywindupbeing“preciselywrong.”

14

ImportantApproximation#1

• SinceweknowwecannotpreciselyimplementDDM,wedothenextbestthing

• Weadapt,adjust,approximate,etc.usingideaswereasonablybelievewillpushuscloserinthedirectionoffairDDMvaluation– Forexample,weconsidervaluebecauseastockthatitmorecloselyalignedwithcurrentearningsismorelikelytobereasonablyalignedwiththestreamoffuturedividends,sincedividendscomefromearnings

15

ImportantApproximation#2

• Wedon’tliterallyhavetorestrictourselvestotherelevantsetofterms:D,G,R(includingRF,RMandB)

• Wecan,instead,substituteotherideasthatareplausiblyrelatedtotheseterms.– Forexample,wecanconsiderfinancialstrengthsincethebalancesheetimpactsthestabilityofprofitabilitywhichimpactsthestabilityofthestock(B)whichinturnimpactsR

16

MakingSenseoftheApproximations• Theapproximations,packagedtogether,makeupthebody

of“fundamentalanalysis”• Thefundamentalratiosetc.thatarewidelyuseddidnot

becomesojustfortheheckofitorbecauseso-and-sosaidso

• Fundamentalratiosgettobewidelyusedbecauseofthewaytheylogicallyconnectustothegoal;alignmentwiththeideal(albeit,sadly,incalculable)DDMideal

• “Empirical”dataminersmighthit,throughluck,onfactorsorformulasthatmeetsuchcriteria.Butoftentheir20-20hindsightleadsthemtorelyonformulasthatdonotqualifyonthisbasis– Useoffactorsthatjusthappenedtohaveworkedwithout

regardtowhethertheySHOULDwork(asperthesecriteria)iswhatcausesthesupersims tofallapartwhenappliedtothefuture

17

Example:ValueandPE

• SubstituteE(EPS)forD–Wecangetawaywiththisbecauseweseethatthemarketbehavesconsistentlywithanexpectationthatallcompanyearningsaccruedirectlytoshareholderandthatthelatterimplicitlychoosetoreinvestalloralotofitbackintothecompany

• Sonow,P=D/(R-G)becomesP=E/(R-G)• Algebraicreshuffling:P/E=1/(R-G)

18

We’veUnlockedtheKeytoValueModeling

• P/E=1/(R-G)– BecauseGisanegativenumberinthedenominator,weknowthatasGgoesup,so,toodoesP/E• TheinspirationforthePEGratio!ButPEGisn’tthewholestory...

– BecauseRisapositivenumberinthedenominator,weknowthatasRgoesup,P/Egoesdown• Interestrates(RF)isamajorcomponentofR;that’swhyfallinginterestratesaregoodforstocks(thepushP/Eup)andviceversa

• B(beta)isalsoamajorcomponentofR,theonlycompany-specificcomponent;that’swhylessriskystocksareoftencriticizedasbeingexpensiveandwhydeepvaluescanbehighrisk– that’showtheyaresupposedtobepriced

19

It’snotjustP/E• Price/Sales

• P/S=Margin/(R-G)– Strongermarginsandbettersalesgrowthpushvaluationsup– SrelatestoEandErelatestoD

• Price/CashFlow• P/CF(orP/FCF)=1/(R-G)• AssumesasimilarrelationshipbetweenCF,FCF,etc.andDasweassumedwithE

• Price/Book• P/B=ROE/(R-G)• ThisisbecauseE=ROE*B

20

ThisisWhatFundamentalsareallabout

• ExtensionsofDDMinrecentslideshavealreadyopenedthedoortoquiteafewfundamentals

• Wekeepgoing– TurnoverisacomponentofROEasismargin– Debt,liquidityratios,etc.relatetoriskandhenceBandthereforeR

– ROE,ROI,andROAmareindicatorsofpersistentprofitgrowthcapabilityandhencerelatetoBandR(andofcourseG)

21

It’smorethanjustfundamentals• Becausewe’relookingintothefuture,wehavetobecreativeinourquestforclues– Sentiment-relatedandtechnicalfactororformulabecomesvaluableifitpassesthistest:thenumberisthewayitisbecause“they”areassuming....• “They”couldbeanalystsand/orthemarketasawhole• Thisisanimportantsourceofinformationbecausewhenwelookintothefuture,weneedtobeopentoqualitativejudgmentsthatcannotbeexpressedintermsofhistoricalfundamentals

22

It’salsoalotmorethan“fairprice”• RobertSchillerandCharlesLee

– P!=V• (Wecan’tassumePriceisequaltoValue)

– P=V+N• PriceisequaltoValueplusNoise

• Ncanbeunderstoodinwayswecanmodel– Nmovesbasedonsentiment(fads,trends),andalsobasedon

informationavailabilityorlackthereofthatmakesithardoreasytocrediblyvalueastock

– Wal Martisprettyclear-cut;itcanbevalued,soNissmall– Biotechmicrocapsarebrutallydifficulttovaluesotheirprices

areallNallthetime• SentimentandTechnicalanalysishelpusgetahandleon

theebbsandflowsofN

23

MakingitWork• There’salotwecanusetohelpusidentifypotentiallygoodstocks

• But...– Weneedtorememberwe’redealingwiththefuture,sowehavetocomeupwithrelationshipswethinkwillbesustainable

– Wealsoneedtoexpressourideasinwaysacomputercanreadandprocess• Wecan’tsimplysaywewantcompanieswithgoodgrowthprospects

• Wehavetospecificallydefine“goodgrowthprospects”inp123language– whichcanincorporatehistoricalfundamentals,sentiment,and/ortechnicals

24

BuildingandTestingaValueModel

• WecanstartwithaValueRankingsystemthatincludesoneormoreratios– sortedassuminglowerisbetter

• Wecouldtrystoppingthereanditmayworkintest,butweknowwe’remissingthings

• Sowe’lladdfactorsrelatingtoG(higherisbetter)andothersthatinfluencefutureB(lowerisbetter)

25

ModelDesign• Whatisintherankingsystemandwhatisinthescreen?– AssumeweusearankingsystemforValue– Assumeweuseascreen(buyrules)toprequalifytheto-be-rankeduniverseonthebasisofcompanieswithgoodgrowthprospectsand/orhighquality(wecananddoassumethathigherqualitycompaniesarelikelytohavemorestableearningsand,hence,lesserfutureBetas)

• Howeverweconfigurethemodel,welookforstockswithratiosthatarelowerthanwethinktheywouldbeifthemarketiscorrectlyunderstandingGandR– Whatwe’rereallydoinghereisinformationarbitrage

26

WhatWeTest- 1

• HavewedoneagoodjobspecifyingG• Sales5YCGr%?• Sales3YCGr%?• (Sales%ChgTTM-Sales5YCGr%)/abs(Sales5YCGr%)?• EPSitems?• LTGrthMean ?• (NextFYEPSMean- NextFYEPS4WkAgo)/NextFYEPS4WkAgo?• AvgRec?• Etc.,etc.,etc.

27

WhatweTest- 2• HavewedoneagoodjobspecifyingValue?

• PEExclXorTTM ?– Arewegettingtoomanybadnumbersduetoinclusionofspecialitems?

• Pr2CashFlTTM?– Itdoesn’tallowforthecapitalspendingequivalentofdepreciation:Isthisaproblem?

• Pr2BookQ?– Withsomanyunquantifiableassetsnowadays,doesthisratiostillwork?

• Etc.,etc.,etc.

28

WhatweTest- 3

• HowshouldwespecifyRisk/Quality?• DoesROEdoit,orarewebetteroffwithROIorROA?• Is5Ygood,orTTM?• ShouldwecomparethemtogetanindicationofROEtrend?– CanROEserveasaproxyforgrowth– thereisarelationship

• Aredebtratiosuseful,orperhapsinterestcoverageandshouldthesebeindustrycomparisons?• Earningsqualityisanindicatorofpersistenceandrisk

29

WhatweTest- 4

• Arewebeingimpactedby“specificationerror?”– WeexpectthatastockwithaPEof12isbettervaluedthanastockwithidenticalGand

RcharacteristicsbutaPEof20– Butmaybenot!

• Whatifthe12PEiscalculatedonthebasisofEPSthatistemporarilyinflatedbyaone-timenevertobeseenagaingainandthatanEPSmorereflectiveofthecompany’sunderlyingfundamentalswouldleadtoaPEof35

• Thissortofthinghappensallthetime– Don’texpectXor tobailyouout;accountingstandardssettershaveprettymuchdefinedXor outof

existence

• Soforus,andcontrarytogeneralstatisticalbestpractices,morefactorscanbebetterthanfewer,aswescreenouthighprobabilitysourcesofspecificationerrorordiversifythemaway(with,say,fivevalueratiosinsteadofone)

– It’simportanttohuntforspecificationerrorbyrunningyourmodelandsamplingpassingcompanies,toseeiftheysatisfythespiritofthelaw

– Specificationerrorisahugesourceofdisappointinglive-moneyresults• It’simpracticaltoexpecttocompletelyeliminateit,butwecanandshouldtrytodothebestwecantoreduceit

30

WhatweTest- Summary• Therearecountlessthingswecantestbuteverythingisrelatedtoa

commongoal:– Haveweeffectivelyspecifiedourideas?

• HavewesucceededintranslatingourideasintoP123language?• Havewecapturedrelationshipslikelytopersistintothefuture?• Havewedealtwithspecificationerrorsasbestwecan?

• Becausethequestionsweaskaresoopen-ended,it’sveryriskytoimplementamodelwithouttestingtoseeifourspecificationisplausible

• It’simportanttounderstandwearenotonanempiricaltreasurehuntforwhatworks.– Thechoiceofwhattotestorwhattodiscardevenwithouttestis

critical.– Wecanonlytestideaswehavereasonstoexpectwillwork,andlook

atthetesttoassesstheefficacyofthespecification– Dataminersgobadtestingthingstheyshouldn’ttest,thekindsof

thingsO'ShaughnessyandTortoriello wouldrejectwithoutevenbotheringtotest

31

WhatwecaninferfromTesting

• Asuccessfultestisonethatallowsustoassumethattherelationshipswespecifiedhaveagoodprobabilityofidentifyingstockswiththepotentialtooutperformthecrowd(howeverwedefine– benchmark– it)– That’swhatweshouldhavebeenlookingatinourtestresults;excessreturn,alpha,etc.

32

WhatweCannotAssumefromaTest

• Wecanneverassumethatthestockwillgotoaparticularprice– Marketactionisthesinglebiggestcomponentofanystockprice,somuchofwhatastockwilldodependsonthemarket

• BecauseP123doesn’thavetoolstopredictfuturemarketprices,wecannotpredicttargetprices– It’snotclearanybodyhasanythingthathelpswiththis– Thisisanimportantreasonwhyanalystsdon’ttake“TargetPrices”seriouslyandwhymanysitesdon’tprovidethem• Theymight,however,beusedtoconstructasentimentindicator

33