Anomaly detection with Machine Learning at the LHC - CERN ...

AnomalydetectionwithMachineLearningattheLHCJudita Mamužić

IFIC / CSIC - University of Valencia 28 April 2021, I Workshop de Computing y Software de la Red Española de LHC

https://indico.cern.ch/event/1019366/

https://indico.cern.ch/event/1019366/

JuditaMamužić|AnomalydetectionwithMachineLearningattheLHC|28May2021|IWorkshopdeComputingySoftwaredelaRedEspañoladeLHC|

Contents

2

•Introduction

•Anomalydetection

•Proofofconceptexamples

•Exampleinphysicsanalyses

•Extendedandfuturetopics

•Discussion

Thisoverviewrepresentsapersonalchoiceofinterestingtopics.Asthefieldisdevelopingveryfast,newexcitingtopicsarisedaily.

IncollaborationwithRobertoRuiz,manythanksforusefulinput!

Mainresources:•TheLHCOlympics2020•DarkMachinesChallenge-UnsupervisedLearning(inpreparation)

https://arxiv.org/abs/2101.08320



Introduction:SearchesforNewPhysics

3

•NosignificantsignofnewphysicsinanyofthesearchesatATLASandCMS.•Highnumberofmodelsandsignaturesconsidered.•Withincreasingcomputingpower,exploredatainnewways(MachineLearning).


Introduction:AnomalyDetection

4

•Anomaly-anydeviationfromtheexpectation.•AnynewphysicsisadeviationfromtheStandardModelexpectation,andrepresentsananomaly.•Conventionalsearches:

•Defineasearchregiondedicatedtoa(oneoragroup)modelassumptions.•Optimisesensitivityforsignaltobackgroundratio(cut-and-count,multi-binfit,machinelearning).•DeterminebackgroundusingMonteCarloand/ordata-drivenbackgrounddeterminationusingcontrolandvalidationregions.•Lookforanexcessinthedata(statisticalinterpretation).

•Changeofparadigm:•Lookforanyanomalyasanydeviationfromthestandardmodel(modelagnostic,consideralargevarietyofmodelsinasingleanalysis).

m

No.

eve

nts

Signal

Background

MET

No.

eve

nts

SignalBackground

q

q


Introduction:AnalysesattheLHC

5

•Withincreasingcomputingpower,MachineLearningtechniquesgainingmomentum.•Machinelearningbeingappliedtoalltypesofanalyses.•Modelagnosticsearchesfornewphysicsshouldbedoneinadditiontoclassicalsearches.

Combined PerformanceTrigger


Introduction:ModelIndependence

6

•AimtohaveleastmodeldependenceonMonteCarloforbothsignalandbackground(analysisdependent).•Background:

•CanbedeterminedfromMonteCarlo•Usingcontrolregionsforscaling•UsingABCDmethod•Usingdatadrivenmethods.

•Signal:•ClassicalsearchestrainusingMCsimulations•Someanalysestrainsignalvsdata(rare)•Generalsearchesmakelittle/noassumptiononthesignal,butrelyonMCforbackgrounddetermination.•AnomalydetectionMLalgorithmsaimtohaveleastdependenceonsignalassumptionsandMonteCarlobackground.

arXiv:2010.14554




AnomalyDetection:GeneralSearches(noML)

7

•Searchfornewphysicsina(quasi)modelindependentway:

•Conventionalsearcheshaveabiasontheselectionduetothemodelassumption(SUSY,exotics)•Fillinthegapifwemissedsomeselection.

•Eventsarefirstclassifiedin704finalstatescategories,consideringelectrons,muons,b-taggedjets,non-b-taggedjets,photonsandMET(type,multiplicity,pt).•UseMCtopredictexpectedbackground.AuxiliarymeasurementsforCRandVR.•Scanondifferentvariables(e.g.Meff)andlookforstatisticallysignificantdeviation.•Calculateprobabilityoffindingthisp-valusingallbinsandallcategories,usingtoys.•ResultsconsistentwiththeSMexpectation.

arXiv:1807.07447

https://atlas.web.cern.ch/Atlas/GROUPS/PHYSICS/PAPERS/EXOT-2016-38/

https://atlas.web.cern.ch/Atlas/GROUPS/PHYSICS/PAPERS/EXOT-2016-38/


AnomalyDetection:LevelofsupervisionforML

8

Semi-supervised:•Labelsofsignalandbackgroundforsomeevents•UsesignalMCtobuildsignalsensitivity,butnotfortesting.

WeaklySupervised:•Labelsarenoisy•Comparetwodatasetswithdifferentamountsofpotentialsignal•Insteadofsignalandbackgroundusepossibly-signal-enrichedandpossibly-signal-depleted.

Unsupervised:•Nolabelinformation•Learndirectlyfrombackgrounddominateddata.

•Supervision-leveloflabelinformation(e.g.signalandbackground)giventothemachinelearningalgorithmduringtraining.

SupervisedMLmethods:•Labelsofsignalandbackgroundforallevents.

Levelofsupervision


AnomalyDetection:Semi-supervised

9

DeepEnsembleAnomalyDetection•Useslabeledsignalandbackground•Hybridsolutionusing:

•ConvolutionalNN,inputrawdata2Dimagesofunclusteredparticlesin

theevent•BDT,inputCNNandkinematicvariablesoffatjets

•~5%increaseBDTclassificationpower.•Methodgoodinidentifyingnewphysicsevents,butnotsogoodinestimatingtheirmass.•Methodnotsomodelindependent.

η − ϕ

arXiv:2101.08320




AnomalyDetection:Weakly-supervised

10

Classificationwithoutlabels(CWoLa):•Insteadofusingsignalandbackground,usemixedsampleswithdifferentproportionofsignal.•AlleventsinmixedsampleM0labeled0,allsamplesinmixedsampleM1labeled1.•TraintheclassifiertodistinguishM0fromM1,thenitwillbealsooptimaltodistinguishsignalfrombackground.•Assumptions:

•Mixedsamplesshouldbestatisticallyidenticalasidefromdifferentclassproportion.

•Physicsanalysisperformedusingasearchfordi-jetresonance.•Trainclassifiersdirectlyondata.

arXiv:2010.14554

M0 M1




PhysicsAnalyses:ATLAS

11

•ATLASdi-jetresonancesearchusingweaksupervision.•FeatureforMLaremassesoffirsttwojets,bumphunt.•Genericsearch:A->BC(tau-leptons,b-quarks,top-quarks,vectorbosons,Higgsbosonandasymmetricdecays),smalltrialfactor.•6SRswithsidebands,NNdifferentforeachSR•NNabletodetecttheinjectedsignal.•Strongestlimitsfordi-jetevents.

q

qABC

Phys.Rev.Lett.125(2020)13

Low-efficiency(signal-like)regionsoftheNNarelocalisedneartheinjectedsignal.

https://inspirehep.net/literature/1794466

https://inspirehep.net/literature/1794466


AnomalyDetection:Unsupervised

12

Autoencoder•Performanomalydetectioninhighdimensionalparameterspace.•DeepNeuralNetworkwithatasktolearntheidentityofamap.•Enforcescompressioninthelatentspace.•TrainingperformedonQCD,reconstructionerrorlargeforsignalordifferentsample.•Usesomemetrictocompareinputandoutput(reconstructionerror),andperformclassification.

arXiv:1808.08992

https://arxiv.org/pdf/1808.08992.pdf

https://arxiv.org/pdf/1808.08992.pdf

JuditaMamužić|AnomalydetectionwithMachineLearningattheLHC|28May2021|IWorkshopdeComputingySoftwaredelaRedEspañoladeLHC|13

UnsupervisedComparison:DarkMachinesChallenge•Inspectunsupervisedanomalydetectionalgorithms.•ResultswillbepublishedsoononarXiv.

DarkMachineschallenge:•UsepublicdatasetatarXiv:2002.12220section23.•Developandtrainthealgorithms(classificationdoneonevent-by-eventbasis,nodensities)todefineananomalyscore,withinDarkMachineseffort.•Consideringanumberofunsupervisedlearningmethods.•Firststudyallmethodsusingknownnewphysicssignals.•Comparemethodsagainsta“secret”dataset.

•Aimtoiteratewiththepublic,invitepublictotraintheiralgorithmsandcompare.•Convergeonanoptimal,globalmodel-independentunsupervisedmethodtobeappliedone.g.ATLAS/CMSdata.

StrongcontributiontoopendataMCgenerationandcalculationusingArtemisacomputingfacility.




UnsupervisedComparison:DarkMachinesChallenge

14

Publicdatasetforthechallenge:•https://zenodo.org/record/3685861•SamplesincludeMadGraph+Pythia8atLOeventgeneration,andDelphes(ATLAScard)forsimulation,preselectionusingobjectthatcouldbetriggered.•O(1B)eventsinCSVformat•Includes:

•Backgroundandsignal(Z’,stop,squarks,gluinos(RPCandRPV),charginos,variousmasspoints)•Datagroupedin4channelswithdifferentpreselection:

1)HT>600GeV,MET>200GeV,MET/HT>0.2,atleast4jetswithpT>200,50,50,50GeV2.a)MET>50GeVandatleast3leptonswithpT>15GeV2.b)MET>50GeV,HT>50GeVandatleast2leptonswithpT>15GeV3)HT>600GeV,MET>100GeV

SM Processes

https://zenodo.org/record/3685861

https://zenodo.org/record/3685861


UnsupervisedComparison:PreliminaryResults#1,#2

Inspectthesignal:==>Notallanomaliesareanomalous,AreaUndertheCurve(AUC)around0.5forchargino-neutralinoproductionmodels,whileveryhighforgluinopairproduction.

Inspecttheamountoftrainingdata(channel/colour):==>Systematicunderperformanceforalgorithmsthattrainonsmallamountofdata,e.g.channel2a-verytightselection(resultsintightcutsandlowstatistics)

ROCcurve

ϵS 1

1-ϵ B

1 Excellent

Poor


UnsupervisedComparison:PreliminaryResults#3,#4,#5Inspectbestperformingmethodbasedondifferentscore:==>Differentoptimalmethodfordifferentchoiceofscore(e.g.overallbestmedianvsabovethreshold).

InspectAUC(AreaUnderROCCurve):==>AUCisnotthebestmeasurewhenconsideringlowsignalefficiencyregioninROC.

Inspecttightbackgroundcuts:==>Nomethodwasabletofindanysignalwithtightbackgroundcuts.

==>Clearpotentialofstudiedalgorithmsandpossibilitytostudyhybridapproaches.Noone-size-fits-all,methodsneedtobestudied.


AnomalyDetection:Trigger

17

•DeployautoencoderalgorithmsatL1triggeronFPGAtotriggerBSMsignatures.•arXiv:1811.10276,IMLslides•Autoencodersverygoodinfindingdifferentnewphysicsscenarios.•Useencoderpartoftheautoencoder,reducetoverysmalllatentspace.Notrunningthedecoder.•Addfigureofmeritinthelatentspace,deviationsfromexpectationwillbethetriggerfornewphysics.•Usesimple𝜒2functiontodeterminethelevelofdeviationfromexpectation(smallfornoise,largefornewphysics).

MLinterfacetoFPGAinPython FPGA


https://indico.cern.ch/event/1025534/contributions/4306062/attachments/2234199/3786641/IML_Meeting.pdf


https://indico.cern.ch/event/1025534/contributions/4306062/attachments/2234199/3786641/IML_Meeting.pdf

https://fastmachinelearning.org/hls4ml/


QuantumTechnology:QuantumAnnealing

18

Nature550(2017)7676

•ProofofprinciplestudyfortheHiggsbosondiscoveryusingquantumannealing.•ConsideringsignalH->𝜸𝜸andSMbackgroundprocesseswithdecaystotwophotons.•AreaunderROCcurve.Solidlinescorrespondtoquantum(green)orsimulated(blue)annealing,anddottedlinestotheDNN(red)orXGB(cyan).•DNNandXGBhaveanadvantageforlargetrainingdatasets,whileannealer-trainednetworksperformbetterforsmalltrainingdatasets.•QuantumcomputingcanfindtheHiggsboson.•Anomalydetectionusingquantumtechnologyshouldbeexploredfurther.

https://www.nature.com/articles/nature24047

https://www.nature.com/articles/nature24047


Summary

19

•Anomalydetectioncomplementaryto“conventional”newphysicssearches.•Proofofconceptforsupervised,weakly-supervisedandunsupervisedlearning.•Comparisonofunsupervisedalgorithms(inpreparation).•Physicsanalysisexamples.•Furthertopicsforanomalydetection.


Discussion

20

1.Whatshouldopendatasamplesinclude?

2.Whatareadvantagesofsupervisedvsunsupervisedapproach?

3.Howgeneralcantheanomalydetectionalgorithmsbe,dependenceontheinputfeatures?

4.WhatimprovementsareneededfordataformatforMLalgorithms?

5.WhatshouldweconsiderforHL-LHCandfuturecolliders?

6.HowshouldtheMLmodelsbesystematicallycollected?

7.Whatotherapplicationsforanomalydetectioncouldbemade(advantagesandlimitations)?

Anomaly detection with Machine Learning at the LHC - CERN ...

Documents

Transcript of Anomaly detection with Machine Learning at the LHC - CERN ...