Generalized dynamic cognitive hierarchy models for strategic driving behavior

Atrisha Sarkar 1, Kate Larson 1, Krzysztof Czarnecki 2

1David R Cheriton School of Computer Science 2Department of Electrical and Computer EngineeringUniversity of Waterloo, Ontario, Canada

atrisha.sarkar, kate.larson, k2czarne@uwaterloo.ca

Abstract

While there has been an increasing focus on the use of gametheoretic models for autonomous driving, empirical evidenceshows that there are still open questions around dealing withthe challenges of common knowledge assumptions as wellas modeling bounded rationality. To address some of thesepractical challenges, we develop a framework of general-ized dynamic cognitive hierarchy for both modelling natu-ralistic human driving behavior as well as behavior plan-ning for autonomous vehicles (AV). This framework is builtupon a rich model of level-0 behavior through the use of au-tomata strategies, an interpretable notion of bounded ratio-nality through safety and maneuver satisficing, and a robustresponse for planning. Based on evaluation on two large nat-uralistic datasets as well as simulation of critical traffic sce-narios, we show that i) automata strategies are well suited forlevel-0 behavior in a dynamic level-k framework, and ii) theproposed robust response to a heterogeneous population ofstrategic and non-strategic reasoners can be an effective ap-proach for game theoretic planning in AV.

1 IntroductionOne of many challenges of autonomous vehicles (AV) in anurban setting is the safe handling of other human road userswho show complex and varied driving behaviors. As AVsintegrate into human traffic, there has been a move from apredict-and-plan approach of behavior planning to a morestrategic approach where the problem of behavior planningof an AV is set up as a non-zero sum game between roadusers and the AV; and such models have shown efficacy insimulation of different traffic scenarios (Fisac et al. 2019;Li et al. 2019; Tian et al. 2018). However, when game the-oretic models are evaluated on naturalistic human drivingdata, studies have shown that human driving behavior is di-verse, and therefore developing a unifying framework thatcan both model the diversity of human driving behavior aswell as plan a response from the perspective of an AV ischallenging (Sarkar and Czarnecki 2021; Sun et al. 2020).

The first challenge is dealing with common knowledge(Geanakoplos 1992) assumptions. Whether in the case ofNash equilibiria based models (rational agents assuming ev-eryone else is a rational agent) (Geiger and Straehle 2021;

Copyright © 2022, Association for the Advancement of ArtificialIntelligence (www.aaai.org). All rights reserved.

Schwarting et al. 2019), in Stackelberg equilibrium basedmodels (common understanding of the leader-follower rela-tionship) (Fisac et al. 2019), or level-k model (the level ofreasoning of AVs and humans), there has to be a consen-sus between an AV planner and other road users on the typeof reasoning everyone is engaging in. How to reconcile thisassumption with the evidence that human drivers engage indifferent types of reasoning processes is a key question.

Whereas the challenge of common knowledge deals withquestions around how agents model other agents, the sec-ond challenge is dealing with bounded rational agents, i.e.modelling sub-optimal behavior in their own response. Withthe general understanding that human drivers are boundedrational agents, the Quantal level-k model (QLk) has beenwidely proposed as a model of strategic planning for AV(Li et al. 2018, 2019; Tian et al. 2018). One way the QLkmodel deals with bounded rationality is by mapping eachagent into a hierarchy (k) of bounded cognitive reasoning.In this model, the choice of level-0 model becomes critical,since the behavior of every agent in the hierarchy dependson the assumption about the behavior of level-0 agents. Themain models proposed for level-0 behavior include simpleobstacle avoidance (Tian et al. 2021), maxmax, and maxminmodels (Sarkar and Czarnecki 2021). Although such ele-mentary models may be acceptable in other domains of ap-plication, it is not clear why human drivers, especially in adynamic game setting, would cognitively bound themselvesto such elementary models. Another way QLk model dealswith bounded rationality is through the use of a precision pa-rameter, where agents instead of selecting the utility maxi-mizing response, make cost proportional errors modelled bythe precision parameter (Wright and Leyton-Brown 2010).While the precision parameter provides a convenient wayto fit the model to empirical data, it also runs the risk ofbeing a catch-all for different etiologies of bounded ratio-nality, including, people being indifferent to choices, uncer-tainty around multiple objectives, as well as the model notbeing a true reflection of the decision process. This impairsthe explainability of the model since it is hard to encode andcommunicate such disparate origins of bounded rationalitythrough a single parameter.

The primary contribution of our work is a framework thataddresses the aforementioned challenges by unifying mod-eling of heterogeneous human driving behavior with strate-

arX

iv:2

109.

0986

1v1

[cs

.AI]

20

Sep

2021

Figure 1: Schematic representation of the dynamic game.Each node is embedded in a spatio-temporal lattice andnodes are connected with a cubic spline trajectory.

gic planning for AV. In this framework, behavior modelsare mapped into three layers of increasing capacity to rea-son about other agents’ behavior – non-strategic, strategic,and robust. Within each layer, the possibility of differenttypes of behavior models lends support for a populationof heterogeneous behavior, with a robust layer on top ad-dressing the problem of behavior planning with a relaxedcommon knowledge assumptions. Standard level-k type andequilibrium models are nested within this framework, andin the context of those models, secondary contributions ofour work are a) the use of automata strategies as a modelof level-0 behavior in dynamic games, resulting in behav-ior that is rich enough to capture naturalistic human driving(dLk(A) model), and b) an interpretable support for boundedrationality based on different modalities of satisficing —safety and maneuver. Finally, the efficacy of the approach isdemonstrated with evaluation on two large naturalistic driv-ing datasets as well as simulation of critical traffic scenarios.

2 Game tree, utilities, and agent typesThe dynamic game is constructed as a sequence of simul-taneous move games played starting at time t = 0 at aperiod of ∆tp secs. over a horizon of ∆th secs. Each ve-hicle i ∈ {1, 2, .., N}’s state at time t is a vector Xi,t =[x, y, vx, vy, vx, vy, θ] representing positional co-ordinates(x, y) on R2, lateral and longitudinal velocity (vx, vy) in thebody frame, acceleration (vx, vy), and yaw (θ). The nodes ofthe game tree Xt = XN

i,t are the joint system states embed-ded in a spatio-temporal lattice (Ziegler and Stiller 2009),and the actions are cubic spline trajectories (Al Jawahiri2018; Kelly and Nagy 2003) generated based on kinematiclimits of vehicles with respect to bounds on lateral and lon-gitudinal velocity, acceleration, and jerk (Bae, Moon, andSeo 2019). A history ht of the game consists of sequence ofnodes X0..Xt traversed by both agents along the game treeuntil time t. We also use a hierarchical approach in the tra-jectory generation process (Fisac et al. 2019; Michon 1985;Sarkar and Czarnecki 2021), where at each node, the tra-jectories are generated with respect to high-level maneu-vers, namely, wait and proceed maneuvers. For wait ma-neuver trajectories, a moving vehicle decelerates (or remainstopped if it is already stopped), and for proceed maneu-vers, a vehicle maintains its moving velocity or acceleratesto a range of target speeds. Strategies are presented in thebehavior strategy form, where πi(ht) ∈ Ti(Xt) is a purestrategy response (a trajectory) of an agent i that maps ahistory ht to a trajectory in the set Ti(Xt), which is theset of valid trajectories that can be generated at the nodeXt. The associated maneuver for a trajectory is represented

as m(πi(h)) ∈ {wait , proceed}. Depending on the contextwhere the response depends on only the current node insteadof the entire history, we use the notation πi(Xt); we alsodrop the time subscript t on the history when a formulationholds true for all t. The overall strategy of the dynamic gameis the cross product of behavior strategies along all possiblehistories of the game σ : π1(h)× π2(h)× ..πN (h);∀h. Weuse the standard game-theoretic notation of i and−i to referto an agent and other agents respectively in a game. A glos-sary of all notations along with specific parameter values isincluded in the technical appendix.

The utilities in the game are formulated as multi-objective utilities consisting of two components — safetyus,i(πi(h), π−i(h)) ∈ [−1, 1] (modelled as a sigmoidalfunction that maps the minimum distance gap betweentrajectories to a utility interval [-1,1]) and progressup,i(πi(h), π−i(h)) ∈ [0, 1];∀i,−i (a function that mapsthe trajectory length in meters to a utility interval [0,1]).In general, these two are the main utilities (often referredto as inhibitory and excitatory utilities, respectively) uponwhich different driving styles are built (Sagberg et al. 2015).Agent types γi ∈ Γ are numeric in the range [-1,1] rep-resenting each agent’s risk tolerance, with lower values in-dicating lower tolerance. This construction is motivated bytraffic behavior models such as Risk Monitoring Model (Vaa2011), Task Difficulty Homeostasis theory (Fuller 2008),where based on traffic situations, drivers continually com-pare each possible action to their own risk tolerance thresh-old and make decisions accordingly. To avoid dealing withcomplexities that arise out of agent types being continuous,for the scope of this paper, we discretize the types by incre-ments of 0.5 for our experiments. Based on their type (i.e.their risk tolerance) how each agent selects specific actionsat each node depends on the particular behavior model, andis elaborated in Sec. 3 when we discuss the specifics of eachbehavior model.

Unless mentioned otherwise, the utilities (both safety andprogress) at a node with associated history ht are calculatedas discounted sum of utilities over the horizon of the gameconditioned on the strategy σ, type γi and discount factor δas followsb∆th−t\∆tpc∑

k=1

δkui(πi(ht+k−1), π−i(ht+k−1);σ, γi) +Nui,C

where ui,C are the continuation utilities beyond the horizonof the game and are estimated based on agents continuingon with the same chosen trajectory as undertaken in the lastdecision node of the game tree for another ∆th seconds, andN is a normalization constant to keep the sum of utilities inthe same range as the step utilites.

3 Generalized dynamic cognitive hierarchymodel.

In the generalized dynamic cognitive hierarchy model,agents belong to one of three layers: a) non-strategic, whereagents do not reason about the other agents’ strategies, b)strategic, where agents can reason about the strategies ofother agents, and c) robust, where agents not only reason

Figure 2: (a) Organization of models in the generalized dy-namic cognitive hierarchy framework. Dashed arrows indi-cate agents’ belief about the population. (b) Automata AAC

(accomodating) and ANAC (non-accomodating)

about the strategies of other agents but also the behaviormodel (Wright and Leyton-Brown 2020) that other agentsmay be following (Fig. 2a). All of these layers operate in asetting of a dynamic game and we present the models andthe solution concepts used in each layer in order.

Non-strategic layerSimilar to level-0 models in the standard level-k reasoning(Camerer, Ho, and Chong 2004), non-strategic agents formthe base of the cognitive hierarchy in our model. However,in our case, we extend the behavior for the dynamic game.The main challenge of constructing the right level-0 modelfor a dynamic game is that it has to adhere to the formal con-straints of non-strategic behavior, i.e. not reason over otheragents’ utilities (Wright and Leyton-Brown 2020), while atthe same time cannot be too elementary for the purpose ofmodeling human driving, which implies allowing for agentsto change their behavior over time in the game based on thegame state.

We propose that automata strategies, which were intro-duced as a way to address bounded rational behavior thatresults from agents having limited memory capacity to rea-son over the entire strategy space in a dynamic game (Marks1990; Rubinstein 1986), address the above problem by strik-ing a balance between adequate sophistication and non-strategic behavior. To this end, we extend the standard level-k model for a dynamic game setting with level-0 behav-ior mediated by automata strategies (referred to as dLk(A)henceforth in the paper). In this framework, a level-0 agenthas two reactive modes of operation, each reflecting a spe-cific driving style modeled as automata strategies; accomo-dating (AAC) and non-accomodating (ANAC) (Fig. 2b). Anagent i playing AAC as type γAC

i ∈ Γ in state WAC random-ize uniformly between trajectories belonging to wait ma-neuver that have step safety utility at least γAC

i . If no suchwait trajectories are available to them, they move to statePAC and randomizes uniformly between trajectories belong-ing to proceed maneuver. ANAC is similar, but the states arereversed thereby resulting in a predisposition that prefersthe proceed state. The switching between the states of theautomata is mediated by the preference conditions φAC

i andφNACi shown in the equations below.

φACi := max

∀π(Xt)∈Ti(Xt)|Wu

∆tps,i (π(Xt)) > γAC

i (1)

φNACi := max

∀π(Xt)∈Ti(Xt)|Pu

∆tps,i (π(Xt)) > γNAC

i (2)

φACi is true when there is at least one wait trajectory (and

conversely, at least one proceed trajectory in the case ofφNACi ) available whose step safety utility (u∆tp

s,i ) at the cur-rent node Xt is above γAC

i (and γNACi for φNAC

i ). Recall thatagent types are in the range [-1,1] and are reflective of risktolerance.

Along with switching between states, agents are also freeto switch between the two automata; however, we leave openthe question of modeling the underlying process and thecausal factors. We envision them to be non-deterministic anda function of the agent’s affective states, such as impatience,attitude, etc., which are an indispensable component of mod-eling driving behavior (Sagberg et al. 2015). For example,one can imagine a left turning vehicle approaching the inter-section starting with an accommodating strategy, but alongthe game play, changes its strategy to non-accommodatingon account of impatience or other endogenous factors. Al-though leaving open the choice of mode switching leads tothe level-0 model in this paradigm to be partly descriptiverather than predictive, as we will see in the next section,such a choice does not compromise the ability of higher levelagents’ to form consistent beliefs about the level-0 agent andrespond accordingly to their strategies. This richer model oflevel-0 behavior not only imparts more realism in the con-text of human driving, but also allows for level-0 agent toadapt their behavior based on the game situation in a dy-namic game.

Strategic layerThe difference between strategic and non-strategic modelsis that the latter adhere to the two properties of other re-sponsiveness, i.e. reasoning over the utilities of the otheragents, and dominance responsiveness, i.e. optimizing overtheir own utilities (Wright and Leyton-Brown 2020). Agentsin the strategic layer in our framework adhere to the abovetwo properties and we include three models of behavior,namely, level-k (k > 1) behavior based on the dLk(A)model and two types of bounded rational equilibrium be-havior, i.e. SSPE (Safety satisfied perfect equilibria) andMSPE (Maneuver satisfied perfect equilibria). We note thatthe choice of models in the strategic layer is not exhaustive,however, we select popular game-theoretic models (level-kand equilibrium based) that have been used in the context ofautonomous driving and address some of the gaps within theuse of those models as secondary contributions.

For the strategic models to use the two properties andcompute a game strategy, the two multi-objective utilitiespresented in Sec. 2 need to be combined into one utility.To that end, we use a lexicographic thresholding approachto combine the two multi-objective utilities (Li and Czar-necki 2019), and furthermore, we connect the lexicographicthreshold to an agent’s type γi. Specifically, the combinedutility ui(πi(h), π−i(h)) is equal to us,i(πi(h), π−i(h)),i.e., the safety utility when us,i(πi(h), π−i(h)) 6 −γi, andotherwise ui(πi(h), π−i(h)) = up,i(πi(h), π−i(h)), i.e., theprogress utility. The threshold is negated to align lower risktolerance with lower value of γi. In the following sections,we also index the agent types with the specific models for

clarity.

dLk(A) model (k > 1 behavior)A level-1 agent 1 believes that the population consists solelyof level-0 agents, and generates a best response to those(level-0) strategies. In a dynamic setting, however, a level-1 agent has to update it’s belief about level-0 agent basedon observation of the game play. This means that in order tobest respond to level-0 strategy, a level-1 agent should forma consistent belief based on the observed history ht of thegame play about the type (γAC

−i , γNAC−i ) a level-0 agent plays

in each automaton.

Definition 1. A belief Bl1 = {ΥACl0 , Υ

NACl0 } of a level-

1 agent at history ht is consistent iff ∀γAC−i ∈ ΥAC

l0 ,π−i(X0)...π−i(Xt) ∈ T (AAC; γAC

−i, X0..Xt) and ∀γNAC−i ∈

ΥNACl0 , π−i(X0)...π−i(Xt) ∈ T (ANAC; γNAC

−i , X0..Xt) whereΥACl0 ⊆ Γ, ΥNAC

l0 ⊆ Γ, and (dropping the superscripts)T (A; γ,X0..Xt) is the trace of the automaton, defined asthe set of all valid sequence of actions generated by an level-0 agent playing an automatonA as type γ and encounteringthe nodes X0..Xt.

Before the level-1 agent has observed any action by thelevel-0 agent, their estimates for ΥAC

l0 and ΥNACl0 is the de-

fault set of types, i.e. ΥACl0 = ΥNAC

l0 = Γ with range 2 (recallΓ = [−1, 1]). However, over time with more observationsof level-0 actions, level-1 agent forms a tighter estimate, i.e.|max(ΥAC

l0 ) − min(ΥACl0 )| 6 2, of level-0’s type. The fol-

lowing theorem formulates the set of consistent beliefs Bl1based on observed level-0 actions with history h more for-mally.

Theorem 1. For any consistent belief γAC−i and γNAC

−i , whereγAC−i ∈ ΥAC

l0 and γNAC−i ∈ ΥAC

l0 the following inequalities holdtrue for any history h of the game,

γAC−i

> max∀X∈h[P] max∀π(X)∈T−i(X)|W u∆tp

s,l0 (π(X))

6 min∀X∈h[W] max∀π(X)∈T−i(X)|W u∆tp

s,l0 (π(X))

γNAC−i

> max∀X∈h[W] max∀π(X)∈T−i(X)|P u∆tp

s,l0 (π(X))

6 min∀X∈h[P] max∀π(X)∈T−i(X)|P u∆tp

s,l0 (π(X))

where h[P] and h[W] are set of game nodes where level-0agent chose a proceed and wait maneuver respectively, andT (X)|P ,T (X)|W are the available trajectories at node Xbelonging to the two maneuvers, proceed and wait, respec-tively.

Proof. There are two parts to the equation, one for γAC−i and

another γNAC−i . We prove the bounds of γAC

−i correspondingto automata AAC, and the proof for γNAC

−i corresponding toautomata ANAC follows in an identical manner.

By construction of automata AAC, proceed trajectoriesare only generated in state PAC, which follows one of the

1 We focus on k = 1 behavior in this section as well as later inthe experiments, but the best response behavior can be extended tok > 1 similar to a standard level-k model.

three transitions {inp.|PAC|WAC}¬φAC

i (Xt)−−−−−−→ PAC. There-fore, ∀Xt ∈ h[P], φAC

i := ⊥ for the transition to happen.Based on eqn. 1, this means that ∀Xt ∈ h[P],

γACi > max

∀π(Xt)∈T−i(Xt)|Wu

∆tps,i (π(Xt)) (A.1)

Since γACi stays constant throughout the play of the game,

for γACi to be consistent for the set of all nodes Xt ∈ h[P],

the lower bound (i.e., at least as high to be true for all nodesXt ∈ h[P]) of γAC

i based on eqn A.1 is

γACi > max

∀Xt∈h[P]max

∀π(Xt)∈T−i(Xt)|Wus,l0(π(X)) (A.2)

Similarly, by construction of automata, wait trajectoriesare only generated in state WAC, which follows one of the

three transitions {inp.|PAC|WAC}φACi (Xt)−−−−−→WAC. Therefore,

∀Xt ∈ h[W], φACi := > for the transition to happen. Based

on eqn. 1, this means that ∀Xt ∈ h[W],

γACi 6 max

∀π(Xt)∈T−i(Xt)|Pu

∆tps,i (π(Xt)) (A.3)

and the upper bound (i.e., at least as low to be true for allnodes Xt ∈ h[W]) of γAC

i based on eqn A.3 ∀Xt ∈ h[W]

γACi 6 min

∀Xt∈h[W]max

∀π(Xt)∈T(Xt)|Wus,l0(π(X)) (A.4)

Since h[P]⋂h[W] = ∅ and h = h[P]

⋃h[W], equations

A.2 and A.4 in conjunction proves the case for γACi bounds.

The proof for γNACi follows in the identical manner as γAC

i ,but with the condition reversed based on the P and W states.

The above theorem formalizes the idea that looking atmaneuver choices made by a level-0 agent at each node inthe history, as well as the range of step safety utility at thatnode for both maneuvers (recall that preference conditionsof the automata are based on step safety utility), a level-1agent can calculate ranges for γAC/NAC

−i from Eqns 1 and 2,for which the observed actions were consistent with the cor-responding automata. With respect to the set of consistentbelief Bl1 about level-0 agent’s strategy, level-1 agent nowneeds to generate a best response that is consistent with Bl1.Dropping the AC/NAC superscripts Let π(Xt;A, Υl0) =

{π(Xt;A, γ−i);∀γ−i ∈ Υl0} be the union of all actionswhen the automata A is played by the types in Υl0, thenΠl0(Xt,Bl1) = π(Xt;AAC, ΥAC

l0 )⋃π(Xt;ANAC, ΥNAC

l0 ) isthe set of all actions that level-0 agent can play based onlevel-1’s consistent belief Bl1. The response to those actionsby level-1 agent (indexed as i) is as follows.

πi(ht;Bl1) = argmax∀π(h)∈Ti(Xt)

∀π−i(Xt)∈Πl0(Xt,Bl1)

ui(πi(h), π−i(Xt)|γl1i ) (3)

where γl1i ∈ Γ is the type of the level-1 agent. Note that thestrategy of the level-1 agent, unlike level-0 agent depends onthe history ht instead of just the state of the node Xt; sincethe history influences the belief Bl1 which in turn influencesthe response.

Equilibrium models.Along with the level-k (k> 1) behavior, another notion ofstrategic behavior that have been proposed as a model ofbehavior planning in AV (Pruekprasert et al. 2019; Schwart-ing et al. 2019) are based on an equilibrium. However, whenit comes to modelling human driving behavior, an equilib-rium model needs to accommodate bounded rational agentsin a principled manner, and ideally should provide a rea-sonable explanation of the origin of the bounded rationality.Based on the idea that drivers are indifferent as long as anaction achieves their own subjective risk tolerance thresh-old (Lewis-Evans 2012), we use the idea of satisficing as themain framework for bounded rationality in our equilibirummodels (Stirling 2003). Specifically, we develop two notionsof satisficing; one based on safety satisficing (SSPE), whereagents choose actions close to the Nash equilibria as longas the actions are above their own risk tolerance threshold,and another based on maneuver satisficing (MSPE), whereagents chose actions close to the Nash equilibria as long asthe actions are of the same high-level maneuver as the opti-mal action.

Safety-satisfied perfect equilibrium (SSPE). The mainidea behind satisficing is that a bounded rational agent, in-stead of always selecting the best response action, selects aresponse that is good enough, where good enough is definedas an aspiration level where the agent is indifferent betweenthe optimal response and the response in question. In thecase of Safety-satisfied perfect equilibrium (SSPE), we de-fine a response good enough for agent i if the response isabove their own safety threshold determined by their type.A more formal definition is as followsDefinition 2. A strategy σ; (γSS

i , γSS−i) :

∏∀i∈N

πi(h), is in

safety satisfied perfect equilibria for a combination of agenttypes (γSS

i , γSS−i) ∈ ΓN if for every history h of the game and

∀i ∈ N

us,i(πi(h), π∗−i(h)) > min(u∗s,i(h), γSSi )

where σ∗; (γSSi , γ

SS−i) :

∏∀i∈N

π∗i (h), is a subgame perfect

Nash equilibrium in pure strategies of the game for agentswith type (γSS

i , γSS−i) and u∗s,i(h) = us,i(π

∗i (h), π∗−i(h)). 2

Based on the above definition, if the safety utility of thebest response of agent i to agent −i’s subgame perfect Nashequilibrium (SPNE) strategies at history h is less than agenti’s own safety threshold as expressed by their type γSS

i , thenthe SSPE response is any trajectory that matches the safetyutility of the SPNE response. However, if the SPNE responseis higher than their safety threshold, then any suboptimal re-sponse that has safety utility higher than γSS

i is a satisfiedresponse, and thus in SSPE.

Maneuver-satisfied perfect equilibrium (MSPE). In thismodel of satisficing, agents choose actions that belong tothe same maneuver as that of the equilibrium action, with2 We calculate the SPNE using backward induction with the com-bined utilities (lexicographic thresholding) in a complete informa-tion setting where agent types are known to each other.

some additional constraints. Illustrated with a simple exam-ple, at any node, if the trajectory of the equilibrium actionbelong to wait maneuver, then all trajectories belonging towait maneuver will be in MSPE. However, to avoid selec-tion of the wait trajectories that have utility lower than anon-equilibrium maneuver (in this case, a proceed trajec-tory), we add the constraint that the utility of a MSPE (inthis case, a wait trajectory) has to be higher than that of allproceed trajectories. A formal definition is as follows.Definition 3. A strategy σ; (γMS

i , γMS−i ) :

∏∀i∈N

πi(h), is in

maneuver satisfied perfect equilibria for a combination ofagent types (γMS

i , γMS−i ) ∈ ΓN if for every history h of the

game and ∀i ∈ N , m(πi(h)) = m(π∗i (h)) and

us,i(πi(h), π∗−i(h)) > max

π′i(h)∈Ti(X)\m∗

ui(π′i(h), π

∗−i(h))

where σ∗; (γMSi , γMS

−i ) :∏∀i∈N

π∗i (h), is a subgame perfect

equilibrium in pure strategies of the game with agent types(γMSi , γMS

−i ), and Ti(X) \ m∗ = {πi(h) : m(πi(h)) 6=m(π∗i (X))} or in other words, the set of available trajec-tories at node X that do not belong to the maneuver corre-sponding to the equilibrium trajectory m(π∗i (h)) and X isthe last node in the history h.

Robust layerWhile the presence of multiple models in the strategic lay-ers allow for a population of heterogeneous reasoners, anagent following one of those models still has specific as-sumptions about the reasoning process of other agents, e.g.level-1 agents believing that the population consists of level-0 agents and equilibrium responders believing that otheragents adhere to a common knowledge of rationality. Basedon the position that a planner for AV should not hold suchstrict assumptions, we develop the robust layer. What dif-ferentiates the robust layer from the strategic layer is thatalong with the two properties of other responsiveness anddominance responsiveness for the strategic layer, agents inthe robust layer also adhere to the property of model respon-siveness, i.e., the ability to reason over the behavior mod-els of other agents. This gives them the ability to reasonabout (forming beliefs about and responding to) a popula-tion of different types of reasoners including strategic, non-strategic, as well as agents following different models withineach layer. The overall behavior of a robust agent can be bro-ken down into three sequential steps as follows.i. Type expansion: Since the robust agent not only has toreason over the types of other agents, but also the possiblebehavior models, we augment the initial set of agent typesΓ that were based on agents’ risk tolerance with the corre-sponding agent models. Let Γ+ :M× Γ be the augmentedtype of an agent, whereM is the set of models presented ear-lier, i.e., {accomodating, non-accomodating, level-1, SSPE,MSPE} and Γ is the (non-augmented) agent type agent type(γACi , γNAC

i , γl1i , γ

SSi , γ

MSi ) within each model.

ii. Consistent beliefs: Similar to strategic agents, based onthe observed history h of the game, a robust agent forms abelief βh ⊆ Γ+ such that the observed actions of the other

agent in the history h is consistent with the augmented types(i.e., model as well as the agent type) in βh. The process ofchecking whether a history is consistent with a combinationof a model and agent type was already developed earlier fortwo non-strategic models (Def. 1). For level-1 models, thehistory is consistent if at each node in history, the response ofthe other agent adheres to equation 3; and for the equilibriummodels, a history h is consistent if based on definitions 2 and3 for SSPE and MSPE respectively, the actions are along theequilibrium paths of the game tree. Assuming that in drivingsituations agents behave truly according to their types, βh isthen constructed as an union of all the consistent beliefs foreach model.iii. Robust response: The idea of a robust response to het-erogeneous models is along the lines of the robust game the-ory approach of (Aghassi and Bertsimas 2006). The beliefset βh represents the uncertainty over the possible modelsthe other agents’ may be using along with the correspond-ing agent types within those models. A robust response tothat is the optimization over the worst possible outcome thatcan happen with respect to that set. Eqn. 4 formulates thisresponse of a robust agent playing as agent i.

πi(h;βh) = argmax∀π(h)

inf∀β∈βh

max∀π−i(h;β)

ui(π(h), π−i(h;β); γRi )

(4)where π−i(h;β) are the possible actions of the other agentbased on the augmented type β ∈ βh and γR

i ∈ Γ is therobust agent’s own type. In this response, the minimiza-tion happens over the agent types (inner inf operator), ratherthan over all the actions π−i(h) as is common in a maxminresponse. Since driving situations are not, in most cases,purely adversarial, this is a less conservative, yet robust, re-sponse compared to a maxmin response.

4 Experiments and evaluation

(a) Snapshot of naturalistic datasets (WMA and inD)

(b) Simulation of critical scenarios: intersection clearance, mergebefore intersection, parking pullout.

Figure 3: Evaluation setups

In this section we present the evaluation of the modelsunder two different experiment setups. First, we comparethe models with respect to large naturalistic observational

driving data using a) the Intersection dataset from the Wa-terloo multi-agent traffic dataset (WMA) recorded at a busyCanadian intersection (Sarkar and Czarnecki 2021), and b)the inD dataset recorded at intersections in Germany (Bocket al. 2019) (Fig. 3a). From both datasets, which includearound 10k vehicles in total, we extract the long durationunprotected left turn (LT) and right turn (RT) scenarios, andinstantiate games between left (and right) turning vehiclesand oncoming vehicles with ∆th = 6s and ∆tp = 2s, re-sulting in a total of 1678 games. The second part of the eval-uation is based on simulation of three critical traffic scenar-ios derived from the NHTSA pre-crash database (Najm et al.2007), where we instantiate agents with a range of risk toler-ances as well as initial game states, and evaluate the outcomeof the game based on each model. All the games in the ex-periments are 2 agent games with the exception of one ofthe simulation of critical scenario (intersection clearance),which is a 3 agent game.Baselines. We select multiple baselines depending onwhether a model is strategic or non-strategic. For non-strategic models, we compare the automata based modelwith a maxmax model, shown to be most promising froma set of alternate elementary models with respect to nat-uralistic data (Sarkar and Czarnecki 2021). For the strate-gic models (level-1 in dLk(A), SSPE, MSPE), we select aQLk model used in multiple works within the context of au-tonomous driving (Li et al. 2018, 2019; Tian et al. 2018,2021). We use the same parameters used in (Tian et al. 2021)for the precision parameters in the QLk model.

Naturalistic data We evaluate the model performance onnaturalistic driving data based on accuracy measure, i.e. thenumber of games where the observed strategy of the humandriver matched a model’s strategy divided by the total num-ber of games in the dataset. More formally, letD be the set ofgames in the dataset, an indicator function I : D → {0, 1}is 1 if in the game g, there exists a combination of agenttypes (γi, γ−i), such that the observed strategy is in the setof strategies as predicted by the model, or 0 otherwise. Theoverall accuracy of a model is given by

∑∀g∈D I/|D|. QLk

(baseline) models being mixed strategy models, we count amatch if the observed strategy is assigned a probability of> 0.5. Table 1 shows the accuracy of each model for eachdataset and scenario. It also shows in parenthesis the meanγi, i.e. the agent type value for each model when the strategymatched the observation. The overall numbers in the tableare in line with the converging consensus from recent litera-ture that there is heterogeneity in driving behavior (Sarkarand Czarnecki 2021; Sun et al. 2020). However, a majortakeaway is that for non-strategic models, automata modelsshow much higher accuracy thereby reflecting high align-ment with human driving behavior compared to the maxmaxmodel. In fact, as we can see from the table that the entriesfor AC and NAC sum up to 1, combination of AC and NACalthough being non-strategic, can capture all observed driv-ing decisions in the dataset, which indicate that automatamodels are very well suited for modelling level-0 behaviorin a dynamic game setting for human driving.

For the strategic models, dLK(A) and SSPE model shows

WMA inDLT (1103) RT (311) LT (187) RT (77)

maxmax 0.33438(-0.02)

0.43023(-0.27)

0.37975(0.2)

0.43506(-0.1)

AC 0.82053(-0.82)

0.90698(-0.84)

0.92089(-0.75)

0.81818(-0.79)

NAC 0.17947(-0.87)

0.09302(-0.82)

0.07911(-0.88)

0.18182(-0.85)

QLk(λ=1) 0.18262(0.07)

0.43265(-0.33)

0.37658(-0.37)

0.43506(0.1)

QLk(λ=0.5) 0.34131(0.03)

0.43023(-0.26)

0.37658(-0.37)

0.43506(0.1)

dLk(A) 0.5529(-0.19)

0.65449(-0.3)

0.51266(-0.51)

0.53247(0.4)

SSPE 0.69144(0.84)

0.90033(0.94)

0.6962(0.86)

0.53247(0.85)

MSPE 0.30479(0.1)

0.44518(0.13)

0.21519(-0.21)

0.27273(0.7)

Robust 0.56045(-0.20)

0.66944(-0.34)

0.51582(-0.51)

0.53247(0.4)

Table 1: Overall accuracy of the models for each dataset andscenario. Mean agent type (γ) noted in parenthesis. LT: Leftturn, RT: Right turn. Number of games noted in the header.

better performance than QLk and MSPE models. However,when we compare the mean agent types, we observe thatwhen SSPE strategies matches the observation, it is basedon agents with very high risk tolerance (reflected in highmean agent type values). If we assume that the populationof drivers on average have moderate risk tolerance, say inthe range [-0.5,0.5] (estimating the true distribution is outof the current scope), dLk(A) is a more reasonable modelof strategic behavior compared to SSPE. We include the ro-bust model comparison for the sake of completeness (andit shows performance comparable to dLk(A) model), but asmentioned earlier, robust model is a model of response ofan AV, and therefore ideally needs to be evaluated on cri-teria beyond just comparison to naturalistic human driving,which we discuss in the next section.

Critical scenarios While evaluation based on a naturalis-tic driving dataset helps in the understanding of how wella model matches human driving behavior, in order to eval-uate the suitability of a model for behavior planning of anAV, the models need to be evaluated on specific scenariosthat encompass the operational design domain (ODD) of theAV (Ilievski 2020). Since the models developed in this paperare not specific to an ODD, we select three critical scenar-ios from ten most frequent crash scenarios in the NHTSApre-crash database (Najm et al. 2007).

Intersection clearance (IC): Left turn across path(LTAP) scenario where the traffic signal has just turned fromgreen to yellow at the moment of the game initiation. Thereis a left turning vehicle is on the intersection and two oncom-ing vehicles from the opposite direction close to the intersec-tion who may chose to speed and cross or wait for the nextcycle. The expectation is that the left turning vehicle shouldbe able to clear the intersection by the end of the game hori-zon without crashing into either oncoming vehicles, and novehicles should be stuck in the middle of the intersection.

Merge before intersection (MBI): Merge scenariowhere a left-turning vehicle (designated as the merging ve-hicle) finds itself in the wrong lane just prior to entering theintersection, and wants to merge into the turn lane in front of

Figure 4: Mean and SD of success for each model in eachscenario across all agent types.

another left-turning vehicle (designated as the on-lane vehi-cle). The expectation is that the on-lane vehicle should allowthe other vehicle to merge.

Parking pullout (PP): Merge scenario where a parkedvehicle is pulling out of a parking spot and merges into traf-fic while there is a vehicle coming along the same direc-tion from behind. The expectation is that the parked vehicleshould wait for the coming vehicle to pass before merginginto traffic.

For each scenario, we run simulations with a range of ap-proach speeds as well as all combination of agent types fromthe set of agent types Γ = {−1,−0.5, 0, 0.5, 1} (parametersand simulation videos are included in the supplementary ma-terial). One way to compare the models is to evaluate thembased on the mean success rate across all initiating states andagent types. Fig. 4 shows the mean success rate (success de-fined as the desired outcome based on expectation defined inthe description for each scenario) for all the strategic and ro-bust models. We see that the mean success rate of the robustand dLk(A) model is higher compared to the equilibriummodels or the QLk model. However, this is only part of thestory. With varying initiation conditions, it may be harderor easier for a model to lead to a successful outcome. Forexample, in the parking pullout scenario a high risk tolerantvehicle coming at a higher speed is almost likely to succeedin all models when facing a low risk tolerant parked vehicleat zero speed. Therefore, to tease out the stability of mod-els across different risk tolerance (i.e. agent type combina-tions), Fig. 4 also plots on y-axis, the standard deviation ofthe mean success rate across different agent types. Ideally, amodel should have a high success rate with low SD acrosstypes indicating that with different combinations of agenttype population (from extremely low risk tolerance to veryhigh), the success rate stays stable. As we see from Fig. 4,the robust and dLk(A) models are broadly in the ideal lowerright quadrant (high mean success rate low SD) for park-ing pullout and merge before intersection scenarios. For ICscenario, however, we observe that the success rate comesat a price of high SD (for all models) as indicated by thelinearly increasing relation between mean success rate andits SD across agent types. This means that the success out-comes are skewed towards a specific combination of agenttypes; specifically, the case where the left turning vehicle has

high risk tolerance. It is intuitive to imagine that in a situa-tion like IC, agents with low risk tolerance would be stuck inthe intersection instead of being able to navigate out of theintersection quickly.

Finally, the failure of models to achieve the expected out-come can also be due to a crash (minimum distance gapbetween trajectories 6 0.1m) instead of an alternate out-come (e.g. getting stuck in the intersection). In all the sim-ulations, for the parking pullout and intersection clearancewe did not observe a crash for any of the models. However,for the merge before intersection, on account of starting at amore riskier situation than the other two in terms of chanceof a crash, the crash rate (ratio of crashes across all simula-tions) for the models across all initial states and agent typeswere as follows: (dlk(A): 0.052, MSPE: 0.022, SSPE: 0.007,QLk(1): 0.026, Robust: 0.053).

Overall, whether or not an AV planner can succeed in theirdesired outcome depends on a variety of factors, such as,the assumption the vehicle and the human drivers hold overeach other, the risk tolerance of each agent, as well as thespecific state of the traffic situation. The analysis presentedabove helps in quantifying the relation between the desiredoutcome and the criteria under which it is possible.

5 ConclusionWe developed a unifying framework of modeling humandriving behavior and strategic behavior planning of AVsthat support heterogeneous models of strategic and non-strategic behavior. We also extended the standard level-kmodel into the dynamic form with a sophisticated yet withinnon-strategic constraints of level-0 behavior through the useof automata strategies (dLk(A)). The evaluation on two largenaturalistic datasets attests to the consensus that there is di-versity in human driving behavior; however, a combinationof a rich level-0 behavior can capture most of the drivingbehavior as observed in naturalistic data. On the other hand,with the awareness that there can be different types of rea-soners in the population, an approach of robust response isnot only effective, but also is stable across a population ofdrivers with different levels of risk tolerance.

ReferencesAghassi, M.; and Bertsimas, D. 2006. Robust game theory.Mathematical programming, 107(1): 231–273.

Al Jawahiri, A. 2018. Spline-based Trajectory Genera-tion for Autonomous Truck-Trailer Vehicles in Low SpeedHighway Scenarios.

Bae, I.; Moon, J.; and Seo, J. 2019. Toward a comfortabledriving experience for a self-driving shuttle bus. Electron-ics, 8(9): 943.

Bock, J.; Krajewski, R.; Moers, T.; Runde, S.; Vater, L.;and Eckstein, L. 2019. The inD Dataset: A Drone Datasetof Naturalistic Road User Trajectories at German Intersec-tions.

Camerer, C. F.; Ho, T.-H.; and Chong, J.-K. 2004. A cog-nitive hierarchy model of games. The Quarterly Journal ofEconomics, 119(3): 861–898.

Fisac, J. F.; Bronstein, E.; Stefansson, E.; Sadigh, D.; Sas-try, S. S.; and Dragan, A. D. 2019. Hierarchical game-theoretic planning for autonomous vehicles. In 2019 Inter-national Conference on Robotics and Automation (ICRA),9590–9596. IEEE.Fuller, R. 2008. Recent developments in driver control the-ory: from task difficulty homeostasis to risk allostasis. InProceedings of the International Conference on Traffic andTransport Psychology, Washington, USA, volume 31.Geanakoplos, J. 1992. Common knowledge. Journal ofEconomic Perspectives, 6(4): 53–82.Geiger, P.; and Straehle, C.-N. 2021. Learning Game-Theoretic Models of Multiagent Trajectories Using ImplicitLayers. Proceedings of the AAAI Conference on ArtificialIntelligence, 35: 4950–4958.Ilievski, M. 2020. WiseBench: A Motion Planning Bench-marking Framework for Autonomous Vehicles. Master’sthesis, University of Waterloo.Kelly, A.; and Nagy, B. 2003. Reactive nonholonomic tra-jectory generation via parametric optimal control. The In-ternational Journal of Robotics Research, 22(7-8): 583–601.Lewis-Evans, B. 2012. Testing models of driver behaviour.University Library Groningen][Host].Li, C.; and Czarnecki, K. 2019. Urban Driving with Multi-Objective Deep Reinforcement Learning. 359–367.Li, N.; Kolmanovsky, I.; Girard, A.; and Yildiz, Y.2018. Game theoretic modeling of vehicle interactions atunsignalized intersections and application to autonomousvehicle control. In 2018 Annual American Control Con-ference (ACC), 3215–3220. IEEE.Li, S.; Li, N.; Girard, A.; and Kolmanovsky, I. 2019. Deci-sion making in dynamic and interactive environments basedon cognitive hierarchy theory, Bayesian inference, and pre-dictive control. In 2019 IEEE 58th Conference on Decisionand Control (CDC), 2181–2187. IEEE.Marks, R. E. 1990. Repeated games and finite automata.Citeseer.Michon, J. A. 1985. A critical view of driver behavior mod-els: what do we know, what should we do? In Human be-havior and traffic safety, 485–524. Springer.Najm, W. G.; Smith, J. D.; Yanagisawa, M.; et al. 2007.Pre-crash scenario typology for crash avoidance research.Technical report, United States. National Highway TrafficSafety Administration.Pruekprasert, S.; Zhang, X.; Dubut, J.; Huang, C.; andKishida, M. 2019. Decision making for autonomous ve-hicles at unsignalized intersection in presence of maliciousvehicles. In 2019 IEEE Intelligent Transportation SystemsConference (ITSC), 2299–2304. IEEE.Rubinstein, A. 1986. Finite automata play the repeated pris-oner’s dilemma. Journal of economic theory, 39(1): 83–96.Sagberg, F.; Selpi; Bianchi Piccinini, G. F.; and Engstrom,J. 2015. A review of research on driving styles and roadsafety. Human factors, 57(7): 1248–1275.Sarkar, A.; and Czarnecki, K. 2021. Solution Conceptsin Hierarchical Games Under Bounded Rationality WithApplications to Autonomous Driving. In Proceedings ofthe AAAI Conference on Artificial Intelligence, volume 35,5698–5708.

Schwarting, W.; Pierson, A.; Alonso-Mora, J.; Karaman, S.;and Rus, D. 2019. Social behavior for autonomous vehicles.Proceedings of the National Academy of Sciences, 116(50):24972–24978.Stirling, W. C. 2003. Satisficing Games and Decision Mak-ing: with applications to engineering and computer science.Cambridge University Press.Sun, L.; Cai, M.; Zhan, W.; and Tomizuka, M. 2020.A Game-Theoretic Strategy-Aware Interaction Algorithmwith Validation on Real Traffic Data. In 2020 IEEE/RSJInternational Conference on Intelligent Robots and Systems(IROS), 11038–11044. IEEE.Tian, R.; Li, S.; Li, N.; Kolmanovsky, I.; Girard, A.; andYildiz, Y. 2018. Adaptive game-theoretic decision makingfor autonomous vehicle control at roundabouts. In 2018IEEE Conference on Decision and Control (CDC), 321–326. IEEE.Tian, R.; Liting, S.; Masayoshi, T.; and David, I. 2021.Anytime Game-Theoretic Planning with Active ReasoningAbout Humans’ Latent States for Human-Centered Robots.In 2021 IEEE International Conference on Robotics andAutomation (ICRA). IEEE.Vaa, T. 2011. Drivers’ information processing, decision-making and the role of emotions: Predictions of the RiskMonitor Model. In Human Modelling in Assisted Trans-portation, 23–32. Springer.Wright, J. R.; and Leyton-Brown, K. 2010. Beyond equilib-rium: Predicting human behavior in normal-form games. InTwenty-Fourth AAAI Conference on Artificial Intelligence.Wright, J. R.; and Leyton-Brown, K. 2020. A formal sepa-ration between strategic and nonstrategic behavior. In Pro-ceedings of the 21st ACM Conference on Economics andComputation, 535–536.Ziegler, J.; and Stiller, C. 2009. Spatiotemporal state lat-tices for fast trajectory planning in dynamic on-road drivingscenarios. In 2009 IEEE/RSJ International Conference onIntelligent Robots and Systems, 1879–1884. IEEE.