Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural LanguageProcessing (HLT/EMNLP), pages 291–298, Vancouver, October 2005. c©2005 Association for Computational Linguistics

Learning What to Talk About in Descriptive Games

Hugo ZaragozaMicrosoft Research

Cambridge, United Kingdomhugoz@microsoft.com

Chi-Ho LiUniversity of Sussex

Brighton, United KingdomC.H.Li@sussex.ac.uk

Abstract

Text generation requires a planning mod-ule to select an object of discourse and itsproperties. This is specially hard in de-scriptive games, where a computer agenttries to describe some aspects of a gameworld. We propose to formalize this prob-lem as a Markov Decision Process, inwhich an optimal message policy can bedefined and learned through simulation.Furthermore, we propose back-off poli-cies as a novel and effective technique tofight state dimensionality explosion in thisframework.

1 Introduction

Traditionally, text generation systems are decom-posed into three modules: the application modulewhich manages the high-level task representation(state information, actions, goals, etc.), the text plan-ning module which chooses messages based on thestate of the application module, and the sentencegeneration module which transforms messages intosentences. The planning module greatly dependson the characteristics of both the application andthe generation modules, solving issues in domainmodelling, discourse and sentence planning, and tosome degree lexical and feature selection (Cole etal., 1997). In this paper we concentrate on oneof the most basic tasks that text planning needs tosolve: selecting the message content, or more sim-ply, choosing what to talk about.

Work on text-generation often assumes that anobject or topic has been already chosen for discus-sion. This is reasonable for many applications, butin some cases choosing what to talk about can beharder than choosing how to. This is the case in thetype of text generation applications that we are in-terested in: generating descriptive messages in com-puter games. In a modern computer game at anygiven moment there may be an enormous numberof object properties that can be described, each withvarying importance and consequences. The outcomeof the game depends not only on the skill of theplayer, but also on the quality of the descriptive mes-sages produced. We refer to such situations as de-scriptive games.

Our goal is to develop a strategy to choose themost interesting descriptive messages that a particu-lar talker may communicate to a particular listener,given their context (i.e. their knowledge of the worldand of each-other). We refer to this as message plan-ning.

Developing a general framework for planning isvery difficult because of the strong coupling be-tween the planning and application modules. Wepropose to frame message planning as a Markov De-cision Process (MDP) which encodes the environ-ment, the information available to the talker and lis-tener, the consequences of their communicative andnon-communicative acts, and the constraints of thetext generation module. Furthermore we propose touse Reinforcement Learning (RL) to learn the op-timal message policy. We demonstrate the overallprinciple (Section 2) and then develop in more de-tail a computer game setting (Section 3).

291

One of the main weaknesses of RL is the problemof state dimensionality explosion. This problem isspecially acute in message planning, since in typicalsituations there can be hundreds of thousands of po-tential messages. At the same time, the domain ishighly structured. We propose to exploit this struc-ture using a form of the back-off smoothing princi-ple on the state space (Section 4).

1.1 Related Work

Our problem setting can be seen as a generalisationof the content selection problem in the generation ofreferring expressions in NLG. In the standard set-ting of this problem (see for example (van Deemterand Krahmer, to appear)) an algorithm needs to se-lect the distinguishing description of an object in ascene. This description can be seen as a subset ofscene properties which i) uniquely identifies a giventarget object, and ii) is optimal in some sense (min-imal, psychologically plausible, etc.) van Deemterand Krahmer show that most content selection algo-rithms can be described as different cost functionsover a particular graph representation of the scene.Minimising the cost of a subgraph leads to a distin-guishing description.

Some aspects of our work generalise that of con-tent selection: i) we consider the target object is un-known, ii) we consider scenes (i.e. world states) thatare dynamic (i.e. they change over time) and reac-tive (i.e. utterances change the world), and iii) weconsider listeners that have partial knowledge of thescene. This has important consequences. For exam-ple, the cost of a description cannot be directly eval-uated; instead, we must play the game, that is, gener-ate utterances and observe the rewards obtained overtime. Also identical word-states may lead to differ-ent optimal messages, depending on the listener’spartial knowledge. Other aspects of our work arevery simplistic compared to current work in con-tent selection, for example with respect to the useof negation and of properties that are boolean, rel-ative or graded (van Deemter and Krahmer, to ap-pear). We hope to incorporate these ideas into ourwork soon.

Probabilistic dialogue policies have been previ-ously proposed for spoken dialogue systems (SDS)(see for example (Singh et al., 2002; Williams etal., 2005) and references therein). However, work in

SDS focus mainly on coping with the noise and un-certainty resulting from speech recognition and sen-tence parsing. In this context MDPs are used to inferfeatures and plan communicative strategies (modal-ity, confusion, initiative, etc.) In our work we do notneed to deal with uncertainty or parsing; our mainconcern is in the selection of the message content.In this sense our work is closer to (Henderson et al.,2005), where RL is used to train a SDS with verymany states encoding message content.

Finally, with respect to the state-explosion prob-lem in RL, related work can be found in the areas ofmulti-task learning and robot motion planning (Diet-terich, 2000, and references therein). In these worksthe main concern is identifying the features that arerelevant to specific sub-tasks, so that robots maylearn multiple loosely-coupled tasks without incur-ring state-explosion. (Henderson et al., 2005) alsoaddresses this problem in the context of SDS andproposes a semi-supervised solution. Our approachis related to these works, but it is different in thatwe assume that the feature structure is known in ad-vance and has a very particular form amenable to aform of back-off regularisation.

2 Message planning

Let us consider an environment comprising a worldwith some objects and some agents, and some dy-namics that govern their interaction. Agents can ob-serve and memorize certain things about the world,can carry out actions and communicate with otheragents. As they do so, they are rewarded or pun-ished by the environment (e.g. if they find food, ifthe complete some goal, if they run out of energy,etc.)

The agents’ actions are governed by a policy. Wewill consider separately the physical action policy(π), which decides which physical action to takegiven the state of the agent, and the message actionpolicy (µ), which decides when to communicate, towhom, and what about. Our main concern in thispaper will be to learn an optimal µ. Before we de-fine this goal more precisely, we will introduce somenotation.

A property is a set of attribute-value pairs. Anobject is a set of properties, with (at least) attributesType and Location. A domain is a set of objects. Fur-

292

thermore, we say that s′ is a sub-domain of s if s′ canbe obtained by deleting property–value pairs froms (while enforcing the condition that remaining ob-jects must have Type and Location). Sub(s) is the setcontaining s, all sub-domains of s, and the emptydomain ∅.

A world state can be represented as a domain,noted sW . Any partial view of the world state canalso be represented as a domain s ∈ Sub(sW ). Sim-ilarly the content of any descriptive message aboutthe world, noted m, can be represented as a partialview of it. An agent is the tuple:

A :=(sA, πA, {µAA′ , sAA′}A′ �=A

)

• sA ∈ Sub(sW ): knowledge that A has aboutthe state of the world.

• sAA′ ∈ Sub(sA ∩ s′A): knowledge that Ahas about the knowledge that A′ has about theworld.

• πa := P (c|sA) is the action policy of A, and cis a physical action.

• µAA′ := P (m ∈ M(sA)|sA, sAA′) is the mes-sage policy of A for sending messages to A′,and M(sA) are all valid messages at state sA(discussed in Section 2.3).

When an agent A decides to send a message to A′,it can use its knowledge of A′ to choose messageseffectively. For example, A will prefer to describethings that it knows A′ does not know (i.e. not insAA′). This is the reason why the message policyµA depends on both sA and sAA′ . After a message issent (i.e. realised and uttered) the agent’s will updatetheir knowledge states sA′ , sA′A and sAA′ .

The question that we address in this paper is thatof learning an optimal message policy µAA′ .

2.1 Talker’s Markov Decision Process

We are going to formalize this problem as a stan-dard Markov Decision Process (MDP). In general aMDP (Sutton and Barto, 1998) is defined over someset of states S := {si}i=1..K and actions associatedto every state, A(si) := {aij}j=1..Ni

. The envi-ronment is governed by the state transition functionPa

ss′ := P (s′|s, a). A policy determines the likeli-hood of actions at a given state: π(s) := P (a|s). At

each state transition a reward is generated from thereward function Ra

ss′ := E{r|s, s′, a}.MDPs allow us to define and find optimal poli-

cies which maximise the expected reward. ClassicalMDPs assume that the different functions introducedabove are known and have some tractable analyti-cal form. Reinforcement Learning (RL) in as ex-tension of MDPs in which the environment functionPa

ss′ is unknown or complex, and so the optimal pol-icy needs to be learned online by directly interactingwith the environment. There exist a number of algo-rithms to solve a RL problem, such as Q-Learningor SARSA (Sutton and Barto, 1998).

We can use a MDP to describe a full descrip-tive game, in which several agents interact with theworld and communicate with each-other. To do sowe would need to consider composite states con-

taining sW , {sA}A, and{{sAA′}A′ �=A

}A

. Simi-

larly, we need to consider composite policies con-

taining {πA}A and{(µAA′)A′ �=A

}A

. Finally, we

would consider the many constrains in this model;for example: only physical actions affect the stateof the world, only message actions affect believes,and only believe states can affect the choice of theagent’s actions.

MDPs provide us with a principled way to dealwith these elements and their relationships. How-ever, dealing with the most general case results inmodels that are very cumbersome and which hidethe conceptual simplicity of our approach. For thisreason, we will limit ourselves in this paper to oneof the simplest communication cases of interest: asingle all-knowing talker, and a single listener com-pletely observed by the talker. We will discuss laterhow this can be generalized.

2.2 The Talking God Setting

In the simplest case, an all-knowing agent A0 sits inthe background, without taking any physical actions,and uses its message policy (µ01) to send messagesto a listener agent A1. The listener agent cannot talkback, but can interact with the environment usingits physical action policy π1. Rewards obtained byA1 are shared by both agents. We refer to this set-ting as the talking God setting. Examples of suchsituations are common in games, for example whena computer character talks to its (computer) team-

293

w w’

s1 s’1

m0

a1

s s’

a

r

Figure 1: Talking God MDP.

mates, or when a mother-ship with full informationof the ground sends a small simple robot to do a task.Another example would be that of a teacher talkingto a learner, except that the teacher may not have fullinformation of the learners head!

Since the talker is all-knowing, it follows thats0 = sW and s01 = s1. Furthermore, since thetalker does not take physical actions, π0 does notneed to be defined. Similarly, since the listener doesnot talk we do not need to define µ10 or s10. Thiscase is depicted in Figure 1 as a graphical model.By grouping states and actions (dotted lines) we cansee that this is can be modelled as a standard MDP.If all the probability distributions are known analyt-ically, or if they can be sampled, optimal physicaland message policies can be learnt (thick arrows).

Several generalizations of this model are possible.A straight forward generalization is to consider morethan one listener agent. We can then choose to learna single policy for all, or individual policies for eachagent.

A second way to generalize the setting is to makethe listeners mind only partially observable to thetalker. In this case the talker continues to know theentire world (s0 = sW ), but does not know ex-actly what the listener knows (s01 �= s0). This ismore realistic in situations in which the listener can-not talk back to the talker, or in which the talkersmind is not observable. However, to model this weneed a partially observable MDP (POMDP). Solv-ing POMDPS is much harder than solving MDPs,but there have been models proposed for dialogue

management (Williams et al., 2005).In the more general case, the talker would have

partial knowledge of the world and of the listener,and would itself act. In that case all agents are equaland can communicate as they evolve in the envi-ronment. The other agents minds are not directlyobservable, but we obtain information about themfrom their actions and their messages. This can allbe in principle modelled by POMDPs in a straight-forward manner, although solving these models ismore involved. We are currently working towardsdoing so.

Finally, we note that all the above cases havedealt with worlds in which objects are static (i.e.information does not become obsolete), agents donot gain or communicate erroneous information, andcommunication itself is non-ambiguous and loss-less. This is a realistic scenario for text generation,and for communication between computer agents ingames, but it is far removed from the spoken dia-logue setting.

2.3 Generation Module and Valid Messages

Generating descriptive sentences of domains can bedone in a number of ways, from template to feature-based systems (Cole et al., 1997). Our frameworkdoes not depend on a particular choice of generationmodule, and so we do not need to discuss this mod-ule. However, our message policy is not decoupledof the generation module; indeed, it would not makesense to develop a planning module which plansmessages that cannot be realised! In our framework,the generation module is seen simply as a fixed andknown filter over all possible the messages.

We formalize this by representing an agent’s gen-eration module as a function ΓA(m) mapping a mes-sage m to a NL sentence, or to ∅ if the module can-not fully realise m. The set of available messagesto an agent A in state sA is therefore: M(sA) :={m |m ∈ Sub(sA) , ΓA(m) �= ∅}.

3 A Simple Game Example

In this section we will use a simple computer gameto demonstrate how the proposed framework can beused to learn message policies.

The game evolves in a grid-world. A mother-ship sends a scout, which will try to move from its

294

Figure 2: Example of a Simple Game Board.

starting position (top left corner) to a target (bot-tom right). There are two types of objects on theboard, Type := {bomb, tree}, with a property Size :={big, small} in addition of Location. If a scout at-tempts to move into a big tree, the move is blocked;small trees have no effect. If a scout moves intoa bomb the scout is destroyed and a new one iscreated at the starting position. Before every stepthe mother-ship may send a message to the scout.Then the scout moves one step (horizontal or ver-tical) towards the target choosing the shortest pathwhich avoids hazards known by the scout (the A*algorithm is used for this). Initially scouts have noknowledge of the objects in the world; they gain thisknowledge by stepping into objects or by receivinginformation from the mother-ship.

This is an instance of the talking god model dis-cussed previously. The scout is the listener agent(A1), and the mother-ship the talker (A0). Thescouts action policy π1 is fixed (as described above),but we need to learn the message policy µ01.

Rewards are associated with the results of phys-ical actions: a high positive reward (1000) is as-signed to reaching the destination, a large negativereward (-100) to stepping in a bomb, a medium neg-ative reward (-10) to being blocked by a big tree, asmall negative reward to every step (-1). Further-more, sending a message has a small negative re-ward proportional to the number of attributes men-tioned in the message (-2 per attribute, to discouragethe talker from sending useless information). Themessage ∅ is given zero cost; this is done in order to

200 500 1000 1500 2000 2500 3000 3500 4000 4500 5000250

300

350

400

450

500

550

600

650

Training Cycles

Rew

ard

optimal propertiesall propertiesTYPE only

Figure 3: Simple Game Learning Results

State Best Action Learnt(and possible sentence realisation)

{ TREE-BIG-LEFT } ∅-SILENCE-

{ BOMB-BIG-FRONT } BOMB-FRONTThere is a bomb in front of you

{ TREE-SMALL-LEFT, TREE-BIG-RIGHTTREE-BIG-RIGHT } There is a big tree to your right{ BOMB-BIG-FRONT,BOMB-SMALL-LEFT, TREE-BIG-RIGHTTREE-BIG-RIGHT, There is a big tree to your rightTREE-SMALL-BACK }

Table 1: Examples of learnt actions.

learn when not to talk.

Learning is done as follows. We designed fivemaps of 11 × 11 cells, each with approximately 15bombs and 20 trees of varying sizes placed in strate-gic locations to make the scouts task difficult (oneof these maps is depicted in Figure 2; an A* pathwithout any knowledge and one with full knowl-edge of the board are shown as dotted and dashed ar-rows respectively). A training epoch consists of ran-domly drawing one of these maps and running a sin-gle game until completion. The SARSA algorithmis used to learn the message policy, with ε = 0.1and γ = 0.9. The states sW and s1 are encodedto represent the location of objects surrounding thescout, relative to its direction (i.e. objects directly infront of the agent always receive the same locationvalue). To speed up training, we only consider the 8cells adjacent to the agent.

Figure 3 shows the results of these experiments.For comparison, we note that completing the game

295

with a uniformly random talking policy results in anaverage reward of less than −3000 meaning that onaverage more than 30 scouts die before the target isreached. The dashed line indicates the reward ob-tained during training for a policy which does notuse the size attribute, but only type and location.This policy effectively learns that both bombs andtrees in front of the agent are to be communicated,resulting in an average reward of approximately 400,and reducing the average number of deaths to lessthan 2. The solid line represents the results obtainedby a policy that is forced to use all attributes. De-spite the increase in communication cost, this pol-icy can distinguish between small and large trees,and so it increases the overall reward two-fold. Fi-nally, the dotted line represents the results obtainedby a policy that can choose whether to use or not thesize attribute. This policy proves to be even moreeffective than the previous one; this means that ithas learnt to use the size attribute only when it isnecessary. Some optimal (state,action) pairs learntfor this policy are shown in Table 1. The first threeshow correctly learnt optimal actions. The last is anexample of a wrongly learnt action, due to the statebeing rare.

These are encouraging results, since they demon-strate in practice how optimal policies may be learntfor message planning. However, it should be clearform this example that, as we increase the numberof types, attributes and values, this approach will be-come unfeasible. This is discussed in the next sec-tion.

4 Back-Off Policies

One of the main problems when using RL in prac-tical settings (and, more generally, using MDPs) isthe exponential growth of the state space, and con-sequently of the learning time required. In our case,if there are M attributes, and each attribute pi hasN(pi) values, then there are S =

∏Mi=1 N(pi) pos-

sible sub-domains, and up to 2S states in the statespace. This exponential growth, unless addressed,will render MDP learning unfeasible.

NL domains are usually rich with structure, someof it which is known a priori. This is the case intext generation of descriptions for computer games,where we have many sources of information about

the objects of discourse (i.e. world ontology, dy-namics, etc.) We propose to tackle the problem ofstate dimensionality explosion by using this struc-ture explicitly in the design of hierarchical policies.

We do so by borrowing the back-off smoothingidea from language models. This idea can be statedas: train a set of probability models, ordered by theirspecificity, and make predictions using the most spe-cific model possible, but only if there is enoughtraining data to support its prediction; otherwise,back-off to the next less-specific model available.

Formally, let us assume that for every states we can construct a sequence of K embeddedpartial representations of increasing complexity,(s[1], . . . , s[k], . . . , s[K]). Let us denote π̂[k] a se-quence of policies operating at each of the partialrepresentation levels respectively, and let each ofthese policies have a confidence measurement ck(s)indicating the quality of the prediction at each state.Since k indicates increasingly complex, we requirethat ck(s) ≥ ck′(s) if k < k′. Then, the most spe-cific policy we can use at state s can be written as:

k∗s := arg maxk

{k · sign (ck(s)− θ)} (1)

A back-off policy can be implemented by choosing,at every state s the most specific policy available:

π(s) = π̂[k∗s ](s[k∗

s ]) (2)

We can use a standard off-policy learning algo-rithm (such as Q-learning or SARSA) to learn all thepolicies simultaneously. At every step, we draw anaction using (2) and update all policies with the ob-tained reward1. Initially, the learning will be drivenby high-level (simple) policies. More complex poli-cies will kick-in progressively for those states thatare encountered more often.

In order to implement back-off policies for oursetting, we need to define a confidence function ck.A simple confidence measure is the number of timesthe state s[k] has been previously encountered. Thismeasure grows on average very quickly for small kstates and slowly for high k states. Nevertheless, re-occurring similar states will have high visit counts

1An alternative view of back-off policies is to consider that asingle complete policy is being learnt, but that actions are beingdrawn from regularised versions of this policy, where the regu-larisation is a back-off model on the features. We show this inAppendix I

296

0 1000 2000 3000 4000 5000500

550

600

650

700

750

800

850

training epochs

Ave

rage

Tot

al R

ewar

d (1

00 R

uns)

Full State, without noise objectsFull State, with 40 noise objectsSimple State, without noise objectsSimple State, with 40 noise objectsBack−Off with 40 noise objects

Figure 4: Back-Off Policy Simulation Results.

for all k values. This is exactly the kind of behav-iour we require.

Furthermore, we need to choose a set of repre-sentations of increasing complexity. For example,in the case of n-gram models it is natural to chooseas representations sequences of preceding words ofincreasing size. There are many choices open to usin our application domain. A natural choice is to or-der attribute types by their importance to the task.For example, at the simplest level of representationobjects can be represented only by their type, at asecond level by the type and colour, and at a thirdlevel by all the attributes. This same technique couldbe used to exploit ontologies and other sources ofknowledge. Another way to create levels of repre-sentation of increasing detail is to consider differentperceptual windows. For example, at the simplestlevel the agent can consider only objects directly infront of it, since these are generally the most im-portant when navigating. At a second level we mayconsider also what is to the left and right of us, andfinally consider all surrounding cells. This could bepursued even further by considering regions of in-creasing size.

4.1 Simulation Results

We present here a series of experiments based onthe previous game setting, but further simplified topinpoint the effect of dimensionality explosion, andhow back-off policies can be used to mitigate it.

We modify the simple game of Section 3 as fol-lows. First, we add a new object type, stone, and a

new property Colour := {red, green}. We let all treesbe green and big and all bombs red and small, andfurthermore we fix their location (i.e. we use onemap instead of five). Finally we change the worldbehaviour so that an agent that steps into a bomb re-ceives the negative reward but does not die, it contin-ues until it reaches the target. All these changes aredone to reduce the variability of our learning base-line.

At every game we generate 40 stones of randomlocation, size and colour. Stepping on stones has nophysical effect to the scout and it generates the samereward as moving into an empty cell, but this is un-known to the talker and will need to be learnt. Thesestones are used as noise objects, which increase thesize of the state space. When there are no noise ob-jects, the number of possible states is 38 ≈ 6.5K(the actual number of states will be much smallersince there is a single maze). Noise objects can take2 × 2 = 4 possible forms, so the total number ofstates with noise objects is (3 + 4)8 ≈ 6M . Evenwith such a simplistic example we can see how dras-tic the state dimensionality problem is. Despite thefact that the noise objects do not affect the rewardstructure of our simple game, reinforcement learn-ing will be drastically slowed down by them.

Simulation results2 are shown in Figure 4. Firstlet us look at the results obtained using the full staterepresentation used in Section 3 (noted Full State).Solid and dotted lines represent runs obtained withand without noise objects. First note that learningwithout noise objects (dotted circles) occurs mostlywithin the first few epochs and settles after 250epochs. When noise objects are added (solid cir-cles) learning greatly slows down, taking over 5Kepochs. This is a typical illustration of the effect thatthe number of states has on the speed of learning.

An obvious way to limit the number of states isto eliminate features. For comparison, we learneda simple representation policy with states encod-ing only the type of the object directly in front ofthe agent, ignoring its colour and all other locations(noted Simple State). Without noise, the performance(dotted triangles) is only slightly worse than that ofthe original policy. However, when noise objects

2Every 200 training epochs we run 100 validation epochswith ε = 0. Only the average validation rewards are plotted.

297

are added (solid triangles) the training is no longerslowed down. In fact, with noise objects this policyoutperforms the original policy up to epoch 1000:the performance lost in the representation is madeup by the speed of learning.

We set up a back-off policy with K = 3 as fol-lows. We use the Simple representation at k = 1,plus a second level of representation where we rep-resent the colour as well as the type of the object infront of the agent, and finally the Full representationas the third level. As the ck function we use statevisit counts as discussed above and we set θ = 10.Before reaching the full policy (level 3), this policyshould progressively learn to avoid bombs and treesdirectly in front (level 1), then (level 2) not avoidsmall trees directly in front. We plot the perfor-mance of this back-off policy (stars) in Figure 4. Wesee that it attains very quickly the performance ofthe simple policy (in less than 200 epochs), but thecontinues to increase in performance settling within500 epochs with a performance superior to that ofthe full state representation, and very close to that ofthe policies operating in the noiseless world.

Despite the small scale of this study, our resultsclearly suggest that back-off policies can be usedeffectively to control state dimensionality explosionwhen we have strong prior knowledge of the struc-ture of the state space. Furthermore (and this may bevery important in real applications such as game de-velopment) we find that back-off policies produce anatural to feel to the errors incurred while learning,since policies develop progressively in their com-plexity.

5 Conclusion

We have developed a formalism to learn interac-tively the most informative message content giventhe state of the listener and the world. We formalisedthis problem as a MDP and shown how RL may beused to learn message policies even when the envi-ronment dynamics are unknown. Finally, we haveshown the importance of tackling the problem ofstate dimensionality explosion, and we have pro-posed one method to do so which exploits explicita priori ontological knowledge of the task.

ReferencesR. Cole, J. Mariani, H. Uszkoreit, A. Zaenen, and V. Zue.

1997. Survey of the State of the Art in Human Lan-guage Technology. Cambridge University Press.

T. G. Dietterich. 2000. Hierarchical reinforcement learn-ing with the MAXQ value function decomposition.Journal of Artificial Intelligence Research, 13:227–303.

J. Henderson, O. Lemon, and K. Georgila. 2005. Hybridreinforcement/supervised learning for dialogue poli-cies from communicator data. In 4th IJCAI Workshopon Knowledge and Reasoning in Practical DialogueSystems.

S. Singh, D. Litmanand, M. Kearns, and M. Walker.2002. Optimizing dialogue management with re-inforcement learning: Experiments with the njfunsystem. Journal of Artificial Intelligence Research,16:105–133.

R. S. Sutton and A. G. Barto. 1998. ReinforcementLearning. MIT Press.

K. van Deemter and E. Krahmer. (to appear). Graphs andbooleans. In Computing Meaning, volume 3 of Stud-ies in Linguistics and Philosophy. Kluwer AcademicPublishers.

J. D. Williams, P. Poupart, and S. Young. 2005. Fac-tored partially observable markov decision processesfor dialogue management. In 4th IJCAI Workshop onKnowledge and Reasoning in Practical Dialogue Sys-tems.

6 Appendix I

We show here that the expected reward for a partialpolicy πk after an action a, noted Qπk(s, a), can beobtained from the expected reward of the full pol-icy Qπ(s, a) and the conditional state probabilitiesP (s|s[k]). We may use this to compute the expectedrisk of any partial policy Rπk(s) from the full policy.

Let Tk(s) :={s′ ∈ S | s′[k] = s[k]

}be the sub-

set of full states which map to the same value of s.Given a state distribution P (s) we can define distri-butions over partial states:

P (s[k], s[j]) =∑

s′∈Tk(s)∩Tj(s)

P (s′) . (3)

Since∑

s′∈Tk(s) P (s′|s[k]) = 1, we haveP (A|s[k]) =

∑s′∈Tk(s) P (A|s′)P (s′|s[k]), and so:

Qπk(s, a) =∑

s′∈Tk(s)

P (s′|s[k])Qπ(s′, a) . (4)

298