An Examination of Criticality-Sensitive Approaches to Coordination

Pedro Szekely, Rajiv T. Maheswaran, Robert Neches, Craig M. Rogers and Romeo SanchezUniversity of Southern California, Information Sciences Institute

4676 Admiralty Way, Suite 1001, Marina Del Rey, CA 90292

Marcel Becker and Stephen FitzpatrickKestrel Institute

3260 Hillview Avenue, Palo Alto, CA 94304

Gergely Gati, David Hanak, Gabor Karsai and Chris van BuskirkVanderbilt University, Institute for Software Integrated Systems,

Box 1829, Station B, Nashville, TN 37235

Abstract

In this work, we address the problem of coordinating the dis-tributed execution of plans and schedules by multiple agentssubject to a number of different execution uncertainties. Thecoordination of multi-agent teams in uncertain, dynamic do-mains is a challenging problem requiring the fusion of tech-niques from many disciplines. We describe an approachbased on the dynamic and selective use of a family of differ-ent problem-solving strategies that combine stochastic stateestimation with repair-based and heuristic-guided planningand scheduling techniques. This approach is implementedas a cognitive problem-solving architecture that combines (i)a deliberative scheduler, which performs partially-centralizedsolution repair, (ii) an opportunistic scheduler, which locallyoptimizes resource utilization for plan enhancement, and (iii)a downgrader, which proactively guides the execution intoregions of higher likelihood of success. This paper charac-terizes the complexity of the problem through examples andexperiments, and discusses the advantages and effectivenessof the implemented solution.

IntroductionThis work addresses the collaborative and coordinated ex-ecution of activities of a multi-agent team. Joint opera-tions in military settings, project/personnel management inglobal enterprise settings, and multiple-rover/UAV missionsin science-discovery or search-and-rescue settings are someexamples of complex and dynamic execution environmentswhere effective and efficient coordination is crucial to thesuccessful achievement of a set of pre-defined mission goals.The complexity of the problem is due to (i) uncertainties as-sociated with the execution environment, which causes anexplosion on the number of system states for which coursesof action need to be determined, (ii) dynamism, meaning

Copyright c© 2006, American Association for Artificial Intelli-gence (www.aaai.org). All rights reserved.

that parameters that define the initial models of uncertaintyand constraints may change in the execution phase, thus re-quiring online reasoning and decisions (iii) partial observ-ability of global goals and other agents’ plans, where agents’local policies may not align towards a coherent global strat-egy, and (iv) distributed decision-making authority, whichmay cause conflicts and delays on the selection of the ap-propriate course of action.

A centralized problem-solver eliminates the issues of par-tial observability and conflicting authority, but puts a highcomputational burden on a single agent. When boundedrationality is considered, this agent cannot make decisionsin a timely manner. While we do not address communica-tion failures or delay in this study, centralization will suf-fer when these extensions are added. Traditional AI meth-ods that can model the distributed nature of the problemsuch as SAT or DCR techniques (DCSP, DCOP) cannot cur-rently handle uncertainty without cumbersome encodings.Standard OR methods such as mathematical programmingor decision-theoretical approaches are built to handle uncer-tainty yet they cannot be immediately utilized for our prob-lem without addressing the partial-observability and multi-agent decision-making issues. Developing techniques thathandle all the aspects of these settings is an emerging area ofresearch. This paper takes a step towards that goal while alsoaddressing the reasoning limitations that are brought aboutby scale, dynamism and bounded rationality which requiremanageable state spaces and quick reasoning.

We combine various elements of these techniques to con-struct a hybrid-scheme with multiple reasoning components.To address uncertainty, each agent constructs local estimatesof the likelihood of success for its potential activities, andpropagates them to relevant agents whose activities may beaffected. These likelihoods are utilized to calculate the im-portance of various activities. We construct a local stateas the union of non-probabilistic classification and a low-

dimensional probabilistic state vector denoted as a profile.The reasoning is performed by three components: (1) thedeliberative scheduler, (2) the opportunistic scheduler, and(3) the downgrader. The deliberative scheduler performs dy-namic partial-centralization of the relevant agents when aproblem is detected in some substructure of the team reward.It then extracts potential solutions and proposes modifica-tions to remedy the problem. Due to the complexity of thistask and the nature of the anticipated failure, this processoccurs over several decision epochs with decision windowsthat begins slightly in the future. The opportunistic sched-uler performs schedule modifications in the near-term. Us-ing only local information, it is capable of making decisionson a faster time-scale, and attempts to increase the qualityand probability of success by utilizing free resources. Thedowngrader complements the previous two components byfreeing resources both in the near-term and the future basedon the evolution of the system up to the present.

ModelWe model our problem with CTAEMS, which is an adap-tation of the TAEMS formulation (Decker & Lesser 1993)representing distributed multi-agent coordination problems.In this formulation, each agent has the potential to execute aset of activities or methods over a finite time horizon brokenup into decision epochs. Once an agent starts a method, itcan either abort it or wait for its completion in order to startanother method. Each method produces some quality andtakes a duration. These quantities are drawn from probabil-ity distributions. Failure occurs when a method yields zeroquality. A failure is modeled when quality zero is listed inits probability distribution with non-zero probability. Oth-erwise, it is an unmodeled failure. Each method belongs toa window that prescribes the earliest time at which it maybegin and the latest time at which it may complete in orderto obtain positive quality.

The qualities achieved by the executed methods are ag-gregated through a tree of quality accumulation functions(QAFs) to yield the reward for the multi-agent team. Thistree consists of a hierarchy of nodes, where each leaf noderepresents a method associated to a single agent that can ex-ecute it. The non-leaf nodes are tasks with associated QAFsthat define their quality as a function of the qualities of thechildren. The children of a task can be methods or tasks.This hierarchy defines how the quality of the root node, theteam reward, is computed from the individual methods thatagents execute.

The QAFs include: (i) MIN, which yields the minimumvalue of qualities of the children, used to model situationswhere all children must succeed in order for the parent toaccumulate quality. (ii) MAX, which yields the maximumvalue of qualities of the children, used to model situationswhere at least one of the children must achieve positive qual-ity. (iii) SUM, which adds the qualities of the children, alsoused to model situations where some children must suc-ceed. (iv) SyncSUM, which adds the qualities of the chil-dren whose start time is the earliest start time of all children.This encourages child nodes to align their execution to beginat the same time.

An additional complexity is the possibility of non-localeffects (NLEs) between nodes. An NLE is an enabling cou-pling between nodes, a directed link between a source and atarget. Targets started before all sources have accumulatedquality will fail and accumulate zero quality regardless ofthe their probability distributions. If the target is a task, theNLE applies to all descendant nodes.

The challenge for the multi-agent team is to make the ap-propriate choices of methods to execute that yield the high-est quality at the root of the CTAEMS hierarchy. Effectivepolicies must react dynamically to the uncertainties in thedurations and qualities of methods in an initial schedule andcreate new schedules as the sample path unfolds. Reactingto the state of the system is made more difficult by the factthat agents cannot see the entire CTAEMS hierarchy. Agentshave a subjective view consisting of the methods they canexecute and their direct ancestry to the root node along withnodes that are one-step removed due to NLEs. This mod-els the partial-observability inherent in multi-agent systemswhere an agent cannot monitor the entire system and oftenis not aware of certain parts of the team. This aspect occursto a greater extent in large-scale settings.

ExampleIn order to illustrate our model, and part of the complexityof the problem, we discuss an example in this section. TheCTAEMS structure for our example can be seen in Figure 1.

This problem consists of three phases (or windows), de-noted as setup, deploy and engage modeled using SUMQAFs. Each window has two different MAX tasks, eachwith two children methods. The problem has two agents,alpha and bravo. Each method belongs either to plan A orplan B. The root, S1, is a MIN QAF, requiring each phase(window) to produce quality in order for the overall prob-lem to succeed. Figure 1 shows the enabling relationshipsamong different methods. For example, method p1aA en-ables method p2aA, which implies that p1aA must executesuccessfully before method p2aA is worth starting.

The layout of Figure 2 illustrates some of the complexi-ties of the problem. For simplicity, each method has beenlabeled with the information regarding the schedule whereit participates, and the agent who executes it. Bars corre-spond to the probability distribution for the durations of eachmethod, here having three outcomes. Plan A is the initialschedule, which leads to a higher accumulated quality but itis somewhat risky, since its activities may take longer. Onthe other hand, plan B produces less quality but it is safer.Note that uncertainty in the method durations can lead tofailures. Furthermore, enabling dependencies increase theprobability of failure for future phases of the problem. Wecan see an example of such failing conditions in Figure 3.Here, Plan A gets delayed given that the setup phase ac-tivities take their maximum duration. Consequently, agentalpha cannot complete its method by the deadline duringthe deploy phase. The failure in the deploy phase along withother enabling conditions preclude the activities in the finalengage phase of the problem which prevents the team fromaccumulating quality.

Figure 1: CTAEMS Example Structure

Figure 2: Time Uncertainty in Example

Our system can predict failures, and trigger timely sched-ule repairs. Determining when to act in uncertain and dis-tributed domains is inherently complex. The general idea isto detect potential problems in the schedule early, in order tohave enough lead time to schedule and execute the enablersof methods needed to repair problems later on. Consideragain Figure 3. Once the method in the setup window hitsits maximum duration, the probability that the alpha agentcompletes its method on time during the deploy phase de-creases (i.e, only two outcomes will allow completion beforethe deadline). Note that this has a cascading effect, since theprobabilities of success for the activities in the engage phasedepend on the success of activities in the deploy phase. Asolution to our example is to anticipate the potential failurein plan A and switch to plan B during the second phase asillustrated in Figure 4.

Figure 3: Failing Execution Trace

In the following sections, we present our approach for dy-namically detecting and repairing potential failures on theschedule based on the probability of success of each method,and its importance with respect to the success of the overallplan.

ApproachOur approach involves the selective combination of differentproblem solving strategies, including partially-centralizedsolution repair (deliberative scheduler), locally optimizedresource allocation (opportunistic scheduler) and proactivere-prioritization (downgrader). These higher-level reason-ing components are triggered and supported by a state man-ager which holds schedules and profiles and an executioncontroller which interacts with the environment and trans-

Figure 4: Execution Trace of Plan B

lates our strategies into actions. The architecture of the CSCcoordinator can be seen in Figure 5.

State Manager and Execution ControllerState Manager The purpose of the state manager is togive the reasoning components the information they needto act. There are essentially two types of such informa-tion: probabilities and importance, denoted by p and α re-spectively in Figure 5. Both types of information are keptabout the current schedule and about potential schedulesthat the agents may engage. The probabilities about thecurrent schedule enable agents to determine whether theyshould change the current schedule. The importance allowsthem to determine the marginal contribution that any indi-vidual method has on the success probability of the currentschedule. The potential probabilities allow agents to reasonabout the probability improvements that could be achievedby engaging new options. The potential importance allowsagents to determine whether methods can still contribute tothe overall quality of the schedule.

The probability and importance of current and poten-tial schedules are global quantities that, in general, dependon the local state of all agents. We have developed dis-tributed algorithms that compute approximations of thesevalues based on local information and approximations re-ceived from other agents. The state manager uses these al-gorithms and the protocols to share local information withother agents.

Schedule The system stores the current schedule in a dis-tributed manner as part of the state graphs maintained by theState Manager instance on each agent. Every method has anassociated scheduled start window and a priority. At the be-ginning of a simulation, these are determined based on theinitial schedule contained in the initial subjective view of anagent. At runtime, the downgrader, the opportunistic anddeliberative schedulers alter the current schedule by modi-fying the start windows and priorities of scheduled methodsor by adding new methods to the schedule. The first twoschedulers operate only on local methods while the delib-

erative scheduler can create a set of schedule changes thatare installed on remote agents. To reduce the potential con-flicts during remote schedule updates running in parallel, alocking mechanism is in place that serializes the distributedinstallation of schedules.Profiles Profiles are designed to reason about uncertainty,and contain (i) a pair representing probability and busy-cycles of a CTAEMS node and (ii) an importance value.These measures are used to evaluate the current and potentialschedules. For the current schedule, probabilities for meth-ods are calculated using information about the distributionsof durations and their execution windows. In addition, wefactor in the probabilities of enabling methods. The busy-cycles are a resource consumption measure also calculatedfrom the duration distributions. These pairs are propagatedthroughout the system via nearest neighbor communication.Using these probabilities, we can calculate the importanceof each node to the overall system performance based on theimportance values in the local view of each node, again in adistributed manner. The potential profiles consist of a singlepair for the methods denoting the likelihood and costs inde-pendent of the current schedule and a set of pairs for highernodes, which represent potential solutions for achieving pos-itive quality at that node. We use these profiles to obtainpotential importance, which determines if a method or nodecan still contribute to helping the team goal succeed. As inthe case of the scheduled profiles and importance, the poten-tial measures are propagated in a distributed manner throughcommunication to relevant neighbors.Execution Controller The execution controller managesthe flow of data between the agent and the underlying execu-tion or simulation environment, including inter-agent mes-saging. It converts execution environment events, such asmethod start and stop, into a platform-neutral representa-tion for portability. It runs at a higher priority level than therest of the agent and avoids priority inversions when access-ing shared data, allowing it to continue to execute scheduledmethods even when the rest of the agent lags behind the paceof execution environment events.

The execution controller contains its own distributedwindow-sliding scheduler. A scheduled method’s executionwill be delayed until its enabling conditions are met, evenif some enabling conditions depend upon the state of meth-ods residing in other agents; the necessary inter-agent com-munications will take place autonomously between execu-tion controller instances. A method will be dropped fromthe schedule when one or more of its enabling conditionshas failed. When a set of methods is to be initiated syn-chronously on a set of agents, the execution controllers willcommunicate to ensure that synchronized execution takesplace in the execution environment once all enabling con-ditions have been met.

Higher-Level Reasoning Components

Deliberative Scheduler The CTAEMS task structure pro-vides a hierarchical representation for the entire family ofpossible schedules for a particular input scenario or prob-

P, α

P, α

P, α

Downgrader

Options

LocalState

Remote

Schedule

DeliberativeScheduler

OpportunisticScheduler

P, α

P, αP, α

Coordination Reasoners

State Manager

Execution Controller

Environment

Figure 5: CSC Architecture

lem. The scheduler component is responsible for traversingthis hierarchy and selecting the set of methods, or activities,that should be executed by the different agents with the goalof maximizing global task quality. Method execution is con-strained by feasibility intervals, scenario specific restrictionson earliest method start and latest method finish times; sin-gle threadedness, each agent can only execute one methodat a time; method interactions, dependencies on the order ofexecution of certain methods.

The scheduler is implemented as a configurable, heuristic-guided search engine that traverses the task hierarchy de-fined in the CTAEMS problem description provided as in-put, selects a set of methods to execute, and assigns start-time windows for methods in this set. The search tree ex-panded by the search engine follows the topology of the taskstructure defined in the CTAEMS structure: starting from apre-defined root node, the algorithm recursively expands thesearch tree for the children of the selected node. The or-der in which the children of a particular node are selected,the strategy used to expand the sub-tree, the allocation ofagent’s time for method execution, and the metric used toselect the results for different types of nodes is parameteriz-able. The cognitive problem solving mechanism uses severaldifferent pre-configured search strategies: fast repair-basedstrategies that only try to fill existing gaps in agent’s avail-ability, load balance techniques that try to guarantee mini-mum amounts of positive quality for certain tasks, robust-ness enhancement strategies that schedules redundant meth-ods to increase probability of positive quality achievement,

and schedule compression techniques combined with someother strategy to reduce agent’s idle time and increase ex-pected quality.

The different agents participating in the execution do nothave a complete view of the task network. Each agent ownsa subset of all the methods and tasks, and can only see theconstraints and nodes directly affecting the owned node. Inresponse to scheduling decisions, or execution events, eachagent perform propagation on its local constraint network.This will cause modifications to both owned and visiblenodes. It then sends to other remote agents the changes onits owned nodes. When receiving this information, remoteagents update their local views or the other agent’s node,and perform constraint propagation to keep the state of theirowned nodes consistent. The updated state of owned nodesfor each agent is then sent to other agents, until there are notmore modifications in the state of owned nodes. If, duringpropagation, an agent detects conflicts involving one of itsowned nodes, this agent is responsible for resolving the con-flict. At this point, the agent may then trigger more complexproblem-solving components that would require the creat-ing of a cluster of agent to solve the problem in a partiallycentralized fashion.

Opportunistic Scheduler The opportunistic scheduler isinstantiated whenever there is a gap where an agent is notperforming a high priority method. At this point, the agentestimates the window of available resource, namely execu-tion time before the next high priority method is scheduledto begin. The agent then may choose to begin executing a

method that is not part of the current schedule. The goalis to choose the best method to execute given the resourcerestrictions, without harming the existing schedule. Usinglocal knowledge of the reward function and methods sched-uled to be executed, the agent dismisses methods that maycause harm should they get aborted by higher priority meth-ods introduced by the deliberative scheduler. Then, the op-portunistic scheduler utilizes the probabilities and impor-tance measures in the profile to determine the method withthe greatest likelihood of helping the team. This method isinserted with medium priority such that any (existing or fu-ture) alterations to the schedule by the deliberative schedulerare not affected. Every agent runs an opportunistic schedulerto maximize the use of its execution capabilities.

Downgrader The downgrader utilizes the potential im-portance to determine whether methods in the schedule canstill contribute to the team goal. The priorities of methodsare reduced if it is determined that they no longer can con-tribute, freeing agent resources for other scheduler compo-nents that schedule methods at higher priority. Downgrad-ing gives methods installed by the deliberative scheduler ahigher likelihood of succeeding by removing unimportantmethods that start earlier, and gives the opportunistic sched-uler more chances to be instantiated. The preceding effectsserve to enhance the robustness of the system.

ExperimentsWe have described in previous sections the main compo-nents of our system to handle distributed scheduling prob-lems under uncertainty, dynamism, temporal constraints andpartial observability. One remaining challenge is to producea set of experiments that could meaningfully test our sys-tem, despite the complexity and quantity of interactions inthe problem. Our aim is to introduce a set of problems thatshare some systematic structure in order to present somepreliminary evaluation of our work. In particular, we havedesigned our problems taking into account two importantfactors: NLE chains and failure rate. Each factor has beencategorized into four subclasses: empty, low, medium andhigh. Each subclass relates to the number of NLE chainsor failures introduced into a particular problem. The entireproblem space is then partitioned as a cross product of thefactors and their subclasses. Finally, each subspace of prob-lems is built using an increasing degree of window overlap.NLEs and failures are introduced randomly for each prob-lem, all of which have four agents for the purpose of thisevaluation.

Our testing platform includes a cluster of distributed 32-bit Intel Linux systems with at least 1.5GB of memory. Oneof them serves as the simulation controller and the rest areassigned to agents.

Figure 6 displays a plot with our solutions. The plotshows the difference in quality between our base system andour higher-level reasoning enhancements. The data pointsrepresent average results over ten trials per problem, un-der different random seeds. Each problem is provided withan initial schedule, that may fail in accordance to a failureprobability distribution. Negative data points illustrate those

problems in which our system produces a reduction in qual-ity over the base system. However, we can see that for mostof the problems our system consistently returns better qual-ity solutions, anticipating and recovering from failures, andgenerating alternative schedules.

Figure 6: Quality Improvement Over Base System

Related WorkThe coordination of multi-agent systems in dynamic,stochastic, temporally-constrained and partially observabledomains is a challenging problem. The most generaldecision-theoretic models for this problem have been provedto be extremely complex (NEXP-complete) (Shen, Becker,& Lesser 2006). In order to lower complexity, some mod-els restrict the types of interactions allowed in the sys-tem, or the features they are able to address. One ofsuch models is DEC-MDP (Goldman & Zilberstein 2003;Beynier & Mouaddib 2005). The decentralized MDP modelusually bases its computation on local policies, with very lit-tle or no communication among agents (Beynier & Mouad-dib 2005). Although, our approach also computes local es-timates of the probability of success for each activity, suchestimates are propagated in a distributed manner to the in-teracting agents.

Another way of modeling the multi-agent coordinationproblem is through traditional AI methods. Distributed Con-straint Optimization Problems (DCOP) have been adaptedto capture the locality of interactions among agents with asmall number of neighbors (Cox, Durfee, & Bartold 2005),but it fails to capture the uncertainty factor in the generalproblem. Network Distributed POMDPs have been pro-posed to address these issues (Nair et al. 2005). Unfortu-nately, they solve only small problems.

Decision-theoretic planning can also be used to modelour problem (Littman, Goldsmith, & Mundhenk 1998). Inthis model, execution monitoring of the system and replan-ning are very important. Conditional plans are generatedin order to deal with contingencies during execution. Re-planning is invoked when an agent identifies unsatisfied,

or likely to fail conditions. However, the requirements fora rich model for actions and time are generally problem-atic for planning techniques based on MDP, POMDP, SAT,CSP, planning graphs or state space encodings (Bresina et al.2002). Finally, scalability is also an issue given the size ofthe problem and the number of potential contingencies. Ourapproach tries to alleviate these problems by being proac-tive, and focusing on activities with high probability of fail-ure. Some other techniques that follow similar reasoningare Just-In-Case (JIC) contingency scheduling (Drummond,Bresina, & Swanson 1994) and Mahinur (Onder & Pollack1999).

ConclusionPreliminary evaluation indicates that the full CSC systemsignificantly outperforms the baseline both in terms of theaverage quality of schedule execution (see Figure 6) andin terms of avoidance of schedule execution failures. Thepreliminary ablation studies show no significant differencein the scores obtained using the deliberative scheduler only,the opportunistic scheduler only or the deliberative and op-portunistic scheduler together. The deliberative and oppor-tunistic schedulers use very different approaches, so we aresurprised to see no significant difference in their scores. Wespeculate that the structure of the problems is such that bothapproaches identify similar solutions. In order to verify thishypothesis, we are now conducting experiments to analyzethe detailed behavior of the system on specific problem in-stances. We are also working on a range of evaluation toolsranging from problem generators, visualization tools andcentralized algorithms to compute benchmark scores.

AcknowledgmentsThe work presented here is funded by the DARPA COOR-DINATORS Program under contract FA8750-05-C-0032.The U.S.Government is authorized to reproduce and dis-tribute reports for Governmental purposes notwithstandingany copyright annotation thereon. The views and conclu-sions contained herein are those of the authors and shouldnot be interpreted as necessarily representing the officialpolicies or endorsements, either expressed or implied, ofany of the above organizations or any person connected withthem.

ReferencesBeynier, A., and Mouaddib, A. 2005. A polynomial al-gorithm for decentralized markov decision processes withtemporal constraints. In Proceedings of the 4th Interna-tional Joint Conference on Autonomous Agents and MultiAgent Systems(AAMAS-05).Bresina, J.; Dearden, R.; Meuleau, N.; Smith, D.; andWashington, R. 2002. Planning under continuous time andresource uncertainty: A challenge for ai. In Proceedings ofUAI.Cox, J.; Durfee, E.; and Bartold, T. 2005. A dis-tributed framework for solving the multiagent plan coor-dination problem. In Proceedings of the 4th International

Joint Conference on Autonomous Agents and Multi AgentSystems(AAMAS-05), 821–827.Decker, K., and Lesser, V. 1993. Quantitative modeling ofcomplex computational task environments. In Proceedingsof the 11th National Conference on Artificial Intelligence(AAAI-93), 217–224.Drummond, M.; Bresina, J.; and Swanson, K. 1994. Just-in-case scheduling. In Proceedings of the 12th NationalConference on Artificial Intelligence (AAAI-94), 1098–1104.Goldman, C., and Zilberstein, S. 2003. Optimizing infor-mation exchange in cooperative multi-agent systems. InProceedings of the Second International Joint Conferenceon Autonomous Agents and Multi Agent Systems(AAMAS-03), 137–144.Littman, M.; Goldsmith, J.; and Mundhenk, M. 1998. Thecomputational complexity of probabilistic planning. Artifi-cial Intelligence Research (JAIR) 9:1–36.Nair, R.; Varakantham, P.; M.‘Tambe; and Yokoo, M.2005. Networked distributed pomdps: A synthesis ofdistributed constraint optimization and pomdps. In Pro-ceedings of the 4th International Joint Conference on Au-tonomous Agents and Multi Agent Systems(AAMAS-05).Onder, N., and Pollack, M. 1999. Conditional, proba-bilistic planning: A unifying algorithm and effective searchcontrol mechanisms. In Proceedings of the 16th NationalConference on Artificial Intelligence (AAAI-99), 577–584.Shen, J.; Becker, R.; and Lesser, V. 2006. Agent Interactionin Distributed MDPs and its Implications on Complexity.In Proceedings of the Fifth International Joint Conferenceon Autonomous Agents and Multi-Agent Systems.