2 1541-1672/06/$20.00 © 2006 IEEE IEEE INTELLIGENT SYSTEMSPublished by the IEEE Computer Society

M a c h i n e E t h i c s

An Approach toComputing EthicsMichael Anderson, University of Hartford

Susan Leigh Anderson, University of Connecticut

Chris Armen, Amherst College

I t might seem impossible to “compute” ideas that humans feel most passionately about

and have such difficulty codifying: their ethical beliefs. Despite this, our interdisci-

plinary team of an ethicist and computer scientists believe that it’s essential that we try,

since there will be benefits not only for the field of artificial intelligence, but ethics as

well. We’ve been attempting to make ethics com-putable for three reasons. First, to avert possibleharmful behavior from increasingly autonomousmachines, we want to determine whether one canadd an ethical dimension to them. Second, we wantto advance the study of ethical theory by making itmore precise. Finally, we want to solve a particularproblem in ethical theory—namely, to develop adecision procedure for an ethical theory that involvesmultiple, potentially competing, duties.

We’ve adopted the action-based approach to eth-ical theory, where the theory tells us how we shouldact in ethical dilemmas. This approach lends itselfto machine implementation by giving the agent eithera single principle or several principles to guide itsactions, unlike other approaches that don’t clearlyspecify the correct action in an ethical dilemma. Agood action-based ethical theory should have thesequalities:1

• Consistency. The theory shouldn’t contradict itselfby saying that a single action in a given set of cir-cumstances is simultaneously right and wrong.

• Completeness. It should tell us how to act in anyethical dilemma in which we might find ourselves.

• Practicality. We should be able to follow it.• Agreement with intuition. The actions it requires

and forbids should agree with expert ethicists’intuition.

The approach to computing ethics that we describein this article is illustrated by MEDETHEX,2 a systemthat uses machine learning to resolve a biomedical

ethical dilemma. As you’ll see, for a dilemma involv-ing three ethical duties and 18 possible cases, MED-ETHEX needed only four training sets to create anethically significant decision principle that coveredthe remaining cases. This, we believe, lends supportfor our approach to computing ethics.

One approach to computing ethicsWe started our project by programming the one

theory that clearly attempts to make ethics com-putable: Hedonistic Act Utilitarianism. According toone of its creators, Jeremy Bentham, HAU simplyinvolves doing “moral arithmetic.”3 HAU maintainsthat an action is right when, of all the possible actionsopen to the agent, it will likely result in the greatestnet pleasure, or happiness, taking all those affectedby the action equally into account. HAU involvesfirst calculating the units of pleasure and displeasurethat each person affected will likely receive fromeach possible action. It then subtracts the total unitsof displeasure from the total units of pleasure foreach of those actions to get the total net pleasure.The action likely to produce the greatest net pleasureis the correct one. If the calculations end in a tie,where two or more actions are likely to result in thegreatest net pleasure, the theory considers theseactions equally correct.

The program JEREMY4 is our implementation ofHAU with simplified input requirements. JEREMY

presents the user with an input screen that promptsfor an action’s description and the name of a personthat action affects. It also requests a rough estimateof the amount (very pleasurable, somewhat pleasur-

This approach

employs machine

learning to help

determine the ethically

correct action when

different ethical duties

give conflicting advice.

able, not pleasurable or displeasurable, some-what displeasurable, or very displeasurable)and likelihood (very likely, somewhat likely,or not very likely) of pleasure or displeasurethat the person would experience from thisaction. The user enters this data for each per-son affected by the action and for each actionunder consideration. When data entry is com-plete, JEREMY calculates the amount of netpleasure each action achieves. (It assigns 2,1, 0, –1, or –2 to pleasure estimates and 0.8,0.5, or 0.2 to likelihood estimates, and sumstheir product for each individual affected byeach action.) It then presents the user withthe action or actions achieving the greatestnet pleasure.

An ideal version of a system such asJEREMY might well have an advantage over ahuman being in following HAU because youcan program it to do the arithmetic strictly(rather than simply estimate), be impartial,and consider all possible actions. We con-clude, then, that machines can follow HAU atleast as well as human beings and perhapseven better, given the same data that humanbeings would need to follow the theory.

Even though HAU is consistent, complete,and can be made practical, most ethicistsbelieve that it fails the test of agreement withintuition. Despite John Stuart Mill’s heroicattempt in chapter five of Utilitarianism toshow that considerations of justice can besubsumed under the utilitarian principle,5

ethicists generally believe that HAU canallow for the violation of individual rights ifthis will likely result in the greatest net goodconsequences, taking everyone affected intoaccount. One could, for instance, construct acase to show that HAU permits killing oneunimportant person to save the lives of fiveimportant persons. This violates the intuitionthat it’s wrong to kill one person to save sev-eral persons.

We have, however, adopted an aspect ofHAU in our current approach to ethicaldecision making. When applying an ethicalduty to a particular dilemma, we considersuch factors as the duty’s intensity and dura-tion and the number of persons affected—which we have initially combined as thelevel of satisfaction or violation of the dutyinvolved.

A more comprehensive ethical theory

In agreement with W.D. Ross,6 we believethat all single-principle, absolute-duty ethi-cal theories (such as HAU and Kant’s Cate-

gorical Imperative, a principle that requiresyou to act in a way that can be universalized)are unacceptable because they don’t appre-ciate the complexity of ethical decision mak-ing and the tensions that arise from differentethical obligations pulling us in differentdirections.

Ross’s theory consists of seven prima facieduties. A prima facie duty is an obligationthat we should try to satisfy but that can beoverridden on occasion by another, strongerduty. Ross’s suggested list of prima facieduties (which he says can be altered) capturesthe best of several single-principle ethicaltheories, while eliminating defects by allow-ing for exceptions. His suggested duties arethose of

• fidelity—you should honor promises andlive up to agreements that you’ve volun-tarily made,

• reparation—you should make amends forwrongs you’ve done,

• gratitude—you should return favors,• justice—you should treat people as they

deserve to be treated, in light of their pastbehavior and rights they might have,

• beneficence—you should act so as to bringabout the most amount of good,

• nonmaleficence—you should act so as tocause the least harm, and

• self-improvement—you should developyour talents and abilities to the fullest.

The first four duties are Kantian in spirit. Thenext two duties—beneficence and non-maleficence—derive from the single utili-tarian principle. However, they reflect Ross’sinsight that you must separate the possiblegood consequences and the likely harm thatcan be caused. The duty of nonmaleficenceis stronger than that of beneficence, to ac-

count for our intuition that it’s wrong to killone person to save five. Finally, the last duty,that of self-improvement, captures the bestof “ethical egoism” by acknowledging thatwe have a special duty to ourselves that wedon’t have to others.

While everyone agrees that Ross’s dutiesseem intuitively plausible, he doesn’t tell ushow to determine the ethically correct actionwhen the duties give conflicting advice,beyond saying that you should use your intu-ition to resolve the conflict. Unfortunately,this would let you rationalize doing whateveryou feel like doing, by maintaining that aduty that supported that action is the mostimportant one in the dilemma.

Without an objective decision procedure,furthermore, the theory can fail all therequirements of an acceptable action-basedethical theory. In a given ethical dilemma,one of Ross’s duties could tell us that a par-ticular action is right, while another couldtell us that the same action is wrong, makingthe theory inconsistent. By not giving us asingle ethically correct action in thatdilemma, so that we don’t know what weought to do, the theory could also be consid-ered incomplete and impractical. Finally,because you could rationalize doing an actionthat an ethical expert, and most of us, wouldconsider wrong, the theory could fail the testof agreement with intuition.

We’ve concluded that the ideal ethical the-ory incorporates multiple prima facie duties,like Ross’s theory, with some sort of a deci-sion procedure to determine the ethically cor-rect action in cases where the duties give con-flicting advice.

A decision procedure for competing duties

We’ve formulated a method that couldhelp make a multiple prima facie duty the-ory, like Ross’s, workable. Our methodessentially adopts John Rawls’ reflectiveequilibrium approach to creating and refiningethical principles, which goes back and forthbetween particular cases and principles.7

First, we find or create ethical dilemmaswhere tension exists between the prima facieduties and where ethicists have reached aconsensus as to the correct action. (The sys-tem can learn a decision principle only to theextent that ethical experts agree on theanswers to particular dilemmas.) We then usemachine learning to abstract a general deci-sion principle from those cases. Finally, wetest this principle on further cases and refine

JULY/AUGUST 2006 www.computer.org/intelligent 3

In a given ethical dilemma, one of

Ross’s duties could tell us that a

particular action is right, while

another could tell us that the same

action is wrong, making the theory

inconsistent.

M a c h i n e E t h i c s

4 www.computer.org/intelligent IEEE INTELLIGENT SYSTEMS

it as needed to reflect ethicists’ intuitionsabout the correct action in these other cases.

Our method uses a trainer (see figure 1) todevelop the decision principle. It prompts theexpert ethicist for an action’s description andan estimate of each of the affected duties’sat-isfaction or violation level (very satisfied,somewhat satisfied, not involved, somewhatviolated, or very violated). The expert entersthis data for each action under consideration.When data entry is complete, the trainerseeks the intuitively correct action from theexpert. It combines this information with theinput case to form a new training examplethat it stores and uses to refine the decisionprinciple. After such training, the decisionprinciple can be used to provide the correctaction for this case, should it arise in the future,as well as for all previous cases encountered.Furthermore, because the decision principlelearned is the least-specific one required tosatisfy cases seen so far, it might be generalenough to be used to determine correct actionsin previously unseen cases as well.

To capture expert ethical opinion, we usemachine learning, currently inductive-logicprogramming, to learn the relationshipsbetween the duties involved in a particulardilemma.

ILP is a machine learning technique thatinductively learns relations represented asfirst-order Horn clauses, classifying positiveand negative examples of a relation.8 To traina system using ILP, you present it with exam-ples of the target relations, indicatingwhether they’re positive (true) or negative(false). The object of training is for the sys-tem to learn a new hypothesis that, in rela-tion to all input cases, is complete (covers allpositive cases) and consistent (covers no neg-ative cases).

We chose this machine learning techniquefor a number of reasons. First, the properties

of set of prima facie duties aren’t clear. Forinstance, do they form a partial order? Arethey transitive? Do subsets of duties have dif-ferent properties than other subsets? Simplyassigning linear weights to the duties isn’tsufficiently expressive to capture the rela-tionships between those duties.2 ILP providesa rich representation language that’s morelikely to express potentially nonclassical relationships. Furthermore, representing therelationships as Horn clauses lets us auto-matically confirm a decision principle’s con-sistency regarding the relationships betweenduties across all cases. Finally, ILP’s declar-ative representation language lets us morereadily express, consult, and update com-monsense background knowledge regardingduty relationships.

The decision principle learned is based ona predicate, supersedes(Action1, Action2),that’s true when the first of the two actionsthat it is given is ethically preferable to thesecond. We represent each action as anordered collection of values specifying thelevel of satisfaction or violation for each dutyinvolved. We use this range of values: –2 rep-resents a serious violation, –1 represents aless serious violation, 0 indicates that theduty is neither satisfied nor violated, +1 indi-cates minimal satisfaction, and +2 indicatesmaximal satisfaction.

A decision procedure determines the cor-rect action for a particular case. The satis-faction or violation values for the duties ofeach possible action in a given case are pro-vided to the decision procedure, which thendetermines the ethically preferable action (ifany) according to the learned decision prin-ciple. The decision procedure uses resolu-tion, an automatic method of theorem prov-ing, to test the supersedes predicate. Forexample, if supersedes(Action1, Action2) istrue, Action1 supersedes Action2 according

to the decision principle and will be outputby the decision procedure as the ethicallypreferable action.

Ethical advisor systemsA good first step toward the eventual goal

of developing machines that can follow eth-ical principles is creating programs thatenable machines to act as ethical advisors tohuman beings.2 We begin this way for fourpragmatic reasons.

First, one could start by designing an advi-sor that gives guidance to a select group ofpersons in a finite number of circumstances,thus reducing the assignment’s scope.

Second, the general public will probablymore easily accept machines that just advisehuman beings than machines that try to behaveethically themselves. In the first case, it’shuman beings who will make ethical decisionsby deciding whether to follow the machine’srecommendations, preserving the idea thatonly human beings will be moral agents. Thenext step in the Machine Ethics project is likelyto be more contentious: creating machines thatare autonomous moral agents.

Third, a problem for AI in general, and sofor this project too, is how to get neededdata—in this case, the information fromwhich to make ethical judgments. With anethical advisor, human beings can beprompted to supply the needed data.

Finally, ethical theory hasn’t advanced tothe point where there’s agreement, even byethical experts, on the correct answer for allethical dilemmas. An advisor can recognizethis fact, passing difficult decisions that mustbe made in order to act onto the human user.

Figure 2 depicts a general architecture foran ethical advisor system. A user inputsdetails of a particular case into the systemand is presented with the ethically preferableaction in accordance with the decision prin-ciple. A knowledge-based interface providesguidance in selecting the duties involved andtheir satisfaction or violation levels for thecase. The interface uses knowledge derivedfrom ethicists concerning the dimensions andduties of the particular ethical dilemma.Knowledge is represented as finite-stateautomata (FSA) for each duty entailed. Ques-tions pertinent to the dilemma serve as startand intermediate states, and intensities ofduty satisfaction or violation levels (as wellas requests for more information) are finalstates. The input to the interface is the user’sresponses to the questions posed; the outputis a case with duty satisfaction or violation

Ethical dilemma withconsensus view

Trainer

Storedcases

Decisionprinciple

Trainingmodule

Figure 1. Developing a decision principle.

levels corresponding to these responses. Thisinterface provides the experienced guidancenecessary to navigate the subtleties of deter-mining satisfaction or violation levels ofduties in particular cases.

Given the details of the case from theknowledge-based interface, the decision pro-cedure consults the decision principle anddetermines whether one action supersedes allothers in the current case. If it discovers suchan action, it outputs that action as the ethicallycorrect action in this case—that is, the actionthat’s consistent with the system’s training.

An ethical advisor: MEDETHEX

We decided to create an ethical advisor,applying our method of creating a decisionprinciple, using a theory that has only fourprima facie duties, a constrained domain, andlacks a decision procedure: Tom Beauchampand James Childress’s Principles of Biomed-ical Ethics.9 PBE uses Ross’s duties of benef-icence, nonmaleficence, and justice and addsthe principle of respect for autonomy. This lastprinciple reflects the recent shift from a pater-nalistic model of the healthcare worker-patientrelationship to one giving the patient a moreactive role. For a patient’s healthcare decisionto be fully autonomous,10 it must meet threecriteria. First, it must be based on sufficientunderstanding of his or her medical situationand the likely consequences of forgoing treat-ment. Second, it must be sufficiently free ofexternal constraints (for example, pressure byothers or external circumstances, such as alack of funds). Finally, it must be sufficientlyfree of internal constraints (for example, painor discomfort, the effects of medication, orirrational fears or values that will likely changeover time).

We chose PBE and biomedical ethicaldilemmas for five reasons. First, PBE uses amore manageable total of four duties, insteadof Ross’ seven. Second, one member of ourresearch team has a biomedical-ethics back-ground. Third, healthcare workers will likelyhave the information needed to judgewhether a particular duty is involved in anethical dilemma and to judge that duty’sintensity. Fourth, more agreement existsamong biomedical ethicists as to the ethicallypreferable action than in other areas ofapplied ethics. Finally, there’s a pressingneed for ethical advice in this area, as bio-medical research introduces new, challeng-ing ethical dilemmas and as baby boomersbegin to age (many ethical dilemmas involveend-of-life care).

MEDETHEX offers guidance on the follow-ing type of biomedical ethical dilemma:

A healthcare professional has recom-mended a particular treatment for her com-petent adult patient, but the patient hasrejected it. Should the healthcare profes-sional try to change the patient’s mind oraccept the patient’s decision as final?

The dilemma arises because, on the onehand, the healthcare professional shouldn’tchallenge the patient’s autonomy unneces-sarily. On the other hand, the healthcare pro-fessional might have concerns about why thepatient is refusing the treatment—that is,whether the decision is fully autonomous.This dilemma is constrained to three of thefour duties of PBE (nonmaleficence, benef-icence, and respect for autonomy) andinvolves only two possible actions in eachcase. We’ve drawn on the intuitions of AllenBuchanan and Dan Brock11 and our project’sethicist (whose views reflect a general con-

sensus) to determine the correct actions inparticular cases of this type of dilemma.

In the type of dilemma we consider, wecan assign specific meanings to each duty’spossible values. For nonmaleficence,

• –2 means that this action will likely causesevere harm to the patient that could havebeen prevented,

• –1 means that this action will likely causesome harm to the patient that could havebeen prevented,

• 0 means that this action isn’t likely tocause or prevent harm to the patient,

• +1 means that this action will likely pre-vent harm to the patient to some degree,and

• +2 means that this action will likely pre-vent severe harm to the patient.

For beneficence,

• –2 means that the other action would likelyhave improved the patient’s quality of lifesignificantly,

• –1 means that the other action would likelyhave improved the patient’s quality of lifesomewhat,

• 0 means that neither action is likely toimprove the patient’s quality of life,

• +1 means that this action will likelyimprove the patient’s quality of life some-what, and

• +2 means that this action will likely improvethe patient’s quality of life significantly.

For respect for autonomy,

• –1 means not immediately acquiescing to

JULY/AUGUST 2006 www.computer.org/intelligent 5

Decision procedure

User

Knowledge-basedinterface

Seeks advice on anethical dilemna

Storedcases

Decisionprinciples

Figure 2. An architecture for an ethical advisor system.

Given the details of the case from

the knowledge-based interface,

the decision procedure consults the

decision principle and determines

whether one action supersedes all

others in the current case.

M a c h i n e E t h i c s

6 www.computer.org/intelligent IEEE INTELLIGENT SYSTEMS

the patient’s wishes but trying again tochange the patient’s mind,

• +1 means that the healthcare worker actsaccording to the patient’s wishes butbelieves that the patient’s decision isn’tfully autonomous, and

• +2 means that the healthcare worker actsaccording to the patient’s wishes andbelieves that the patient’s decision is fullyautonomous.

(Because this dilemma always involves auton-omy, but never to the extent of forcing a treat-ment on the patient, 0 and –2 aren’t options.)

As an example, consider a specific case ofthe type of dilemma we’re considering. Apatient refuses to take an antibiotic that’salmost certain to cure an infection that wouldotherwise likely lead to his death. He decidesthis on the grounds of long-standing religiousbeliefs that forbid him to take medications.The correct action in this case is for thehealthcare worker to accept the patient’sdecision as final because, although severeharm (his death) will likely result, his deci-sion can be seen as being fully autonomous.The healthcare worker must respect a fullyautonomous decision of a competent adultpatient, even if he or she disagrees with it,because the decision concerns the patient’sbody and a patient should have control overwhat is done to his or her body. This dilemmaappears as training case 1 in table 1. In thiscase, the predicate supersedes(Accept, TryAgain) would be true and supersedes(TryAgain, Accept) would be false.

Training MEDETHEX

We presented MEDETHEX with four posi-tive training cases, drawn from the 18 possi-ble cases, and four negative cases derived by

simply exchanging the actions of the posi-tive training examples. Training case 1 wasjust described in the previous paragraph.

In training case 2, a patient won’t considertaking medication that could only help alle-viate some symptoms of a virus that must runits course. He refuses the medication becausehe has heard untrue rumors that the medica-tion is unsafe. Even though the decision isless than fully autonomous, because it’sbased on false information, the little goodthat could come from taking the medicationdoesn’t justify trying to change his mind. So,the doctor should accept his decision.

In training case 3, a patient with incurablecancer refuses further chemotherapy that willlet him live a few months longer, relativelypain free. He refuses the treatment because,ignoring the clear evidence to the contrary,he’s convinced himself that he’s cancer-freeand doesn’t need chemotherapy. The ethi-cally preferable answer is to try again. Thepatient’s less than fully autonomous decisionwill lead to some harm (dying sooner) anddeny him the chance of a somewhat longerlife (a violation of the duty of beneficence),which he might later regret.

In training case 4, a patient, who has suf-fered repeated rejection from others due to avery large noncancerous abnormal growth onhis face, refuses to have simple and safe cos-metic surgery to remove the growth. Eventhough this has negatively affected his careerand social life, he’s resigned himself to beingan outcast, convinced that this is his lot inlife. The doctor is convinced that his rejec-tion of the surgery stems from depression dueto his abnormality and that having thesurgery could vastly improve his entire lifeand outlook. The doctor should try again toconvince him because so much of an

improvement is at stake and his decision isless than fully autonomous.

Table 1 summarizes the levels of duty sat-isfaction or violation for both of the possibleactions in all four training cases and indicatesthe correct action in each case.

We can more succinctly characterize thecases using the difference between the val-ues for duties in the ethically preferableaction and the values for correspondingduties in the less preferable action. For exam-ple, in training case 1, the differencesbetween the duties of the Accept and the TryAgain actions are –4, –4, 3. Positive differ-ences signify duties that are favored in theethically preferable action (respect for auton-omy in this example); negative differencessignify duties that are favored in the lesspreferable action (nonmaleficence and benef-icence in this example).

Figure 3 denotes each possible case in thedilemma under consideration. It representseach case as a triple denoting the differencesin corresponding duties, depicted as a pointin a 3-space. The space as a whole representsall 729 discrete, three-duty cases possiblegiven the range of values permitted for eachduty’s level of satisfaction or violation. Thehighest, right-most point (4, 4, 4) representsthe case in which each duty is maximally infavor of the ethically preferable action. Thelowest, left-most point (–4, –4, –4) representsthe case in which each duty is maximally infavor of the less ethically preferable action.Blue points signify positive training cases,red points signify negative training cases, andwhite points signify the remaining 14 possi-ble positive cases.

The learning task is to find a set of clausesthat covers all the positive training exampleswhile not covering any negative trainingexamples. Figure 4 illustrates the set of clausesdefining the supersedes(Action1, Action2)predicate learned by MEDETHEX, whereD<duty> denotes the difference betweenAction1’s <duty> value and Action2’s <duty>value.

Each clause specifies a lower bound foreach of the three duty differentials that musthold for that clause to be true. As each clauseis joined to the others disjunctively, any onetrue clause will cause the supersedes predi-cate to be true. For example, the third clausestates that in order for it to consider Action1ethically preferable to Action2,

• the value for nonmaleficence must be 1 ormore in favor of Action1,

Table 1. The levels of duty satisfaction or violation for the two possible actions in fourMEDETHEX training cases. A check mark indicates the ethically correct action in each case.

Training case no. Nonmaleficence Beneficence Autonomyand action value value value

Case 1Try Again +2 +2 –1

✓ Accept –2 –2 +2

Case 2Try Again 0 +1 –1

✓ Accept 0 –1 +1

Case 3✓ Try Again +1 +1 –1

Accept –1 –1 +1

Case 4✓ Try Again 0 +2 –1

Accept 0 –2 +1

• the value for beneficence can be any value(as –4 is the lowest possible bound), and

• the value for respect for autonomy can bein favor of Action2 by no more than 2.

This set of clauses, in relation to the typeof dilemma under consideration, representsa decision principle that states that a health-care worker should challenge a patient’sdecision if it isn’t fully autonomous andthere’s either any violation of nonmalefi-cence or a severe violation of beneficence.This philosophically interesting result givescredence to Rawls’method of reflective equi-librium. Through abstracting a principle fromintuitions about particular cases and thentesting that principle on further cases, we’veproduced a plausible principle that tells uswhich action is ethically preferable whenspecific duties pull in different directions ina particular ethical dilemma. Furthermore,this abstracted principle supports Ross’sinsight that violations of nonmaleficenceshould carry more weight than violations ofbeneficence.

Visualizing the learned decisionprinciple

Figure 5 graphically represents the seman-tics of each clause that MEDETHEX learned.Each blue point represents the training casefor which the clause was generated, whitepoints represent the remaining positive casesthat the clause covers, red points representthe negative training cases, and the volumerepresents the space assumed to contain onlypositive cases.

Figure 5a represents the first clause, gen-erated to cover training case 1. It covers eightof the remaining positive cases and showshow the learning algorithm greedily occupiesspace surrounding a training example thatdoesn’t contain a negative example. In thiscase, the resulting volume simply representsthe commonsense inference that if any dutydifferential for Action1 in a case is equal toor greater than a positive training case’s dutydifferential, that case will also be positive.

Figure 5b represents the second clause, fortraining case 2. It covers one of the remain-ing positive cases and shows how the spaceoccupied by the learning algorithm can besomewhat speculative. Although this spacecontains the same commonsense inferenceas that for training case 1, it extends itsboundaries to lower values than those of itstraining case. The intuition is that, becausethis space contains no negative cases, we can

safely assume that the cases this space cov-ers are positive until we receive evidence tothe contrary. If and when the training modulefinds a negative case in this space, it willmodify the clause to account for it. This is inthe spirit of reflective equilibrium, whichadvocates abstracting principles from caseswhile permitting modification of these prin-ciples when new cases require it.

Figure 5c represents the third clause, fortraining case 3. It covers five of the remainingpositive cases and, like the space generatedfor training case 2, extends its boundarieslower than those of its training case. Finally,figure 5d represents the fourth clause, for

training case 4. It covers the last remainingpositive cases and extends its boundarieslower than its training case as well.

Clearly, other partitions of this space cancover all positive cases. Figures 5a, 5c, and5d reflect this particular implementation’sbias toward constraining the fewest numberof duty differentials. Figure 5a shows thatonly the duty differential for respect forautonomy has any bearing on the firstclause’s truth value, permitting two degreesof freedom in the graph—the duty differen-tials of beneficence and nonmaleficence.Likewise, figures 5c and 5d show the degreesof freedom permitted in the last two

JULY/AUGUST 2006 www.computer.org/intelligent 7

4

Auto

nom

y

3

2

1

0

–1

–2

–3

–44321

0–1–2

–34/–4

Nonmaleficence

Positivetraining case

Negativetraining case

Remainingpositive case

Beneficence32

10

–1–2

–3–4

–4, –4, 3

0, –4, 2–2, –2, 2

0, –2, 2

0, 4, –20, 2, –2

4, 4, –3

2, 2, –2

Figure 4. The set of clauses defining the supersedes(Action1, Action2) predicatelearned by MEDETHEX.

∆nonmaleficence –4 ∧ ∆beneficence –4 ∧ ∆autonomy 3∨∆nonmaleficence –1 ∧ ∆beneficence –3 ∧ ∆autonomy –1∨∆nonmaleficence 1 ∧ ∆beneficence –4 ∧ ∆autonomy –2∨∆nonmaleficence –4 ∧ ∆beneficence 3 ∧ ∆autonomy –2

Figure 3. All 18 possible cases of the biomedical ethical dilemma. We represent eachcase as a triple denoting the differences in corresponding duties, depicted as a point ina 3-space.

clauses—the duty differentials of benefi-cence and nonmaleficence, respectively.

Figure 6 represents the union of figures 5athrough 5d. As such, it represents the totalspace of duty differentials in the dilemmaunder consideration where MEDETHEX

learned supersedes(Action1, Action2) as true.We can further verify (or refine) this space bydeveloping other positive and negative cases

and determining where they fall within it. Ifa positive case falls outside the space or a neg-ative case falls within it, we must refine thespace to account for it. If we can find no suchexamples, the space has been further verified.

Future researchWe plan to develop MEDETHEX further to

see whether the learned decision principle

works in other dilemmas involving the samethree duties. It will be interesting to add thefourth duty, justice, to see to what extentthere’s a consensus among bioethicists incases where this duty is involved from whichwe can abstract a decision principle. There’sdisagreement about what is just among thoseworking in ethics in other domains, but theremight not be disagreement among bioethicists.

M a c h i n e E t h i c s

8 www.computer.org/intelligent IEEE INTELLIGENT SYSTEMS

4

Auto

nom

y

3

2

1

0

–1

–2

–3

–443210

–1–2–34/–// 4

Nonmaleficence3

21

0–1

–2–3

–4––

–4,– ––

Positive training case Negative training case Remaining positive case

4

Auto

nom

y

3

2

1

0

–1

–2

–3

–443210

–1–2–34/–// 4

Nonmaleficence3

21

0–1

–2–3

–4

2

BeneficenceBeneficence

4

Auto

nom

y

3

2

1

0

–1

–2

–3

–443210

–1–2–34/–// 4

Nonmaleficence3

21

0–1

–2–3

–4––

–

Beneficence

4

Auto

nom

y

3

2

1

0

–1

–2

–3

–443210

–1–2–34/–// 4

Nonmaleficence3

21

0–1

–2–3

–4 4,

Beneficence

(b)(a)

(d)(c)

Figure 5. The duty differential space covered by (a) the first clause of the decision principle that MEDETHEX learned, (b) the secondclause, (c) the third clause, and (d) the fourth clause.

Furthermore, we would like to see if ourapproach to learning decision principles willprove viable for other sets of duties, includ-ing sets of higher cardinality, and in otherdomains. It’s reasonable to believe that eachspecific applied ethics domain (legal ethics,business ethics, journalistic ethics, and so on)involves juggling a set of prima facie dutiesthat’s specific to that domain. In each case,there will be the problem of abstracting a deci-sion principle to determine the correct actionwhen the duties conflict. We plan, therefore, tolook at other domains to see whether ourapproach to creating an ethical-advisor sys-tem might be helpful in solving ethical dilem-mas for those who work in those domains.

Our long-term goal, of course, is to have amachine follow ethical principles itself, ratherthan simply advising human beings as to howto behave ethically. We believe, though, thatthe first step in the development of machineethics must be to work on making ethics com-putable. If that task can’t be accomplished, atleast to the extent to which ethics experts arein agreement as to what’s ethically right, thencreating a machine that behaves ethically willbe impossible. Creating ethical-advisor sys-tems lets us explore the extent to which ethicscan be computed in specific domains. Onceethics experts are comfortable with theresults, then an ethical dimension can, at leastin principle, be incorporated into machinesthat function in those domains. This shouldnot only avert unethical behavior on the partof machines, but also allow them to do tasksthat we would have previously thought onlyhuman beings should do.

The process of making an ethical the-ory precise enough to be computed

will likely sharpen and revise the theoryitself. This research provides an opportunityfor applying AI techniques in a new domainand developing new areas of applied ethics,as well as making a contribution to ethicaltheory itself.

Our results demonstrate that a problem inethical theory—devising a decision proce-dure for an ethical theory involving multipleprima facie duties—can be solved at least ina constrained domain and that AI techniquescan help solve it. So, we believe that not onlycan you train a machine to make ethical deci-sions but also that machines can help humanbeings codify the principles that should guidethem in ethical decision making.

References1. S. Anderson, “We Are Our Values,” Ques-

tioning Matters, D. Kolak, ed., Mayfield,1999, p. 599.

2. M. Anderson, S. Anderson, and C. Armen,“Toward Machine Ethics: Implementing TwoAction-Based Ethical Theories,” Proc. AAAI2005 Fall Symp. Machine Ethics, AAAIPress, 2005, pp. 1–16.

3. J. Bentham, An Introduction to the Principlesand Morals of Legislation, Oxford Univ.Press, 1799.

4. M. Anderson, S. Anderson, and C. Armen,“MedEthEx: Toward a Medical Ethics Advi-sor,” Proc. AAAI 2005 Fall Symp. CaringMachines: AI in Eldercare, AAAI Press,2005, pp. 9–16.

5. J.S. Mill, “Utilitarianism,” Utilitarianism andOther Writings, M. Warnock, ed., New Amer-ican Library, 1974, chapter 5.

6. W.D. Ross, The Right and the Good, Claren-don Press, 1930.

7. J. Rawls, “Outline for a Decision Procedurefor Ethics,” Philosophical Rev., vol. 60, 1951.

8. N. Lavrac and S. Dzeroski, Inductive LogicProgramming: Techniques and Applications,Ellis Harwood, 1997.

9. T.L. Beauchamp and J.F. Childress, Princi-ples of Biomedical Ethics, Oxford Univ.Press, 1979.

10. T.A. Mappes and D. DeGrazia, BiomedicalEthics, 5th ed., McGraw-Hill, 2001, pp. 39–42.

11. A.E. Buchanan and D.W. Brock, Deciding forOthers: The Ethics of Surrogate DecisionMaking, Cambridge Univ. Press, 1989, pp.48–57.

JULY/AUGUST 2006 www.computer.org/intelligent 9

4

Auto

nom

y

3

2

1

0

–1

–2

–3

–44321

0–1–2

–34/–// 4

Nonmaleficence

32

10

–1–2

–3–4

Positivetraining case

Negativetraining case

Remainingpositive case

Beneficence

–4, –– –––

Figure 6. The duty differential space covered by the combined clauses.

T h e A u t h o r sMichael Andersonis an associate profes-sor of computer sci-ence at the Universityof Hartford. Contacthim at anderson@hartford.edu.

Susan Leigh Ander-son is a professor ofphilosophy at the Uni-versity of Connecticut,Stamford. Contact herat susan.anderson@uconn.edu.

Chris Armen is a vis-iting assistant profes-sor at Amherst Col-lege’s Department ofMathematics and Com-puter Science. Contacthim at carmen@amherst.edu.