Filtering Out Bad Answers with Semantic Relations in a Web-Based Question-Answering System

22 avril 2004 TALN-2004, Atelier QR 1

Leila Kosseim and Jamileh YousefiCLaC Laboratory

Concondia UniversityMontréal

Filtering Out Bad Answers with Semantic Relations in a Web-Based Question-Answering System


Outline1.Introduction2.The original Web-QA component

3.Filtering out bad answers4.Evaluation5.Discussion and Future work


1. Introduction

Quantum-QA: The Web-QA Component provides Quantum with candidate answers extracted from documents found on the web using a search engine (Yahoo).

QuantumWeb-QAComponent

The Web Trec Collection

data

1 3

2

Candidate answers


1. Introduction(cont.)

Conjunction of Web-QA Component with Quantum:

Yields an interesting improvement, but Its results are very noisy.The right answer is there, but hidden alongside many wrong answers. Without the core Quantum system, no

distinction between right and wrong answers is possible.


2. The original Web-QA component

Example: Who is the fastest swimmer in the world?

1. Produce a formulation such as: The fastest swimmer in the world is <PERSON-NAME>.

2. Search the Web for the exact formulation 3. Extract the noun phrase following the

formulation 4. Apply simple syntactic and semantic

checks to ensure that the noun phrase is a PERSON-NAME.


Question reformulation

To formulate an answer formulation from a question: use a set of hand-made templates templates are composed of 2 sets of patterns:

Question pattern: Defines what the question must look like: When did < ANY-SEQUENCE-WORDS> <VERB-simple> ?

Set of answer patterns: Defines a set of possible answer formulations<DATE> < ANY-SEQUENCE-WORDS> <VERB-past>

the patterns take into account : Specific keywords (e.g. When) Strings of characters (e.g. ANY-SEQUENCE-WORDS) Part-of-speech tags (e.g. VERB-simple).


Evaluation of Web-QA component alone

Corpus No of questions

No of questions with a reformulation

No of questionswith at least one candidate answer

No of questions with a correct candidate answer in the top 5 candidates

Precision

TREC-9 694 624 (89.9%)

63 (9.1%) 17 (2.4%) 0.270

TREC-10

499 450 (90.2%)

256 (51.3%)

153 (30.7%) 0.598


In this paper

We present a method to: Filter out noisy candidate answers

Re-rank candidate answers by using semantic relations


3. Filtering and re-ranking

Our methodology is composed of five steps:

1. Run the Web-QA component and retrieve its top 200 candidates,

2. Run a named-entity tagger,3. Extract the semantic relation and

arguments for each candidate,4. Validate the semantic relation,5. Re-rank the candidate answers by

frequency.


Step 1: Initial Run

Run the Web-QA component and retrieve its top 200 candidates

Example: Who founded American Red Cross?

Jane Delano Who former schoolteacher Activism Exit RW ONLINE WhoWho ReallyClara Barton Christopher Blake…


Step 2: Named-entity tagger

tag candidates with the GATE-NE named-entity tagger

keep only candidates that satisfy the predicted constraints of the answer

Example: Who founded American Red Cross? Jane Delano

Clara Barton Christopher Blake…


Step 3: Find the semantic relation

Decompose the original question into two parts:

1. the main semantic relation relation = the main verb of the question

2. the argument expressed in the question argument1 = first noun phrase of the question

Use the candidate answer as argument2 Example: Who founded American Red Cross? relation: found

argument1: American Red Cross argument2: ANSWER

Found (American Red Cross, ANSWER)


Simplification Assumptions

Relations holding between more than two arguments, are considered as a binary relation:example: When did ARG1 sold ARG2?

relation : giveargument1: ARG1argument2: Answer

If the main verb is TO-BE, then we don’t take the semantic relation into account: example: Who is the author of John Christoph?

relation : noneargument1: John Christophargument2: Answer


Step 3 (cont.): Argument tuples

create a set of argument tuples from: the argument of the question and the candidates answers:

(American Red Cross, Jane Delano )(American Red Cross, Clara Barton )(American Red Cross, Christopher Blake)

submit all tuples to the document collection to find paragraphs that contain both elements

extract the paragraphs where both elements are within a distance of N words of each other

pre-window American Red Cross in-window Jane Delano post-window

≤ N words argument 1 N words ≤ argument 2 N ≤words


Step 4: Validate the semantic relation

The semantic relation may occur as a verb or noun or adjective in the context window.

Example: X was founded by Y Y is the founder of X


Verbs with equivalent semantic relation

check if a verb in any context window is a synonym, a hypernym or a hyponym of the relation in the question.

here, we use a part-of-speech tagger and WordNet.. . . 1881 In History, Event: American Red Cross founded by Clara

Barton. Related Year: 1881 Related Events: . . .

In addition, the American Red Cross organization was formed in 1881, Clara Barton served as its first president . . .

. . . Maryland, house where Clara Barton organized and directed the American Red Cross efforts. Hours, history of the . . .


Nouns and adjectives with equivalent semantic relation

if no verb has an equivalent semantic relation, try to validate nouns and adjectives

check if it has the same stem as the original relation or one of its synonyms.

here, we use the Porter stemmer. . . and of active volunteerism than Clara Barton, founder of the

American Red Cross.

. . . Clara Barton, American humanitarian, organizer of the American Red Cross, b. North Oxford . . .


Step 5: Re-rank candidate answers

throw away tuples that have no similar semantic relation in the question and in the documents

re-rank remaining candidates according to the number of passages in the collection containing the same relation.

Example: Argument tuple: (American Red Cross, Clara Barton). Nb of passages found: 110 passages Nb of passages with the relation kill within 5 words: 24 passages

Rank of argument tuple: 24/110 apply this to all the argument tuples, then select the five best ranked tuples


4. Evaluation

Comparison of the original and the new WebQA components

Corpus System No of questions

No of questionswith at least one candidate answer

No of questions with a correct candidate answer in the top 5 candidates

Precision

TREC-9 OriginalNew

694694

63 (9.1%)28 (4.0%)

17 (2.4%)20 (2.9%)

0.2700.714

TREC-10

OriginalNew

499499

255 (51.1%)189 (37.8%)

153 (30.7%)152 (30.5%)

0.5980.804

On the questions in the TREC-9 and TREC-10 collection Comparison using the pp-eval program provided by NIST


4. Evaluation (cont.)

Nb of questions with candidate answers is inferior in the new system but, Nb of correct candidates is greater or similar, the precision

is higher. although we provide less candidates, they are more likely to constitute correct answers than before.

In TREC-10: MRR is increased by 12% 31% of correct answers were ranked better by 4.5 on average. 9% ranked worse by 3.5 on average. Only one good answer was lost during the process.

In TREC-9: MRR is increased by 40% 25% of correct answers were ranked better; moved up the list by

2.2 on average. 5% of correct answers got worse ranks; moved down by 6 on

average. 5% of good answers were removed.


5. Discussion

Our approach: improves the precision of our Web-QA module less strict than looking for exact phrases as it looks for reinforcement of the semantic relation between the arguments

but, more sensitive to mistakes and wrong interpretations e.g. negations and other modal words

if we look in a much smaller corpus, such as in closed-domain QA, looking for semantic equivalences may be more fruitful.


6. Future work currently, a large number of questions that are reformulated retrieve no candidate answer

so our next goal is to look at generating better reformulations automatically

Filtering Out Bad Answers with Semantic Relations in a Web-Based Question-Answering System

Documents

Transcript of Filtering Out Bad Answers with Semantic Relations in a Web-Based Question-Answering System