Evaluate Q&A¶

Author(s)	Renato Leite (renatoleite@), Egon Soares (egon@)
Last updated	09/05/2023

Exact Match¶

$ EM(Truth, Prediction) = \begin{cases} 0 & \quad \text{if } Truth \neq Prediction\\ 1 & \quad \text{if } Truth = Prediction \end{cases} $

In [1]:

def exact_match(ground_truth:str , answer:str) -> int:
    return 1 if ground_truth == answer else 0

def exact_match(ground_truth:str , answer:str) -> int: return 1 if ground_truth == answer else 0

Case 1 - There is an answer¶

In [2]:

ground_truth_1 = "Google was founded on September 4, 1998, by American computer scientists Larry Page and Sergey Brin while they were PhD students at Stanford University in California."
answer_1_a = ""
answer_1_b = "Google was founded on September 4, 1998, by American computer scientists Larry Page and Sergey Brin while they were PhD students at Stanford University in California."
answer_1_c = "Google was founded on September 4, 1998, by American computer scientists Larry Page and Sergey Brin."

ground_truth_1 = "Google was founded on September 4, 1998, by American computer scientists Larry Page and Sergey Brin while they were PhD students at Stanford University in California." answer_1_a = "" answer_1_b = "Google was founded on September 4, 1998, by American computer scientists Larry Page and Sergey Brin while they were PhD students at Stanford University in California." answer_1_c = "Google was founded on September 4, 1998, by American computer scientists Larry Page and Sergey Brin."

In [3]:

print(f"Ground Truth: {ground_truth_1}")
print(f"\nAnswer a: {answer_1_a}\nEM: {exact_match(ground_truth_1, answer_1_a)}")
print(f"\nAnswer b: {answer_1_b}\nEM: {exact_match(ground_truth_1, answer_1_b)}")
print(f"\nAnswer c: {answer_1_c}\nEM: {exact_match(ground_truth_1, answer_1_c)}")

print(f"Ground Truth: {ground_truth_1}") print(f"\nAnswer a: {answer_1_a}\nEM: {exact_match(ground_truth_1, answer_1_a)}") print(f"\nAnswer b: {answer_1_b}\nEM: {exact_match(ground_truth_1, answer_1_b)}") print(f"\nAnswer c: {answer_1_c}\nEM: {exact_match(ground_truth_1, answer_1_c)}")

Ground Truth: Google was founded on September 4, 1998, by American computer scientists Larry Page and Sergey Brin while they were PhD students at Stanford University in California.

Answer a: 
EM: 0

Answer b: Google was founded on September 4, 1998, by American computer scientists Larry Page and Sergey Brin while they were PhD students at Stanford University in California.
EM: 1

Answer c: Google was founded on September 4, 1998, by American computer scientists Larry Page and Sergey Brin.
EM: 0

Case 2 - There is no answer¶

In [4]:

ground_truth_2 = ""
answer_2_a = ""
answer_2_b = "Google and YouTube are the two most visited websites worldwide followed by Facebook and Twitter."
answer_2_c = "No answer found."

ground_truth_2 = "" answer_2_a = "" answer_2_b = "Google and YouTube are the two most visited websites worldwide followed by Facebook and Twitter." answer_2_c = "No answer found."

In [5]:

print(f"Ground Truth: {ground_truth_2}")
print(f"\nAnswer a: {answer_2_a}\nEM: {exact_match(ground_truth_2, answer_2_a)}")
print(f"\nAnswer b: {answer_2_b}\nEM: {exact_match(ground_truth_2, answer_2_b)}")
print(f"\nAnswer c: {answer_2_c}\nEM: {exact_match(ground_truth_2, answer_2_c)}")

print(f"Ground Truth: {ground_truth_2}") print(f"\nAnswer a: {answer_2_a}\nEM: {exact_match(ground_truth_2, answer_2_a)}") print(f"\nAnswer b: {answer_2_b}\nEM: {exact_match(ground_truth_2, answer_2_b)}") print(f"\nAnswer c: {answer_2_c}\nEM: {exact_match(ground_truth_2, answer_2_c)}")

Ground Truth: 

Answer a: 
EM: 1

Answer b: Google and YouTube are the two most visited websites worldwide followed by Facebook and Twitter.
EM: 0

Answer c: No answer found.
EM: 0

In [ ]: