RetrievalEvaluator (OpenIMAJ master project 1.3.10 API)

java.lang.Object
- org.lemurproject.ireval.RetrievalEvaluator

```
public class RetrievalEvaluator
extends Object
```
A retrieval evaluator object computes a variety of standard information retrieval metrics commonly used in TREC, including binary preference (BPREF), geometric mean average precision (GMAP), mean average precision (MAP), and standard precision and recall. In addition, the object gives access to the relevant documents that were found, and the relevant documents that were missed.

BPREF is defined in Buckley and Voorhees, "Retrieval Evaluation with Incomplete Information", SIGIR 2004.

Author:

Trevor Strohman, Jonathon Hare (jsh2@ecs.soton.ac.uk)

Nested Class Summary

Nested Classes
Modifier and Type	Class and Description
`static class`	`RetrievalEvaluator.Document` This class represents a document returned by a retrieval system.
`static class`	`RetrievalEvaluator.Judgment` This class represents a relevance judgment of a particular document for a specific query.

Constructor Summary

Constructors
Constructor and Description
`RetrievalEvaluator(String queryName, List<RetrievalEvaluator.Document> retrieved, Collection<RetrievalEvaluator.Judgment> judgments)` Creates a new instance of RetrievalEvaluator

Method Summary

All Methods Static Methods Instance Methods Concrete Methods
Modifier and Type	Method and Description
`double`	`averagePrecision()` Returns the average precision of the query.
`double`	`binaryPreference()` The binary preference measure, as presented in Buckley, Voorhees "Retrieval Evaluation with Incomplete Information", SIGIR 2004.
`static int[]`	`getFixedPoints()`
`double[]`	`interpolatedPrecision()`
`ArrayList<RetrievalEvaluator.Document>`	`irrelevantRetrievedDocuments()` This method returns a list of all documents that were retrieved but assumed to be irrelevant.
`ArrayList<RetrievalEvaluator.Document>`	`judgedIrrelevantRetrievedDocuments()`
`protected double`	`normalizationTermNDCG(int documentsRetrieved)`
`double`	`normalizedDiscountedCumulativeGain()` Normalized Discounted Cumulative Gain
`double`	`normalizedDiscountedCumulativeGain(int documentsRetrieved)` Normalized Discounted Cumulative Gain
`double`	`precision(int documentsRetrieved)` Returns the precision of the retrieval at a given number of documents retrieved.
`double[]`	`precisionAtFixedPoints()`
`String`	`queryName()`
`double`	`recall(int documentsRetrieved)` Returns the recall of the retrieval at a given number of documents retrieved.
`double`	`reciprocalRank()` Returns the reciprocal of the rank of the first relevant document retrieved, or zero if no relevant documents were retrieved.
`ArrayList<RetrievalEvaluator.Document>`	`relevantDocuments()` Returns a list of all documents judged relevant, whether they were retrieved or not.
`ArrayList<RetrievalEvaluator.Document>`	`relevantMissedDocuments()` Returns a list of documents that were judged relevant that were not retrieved.
`int`	`relevantRetrieved(int documentsRetrieved)` The number of relevant documents retrieved at a particular rank.
`ArrayList<RetrievalEvaluator.Document>`	`relevantRetrievedDocuments()` Returns a list of retrieved documents that were judged relevant, in the order that they were retrieved.
`ArrayList<RetrievalEvaluator.Document>`	`retrievedDocuments()`
`double`	`rPrecision()` Returns the precision at the rank equal to the total number of relevant documents retrieved.

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Constructor Detail
  - RetrievalEvaluator
```
public RetrievalEvaluator(String queryName,
                          List<RetrievalEvaluator.Document> retrieved,
                          Collection<RetrievalEvaluator.Judgment> judgments)
```
    Creates a new instance of RetrievalEvaluator
    
    Parameters:
    
    queryName -
    
    retrieved - A ranked list of retrieved documents.
    
    judgments - A collection of relevance judgments.
- Method Detail
  - queryName
```
public String queryName()
```
    Returns:
    
    the name of the query represented by this evaluator.
  - getFixedPoints
```
public static int[] getFixedPoints()
```
    Returns:
    
    the fixed points (number of retrieved docs) at which precision is evaluated
  - precisionAtFixedPoints
```
public double[] precisionAtFixedPoints()
```
    Returns:
    
    the precision at the fixed points specified by getFixedPoints().
  - interpolatedPrecision
```
public double[] interpolatedPrecision()
```
    Returns:
    
    the interpolated precision at 10% recall intervals
  - precision
```
public double precision(int documentsRetrieved)
```
    Returns the precision of the retrieval at a given number of documents retrieved. The precision is the number of relevant documents retrieved divided by the total number of documents retrieved.
    
    Parameters:
    
    documentsRetrieved - The evaluation rank.
    
    Returns:
    
    the precision at the given number of retrieved documents.
  - recall
```
public double recall(int documentsRetrieved)
```
    Returns the recall of the retrieval at a given number of documents retrieved. The recall is the number of relevant documents retrieved divided by the total number of relevant documents for the query.
    
    Parameters:
    
    documentsRetrieved - The evaluation rank.
    
    Returns:
    
    the recall at the given number of retrieved documents.
  - rPrecision
```
public double rPrecision()
```
    Returns the precision at the rank equal to the total number of relevant documents retrieved. This method is equivalent to precision( relevantDocuments().size() ). If R is greater than the number of documents retrieved, the non-retrieved documents are assumed to be non-relevant (cf trec_eval 8).
    
    Returns:
    
    the r-precision
  - reciprocalRank
```
public double reciprocalRank()
```
    Returns the reciprocal of the rank of the first relevant document retrieved, or zero if no relevant documents were retrieved.
    
    Returns:
    
    the reciprocal rank
  - averagePrecision
```
public double averagePrecision()
```
    Returns the average precision of the query.
    Suppose the precision is evaluated once at the rank of each relevant document in the retrieval. If a document is not retrieved, we assume that it was retrieved at rank infinity. The mean of all these precision values is the average precision.
    
    Returns:
    
    the average precision
  - binaryPreference
```
public double binaryPreference()
```
    The binary preference measure, as presented in Buckley, Voorhees "Retrieval Evaluation with Incomplete Information", SIGIR 2004. This implemenation is the 'pure' version, which is the one used in Buckley's trec_eval (v 8 with bpref bugfix).
    
    The formula is: 1/R \sum_{r} 1 - |n ranked greater than r| / min(R, N) where R is the number of relevant documents for this topic, N is the number of irrelevant documents judged for this topic, and n is a member of the set of first R judged irrelevant documents retrieved.
    
    Returns:
    
    the binary preference.
  - normalizedDiscountedCumulativeGain
```
public double normalizedDiscountedCumulativeGain()
```
    Normalized Discounted Cumulative Gain
    This measure was introduced in Jarvelin, Kekalainen, "IR Evaluation Methods for Retrieving Highly Relevant Documents" SIGIR 2001. I copied the formula from Vassilvitskii, "Using Web-Graph Distance for Relevance Feedback in Web Search", SIGIR 2006. Score = N \sum_i (2^{r(i)} - 1) / \log(1 + i) Where N is such that the score cannot be greater than 1. We compute this by computing the DCG (unnormalized) of a perfect ranking.
    
    Returns:
    
    the normalized discounted cumulative gain (ndcg).
  - normalizedDiscountedCumulativeGain
```
public double normalizedDiscountedCumulativeGain(int documentsRetrieved)
```
    Normalized Discounted Cumulative Gain
    This measure was introduced in Jarvelin, Kekalainen, "IR Evaluation Methods for Retrieving Highly Relevant Documents" SIGIR 2001. I copied the formula from Vassilvitskii, "Using Web-Graph Distance for Relevance Feedback in Web Search", SIGIR 2006. Score = N \sum_i (2^{r(i)} - 1) / \log(1 + i) Where N is such that the score cannot be greater than 1. We compute this by computing the DCG (unnormalized) of a perfect ranking.
    
    Parameters:
    
    documentsRetrieved -
    
    Returns:
    
    the normalized discounted cumulative gain (ndcg).
  - normalizationTermNDCG
```
protected double normalizationTermNDCG(int documentsRetrieved)
```
  - relevantRetrieved
```
public int relevantRetrieved(int documentsRetrieved)
```
    The number of relevant documents retrieved at a particular rank. This is equivalent to n * precision(n).
    
    Parameters:
    
    documentsRetrieved - the rank
    
    Returns:
    
    the number of relevant docs at the rank.
  - retrievedDocuments
```
public ArrayList<RetrievalEvaluator.Document> retrievedDocuments()
```
    Returns:
    
    The list of retrieved documents.
  - judgedIrrelevantRetrievedDocuments
```
public ArrayList<RetrievalEvaluator.Document> judgedIrrelevantRetrievedDocuments()
```
    Returns:
    
    The list of all documents retrieved that were explicitly judged irrelevant.
  - irrelevantRetrievedDocuments
```
public ArrayList<RetrievalEvaluator.Document> irrelevantRetrievedDocuments()
```
    This method returns a list of all documents that were retrieved but assumed to be irrelevant. This includes both documents that were judged to be irrelevant and those that were not judged at all. The list is returned in retrieval order.
    
    Returns:
    
    the list of all retrieved irrelevant documents.
  - relevantRetrievedDocuments
```
public ArrayList<RetrievalEvaluator.Document> relevantRetrievedDocuments()
```
    Returns a list of retrieved documents that were judged relevant, in the order that they were retrieved.
    
    Returns:
    
    the list of all retrieved relevant documents
  - relevantDocuments
```
public ArrayList<RetrievalEvaluator.Document> relevantDocuments()
```
    Returns a list of all documents judged relevant, whether they were retrieved or not. Documents are listed in the order they were retrieved, with those not retrieved coming last.
    
    Returns:
    
    the list of all relevant documents
  - relevantMissedDocuments
```
public ArrayList<RetrievalEvaluator.Document> relevantMissedDocuments()
```
    Returns a list of documents that were judged relevant that were not retrieved.
    
    Returns:
    
    the relevant documents that were missed by the search engine

Class RetrievalEvaluator

Nested Class Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Constructor Detail

RetrievalEvaluator

Method Detail

queryName

getFixedPoints

precisionAtFixedPoints

interpolatedPrecision

precision

recall

rPrecision

reciprocalRank

averagePrecision

binaryPreference

normalizedDiscountedCumulativeGain

normalizedDiscountedCumulativeGain

normalizationTermNDCG

relevantRetrieved

retrievedDocuments

judgedIrrelevantRetrievedDocuments

irrelevantRetrievedDocuments

relevantRetrievedDocuments

relevantDocuments

relevantMissedDocuments