org.lemurproject.ireval

Class IREval

java.lang.Object

org.lemurproject.ireval.IREval

public class IREval
extends Object

A trec_eval style tool in pure java.

Author:

Trevor Strohman, Jonathon Hare (jsh2@ecs.soton.ac.uk)

Constructor Summary

Constructors

Constructor and Description
IREval()

Method Summary

Modifier and Type	Method and Description
static String	comparisonEvaluation(SetRetrievalEvaluator baseline, SetRetrievalEvaluator treatment, String baselineName, String treatmentName) Compare two sets of retrieval results.
static SetRetrievalEvaluator	create(TreeMap<String,ArrayList<RetrievalEvaluator.Document>> allRankings, TreeMap<String,ArrayList<RetrievalEvaluator.Judgment>> allJudgments) Creates a SetRetrievalEvaluator from data from loadRanking and loadJudgments.
static TreeMap<String,ArrayList<RetrievalEvaluator.Judgment>>	loadJudgments(String filename) Loads a TREC judgments file.
static TreeMap<String,ArrayList<RetrievalEvaluator.Document>>	loadRanking(String filename) Reads in a TREC ranking file.
static void	main(String[] args) Tool main method.
static String	singleEvaluation(SetRetrievalEvaluator setEvaluator, boolean showIndividual) Returns an output string very similar to that of trec_eval.
static String	singleQuery(String query, RetrievalEvaluator evaluator) Returns an output string very similar to that of trec_eval.
static void	usage() Print tool usage

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

IREval

public IREval()

Method Detail

loadJudgments

public static TreeMap<String,ArrayList<RetrievalEvaluator.Judgment>> loadJudgments(String filename)
                                                                            throws IOException,
                                                                                   FileNotFoundException

Loads a TREC judgments file.

Parameters:

filename - The filename of the judgments file to load.

Returns:

Maps from query numbers to lists of judgments for each query.

Throws:

IOException

FileNotFoundException

loadRanking

public static TreeMap<String,ArrayList<RetrievalEvaluator.Document>> loadRanking(String filename)
                                                                          throws IOException,
                                                                                 FileNotFoundException

Reads in a TREC ranking file.

Parameters:

filename - The filename of the ranking file.

Returns:

A map from query numbers to document ranking lists.

Throws:

IOException

FileNotFoundException

create

public static SetRetrievalEvaluator create(TreeMap<String,ArrayList<RetrievalEvaluator.Document>> allRankings,
                                           TreeMap<String,ArrayList<RetrievalEvaluator.Judgment>> allJudgments)

Creates a SetRetrievalEvaluator from data from loadRanking and loadJudgments.

Parameters:

allRankings -

allJudgments -

Returns:

the evaluation result

singleQuery

public static String singleQuery(String query,
                                 RetrievalEvaluator evaluator)

Returns an output string very similar to that of trec_eval.

Parameters:

query -

evaluator -

Returns:

the result as a String

singleEvaluation

public static String singleEvaluation(SetRetrievalEvaluator setEvaluator,
                                      boolean showIndividual)

Returns an output string very similar to that of trec_eval.

Parameters:

setEvaluator -

showIndividual -

Returns:

the result as a String

comparisonEvaluation

public static String comparisonEvaluation(SetRetrievalEvaluator baseline,
                                          SetRetrievalEvaluator treatment,
                                          String baselineName,
                                          String treatmentName)

Compare two sets of retrieval results.

Parameters:

baseline -

treatment -

baselineName -

treatmentName -

Returns:

the result as a String.

usage

public static void usage()

Print tool usage

main

public static void main(String[] args)
                 throws IOException

Tool main method.

Parameters:

args -

Throws:

IOException