Terrier Core

Reset problem in Terrier evaluation tool

Details

  • Type: Bug Bug
  • Status: Resolved Resolved
  • Priority: Major Major
  • Resolution: Fixed
  • Affects Version/s: 3.0
  • Fix Version/s: 3.0
  • Component/s: None
  • Description:
    Nicola reported today a bug with the Terrier evaluation tool. For some queries, the system reports an MAP of 2 (yes 2) at precision @1. It appears that it is a resetting problem when multiple evaluation files are used.

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Craig Macdonald added a comment - 16/Sep/09 12:38 PM

Not a reset problem actually. A problem calculating precision at rank. 

if (relevantRetrieved[j].rank <= precisionRank)

should read

if (relevantRetrieved[j].rank < precisionRank)

because ranks are 0 based.

I have also added test cases for the AdhocEvaluation class. The test cases check various measures (hand calculated) for rankings upto size 4. Without the patch, the test fails. With it, the test passes.

Show
Craig Macdonald added a comment - 16/Sep/09 12:38 PM Not a reset problem actually. A problem calculating precision at rank. 
if (relevantRetrieved[j].rank <= precisionRank)
should read 
if (relevantRetrieved[j].rank < precisionRank)
because ranks are 0 based. I have also added test cases for the AdhocEvaluation class. The test cases check various measures (hand calculated) for rankings upto size 4. Without the patch, the test fails. With it, the test passes.
Hide
Permalink
Iadh Ounis added a comment - 16/Sep/09 1:46 PM

Well spotted Craig!

Show
Iadh Ounis added a comment - 16/Sep/09 1:46 PM Well spotted Craig!

People

Dates

  • Created:
    10/Sep/09 3:57 PM
    Updated:
    05/Mar/10 5:04 PM
    Resolved:
    16/Sep/09 12:38 PM
Atlassian JIRA (v4.1#519) | Report a problem | Request a feature | Contact administrators
Powered by a free Atlassian JIRA open source license for Univ of Glasgow. Try JIRA - bug tracking software for your team.