Terrier Core

This issue is even more complicated. The reducer uses the side-effect files for two purposes:

• To determine what document index and metaindex structures need to be merged for its final index
• To determine what the docid offsets should be in inverted index.

This means that all the docids in the shards are global, not local to the inverted index being created by that shard.

For instance, no docid in the second shard index will be less than the number of documents in the first shard index.

The NWayMergers need to account for the inverted index docid problem.

I have two classes in SVN that try to fix this problem for existing indices:

• FixBadReducerIndex copies the index into a new index, fixing the docids in the inverted file, the collection statistics, and selecting only the appropriate parts of the document index and metaindex along the way.
• FixDocumentIndexBadReducer just calculates the correct collection statistics.

Initial version of a patch for the multi reducer problem.

Richard and I checked this, and it does make sense. We're going to try this with for Blogs08 with blocks, as a single reducer doesnt have enough disk space to do this corpus.

Fixed version committed to SVN trunk.