[#TR-38] MapReduce InputFormat for BitPostingIndexInputStream - Terrier JIRA

Description:

Recent core changes have made generic PostingIndex and PostingIndexInputStream objects which give access to IterablePostings. It would be good to have an InputFormat for splitting the reading of a PostingIndex across various map tasks.

Ascending order - Click to sort in descending order

Hide

Permalink

Craig Macdonald added a comment - 07/May/09 8:15 PM

This would have benefits in the following scenarios:

If we use PostingIndex as a LinkServer, then this would allow link analysis index processing to be easily split
Inversion of indices (i.e. Inverted-> Direct; Direct->Inverted) could be split to run in parallel
DirectIndex analysis.

As a PostingIndex can be large, locality should be supported.

Show

Craig Macdonald added a comment - 07/May/09 8:15 PM This would have benefits in the following scenarios:

If we use PostingIndex as a LinkServer, then this would allow link analysis index processing to be easily split
Inversion of indices (i.e. Inverted-> Direct; Direct->Inverted) could be split to run in parallel
DirectIndex analysis.

As a PostingIndex can be large, locality should be supported.

Hide

Permalink

Craig Macdonald added a comment - 07/May/09 8:16 PM

Initial version, untested.

Show

Craig Macdonald added a comment - 07/May/09 8:16 PM Initial version, untested.

Hide

Permalink

Craig Macdonald added a comment - 08/May/09 10:00 AM

Updated version - some files were missing from the patch.

Show

Craig Macdonald added a comment - 08/May/09 10:00 AM Updated version - some files were missing from the patch.

Hide

Permalink

Craig Macdonald added a comment - 16/Jul/09 7:13 PM

Resolved a final version to trunk.

Show

Craig Macdonald added a comment - 16/Jul/09 7:13 PM Resolved a final version to trunk.

Terrier Core