Terrier Core

Most time was spent summing up the memory consumption of the MemoryPostings object:

rank self accum count trace method
1 41.13% 41.13% 32556 300488 uk.ac.gla.terrier.structures.indexing.singlepass.MemoryPostings.getMemoryConsumption

TRACE 300488:
uk.ac.gla.terrier.structures.indexing.singlepass.MemoryPostings.getMemoryConsumption(MemoryPostings.java:128)
uk.ac.gla.terrier.indexing.BasicSinglePassIndexer.checkFlush(BasicSinglePassIndexer.java:287)
uk.ac.gla.terrier.indexing.BasicSinglePassIndexer.indexDocument(BasicSinglePassIndexer.java:320)
uk.ac.gla.terrier.indexing.BasicSinglePassIndexer.createInvertedIndex(BasicSinglePassIndexer.java:224)

This was caused by TREC-43.

Instead of calculating memory usage for every call of getMemoryConsumption(), I instead keep track of consumption as postings objects are added or updated. For WT2G indexing, this takes indexing time (with profiling enabled) down from 989.647 seconds to 388.993 seconds (with profiling enabled).

Changed committed to SVN.

Nicola also reported slow indexing with WT2G. He said he could not achieve the timings mentioned on the Terrier web page on his laptop with Terrier3. Is this due to the same issue?

Indeed, it was he who alerted me that there might be a problem here. I wont tell him how to fix it, as he doesn't have to index too regularly anyway.

I told him to speak to you about the issue. Good (that the bug was fixed). You might be right: we do need some sort of unit testing after all.