Terrier Core

In singlepass indexing, checking for enough free memory is insufficient

Details

  • Type: Improvement Improvement
  • Status: Resolved Resolved
  • Priority: Major Major
  • Resolution: Fixed
  • Affects Version/s: 3.0
  • Fix Version/s: 3.0
  • Component/s: .indexing
  • Description:
    Hide
    Single-pass tries to use as much memory as possible for mini-inverted indices (flushes). It uses some Java code to guess how much memory is left.
    When JVM has allocated all memory, and when only 70% free, flush().

    However, Java's memory management isn't reliable. We can easily get out of memory errors, particularly in Hadoop mode, and often for block indexing. Java6 makes this problem worse as it will throw OutOfMemoryError earlier than Java 5 would.
    Show
    Single-pass tries to use as much memory as possible for mini-inverted indices (flushes). It uses some Java code to guess how much memory is left. When JVM has allocated all memory, and when only 70% free, flush(). However, Java's memory management isn't reliable. We can easily get out of memory errors, particularly in Hadoop mode, and often for block indexing. Java6 makes this problem worse as it will throw OutOfMemoryError earlier than Java 5 would.

Attachments

  1. Text File
    singlepass-used-memory.patch
    (5 kB)
    Craig Macdonald
    29/Jun/09 6:58 PM

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Craig Macdonald added a comment - 29/Jun/09 6:57 PM

Proposed Solution: instead of checking how much is free, check to see how much you know you have used for mini inv index in memory. Set threshold e.g. to 300MB.

Future work: set this as % of max JVM size?

Show
Craig Macdonald added a comment - 29/Jun/09 6:57 PM Proposed Solution: instead of checking how much is free, check to see how much you know you have used for mini inv index in memory. Set threshold e.g. to 300MB. Future work: set this as % of max JVM size?
Hide
Permalink
Craig Macdonald added a comment - 29/Jun/09 6:58 PM

Initial patch for singe-pass and hadoop mode indexing, for Terrier 2. Initial experiments show this to make indexing more resilient.

Show
Craig Macdonald added a comment - 29/Jun/09 6:58 PM Initial patch for singe-pass and hadoop mode indexing, for Terrier 2. Initial experiments show this to make indexing more resilient.
Hide
Permalink
Rodrygo L. T. Santos added a comment - 30/Jun/09 9:55 AM

Which mechanism are you using to measure the available memory? Is it different from this one?

MemoryMXBean mxBean = ManagementFactory.getMemoryMXBean();
MemoryUsage mu = mxBean.getHeapMemoryUsage();
long used = mu.getUsed() / 1048576;
long max = mu.getMax() / 1048576;

Show
Rodrygo L. T. Santos added a comment - 30/Jun/09 9:55 AM Which mechanism are you using to measure the available memory? Is it different from this one? MemoryMXBean mxBean = ManagementFactory.getMemoryMXBean(); MemoryUsage mu = mxBean.getHeapMemoryUsage(); long used = mu.getUsed() / 1048576; long max = mu.getMax() / 1048576;
Hide
Permalink
Craig Macdonald added a comment - 01/Jul/09 10:18 AM

Hi Rodrygo, thanks for your interest.

We currently have a MemoryChecker interface. The implementation I'm using is based on the java.lang.Runtime object:
http://trmaster/cgi-bin/viewvc/trunk/src/uk/ac/gla/terrier/utility/RuntimeMemoryChecker.java

Would you be able to provide an implementation based on the bean interface you have found? I dont know if the statistics that this implementation provides are the same as those from the Runtime interface. I have checked the JDK source, and it's not the case that the bean is just a wrapper for the Runtime object.

C

Show
Craig Macdonald added a comment - 01/Jul/09 10:18 AM Hi Rodrygo, thanks for your interest. We currently have a MemoryChecker interface. The implementation I'm using is based on the java.lang.Runtime object: http://trmaster/cgi-bin/viewvc/trunk/src/uk/ac/gla/terrier/utility/RuntimeMemoryChecker.java Would you be able to provide an implementation based on the bean interface you have found? I dont know if the statistics that this implementation provides are the same as those from the Runtime interface. I have checked the JDK source, and it's not the case that the bean is just a wrapper for the Runtime object. C
Hide
Permalink
Craig Macdonald added a comment - 16/Jul/09 7:13 PM

I committed an improved version to trunk.

Show
Craig Macdonald added a comment - 16/Jul/09 7:13 PM I committed an improved version to trunk.

People

Dates

  • Created:
    29/Jun/09 6:56 PM
    Updated:
    05/Mar/10 4:54 PM
    Resolved:
    16/Jul/09 7:13 PM
Atlassian JIRA (v4.1#519) | Report a problem | Request a feature | Contact administrators
Powered by a free Atlassian JIRA open source license for Univ of Glasgow. Try JIRA - bug tracking software for your team.