armi.bookkeeping.memoryProfiler module

Interface to help diagnose memory issues during debugging/development.

There are many approaches to memory profiling.

1. You can ask psutil for the memory used by the process from an OS perspective. This is great for top-down analysis. This module provides printouts that show info from every process running. This is very fast.

2. You can use asizeof (part of pympler) to measure the size of various individual objects. This will help you pin-point your issue. But it’s slow.

3. You can use gc.get_objects() to list all objects that the garbage collector is tracking. If you want, you can filter it down and get the counts and sizes of objects of interest (e.g. all armi objects).

This module has tools to do all of this. It should help you out.

Note that if psutil is reporting way more memory usage than the asizeof reports, then you probably are dealing with a garbage collection queue. If you free a large number of objects, call gc.collect() to force a garbage collection and the psutil process memory should fall by a lot. Also, it seems that even if your garbage is collected, Windows does not de-allocate all the memory. So if you are a worker and you just got a 1.6GB reactor but then deleted it, Windows will keep you at 1.6GB for a while.

See Also:

http://packages.python.org/Pympler/index.html https://pythonhosted.org/psutil/ https://docs.python.org/2/library/gc.html#gc.garbage

armi.bookkeeping.memoryProfiler.describeInterfaces(cs)

Function for exposing interface(s) to other code

class armi.bookkeeping.memoryProfiler.MemoryProfiler(r, cs)

Bases: armi.interfaces.Interface

Construct an interface.

The r and cs arguments are required, but may be None, where appropriate for the specific Interface implementation.

Parameters

• r (Reactor) – A reactor to attach to

• cs (Settings) – Settings object to use

Raises

RuntimeError – Interfaces derived from Interface must define their name

name = 'memoryProfiler'

interactBOL()

Called at the Beginning-of-Life of a run, before any cycles start.

interactEveryNode(cycle, node)

Called at each time node/subcycle of every cycle.

interactEOL()

End of life hook. Good place to wrap up or print out summary outputs

displayMemoryUsage(timeDescription)

Print out some information to stdout about the memory usage of ARMI.

Makes use of the asizeof utility.

Useful when the debugMem setting is set to True.

Turn these on as appropriate to find all your problems.

_reactorAssemblyTrackingBreakdown()

_checkForDuplicateObjectsOnArmiModel(attrName, refObject)

Scans thorugh ARMI model for duplicate objects

_printFullMemoryBreakdown(startsWith='armi', reportSize=True, printReferrers=False)

looks for any class from any module in the garbage collector and prints their count and size

Very powerful. Also very slow if you reportSize

Parameters

• startsWith (str, optional) – limit to objects with classes that start with a certain string

• reportSize (bool, optional) – calculate size as well as counting individual objects. SLLOOOWW.

Notes

Just because you use startsWith=armi doesn’t mean you’ll capture all ARMI objects. Some are in lists and dictionaries.

static _getSpecificReferrers(klass, ancestorKlass)

Try to determine some useful information about the structure of ArmiObjects and potential orphans.

This takes a class and an expected/nominal parent class, which should both be instances of ArmiObject. It will then locate all instances of klass that are tracked by the GC, igoring those that have an ancestor of ancestorKlass type. A report will be generated containing the counts of the instances of klass that are _not_ part of the ancestor_class along with their referrer class.

This is useful for diagnosing memory leaks, as it points to unexpected referrers to ArmiObjects.

static _getReferrers(obj)

Print referrers in a useful way (as opposed to gigabytes of text

static _discussSkipped(skipped, errors)

class armi.bookkeeping.memoryProfiler.KlassCounter(reportSize)

Bases: object

countObjects(ao)

Recursively find non-list,dict, tuple objects in containers.

Essential for traversing the garbage collector

class armi.bookkeeping.memoryProfiler.InstanceCounter(classType, reportSize)

Bases: object

add(item)

armi.bookkeeping.memoryProfiler._getClsName(obj)

armi.bookkeeping.memoryProfiler._getModName(obj)

class armi.bookkeeping.memoryProfiler.ObjectSizeBreakdown(name, minMBToShowAttrBreakdown=30.0, excludedAttributes=None, initialZeroes=0)

Bases: object

property size

calcSize(obj)

_breakdownAttributeSizes(obj)

class armi.bookkeeping.memoryProfiler.ProfileMemoryUsageAction(timeDescription)

Bases: armi.mpiActions.MpiAction

invokeHook()

This method must be overridden in sub-clases.

This method is called by worker nodes, and has access to the worker node’s operator, reactor, and settings (through self.o, self.r, and self.cs). It must return a boolean value of True or False, otherwise the worker node will raise an exception and terminate execution.

Returns

result – Dependent on implementation

Return type

object

class armi.bookkeeping.memoryProfiler.SystemAndProcessMemoryUsage

Bases: object

class armi.bookkeeping.memoryProfiler.PrintSystemMemoryUsageAction

Bases: armi.mpiActions.MpiAction

property minProcessMemoryInMB

property maxProcessMemoryInMB

invokeHook()

This method must be overridden in sub-clases.

This method is called by worker nodes, and has access to the worker node’s operator, reactor, and settings (through self.o, self.r, and self.cs). It must return a boolean value of True or False, otherwise the worker node will raise an exception and terminate execution.

Returns

result – Dependent on implementation

Return type

object

printUsage(description=None)

This method prints the usage of all MPI nodes.

The printout looks something like:

SYS_MEM HOSTNAME 14.4% RAM. Proc mem (MB): 491 472 471 471 471 470 SYS_MEM HOSTNAME 13.9% RAM. Proc mem (MB): 474 473 472 471 460 461 SYS_MEM HOSTNAME … SYS_MEM HOSTNAME …

armi.bookkeeping.memoryProfiler._getFunctionObject()

Return the function object of the calling function. Useful for debugging

armi.bookkeeping.memoryProfiler._walkReferrers(o, maxLevel=0, level=0, memo=None, whitelist=None)

Walk down the tree of objects that refer to the passed object, printing diagnostics along the way.