Ideas and Plans for Memory Profiling with NYTProf

XXX This mostly a dumping ground for thoughts, and links at the moment. Somewhere to gather the info so it'll be easier to find when someone wants to work on it.

It's also focussed on runtime profiling of total memory usage over time (rather than a detailed analysis of who 'owns' what memory as some particular point in time, like the end of the program).

NYTProf currently only measures time and supports a limited range of "clocks" (realtime vs cputime). For profiling memory we need to add a new kind of "clock" that measures memory usage. Since we're generalizing the concept of what gets measured (and how we get the info from the system) a better name than "clock" would be "probe".

Conjectural terminology:

    "Probe" means some measuring mechanism like get_clock(), times(), getrusage(),
    that may yield multiple pieces of information with a single call.

    "Measure" is one specific item generated by a probe.

    Probe "time"   uses times(), measures: "time.user", "time.user+sys" etc
    Probe "clock"  uses clock_gettime(), measures: "clock.realtime", "clock.monotonic" etc
    Probe "rusage" uses getrusage(), measures: "rusage.majflt", "rusage.nvcsw" etc
    Probe "memory" measures: "memory.bytes", "mem.allocs" etc
    Probe "arena"  measures: "arena.svs", "arena.bytes" etc

Generalize the concepts of probes. Have a structure defining a 'probe' with pointers to functions to get the values, subtract values to get relative ticks, return the tick units etc. Give them names and attributes (cpu, realtime etc). User could then pick a probe by name. By default we'd pick the best available realtime probe. Use the subtraction logic where we currently handle times in the statement and subroutine profilers.

Email threads

"Memory profiling in Devel::NYTProf?: - June 2009 Talking about the problems of measuring memory usage of the whole process re:

"Memory profiling possibilities in NYTProf" - September 2009 This includes a detailed overview of the issues.

"profiling memory" - Dec 2009 Nicholas Clark offers an experimental patch that intercepts malloc and free and makes NYTProf measure memory usage.

Possibly Relevant Perl Modules

Per-process memory information:

Arena, stash, and pad based memory reporters:

Per-object memory size reporters:


Other Items of Interest

"Memory Efficient Perl" slides by jjore

Other profile/memory visualization tools