Tracking a memory leak using .NET tools

This is a two part series:

• Debugging Docker containers running .NET
• Tracking a memory leak using .NET tools

Previously, we looked at some tools we can use to collect a memory dump file which can be used for analysis. This posts follows on from that one. Let's continue.

Memory leaking code sample

namespace MemoryLeakingCode
{
    class Person
    {
        public string Name { get; set; }
    }

    class Program
    {
        public static IList<Person> RootedPeople = new List<Person>();

        static void Main(string[] args)
        {
            RootedPeople.Add(new Person { Name = "Jason" });
            RootedPeople.Add(new Person { Name = "Joe" });
            RootedPeople.Add(new Person { Name = "Jamie" });

            Console.WriteLine($"People count: {RootedPeople.Count}");
        }
    }
}

First peek

Let's get a high level overview of the memory usage and hone in on some interesting parts. This command will list all objects on the managed heap. We're looking for rooted objects.

As shown previously, run the program and collect the memory dump for analysis.

dotnet-dump collect --process-id $(pidof dotnet)

Load the dump in for analysis

dotnet-dump analyze <filename>

Get an overview

dumpheap -stat

This potentially will output a lot of data. Now we want to narrow down the search.

Narrowing it down

Using method table -mt flag

Looking at the output from the dumpheap -stat command, we're interested in the following:

Statistics:
              MT    Count    TotalSize Class Name
00007f1f6e46f6a0        3           72 MemoryLeakingCode.Person

let's inspect the following MT

dumpheap -mt 00007f1f6e46f6a0

Outputs

Address          MT                     Size
00007f1f40008428 00007f1f6e46f6a0       24
00007f1f400084b0 00007f1f6e46f6a0       24
00007f1f400084f0 00007f1f6e46f6a0       24

Statistics:
              MT    Count    TotalSize Class Name
00007f1f6e46f6a0        3           72 MemoryLeakingCode.Person
Total 3 objects

We can see the MemoryLeakingCode.Person type is used 3 times.

Using gcroot

Let's look deeper and see if we can find what code is holding onto MemoryLeakingCode.Person.

Use gcroot to see how the object is rooted:

gcroot -all 00007f1f400084f0

Outputs

Thread 140:
    00007FFCE894D4E0 00007F1F6E3606E6 MemoryLeakingCode.Program.Main(System.String[]) [...\MemoryLeakingCode\MemoryLeakingCode\Program.cs @ 24]
        rbp-68: 00007ffce894d4e8
            ->  00007F1F40008440 System.Collections.Generic.List`1[[MemoryLeakingCode.Person, MemoryLeakingCode]]
            ->  00007F1F40008478 MemoryLeakingCode.Person[]
            ->  00007F1F400084F0 MemoryLeakingCode.Person

    00007FFCE894D4E0 00007F1F6E3606E6 MemoryLeakingCode.Program.Main(System.String[]) [...\MemoryLeakingCode\MemoryLeakingCode\Program.cs @ 24]
        rbp-40: 00007ffce894d510
            ->  00007F1F40008440 System.Collections.Generic.List`1[[MemoryLeakingCode.Person, MemoryLeakingCode]]
            ->  00007F1F40008478 MemoryLeakingCode.Person[]
            ->  00007F1F400084F0 MemoryLeakingCode.Person

    00007FFCE894D4E0 00007F1F6E3606E6 MemoryLeakingCode.Program.Main(System.String[]) [...\MemoryLeakingCode\MemoryLeakingCode\Program.cs @ 24]
        rbp-38: 00007ffce894d518
            ->  00007F1F400084F0 MemoryLeakingCode.Person

    00007FFCE894D4E0 00007F1F6E3606E6 MemoryLeakingCode.Program.Main(System.String[]) [...\MemoryLeakingCode\MemoryLeakingCode\Program.cs @ 24]
        rbp-30: 00007ffce894d520
            ->  00007F1F40008440 System.Collections.Generic.List`1[[MemoryLeakingCode.Person, MemoryLeakingCode]]
            ->  00007F1F40008478 MemoryLeakingCode.Person[]
            ->  00007F1F400084F0 MemoryLeakingCode.Person

    00007FFCE894D4E0 00007F1F6E3606E6 MemoryLeakingCode.Program.Main(System.String[]) [...\MemoryLeakingCode\MemoryLeakingCode\Program.cs @ 24]
        rbp-20: 00007ffce894d530
            ->  00007F1F40008440 System.Collections.Generic.List`1[[MemoryLeakingCode.Person, MemoryLeakingCode]]
            ->  00007F1F40008478 MemoryLeakingCode.Person[]
            ->  00007F1F400084F0 MemoryLeakingCode.Person

HandleTable:
    00007F1FE6D015F8 (pinned handle)
    -> 00007F1F4FFFF038 System.Object[]
    -> 00007F1F40008440 System.Collections.Generic.List`1[[MemoryLeakingCode.Person, MemoryLeakingCode]]
    -> 00007F1F40008478 MemoryLeakingCode.Person[]
    -> 00007F1F400084F0 MemoryLeakingCode.Person

Found 6 roots.

We can see it's referenced in our List in Main. From this point we should evaluate the List code as to why it's rooted! But let's carry on a little further for fun.

Dumping objects

From the HandleTable section of the output, let's dump out and have a look at:

System.Collections.Generic.List`1[[MemoryLeakingCode.Person, MemoryLeakingCode]]

Running:

dumpobj 00007F1F40008440

Outputs

Name:        System.Collections.Generic.List`1[[MemoryLeakingCode.Person, MemoryLeakingCode]]
MethodTable: 00007f1f6e4d1d00
EEClass:     00007f1f6e455f00
Size:        32(0x20) bytes
File:        /usr/share/dotnet/shared/Microsoft.NETCore.App/3.1.12/System.Private.CoreLib.dll
Fields:
MT                Field         Offset     Type VT       Attr       Value             Name
0000000000000000  4001b11        8              SZARRAY  0 instance 00007f1f40008478 _items
00007f1f6e3da0e8  4001b12       10         System.Int32  1 instance                3 _size
00007f1f6e3da0e8  4001b13       14         System.Int32  1 instance                3 _version
0000000000000000  4001b14        8              SZARRAY  0   static dynamic statics NYI  s_emptyArray

We don't really expect to learn anything more at this point apart from maybe that a List encapsulates an array type. Just to finish the analysis let's look at the array instance.

dumpobj 00007f1f40008478

Ouputs

Name:        MemoryLeakingCode.Person[]
MethodTable: 00007f1f6e4d20a8
EEClass:     00007f1f6e3d5488
Size:        56(0x38) bytes
Array:       Rank 1, Number of elements 4, Type CLASS
Fields:
None

Again nothing exciting... except maybe that the number of elements is 4 which is 1 higher than the number of elements we inserted (3). This is the given array length by List. We didn't specify a length when we used a List type. List has made some assumptions based on performance and the expectation that the list will grow. Capacity can change depending on the size of your list.

If we changed the original code to the following and re ran the same analysis:

class Program
{
    public static Person[] RootedPeople = new Person[3];

    static void Main(string[] args)
    {
        RootedPeople[0] = new Person { Name = "Jason" };
        RootedPeople[1] = new Person { Name = "Joe" };
        RootedPeople[2] = new Person { Name = "Jamie" };

        Console.WriteLine($"People count: {RootedPeople.Length}");
    }
}

We get the following output because we used an Person[] array and set a length. One less level of abstraction and 1 element of memory shaved off too. Think of the savings! A whole 8 bytes!

Name:        MemoryLeakingCode.Person[]
MethodTable: 00007fe34500f730
EEClass:     00007fe344f75488
Size:        48(0x30) bytes
Array:       Rank 1, Number of elements 3, Type CLASS
Fields:
None

Extras

Using the -dead and -live flag

The -live and -dead flags list objects that can or cannot be reclaimed at this point. We're interested in the live objects. This can be a useful filter when investigating memory leaks.

Let's see what we the GC is expected to clean up. We dont expect to see our MemoryLeakingCode.Person or IList<RootedPeople>in this output.

dumpheap -dead

Statistics:
              MT    Count    TotalSize Class Name
00007f1f6e3ed9d8        1           32 System.IntPtr[]
00007f1f6e3ec308        2           64 System.RuntimeType[]
00007f1f6e3eacd0        2           64 System.Type[]
00007f1f6e40d508        2           88 System.Int32[]
00007f1f6e3e21c8        3          120 System.SByte[]
00007f1f6e3e0f90        3          144 System.String
00007f1f6e40d948        2          288 System.Collections.Generic.Dictionary`2+Entry[[System.String, System.Private.CoreLib],[System.Object, System.Private.CoreLib]][]
00007f1f6e3d5510        3          528 System.Object[]
Total 18 objects

As we can see our memory leaking code isn't in here.

dumpheap -live

This can take a long while depending on the heap size so be patient. Here's a snippet of my output with some noise taken out:

Statistics:
              MT    Count    TotalSize Class Name
...
00007f1f6e4d1d00        1           32 System.Collections.Generic.List`1[[MemoryLeakingCode.Person, MemoryLeakingCode]]
...
00007f1f6e46f6a0        3           72 MemoryLeakingCode.Person
00007f1f6e3e7f18        1           72 System.Collections.Generic.Dictionary`2[[System.String, System.Private.CoreLib],[System.Object, System.Private.CoreLib]]
00007f1f6e4d20a8        2           80 MemoryLeakingCode.Person[]
...
00007f1f6e3d5510        9        18168 System.Object[]
00007f1f6e3e0f90      134        34124 System.String
Total 303 objects

As we can see System.String appears often and MemoryLeakingCode.Person appears 3 times as expected for the amount of objects we created the object. This flag can be used early on in the progress when narrowing down.

When there's actually no leak...

In different scenarios sometimes there isn't a memory leak even though the memory increases without deallocation for a long period of time which might be unexpected. It's beneficial to understand a little bit about how the garbage collector operates.

From What happens during a garbage collection:

"Ordinarily, the large object heap (LOH) is not compacted, because copying large objects imposes a performance penalty. However, in .NET Core and in .NET Framework 4.5.1 and later, you can use the GCSettings.LargeObjectHeapCompactionMode property to compact the large object heap on demand. In addition, the LOH is automatically compacted when a hard limit is set by specifying either:

• A memory limit on a container.
• The GCHeapHardLimit or GCHeapHardLimitPercent run-time configuration options."

So after inspecting a heapdump and everything looks like it's able to be cleaned up by the GC. It's probably just that it hasn't kicked in yet because it hasn't met those conditions. You can always stick in a GC.Collect() in places as an experiment to somewhat verify but even calling for garbage collection directly doesn't guarantee it will happen when called.