I’ve spent a few days trying to track down a garbage collection problem in some code I’ve been working on.  I finally figured it out. Along the way, I’ve learned more about .NET garbage collection than I ever wanted to know, and discovered a useful tool to help with these kinds of problems.

I have a bunch of SharedConnection objects.  I keep them in a cache, in the form of a Dictionary<string,WeakReference>. The cache is filled with WeakReferences, because I don’t want the cache to keep the SharedConnections from being garbage-collected when nobody else is referencing them anymore.

But it wasn’t working. Unreferenced SharedConnection objects lived forever, even when I explicitly called GC.Collect(). I know, the first rule of GC.Collect() is “Don’t call GC.Collect().” But I have a good reason. And unreferenced SharedConnection objects were not getting collected. Why not?

I found a program called .NET Memory Profiler, which has been very helpful. What it does is takes a snapshot of the entire heap, and then lets you browse around it. You can find all instances of a given type. You can find everything else that the object references, directly or indirectly. And you can find everything that references the object, directly or indirectly. This last one is the one that really helps with sorting out garbage-collection problems.

The best explanation of the .NET garbage collector I’ve found is here. The important thing is that it does not use reference counting. That means it does not get tricked when two objects circularly reference each other. Such objects can still be garbage collected, if there are no other references. Any object that is unreachable from a “root” is a candidate for collection. If an object can’t be reached from a root, then it can’t be reached at all, ever, by anyone, and might as well be collected. That includes little islands of mutually-referencing objects that can’t be reached from anywhere else. They’re gone. Or at least they should be.

But my SharedConnection objects were not being collected. .NET Memory Profiler showed me why. The problem is that my SharedConnection objects include a System.Timers.Timer field, and I have hooked the timer’s Elapsed event. That creates one of those circular-reference structures. I was aware of this, and didn’t think it would be a problem. And indeed, by itself, it isn’t.

But .NET Memory Profiler showed me that my SharedConnection object does still have a root path, which leads via the Timer to an object named System.Threading._TimerCallback. I don’t know what that object is, it’s something internal to .NET and not documented anywhere. But it’s reachable from a root, so I’m reachable from a root. Damn!

So, there you go. If you have a System.Timers.Timer field, and you’ve hooked it’s Elapsed event, then you will never, ever get garbage-collected.

Having realized that, I added this bit of code before the GC.Collect() call:

foreach( WeakReference weak_ref in ExistingConnections.Values ) {
    SharedConnection connection = weak_ref.Target as SharedConnection;
    if( connection != null ) {
        connection.SuspendConnection();
    }
}
GC.Collect();

The call to connection.SuspendConnection() will unhook the Timer.Elapsed (among other things), which should break the root path and finally allow the SharedConnection objects to get garbage-collected.

But it still doesn’t work. WTF?

Taking another .NET Memory Profiler snapshot, I now see there are no root paths keeping the object alive. So why didn’t the garbage collector collect it?

Phase 2

After much more head-scratching, and googling till my eyeballs bled, I finally had the idea to take another Memory Profiler snapshot right in there between the foreach loop and the GC.Collect(). I added a 45 second delay in there to give me time to grab the snapshot.

What this new snapshot showed was new root paths! Paths that did not exist before the function was called, and would not exist again after. But at that point, just before GC.Collect()

Here I’m going into conjecture, I don’t really understand C# enough to know for sure what’s happening. But my guess is that it’s because of the strong reference I created inside the foreach loop. I figured that once I leave the foreach loop, the strong references will go out-of-scope and be destroyed, because that’s what the scoping rules of C# say. But I’m guessing that that’s not the entire story.

Perhaps it’s actually like in unmanaged C/C++: You can have local variables inside nested blocks, and their destructors run when you leave their enclosing scopes. But the stack space for all of them is actually created the moment you call the function, and doesn’t get released until the end of the function.

Perhaps it’s similar in C#: The nested hard-reference becomes inaccessible to C# source code after we leave the foreach block, but perhaps the variable itself continues to exist until the end of the function. That would explain how it is that a hitherto unknown root path now exists when I call GC.Collect().

I finally fixed the problem by adding a connection = null; line at the end of the foreach block:

foreach( WeakReference weak_ref in ExistingConnections.Values ) {
    SharedConnection connection = weak_ref.Target as SharedConnection;
    if( connection != null ) {
        connection.SuspendConnection();
    }
    connection = null;
}
GC.Collect();

It looks like a useless bit of ineffective code, but it makes all the difference. Now the GC.Collect() finally collects my unused SharedConnection objects.

What a struggle. And I have to say, I’d never have figured it out without .NET Memory Profiler.