Share this question

Welcome to Teachnovice Q&A, where you can ask questions and receive answers from other members of the community.

This is a collaboratively edited question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

HttpClientHandler / HttpClient Memory Leak?

0 like 0 dislike
479 views

I have anywhere from 10-150 long living class objects that call methods performing simple HTTPS API calls using HttpClient. Example of a PUT call:

using (HttpClientHandler handler = new HttpClientHandler())
{
    handler.UseCookies = true;
    handler.CookieContainer = _Cookies;

    using (HttpClient client = new HttpClient(handler, true))
    {
        client.Timeout = new TimeSpan(0, 0, (int)(SettingsData.Values.ProxyTimeout * 1.5));
        client.DefaultRequestHeaders.TryAddWithoutValidation("User-Agent", Statics.UserAgent);

        try
        {
            using (StringContent sData = new StringContent(data, Encoding.UTF8, contentType))
            using (HttpResponseMessage response = await client.PutAsync(url, sData))
            {
                using (var content = response.Content)
                {
                    ret = await content.ReadAsStringAsync();
                }

            }
        }
        catch (ThreadAbortException)
        {
            throw;
        }
        catch (Exception ex)
        {
            LastErrorText = ex.Message;
        }
    }
}

After 2-3 hours of running these methods, which include proper disposal via using statements, the program has creeped to 1GB-1.5GB of memory and eventually crashes with various out of memory errors. Many times the connections are through unreliable proxies, so the connections may not complete as expected (timeouts and other errors are common).

.NET Memory Profiler has indicated that HttpClientHandler is the main issue here, stating it has both 'Disposed instances with direct delegate roots' (red exclamation mark) and 'Instances that have been disposed but are still not GCed' (yellow exclamation mark). The delegates that the profiler indicates have been rooted are AsyncCallbacks, stemming from HttpWebRequest.

It may also relate to RemoteCertValidationCallback, something to do with HTTPS cert validation, as the TlsStream is an object further down in the root that is 'Disposed but not GCed'.

With all this in mind - how can I more correctly use HttpClient and avoid these memory issues? Should I force a GC.Collect() every hour or so? I know that is considered bad practice but I don't know how else to reclaim this memory that isn't quite properly being disposed of, and a better usage pattern for these short-lived objects isn't apparent to me as it seems to be a flaw in the .NET objects themselves.


UPDATE Forcing GC.Collect() had no effect.

Total managed bytes for the process remain consistent around 20-30 MB at most while the process overall memory (in Task Manager) continues to climb, indicating an unmanaged memory leak. Thus this usage pattern is creating an unmanaged memory leak.

I have tried creating class level instances of both HttpClient and HttpClientHandler per the suggestion, but this has had no appreciable effect. Even when I set these to class level, they are still re-created and seldom re-used due to the fact that the proxy settings often require changing. HttpClientHandler does not allow modification of proxy settings or any properties once a request has been initiated, so I am constantly re-creating the handler, just as was originally done with the independent using statements.

HttpClienthandler is still being disposed with "direct delegate roots" to AsyncCallback -> HttpWebRequest. I'm starting to wonder if maybe the HttpClient just wasn't designed for fast requests and short-living objects. No end in sight.. hoping someone has a suggestion to make the use of HttpClientHandler viable.


Memory profiler shots: Initial stack indicating that HttpClientHandler is the root issue, having 304 live instances that should have been GC'd

enter image description here

enter image description here

asked Jan 8, 2015 by user1111380  
@ChrisEelmaa I've moved on to just using HttpWebRequest async at this point, which the HttpClient is just a wrapper for, and it has alleviated the memory issues two-fold (e.g. before the application crashed after 3 hours, now it will run for 8). I don't see a solution to this and it appears to be a flaw in the framework, as you suggested - spending more time trying to resolve it is beyond my interest level at this point. Thanks for the suggestions.
@user1111380 HttpClientHandler must have something connected to it, see if the response connection is being closed? Not being able to GC an object is a major issue that would be piked up very fast, the issue must be on your code.
try to add this as your first catch "catch (WebException ex) { ex.Response.Close(); }"
Try to lower down your timeout see if the connection retention is the problem.
@AlexandrNikitin; yep, leaking here.

1 Answer

0 like 0 dislike

Using the repro form Alexandr Nikitin, I was able to discover that this seems to happen ONLY when you have HttpClient be a short lived object. If you make the handler and client long lived this does not seem to happen:

using System;
using System.Net.Http;
using System.Threading.Tasks;

namespace HttpClientMemoryLeak
{
    using System.Net;
    using System.Threading;

    class Program
    {
        static HttpClientHandler handler = new HttpClientHandler();

        private static HttpClient client = new HttpClient(handler);

        public static async Task TestMethod()
        {
            try
            {
                using (var response = await client.PutAsync("http://localhost/any/url", null))
                {
                }
            }
            catch
            {
            }
        }

        static void Main(string[] args)
        {
            for (int i = 0; i < 1000000; i++)
            {
                Thread.Sleep(10);
                TestMethod();
            }

            Console.WriteLine("Finished!");
            Console.ReadKey();
        }
    }
}
answered Jan 8, 2015 by Matt Clark  
...