Reusing HttpClient didn’t solve all my problems
In the previous article I posted about how disposing of HttpClient frequently choked my Api and made server crash under huge load. We looked at better way of reusing HttpClient and not get in to thread safety issues. Reusing HttpClient did improve the stability of the Api but it didn’t improve the performance of my system. By performance I mean firing multiple parallel http api calls and getting individual responses in least amount of time.
Why do we need this?
Let’s run through a hypothetical example where let’s say you are building a Rest End point which needs to provide list of dealers from whom you can order parts for fixing a vehicle.
Let’s assume your Api takes in list of Partids for which you need to find dealers. Your request payload looks like below:
{
"partIds": [1, 2, 3, 4, 5, 6, 7, 8]
}Lets say this is how your response json would look like:
{
"dealers": [{
"dealerId": 1,
"dealerName": "abc dealer",
"parts": [{
"partId": 1,
"partId": 2
}]
}, {
"dealerId": 2,
"dealerName": "def dealer",
"parts": [{
"partId": 3,
"partId": 4,
"partId": 5
}]
}, {
"dealerId": 3,
"dealerName": "xyz dealer",
"parts": [{
"partId": 6,
"partId": 7,
"partId": 8
}]
}]
}To generate this response lets say you have to call inventory Api by dealers and provide it list of part ids for which you need to check availability. So let’s say below is standard request body for dealer inventory api.
{
"parts": [{
"partId": 1,
"partId": 2,
"partId": 3,
"partId": 4,
"partId": 5,
"partId": 6,
"partId": 7,
"partId": 8
}]
}Response received from each dealer is let’s say of below format.
{
"parts": [{
"partId": 1,
"price": 255,
"isAvailable": true
}, {
"partId": 2,
"price": 55,
"isAvailable": true
}, {
"partId": 3,
"price": 155,
"isAvailable": true
}, {
"partId": 4,
"price": 255,
"isAvailable": false
}, {
"partId": 5,
"price": 255,
"isAvailable": false
}, {
"partId": 6,
"price": 255,
"isAvailable": false
}, {
"partId": 7,
"price": 255,
"isAvailable": false
}, {
"partId": 8,
"price": 255,
"isAvailable": false
}]
}As you can see from above example if you have dealers whom you have relationship with then for every request you need to call each dealer and preferably in parallel and process their responses. So for every incoming request you have to open 15 outbound requests. If you are not reusing the HttpClient as described in your previous post you will exhaust the available sockets extremely soon and thus causing request failures and user experience degradation. Pretty much loosing your customer. But even after following the best practice as I wrote earlier it’s not going to decrease the response time for your outbound requests as more and more request get queued on outbound sockets. For this you need to toggle different “ServicePointManager” settings.
ServicePointManager.DefaultConnectionLimit is the key
There are plenty of articles on internet saying DefaultConnection limit is 2 for service point manager is 2 which is true and false both. Looking at the source code documentation and my observation i can safely tell you if you don’t make any changes default connection limit your outbound connections will be throttled to 10 outbound connections max for asp.net pipeline.
If you don’t have any high throughput requirements for outbound connections you will never have to worry about toggling this setting. But for my use case this was severely limiting and I started wondering what other defaults I need to override and what to set.
How did I diagnose the problem
My integration tests which would call one of the endpoints in my api, which would internally make 15 api calls, will work smoothly on running them sequentially. But as soon as I run them in parallel one or other tests will take forever to execute and eventually fail with timeout error codes. If you run fiddler on the server where api is hosted you can see number of outbound connections at a given time.
So with simple math if I ran 10 integration tests in parallel my api will have 150 outbound connections. This issue set me on a path to research and learn form other engineers having similar problems and but literature on internet for these things is extremely light. So I did the digging old way of reading a book to solve a problem rather than skim the SO questions.
One of my favorite book on .Net app tuning is “Writing High-Performance .NET Code by Ben Watson”. This book threw some light on toggling various ServicePointManger settings in the 6th chapter under “Optimize HTTP settings and network communications”. So I followed the recommendations from Ben and to my amazement made a huge difference in my application performance.
Optimal ServicePointManager settings for high throughput and high performance
Here are the properties I have toggled and it has made huge impact on my orchestraion api performance.
ServicePointManager.UseNagleAlgorithm = false;ServicePointManager.Expect100Continue = false;ServicePointManager.DefaultConnectionLimit = int.MaxValue;ServicePointManager.EnableDnsRoundRobin = true;ServicePointManager.ReusePort = true;
DefaultConnectionLimit- The number of connections per end point. Setting this to highest number increased my overall throughput substantially. This setting is running successfully in prod without any instability on any of the servers. You can safely turn this on for your Apis too.
Expect100Continue- This setting was put in place to save bandwidth before sending huge object for post and put request to ensure remote end point is up and running. If you own the api on both sides then you should turn this off to reduce latency between each api communication.
UseNagleAlgorithm- Nagle’s algorithm is a means of improving the efficiency of TCP/IP networks by reducing the number of packets that need to be sent over the network. This can decrease the overall transmission overhead but can cause delay in data packet arrival. You can turn this off safely as well. Modern network communication shouldn’t be relying on such old measures of saving bandwidth.
There are few more properties mentioned in this chapter which could further enhance the performance but I was able to hit my benchmarks successfully with above changes.
I’m experimenting with last two properties (EnableDnsRoundRobin and Reuse Port) as I have not been able to benchmark their contributions to over all picture so take last two with pinch of salt.
One final word of utmost importance make service point manager changes in Global.asax.cs before you make any api calls. ServicePointManager once initialized remains in memory with same configuration throughout the life cycle of the app unless you cherry pick ServicePoint objects for each end point.
Do share if any of these changes helped you too in making your Api faster.
Keep calm there is always a solution for every problem you just need to seek with intensity!!!
