Tag Archives: Performance

Troubleshooting badly behaving IIS application pools

There are many reasons why an application pools’ worker process (W3WP.exe) could be behaving badly.  The best approach is to capture some memory dumps during the problem situation and then analyze them.  The problem with that is the root cause is not always obvious, even after many hours of analysis and investigation.

An alternative to memory dump analysis, or maybe a prequel, is to capture and analyze the behavior of the worker process.  One of the simplest actions to monitor the application pool is to open up Task Manager and look at the PID of the W3WP.exe process, as illustrated in Figure 1.

Figure 1, the w3wp.exe PID

Write down the PID in the morning and then come back a few times during the day to see if the process is restarting.  When/if the worker process restarts, it will get a new PID.  If you do notice that the worker process is being restarted, consider using the “Generate Recycle Event Log Entry” features available via the Advanced Settings… window for the given application pool.  Figure 2 shows the default settings on a Windows Server 2012 IIS 8 machine.

Figure 2, The Generate Recycle Event Log Entry settings

During the troubleshooting phase you might consider enabling all of the attributes to be certain that you capture as much information as possible.  However, once you complete your troubleshooting phase, consider disabling the ones you do not find useful in
your environment.  The description of these attributes are described in the below bulleted list.

  • Application Pool Configuration Changed – Event is logged when the application pool recycles due to a change in its configuration
  • ISAPI Report Unhealthy – Event is logged because an ISAPI extension has reported itself as unhealthy
  • Manual Recycle – Event is logged when the application pool has been manually recycled
  • Private Memory Limit Exceeded – Event is logged when the application pool recycles after exceeding its private memory limit
  • Regular Time Interval – Event is logged when the application pool recycles on its scheduled interval
  • Request Limit Exceeded – Event is logged when the application pool recycles after exceeding its request limit
  • Specific Time – Event is logged when the application pool recycles at a scheduled time
  • Virtual Memory Limit Exceeded – Event is logged when the application pool recycles after exceeding its virtual memory limits

Once configured, you would begin seeing events logged into the System event logs on the IIS server.  Figure 3 shows 3 logged events which caused a recycle of the application pools’ worker process.

  • Event ID 5186, 5080 and 5079
  • Useful link to IIS Application Pool Recycling event ids

Figure 3, IIS Application Pool Recycling events

The recycling of an application pool is in many cases nothing to be alarmed about.  However, this is a place to analyze and track when you are experiencing some unexpected behavior.  At the same time, if the recycling is happening multiple times throughout the day, you should look into what is causing it to happen as the recycle does have a customer impact.  Read this article which digs deeper into appDomain recycles and the impact they can have here.

via: http://blogs.msdn.com/b/benjaminperkins/archive/2013/07/01/troubleshooting-badly-behaving-iis-application-pools.aspx

IIS 7.5: Avoid performance issues when creating isolated application pools for applications

At NowOnline, we recently experienced major performance problems with two of our webservers (Windows 2008 Web Server with IIS7.5). Over the months memory consumption and CPU usage had increased substantially as the number of hosting packages increased to respectively 290 and 120 websites. As a result, websites running on these machines became progressively slower. Worse, the dreaded ‘Out of memory’ exception started showing up periodically as the servers tried to free and allocate RAM to new worker processes. Adding additional RAM to the virtual machines did remedy the problems for a while, but this was not a permanent solution.

Our hosting provider diagnosed the problem initially and attributed it to the sourcecode and the volume of websites (and suggested adding more RAM). So I set out to figure out what was going wrong, really. I initially explored potential memory leaks or bad code but was unable to find a clear pattern here. Some of the larger sites used more memory, obviously. But I couldn’t find clear examples of the telltale sign of memory leaks; an ever increasing memory consumption by a single website. Next, we started moving websites to another webserver to test if the problem was caused by the load of the increasing pool of websites. Turning off some heavy websites did not improve performance and did not reduce memory load significantly. Any freed up RAM was quickly used up by other websites.

A major part of the solution turned out to be in the way we used application pools. When configuring the webservers I had always adhered to the ‘best practice’ of creating a separate application pool per production site (see this article, for example). This is actually the default behavior when creating a new website in IIS. There are clear advantages to this approach:

  • If a code problem causes the website to crash (i.e. infinite loop or a memory leak), only the associated application pool crashes. This means that other websites are not affected by problems in other websites;
  • This approach makes it easier to tweak application pool settings (CPU and RAM allocation) for specific websites;
  • It is easier to diagnose memory leaks by investigating RAM usage by a single application pool;

IIS is quite clever when it comes to managing the application pools. If a website is not getting any traffic for a while, IIS shuts down the associated application pool by default. This frees up resources (RAM and CPU) for other application pools. Should a new visitor arrive, IIS simply starts up the application pool and spawns worker processes. It soon dawned on me that every application pool requires some overhead in resources. And RAM that was allocated to one pool could not be freely used by another pool.

To test if our configuration (one application pool per website) was indeed causing the problems, I started grouping the websites into several shared application pools based on their logical relatedness and .NET framework. For the more critical websites, I decided to stick to isolated application pools just to be sure. I ended up going from 250 pools to about 20. The results speak for themselves:


Picture 1: Memory consumption by alpha-web1 before and after the change (around 4PM)


Picture 2: Memory consumption by alpha-web2 before and after the change (between 7 and 8.30PM)

The results clearly show that RAM consumption dropped dramatically, from a structural 98% to around 45%. The virtual memory (‘Swap’ in the picture) that had to made available to virtually increase RAM also dropped to zero, which had the nice side effect of lowering Disk I/O. Not only had the servers become significantly more responsive, performance for individual websites also increased.

We initially worried about strange behavior, such as duplicate MVC route names, caching conflicts and such, but experienced none of these. IIS does isolate this kind of behavior nicely.

The lesson that I learned from this is that it’s not a good idea to stick too rigidly to a ‘application pool per website’ practice. Instead, try grouping non-critical websites into shared application pools as much as possible, unless you run into problems. If you have a critical production website, creating a separate application pool is still the best possible advice though. I hope this post will be of help to other administrators struggling with similar performance issues.

via: http://www.christiaanverwijs.nl/post/2013/08/28/IIS-75-Avoid-performance-issues-when-creating-isolated-application-pools-for-applications4.aspx

Best practices for creating websites in IIS 6.0

Every time I create an IIS website, I do some steps, which I consider as best practice for creating any IIS website for better performance, maintainability, and scalability. Here’ re the things I do:

Create a separate application pool for each web application

I always create separate app pool for each web app because I can select different schedule for app pool recycle. Some heavy traffic websites have long recycle schedule where low traffic websites have short recycle schedule to save memory. Moreover, I can choose different number of processes served by the app pool. Applications that are made for web garden mode can benefit from multiple process where applications that use in-process session, in memory cache needs to have single process serving the app pool. Hosting all my application under the DefaultAppPool does not give me the flexibility to control these per site.

The more app pool you create, the more ASP.NET threads you make available to your application. Each w3wp.exe has it’s own thread pool. So, if some application is congesting particular w3wp.exe process, other applications can run happily on their separate w3wp.exe instance, running under separate app pool. Each app pool hosts its own w3wp.exe instance.

So, my rule of thumb: Always create new app pool for new web applications and name the app pool based on the site’s domain name or some internal name that makes sense. For example, if you are creating a new website alzabir.com, name the app pool alzabir.com to easily identify it.

Another best practice: Disable the DefaultAppPool so that you don’t mistakenly keep adding sites to DefaultAppPool.

image

First you create a new application pool. Then you create a new Website or Virtual Directory, go to Properties -> Home Directory tab -> Select the new app pool.

image

Customize Website properties for performance, scalability and maintainability

First you map the right host headers to your website. In order to do this, go to WebSite tab and click on “Advanced” button. Add mapping for both domain.com andwww.domain.com. Most of the time, people forget to map the domain.com. Thus many visitors skip typing the www prefix and get no page served.

image

Next turn on some log entries:

image

These are very handy for analysis. If you want to measure your bandwidth consumption for specific sites, you need the Bytes Sent. If you want to measure the execution time of different pages and find out the slow running pages, you need Time Taken. If you want to measure unique and returning visitors, you need the Cookie. If you need to know who is sending you most traffic – search engines or some websites, you need the Referer. Once these entries are turned on, you can use variety of Log Analysis tools to do the analysis. For example, open source AWStats.

But if you are using Google Analytics or something else, you should have these turned off, especially the Cookie and Referer because they take quite some space on the log. If you are using ASP.NET Forms Authentication, the gigantic cookie coming with every request will produce gigabytes of logs per week if you have a medium traffic website.

image

This is kinda no brainer. I add Default.aspx as the default content page so that, when visitors hit the site without any .aspx page name, e.g. alzabir.com, they get the default.aspx served.

image

Things I do here:

  • Turn on Content Expiration. This makes static files remain in browser cache for 30 days and browser serves the files from its own cache instead of hitting the server. As a result, when your users revisit, they don’t download all the static files like images, javascripts, css files again and again. This one setting significantly improves your site’s performance.
  • Remove the X-Powered-By: ASP.NET header. You really don’t need it unless you want to attach Visual Studio Remote Debugger to your IIS. Otherwise, it’s just sending 21 bytes on every response.
  • Add “From” header and set the server name. I do this on each webserver and specify different names on each box. It’s handy to see from which servers requests are being served. When you are trying to troubleshoot load balancing issues, it comes handy to see if a particular server is sending requests.

image

I set the 404 handler to some ASPX so that I can show some custom error message. There’s a 404.aspx which shows some nice friendly message and suggests some other pages that user can visit. However, another reason to use this custom mapping is to serve extensionless URL from IIS. Read this blog post for details.

image

Make sure to set ASP.NET 2.0 for your ASP.NET 2.0, 3.0 and 3.5 websites.

Finally, you must, I repeat you “MUST” turn on IIS 6.0 gzip compression. This turns on the Volkswagen V8 engine that is built into IIS to make your site screaming fast.

via: http://msmvps.com/blogs/omar/archive/2008/10/04/best-practices-for-creating-websites-in-iis-6-0.aspx