Logmatic.io Blog

5 Steps to tech & business analytics
with IIS logs

Hello fellas, it’s Pierre again.

In my previous article I shared how you can monitor your HA Proxy logs to extract useful information on your load balancer’s behavior & performance. I’m back this timsce to talk about IIS web servers and IIS logs.

First things first, some background. Internet Information Services (IIS) is an extensible web server created by Microsoft and is used with the Windows NT family. If you are a .NET dev and a lifelong Windows programmer, then IIS servers are the best solution for you, better than Apache or Nginx. With a friendly user interface and a well-documented scriptability, the IIS developer ecosystem appears very mature.

In this article, I will systematically cover the main things to know about IIS logs: where to locate them, what their format is, and how to collect them. Then I’ll especially focus on how to extract tech & business value out of them with real-life use cases. So let’s jump right into it.

I. Where do I find my IIS logs?

While writing this article, I came to realise that the IIS logs location is not as intuitive as I might have thought. That’s why I’m including a step-by-step “find your IIS logs” explanation below for IIS 7 and above:

Log into your IIS manager, and from there, go to your website setting panel. Check the Logging tab: in the corresponding directory field you will find your logs waiting for you.
It’s important to note that this directory creates one folder per website. This means that logs for site ID 1234 are in the W3SVC1234 folder, log files for site ID 2222 are in the W3SVC2222 folder, and so on…

Now that you know where your IIS logs are, let’s understand what their format is.

II. What is IIS logs format?

Starting with IIS 7 and above, all IIS logs follow the World Wide Web Consortium (W3C) log file format. This setting can be changed by editing the logFormat attribute of IIS to NCSA, or in the Custom section of your website logging settings.

You can also define the information categories of IIS logs by editing the logExtFileFlags attribute. It will allow you to add more context to your logs. Default values are the following:



Date, Time


Date and time on which the activity occurred.


ClientIP


IP address of the client that made the request.
UserName

Browser type that the client used.
ServerIP


IP address of the server on which the log file entry was generated.
Method

Requested action. For example, GET, POST, etc.
UriStem


Universal Resource Identifier (URI) stem information, which is the target of the action.
Uri Query


Query, if any, that the client was trying to perform.
TimeTaken


Amount of time taken for a request to be completed. The time taken is recorded in milliseconds.
HttpStatus
HTTP status code.
Win32Status
Windows status code.
ServerPort


Server port number that is configured for the site.
UserAgentBrowser type that the client used.
HttpSubStatus
Sub-status code of the HTTP error. For example, for the 500.18 HTTP error, status code is 500 and sub-status code is 18.
RefererWebsite last visited by user before being referred to current website.

This set of default metrics are great for troubleshooting and monitoring. Should you have specific issues (cookie analysis, analysing your requests headers…) and wish to boost your monitoring, feel free to refer to this Microsoft article Advanced logging for IIS tutorial.

IIS 8.5 goes one step further. This version of IIS enables you to log custom fields in addition to the standard logged set of information. Be aware that in order to add these custom fields, W3C needs to be the log format used. Get the whole description on IIS customFields article.

Note: For IIS web servers, if the element is configured in both the section and in the section specific to the website. Configuration in the section takes over.

With location out of the way, let’s check out how you can collect your IIS logs.

III. How do I collect my IIS logs smoothly?

The first thing to do is check if logging is enabled for your IIS website(s). This is usually activated by default, but if it’s not, check this LeanSentry article to turn on your IIS logging.

Now, how can you collect & analyze your IIS access logs? Two options are available: tailing IIS log files or using a log management solution. The second solution will bring you way more value than the usual tail/grep method. You can be sure to get a clearer visibility over your stack while saving much time. Ok, I might be partial on this topic, working for Logmatic.io log management tool myself, but in all fairness, I can only advise you to use a log management tool for monitoring your logs.

In order to forward your IIS logs to a log management tool, you will need a reliable log-shipper. The one we recommend for a Windows environment, considering both its forwarding and processing possibilities, is Nxlog.

IV. How do I ship my IIS logs?

Alright, so now your logs are properly set and you know where to find them. We can start using a log-shipper in order to manipulate them.

JSON logging is the best card up our sleeve. Since IIS logs are in a csv-like file, you need to configure your log shipper on how to process and prepare the data before sending it as a JSON object to a log management tool.

We’re moving here with a Nxlog example, Nxlog being our preferred IIS log forwarder, but you could do the same with logstash or fluentD. Luckily, Nxlog has the right module for our purpose. `xm_csv`. Here it goes:

  1. You first need to define the following extension in your Nxlog configuration:
  2. <Extension w3c>
      Module xm_csv
      Fields $date, $time, $s_ip, $cs_method, $cs_uri_stem, $cs_uri_query, $s_port, $cs_username, $c_ip, $cs_User_Agent, $cs_Referer, $sc_status, $sc_substatus, $sc_win32_status, $time_taken
      FieldTypes string, string, string, string, string, string, integer, string, string, string, string, integer, integer, integer, integer
      Delimiter ' '
    </Extension>
    

    The Fields here are the header of your IIS log file; the format we describe is the default IIS log format. If you are using a different format you can actually copy these from the log file header.

  3. Once the extension defined, you need to integrate it into your Nxlog input module:
  4. <Input IIS_Site1>
      Module    im_file
      File    "C:\\inetpub\\logs\\LogFiles\\W3SVC1234\\u_ex*"
      SavePos  TRUE
    
      Exec if $raw_event =~ /^#/ drop();
      else
      {
        w3c->parse_csv();
        $EventTime = parsedate($date + " " + $time);
        $SourceName = "IIS";
        $raw_event = to_json();
      }
    </Input>
    

Then you can wire this input to any output you previously configured and logs will be sent as a nicely wrapped JSON object to anywhere you want.

iis log

V. How do I get additional value from my IIS logs?

Once your logs are nicely formated in a JSON format and shipped to a log management tool, you can start manipulating them and building log analytics. I’m sharing below some of the most common and handy use cases I’ve encountered – I hope they will inspire you.

1) IIS logs for enchanting your tech teams

latency

First, let’s focus on the attribute TimeTaken (time needed for your webserver to answer a client request). This attribute is logged by default with IIS, contrary to Apache or NGINX server logs. As one of my colleagues put it in his article:

So what insights can you extract thanks to TimeTaken?

  • You can rank all your requested URLs to see which one is the slowest.
  • url timetaken

  • You can quantize your different TimeTaken to have a global overview of what’s happening.
  • quantise time taken

  • You can see how some url query parameter can impact your code behavior:
  • query parameter

An analysis based on the Status code attribute can provide you with critical clues to understand what’s going on. Simply classifying your top Uri Stem hit answered by a 4xx or 5xx status will make it easier to decide on future developments.

  • Having 500 errors in your back-office could point out issues with your API. Some of the questions to check are: have you released recently? Or is one of your server down? Could you be under DDOS attack?
  • The 40x errors in the back-office may be showing your partner / user that they’re using a recently deprecated API. By analyzing your IIS access logs, seeing who is using your api and informing them to rectify it – will become a piece of cake. And it will enable you to prevent 40x errors.
  • Let’s detail the case of DDOS attacks further. The main thing to remember is that monitoring your client IP can be very useful. By getting the ranking of your most requested individual IP sources, you’ll find the IPs to ban in no time!

    correlate data

    116.31.116.36 is the most requesting url, but have a pretty regular shape, contrary to 108.31.135.118 which just tried to blast the website with a spike of requests

    I also want to point out the power of correlated analyses. As you can see in the image above, combining two analyses gives you way more information than looking at them separately:
    By combining all former analyses on the same dashboard you get a bird’s eye view over your IIS web server, its behavior and health.

    In the blink of an eye you can clearly see if everything is running smoothly or if something strange is happening.

    webintegration error

    2) IIS logs opening unexpected doors to Marketing teams

    Beyond pure technical monitoring, you can extend IIS log analysis to business metrics. Metrics such as web traffic, most requested URLs, top of referrers, relative repartition of IP addresses and other pieces of user related information such as user-agents.

    seo dashboard

    By combining the analyses suggested above, you are able to tell if your customers behave differently on iOS or Windows, on mobile or tablet. You will have a better understanding of how your customers interact with your IIS websites and how it technically impacts them. This spells out: great visibility for marketing.

    Discover more on access logs in general in one of our fellow articles, written by my colleague, Guillaume – The value that can be found in your access logs. If you haven’t read it yet, I would strongly advise you to take a look.

    Wrapping Up

    Don’t just leave your IIS logs stored in some unknown directory. Bring them to life & extract their value. And the best way of doing this is by configuring them into a neat JSON format and shipping them to a log analytics tool.

    Your tech teams can benefit from IIS logs but, and this may have come as a surprise, business teams too. So go get there and push for data-driven decisions for all – from DevOps, Devs, Sysadmins to Marketing, SEO and Product departments.

    And to become a pro of IIS logging analytics, make sure to trace correlations between analyses. They will unlock unforeseen insights, helping you understand your stack and what is going on with it much quicker.

    Related Posts