FAQ

Here you will find some links to answers and discussions about common questions and other things that may prove useful, educational, amusing, curious, or whatever about matters of response times, performance, site integrity ... .

Measuring Availability and Performance

Are you always able to sample every 30 minutes?

While we would like to be able to say yes, the real answer is that we do everything we can to sample every 30 minutes, on the hour and half hour. However, occasionally it is not possible to completely sample all the URLs we monitor in 30 minutes.

There are several reasons for this, most quite beyond anyone's control. For all its wonders, the Internet is not consistent. Sometimes it is fast and at others, for no readily apparent reason, slow. Sometimes things happen to the backbone connections. A connection will be lost and significant traffic must be rerouted over other connections overloading them and slowing down the Internet. Sometimes things happen to the routers at major Internet hubs and packets get lost resulting in a slowdown. One thing we've learned is that when it comes to the Internet, "stuff happens."

At the end of the day, the Internet is like most other utilities. While it is generally very reliable, occasionally stuff happens and it misbehaves. When it does, we sometimes get caught in the backwash and are unable to sample.

What do you do when you can not sample every 30 minutes?

When this happens, we complete the current sample pass and then decide what to do next. If we are only over the next 30 minute mark a little bit, we start the next sample pass right away. But if we have slipped "to much" we wait until the next 30 minute hack before starting the next sample pass.

This approach allows us to maintain the time coherency needed for our analysis reports.

When do "skips" happen?

Typically, "skips" happen at night between around midnight and 6 AM. This is a common time for maintenance of the Internet. That maintenance often disrupts regular operations. Additionally, many firms engage in bulk data transfers during these hours creating a significant load on the Internet.

What's a timeout?

Timeouts occur under one of two conditions. In the first the URL returns a "Page not found" error. In the second, the URL does not completely download in 30 seconds. This includes all content on the page (images, advertisements, etc.).

Can you measure transaction times?

The answer is a qualified yes. We are offering transaction measurement services to our Monitor³ and Monitor⁴ customers. While these measurements are customized to the needs of specific customers, as a rule we are measuring the following:

Because each transaction time situation is unique, they require special Monitor handling and special report analysis. As a consequence, transaction time measurements are quoted on request.

What other sorts of things have you measured?

Lots! Seriously, if you have a site measurement concern, give us a call and we can talk about it.

How do you measure availability and performance?

We use a variety of tools to measure availability and performance in our consulting work. You can find out more about them here. In our production systems we use a combination of techniques.

Our preferred technique is to use com automation of Internet Explorer. Using some proprietary techniques we developed, we are able to accurately measure page load times at the millisecond level. We have seen a few pages where this approach does not work acceptably. When we encounter such situations we first augment IE com automation with window monitoring. To date that has always worked. In the event that it ever fails we also have available frame "onload" and straight window monitoring.

URLs

What kinds of URLs can I monitor?

As part of the basic Monitor services you can monitor any URL that will respond to a constant HTTP address. Things like:

As a practical matter this is just about all pages that are not transactional. You can find out about transactional pages here.

Are there any URLs you can not monitor?

The flip, but nonetheless correct answer is, so far we can monitor any URL. We've yet to encounter a URL we can not monitor but, of course, we have yet to monitor every URL on every financial industry site. At present we are monitoring URLs on 203 banking Web sites successfully. You can find a list of many of the sites we have monitored here.

If you have any concerns in this area, provide us with the URLs and we'll verify that we can monitor them. You may want to look at Measuring Availability and Performance.

What URLs have you worked with?

We have worked with a very large number of URLs on many financial institution sites. Indeed, as part of our benchmarking activities we are systematically scanning every public URL on every bank and credit union site in the United States. While this may seem like something of a daunting proposition, our advanced technology actually makes it relatively straight forward. You can see some of the institutions we have monitored here.

Are you really measuring every public page on every financial industry web site?

Yes. One sure way to know how the industry as a whole is performing is to measure it. While the number of pages is large, it is entirely within the bounds of what we can do with our Monitor platform. We will be presenting the results to our customers when the scan is complete.

How many URLs should I monitor?

We are often asked about how many URLs should be monitored. The flip, but correct, answer is that it depends on what you are trying to do.

As general guidance, we suggest that institutions track at least those URLs served by different Internet servers or supported by different back end systems. For example, an institution might have its main site hosted on a hosting service, its Internet banking site hosted by another, its check reorder feature outsourced to a partner, and its credit card products outsourced to another partner. We would suggest that this institution monitor a URL on each of these subsystems.

Notwithstanding this guidance, there are many good reasons for monitoring more. If you are concerned, give us a call and we will help work out what's right for you.

What partner URLs should I monitor?

The short answer is the ones you care about. As a practical matter that includes all that provide significant site content or functionality. A small financial institution will have a dozen or so partners. A large institution may have hundreds. As a rule we suggest that you monitor business partners if

What competitor URLs should I monitor?

The flip, but nonetheless correct, answer is the ones you care about. As a practical matter we find that most of our customers monitor a combination of geographic competitors and larger banks. Some of our customers will change the competitors they monitor over time in response to the evolving competitive environment.

Reports

I don't understand the detailed analysis? How do I use it to help me?

The detailed sections of the report are, well, detailed. They may appear somewhat intimidating if you are not familiar with data analysis or performance monitoring. But do not despair. We will be glad to go over the report with you, explain each of the sections and answer your questions.

When you are looking at the report, bear in mind that each of the analyzes answer's a different question about availability and performance. Some are there to provide a broad overall perspective. Others drill into specific measurements.

Grades and ranks

What's the difference between grades and rankings?

The reports contain both URL grades and URL rankings. These are provided both for the URL as a whole, and for individual analysis measurements.

Grades
Grades provide an absolute measure of availability and performance. They are independent of the performance and availability of other URLs at other financial service institutions. Grades are based on a benchmark analysis of over 200 URLs that reflects both user response to performance and what is technically possible. A grade of 100 is perfect.

Rankings
Ranks provide a relative measure of availability and performance during the reporting interval. They measure how a URL compares to other URLs independent of whether or not the actual behavior is good or bad. They are useful in understanding how a given URL performs relative to all other URLs. Rankings are determined by analyzing the performance of over 200 URLs during the report interval. The best URL is given a ranking of 100 and the worst a ranking of 0.

Why are grades and ranks different?

Grade and rank can (and for some analysis metrics often do) vary significantly. A URLs may receive a fair grade and yet rank poorly. This occurs when the URL is performing well in an absolute sense, yet because there are many other URLs performing even better, it's relative ranking is significantly poorer. Since grades and ranks are fundamentally different measurements of "goodness" they often vary with the grade being better than the rank or the rank being better than the grade.

How are overall grades and ranks determined?

Statistical grades and ranks
The overall statistical grades and ranks are based on the weighted sum of the URL's individual grades and ranks in the following areas: average response time, the standard deviation of the response time, median response time, maximum response time, minimum response time, number of timeouts, and response time distribution.

While the weighting function is proprietary, it has the following general characteristics:

Distribution grades and ranks
Distribution grades and ranks are based on the weighted sum of the URL's response time distribution. The weighting function is proprietary. However, it weights faster response times better than slower response times. Timeouts have a significant negative weight.

How are the grade and rank weighting functions determined?

Grade weights are largely driven by human factor studies. Numerous such studies confirm that the average user is generally satisfied with a 2 second response time, not significantly more satisfied with a sub 1 second response time, becomes increasingly dissatisfied with response times greater than 6 seconds, and is highly dissatisfied with response times above 15 seconds. Weights reflect a variety of such studies.

Rank weights are largely driven by an analysis of over 500,000 tests to determine best practice and common practice.

We review the weighting functions from time to time to insure that they track improvements in the performance of the URLs we monitor.

Competitor URLs

Analysis measurements

Why aren't averages, standard deviations, ... a good measure of the sites performance?

Averages, standard deviations, etc. are overused and over abused statistical measurements. The measures are very useful when the underlying distribution is well behaved. But, when it is not, as is far more often the case than generally realized, many statistical measurements are worse than useless. Not only do they not mean much of anything, they mislead the unwary to false conclusions. You can find an interesting discussion of the general problem at http://www.topquark.co.uk/articles/LoA.html.

Among the problems we have seen in our testing are sites with two or more peaks in the distribution of their response times. Averages hide this kind of behavior so you do not know that the site is actually a high performer much of the time but occasionally something is going wrong and the site appears average. In this case the large number of slow responses clearly pointed to a problem with one of the site's content servers.

Because of this kind of problem, we do distribution analysis in addition to making the usual statistical measures.

What is the response time distribution?

What are the response time distribution cells?

How are they chosen?

The response distribution is created by counting the number of responses in time cells. Depending on our analysis purposes we use times ranging from 0.1 seconds to 2 seconds. Some times the cell time width varies. Here is what a 0.1 second distribution to 200 plus financial institution URLs looks like.

What's a good response time?

What's a poor response time?

Beauty may be in the eye of the beholder, but human factor studies indicate the following:

Time	Comment
0 - 2 seconds	Response times under 2 seconds are very good and highly valued by users.
2 - 4 seconds	Users generally accept response times below 4 seconds.
> 4	Above 4 seconds, users generally become increasingly dissatisfied.
> 10 seconds	Few users will tolerate response times above 10 seconds without a compelling reason.
> 20 seconds	Even with a compelling reason, almost no user will tolerate times above 20 seconds.
> 60 seconds	For all practical purposes, the site is unusable.

What does a good response time distribution look like?

What does a bad one look like?

Response time distributions generally follow one of the patterns shown below. The general pattern is a skewed "bell shaped curve". Better URLs are skewed more strongly to the faster response times on the left. As the "skew" moves to the slower response times on the right, the site is performing progressively poorer. Very poor URLs have a generally scattered distribution with a poorly defined maximum. Response time distributions

What's the 90% response time?

A URL can be characterized by the time at which it responds to 90% or more of URL requests. Better URLs have 90% response times under 4 seconds. As the 90% response time increases, the URLs performance becomes increasingly poorer. A useful way to display the 90% response time is to graph the cumulative percent responses versus response time (below). This figure shows that Fifth Third, one of the faster banks on the Internet, has a 90% response time under 4 seconds. Conversely, Citibank has a 90% response time of 15 seconds (very poor). Wachovia falls in between (but, is not especially fast).