That's a pretty dramatic drop. I started going through all the scenarios of what might be happening. Then a couple days later I noticed this...

In other words, at the same time there was a drop in traffic from images.google.com, there was an uptick in traffic from www.google.com/imgres. This means Google is now serving Google Images content off www.google.com.

So I didn't do anything wrong, and my Google Images traffic is just fine... Now, if Google Analytics could only learn how to properly count Google Images traffic we'd be set.

It all started with discrepancies between Google Analytics and Unica's NetInsight - big discrepancies for one particular site where NetInsight was showing about 60% more visits and visitors. You always expect some variation between analytics packages - but not 60%.

We first noticed that the pages with the biggest discrepancies were those that were most likely to get Google Images traffic. Then I noticed that many of the referral sources were roughly similar, but Google Images was way out of whack.

Google Analytics Doesn't Report Pages Framed By Another Site

I had a theory - that Google Analytics wasn't counting pageviews if the page was framed by another site. So I did a test. I set up a Google Analytics profile and made the tags only fire if they were in a frameset. The code looked something like this:

if (top.location != location) {
var framesTracker = _gat._getTracker("UA-287XXX-4");
framesTracker._initData();
framesTracker._trackPageview();
}

The first day I thought my theory had been proven wrong - traffic was showing up in the frames profile, but then it quickly dropped off to zero.

In other words, the first day the profile existed framed pages were counted as page hits. The next day a filter kicked in and, with a few rare exceptions, framed pages never showed up again.

It's not just Google Images traffic that's not being reported, it's any site that frames pages on your site - so things like Yahoo! Image Search and MSN/Live Image Search are affected as well.

Why they would do this, I just don't understand. It's truly baffling. If your web server serves the page, then it should appear in the analytics for your site. Period. This means traffic from things like image and video search is grossly under reported in Google Analytics.

But it gets worse...

Google Reports Some Traffic From Google Images As 'Direct' Traffic

Let's take the example of Google Images (though it happens in other cases). You do a query and get a search results page with a bunch of thumbnails on it (example).

You then click on an image and and get a page that's displayed using a frameset (example).

In the top frame is Google's information on the image. In the bottom frame is the page from the site with the image.

Now, if you click on any link in the bottom frame (that's being served by the site with the image), it does not remove the frameset. The frameset stays in place. Technically, I understand where this is coming from, but it's horrible from a user interface perspective. But remember what we established above - no pages that are framed by another site get reported in Google Analytics. So the person could spend a half hour looking at your site this way and you would never know it. They could purchase something from you and it wouldn't register as an e-commerce transaction in Google Analytics. This is bad - very bad...

Realizing this, I decided to add target="_top" to all of the links on the site to ensure that if the person clicked on a link on our page the frameset would be removed.

That lead to some confusion the following days when we saw a spike in direct traffic and didn't understand where it was coming from...

After a while we realized it was coming from Google Images (or more precisely, from framed pages). Let me explain...

Because Google Analytics doesn't count the first framed page as traffic on your site (even though it's served by your site), that page just doesn't exist in Google Analytics. But it's that page that has Google Images as a referrer. So, by losing that page you lose the fact that Google Images sent you the traffic because the first page that Google Analytics counts is the one that the person went to when they clicked on the link on your framed page. Hence, the first initial referrer according to Google Analytics was the framed page on your site. That appears (in Google Analytics) as direct traffic (not a referral or organic traffic) since your own site can't be a referrer for your site.

Google's help pages are inaccurate when they say:

If frames on the site reside on different domains, the referral information is likely to be inaccurate, since one frame may be recorded as the referring source of another, instead of a previous site being recorded as the referring source.

It's more than the referral information being inaccurate - they're removing much of the traffic completely.

This does not mean you lose a record of all the traffic from Google Images. If the user clicks on a link on a page controlled by Google Images, for example the "remove frame" or "original context" links in the top frame, then since that action took place on a Google Images page, it will come through as a Google Images referral. But that's just a fraction of your actual Google Images traffic. If someone looks at the one framed page and leaves - they'll never show up in Google Analytics at all.

Almost Impossible To Get Search Keywords For Image Search

To make matters worse, even though Google Images is a search engine, it's pretty much impossible to treat it as an organic traffic source. Beyond the fact that you don't see all the traffic, the problem is one of URLs. Let me explain...

For starters it would be nice if we could just add something like this to the tracking code:

pageTracker._addOrganic("images.google.com","q");

In theory that's supposed to take any referral from images.google.com, find the 'q' query parameter and record the contents of that as the search keyword.

But the 'q' parameter is only found in the URL of the search results page. But the search results page is the referrer for the frameset that shows the page from your site at the bottom. The referrer for your page is the frameset, not the search results page. The frameset does, sorta, have the q parameter in it, but it's hidden inside the 'prev' parameter and looks something like this...

&prev=/images%3Fq%3Dharrier%2Bhound%26gbv%3D...

Notice the q%3Dharrier%2Bhound... That's not nearly as neat as it was on the search results page where it was &q=harrier+hound.

Now, some people have written some fairly complex routines to parse the q parameter out of that string, but it only works some of the time. When you consider the fact that the Google Images traffic you see is only a fraction of the real traffic it's questionable whether it's worth it to even bother since it's just bad data and you'll be mixing it with your good organic data from regular web search. Mixing good with bad makes everything bad.

It Doesn't Have To Be This Way

Google Analytics has painted themselves in a corner to an extent by not properly handling framed documents. If they decided to change how they handled framed documents it would significantly impact the analytics of many sites. But I'm not sure that a good excuse for continuing the way things are...

What should happen is that if a page's referrer is a frameset of a known search engine, then the query parameter should be pulled out of the referrer for the frameset (should be possible with Javascript). And it goes without saying that things like Google Images and Google Video should be known to be search engines (they are not at the moment).

You would think Google would at least want to properly reflect the traffic they drive to sites. Right now they're serverly underreporting traffic from Google Images and Google Video.

Hopefully this will get better in the future. I should do a test to see how IndexTools handles the situation since, in time, it will be Yahoo! free competitor to Google Analytics.

Segmentation

One possible way is by using segmentation. Basically segmentation lets you set a variable for each user that you can use to group different types of users. You might have a segmentation variable with values like "employee", "customer", and "general public".

You set the segmentation variable for Google Analytics by adding a line to your tracking code that looks something like:

pageTracker._setVar("segmentation value");  [using new GA tracking code]
__utmSetVar('segmentation value');          [using old GA tracking code]


The other tag-based analytics programs have a similar feature (often more advanced with the ability to set multiple variable values).

The theory behind this approach is that the bots may not actually execute the Javascript, they may see the tracking code on your page, recognize it as a tracking code, grab your Google Analytics ID and do a programmatic call to Google Analytics (as described by Peter van der Graff). In this case, if you've set a segmentation value, they most likely won't be programmed to detect that and as a result they'll show up on your reports as "not set".

That said, at least in some of the cases I've seen, the bots are able to set segmentation variables, which makes me think the bots use embedded browsers in the same manner as the screen capture program WebShots (which is controllable from the command line).

Segmentation + A Honey Pot

A better option is to use segmentation in addition to a honey pot. Traditionally, a honey pot is link a real user would never click on, which is excluded by robots.txt, so no well-behaved spider would click on it either. Only bad bots load honey pot pages. In this case robots.txt exclusion is not necessary, since no "good bot" currently executes Javascript-based analytics tags, and if a good bot ever did do that you could tell it was a good bot by how it identifies itself in it's 'user-agent' string.

Essentially, you're setting up a page and hoping the bad bot goes to the page. When the bad bot goes to the page you set the segmentation variable to something like "bad bot" and you can then see the bad bot in your analytics reports. Because you're using a segmentation variable, just about any report can be broken down by the presence of this value.

You'll want to make sure you don't then overwrite the segmentation variable. If you use a honey pot you may want to only set the segmentation variable on the honey pot page, that way you won't wipe out the value on a subsequent page. Or at least test the cookie with the segmentation value in it before setting other segmentation values.

This solution is far from perfect. For starters, the bad bot may not choose to go to your honey pot page. Some may and some may not. These bad bots are not spiders - they're not trying to find pages on your site. They're programmed to act like humans, so they'll just click on a few links and leave. However, if you see "bad bot" in your segmentation values, it will tell you you may have bigger problems.

A Honey Pot Without Segmentation

You can also use the honey pot concept without segmentation. In this case you'd simply look for the honey pot page in your content reports and see examine entrance paths to the page to determine who's sending you bad traffic.

This may be sufficient. While segmentation variables can be used on many different types of reports, because the segmentation variable won't be set every time a bad bot crawls your site, they may not be as helpful as you might think. The general knowledge that there are bad bots crawling your site and their entrance paths may be as good as it gets.

But do realize that just because a bot says it's coming from xyz.com, doesn't mean it's actually come from xyz.com - that can be faked too. The particulars of your situation will determine how best to interpret what you see in the data.

Segmentation + Honey Pot, Take 2 (best option)

Using both of the approaches above separately is probably the best solution.

To find bad bots that are executing the analytics tags programmatically, set a segmentation variable for all users (bad bots or not), and use the lack of a segmentation value to identify programmatic bad bots. Then you use path analysis on a honey pot to find the bad bots that are executing the Javascript with an embedded browser.

That gives you coverage on both types of possible bad bot implementations.

Wrap Up

Let me reiterate that there's no way to catch every bad bot - at least I can't think of a way. But if you use some of the strategies I've mentioned here and see that bad bots are crawling your site and affecting your analytics, then you're ahead of the game. The worst case scenario is that bad bots cause you to make bad decisions by changing your data and leading you to false conclusions. If you know your data have been affected, you're more likely to think twice and not rely on faulty data to make decisions.

And also remember, as I've discussed in a previous post, there are ways to use segmentation to set up bot-free zones in your web analytics. Segmentation on things like "Paying customers" or "registered users" can all be relatively immune to bot traffic.

Good luck!

Insuring that pages are built correctly

At it's heart, one of the core practices of SEO is to ensure that pages are built properly - that their focus and content can be easily understood by search engine spiders. "Web designers" just want pages to look pretty. "Web programmers" just want pages to be connected to data sources in cool ways or do some other cool trick. Neither of those groups really cares about how the pages perform. SEO will always need to be there to make sure the page can be understood by search engines. With the advent of botnet web analytics corruption, it may be difficult for an SEOer to prove s/he's done her/his job, but it's not impossible...

E-commerce

Sites that have e-commerce as their primary goal will be far less affected than "marketing" sites trying to get market exposure because e-commerce transactions will be difficult to fake. Conversion ratios may be corrupted, but the actual sales (in dollars, pounds, euros, etc.) will still be a concrete measure of success. Further, segmentation by things like "paying customer" will become invaluable.

Registered users

Building on segmentation possible with e-commerce, segmentation based on whether the user is registered will be invaluable as well. Using segmentation, you can look just at the activity of registered users which should be bot free provided you use a good captcha, and possibly e-mail confirmation. (At the very least it will take a very targeted attack to mess with your user segmentation data - it won't be affected by random attacks).

Ranking in SERPs

Your actual ranking in the SERPs is also something that can't be faked. Programs like WebPosition, while hated by the search engines, will be critical in measuring success and validating organic traffic. [If you're getting traffic off the keyword 'art', but don't rank for 'art', then you know you have a bot problem.]

The cost of good analytics

The analytics company that works the hardest to combat botnets will be the big winner here. Since Google Analytics, Microsoft Gatineau, and now IndexTools (Yahoo!), are free packages it's unclear whether Google, Microsoft or Yahoo! really have the incentive to combat the effects of botnets. I mean, look at Google's track record with click fraud in AdWords. Even if you accept the fact that they try to stop it, there's still plenty of it that goes undetected. If the tool that is best able to effectively detect and squash botnets is a paid service, there will be a very real increase in the cost of quality analytics.

It will also be more expensive to interpret analytics since you can't just take the numbers at face value. You'll have to jump through more hoops to come to a conclusion and it's not likely to be able to be done by people without significant experience. Those experts will come at a price.

I can also see 3rd party verification services cropping up that may not engage in SEO, but validate the results of SEO companies. Again, an added cost, though probably only something that would be done for very high-end projects.

Conclusion

Botnets aren't going to destroy the SEO industry. If anything they'll increase the level of professionalism. Word will get out that fraud and deception are possible with SEO and reputation will become absolutely critical. I'm guessing it will knock some smaller players (both customers and providers) out of the game - or at least knock them down to a place where they don't even try to validate results. But in cases where money talks, money will be able to buy at least "decent" analytics and the people who are best qualified to interpret the data.

Still, analytics hacking by botnets will become a much bigger problem in the future, and it will radically change the SEO industry...

It would appear that your search engine traffic has gone down by 2.76%. But that's not actually the case at all. If you drill into search engines you see the following....

Organic traffic has actually gone up 5.7% even though the first graphic said it had gone down 2.76%. The reason is simple - search engine traffic went down relative to the other types of traffic. In other words, direct traffic went up more than 5.7%. The first graphic is the product of a zero sum game - if one area increases, the others have to decrease, even though, on their own they're increasing.

Let's put it another way. If direct traffic and referring sites had gone up 5.7% as well, then the percentages in the first graphic would all have been zero even though all of them went up 5.7%.

The take away is that when interpreting web analytics stats, be careful in your interpretations. Digging deeper can clarify initial misconceptions.

That's both good and bad... One of the wonderful things about IndexTools was how responsive they were - they'd do all sorts of customization/integration for you at a reasonable price. I'm hoping that's not going to go away, but I'm afraid it will... On the other hand they were the analytics tool I really wanted to work with becaue,  like Google Analytics, they just seemed to "get" the issues and have a sensible, but powerful interface. I've had some exposure now to both WebTrends and Unica's NetInsight and can't say the same for either of them.

The bottom line is that it's possible we'll see another free competitor to Google Analytics that has many of the same qualities of Google Analytics in that it has a clear, no-nonsense, to-the-point interface. That's quite exciting.

It is curious timing however. In the middle of a possible merger between Yahoo! and Microsoft, Yahoo! goes and buys an analytics package when Microsoft is already working on Gatineau. What would be horrible is to see a merger where IndexTools gets shut down completely in favor of Gatineau. Let's hope that never happens...

On sites that have the typical business week roller coaster appearance when you look at a lot of data, say over a year, it can be difficult to see trends. For example here's a busy e-commerce site:

You can generally get a feel for what's going on in that graph, but it's much clearer in the graph below which has one data point for each week. (It looks like there were some problems in December)...

Yes, you lose some detail, but when looking at a lot of data you're able to better see the forest rather than the trees. The roller coaster trends you see in analytics tend to be weekly trends with traffic high during the work week and low on the weekend. So a weekly graph smooths out those normal ups and downs and lets you see very clearly how you did on a particular week.

When you get to monthly summary graphs the trends aren't quite as detailed...

I suppose that is useful for businesses who think in terms of performance month-to-month, but thinking that way will hide downturns that are spread over two months and exaggerate ones that are contained completely within a single month. In other words, unlike the weekly graph, it's not smoothing out routine variations - it's just removing details (that may be important).

The weekly and monthly graphs are useful even on sites which don't see routine weekly variation. Here are the graphs from a blog site that has pretty consistent traffic (weekends are about the same as weekdays). We start with the standard daily graph...

That's a much smoother graph than its counter part from the e-commerce site above (the first graph), but this next graph (the weekly graph) makes the trends very clear without losing much detail...

Even though you can see the trends in the daily graph, that weekly graph really does a much better job of smoothing the line out and showing the general trend, though it does hide the very last spike in traffic.

The monthly graph however, once again smooths things out so much that it loses important details...

According to that graph the growth rate of the site is nearly exponential. It loses the recent spikes and makes thing seem a little too optimistic. The accurate message from the data should be "growth is good, but if spikes in traffic aren't maintained traffic may go down somewhat". The monthly graph makes it seem like growth into the future is unlimited.

Theh bottom line is that Google has added another useful tool, but as with all statistical tools you need to be careful that the conclusions drawn from these tools don't hide important trends in the data. In this particular case I believe the weekly graphs are a huge asset, but the monthly graphs aggregate too much data to accurately describe the underlying trends. Of course, as you get years worth of data accumulated, the monthly graphs will be more appropriate to show long-term tending.

There's nothing on the default dashboard that will tell you if Google is done processing data for the previous day. However, it's not hard to figure out... Click on "Visitors" in the nav bar on the left side of the screen. You'll see an overview of visitor data that will include a section that looks something like the one on the right...

If the graphs for Absolute Unique Visitors and New Visits drop to zero on the last day, then Google has not done end-of-day processing, which means it's likely that all of the data are not in and your numbers for the previous day may change.  The screen grab to the right was taken at 11:30am the following day - so it can take Google quite a while to actually finalize the numbers for the previous day.

This had me stumped until I noticed that the number of new visits was flatlined at 0%.

In other words, Google's referral data is based on either the first site a person came from or the most recent site. I'm guessing its the first site since that's a fair number of visits to still be having months after the site is shut down. You'd think they'd find the site some other way and if Google showed the most recent referrer the original referrer would be wiped out by now.

If you look at it from just before the site was shut down you can see that there was a fairly precipitous drop, but then the real converts have hung in there (and not cleared their cookies)...

So if you have links from sites that no longer exist - congratulations - it means you have some real converts to your site...

Having done a little research it seems the major problems started around 6pm (Eastern?) on Saturday the 28th , which means they're almost a full 2 days into the outage at this point.

Google finally put out an announcement telling people about the outage. It took Google nearly a day and a half to say anything about it, which is unfortunate - they usually do better. The announcement said that data from Saturday should be fine now, but that's not what I'm seeing at all - there's just as much missing as there always was.

The weird part is that a trickle of visits did manage to get through. But the really odd part is that it's not like one data center was up - the hits that got through are distributed all around the world, and there's a mix of organic, direct and referrals... Go figure.

I'd love to be a fly on the wall in their office right now, and would hate to be them...

Hopefully they'll get it fixed quickly.

When you pay for analytics with services such as WebTrends and IndexTools you sign contracts and the data clearly belongs to you. With Google Analytics, since it's free the trade off is that Google gets to do pretty much what they want with your data, within the confines of their privacy policy. The same will most likely be true of Microsoft.

The thing is, Google seems to always defend user data from the prying eyes of governments. They've got a good track record. I don't think of Microsoft in the same way. Google may be the new Big Brother, but they're doing it in a very different way than Microsoft was Big Brother in the past - Google just seems more responsible about the obligations that come with power.

Analytics data is pretty sensitive information. I'm just not sure Microsoft will guard my data as vigorously as Google would. Not that I'm intending to start up any terrorist web sites, but it's more about the principle of it...

But what's happening now is far worse. From my estimations there's a huge delay and no notice posted on the Google Analytics blog. Here's some evidence from a variety of sites...

Slicksurface.com:

AnimalHavenShelter.org:

A consumer health site:

The personal blog of a someone I know:

Another personal blog:

These aren't sites with insignificant traffic and they're about as broad a spectrum of sites as you get and they're all showing exactly the same trend - no data since 2 or 3am on the 29th, and substantial under reporting starting at the morning of the 28th for some sites and the evening of the 28th for other sites.

The last day shown in each of those graphs is yesterday - 7/29. Which means they were down all day yesterday, a substantial portion of 7/28, and so far today...

What bothers me the most is the lack of communication on the part of Google. Hopefully they're still collecting data and it's just a matter of time before it all shows up again in the reports.

UPDATE (1pm ET): Google has posted something on their blog acknowledging the problem and saying it's just a reporting delay... Let's hope.