I asked myself this question the first time I saw a survey invitation with the following warning:
Please note, this survey contains media that is not compatible with Firefox Internet Browser
The invitation continued with instructions to copy and paste the link into an Internet Explorer window if Firefox is my default browser.
Let’s look at this in more detail. To dispose of the title question first, the only obvious logical reason why someone fielding online surveys wouldn’t provide support for Firefox users would be if they were surveying people who don’t use it. Perhaps even that isn’t exactly logical, but at least it’s a reasonable excuse. If you are creating something that requires significant development effort, and you are screening for Internet Explorer users, why bother with Firefox?
Unfortunately, that theory doesn’t fit the situation. I’ve seen invitations with this warning for over a year, covering Consumer Package Goods and Retail Stores. I have yet to come up with a good reason, and the research company hasn’t offered me one.
But why is it such a bad idea?
First, Sample Bias. Systematically excluding a segment of the overall population you want to survey is generally a bad practice. It is easy to gather results that are biased, for reasons that may be obvious or less so.
Remember the days of telephone surveys? (I know, we are still collecting data via the telephone, but many people are only familiar with online surveys.) Best practices include calling at random times of the day and night, and also letting the phone ring for quite a while. Why? To increase the chances of the respondent being a person who works, and also to increase the coverage of people who might be elderly or infirm – and who might take longer to reach the phone. Without these measures, you might end up with a disproportionately large number of fit stay-at-home respondents. Some corrections could be done with weighting, but this adds unnecessary complexity versus just improving the representivity of the sample in the first place.
In the case of eliminating surveys from Firefox users, it would probably be a good idea to understand the potential impact through browser share numbers. Unfortunately, this isn’t quite as easy as it might seem, which is probably why we see percentages ranging from 14% to several times higher for Firefox usage in the US. These differences are caused by data collection methodologies and also browser behavior, but this article isn’t about browser share so let’s just settle on an approximation of 20% user share for Firefox. So these surveys are systematically excluding about one fifth of the US population. I could easily come up with some imagined differences between Firefox users and users of other browsers, but fortunately there is some real research out there. comScore reported in 2007 on a study that looked into the differences between Firefox and Internet Explorer users. The results showed that Firefox users were more likely to be younger, higher income, and male than the average Internet user. Would this impact a project covering food items in the grocery store? You bet. comScore’s study also showed that Firefox users are more likely to have a broadband connection and that their site visitation profile varied from the average – which could impact advertising placement and content.
The other impact for concern, although probably a lesser concern in this case than sample bias, is that of Lower Response Rates. Without hard evidence we can only speculate on the impact, but it seems likely that some people who receive an invitation excluding Firefox might decide not to participate even though they could do so fairly easily by starting Internet Explorer and pasting the link. The additional steps involved are a deterrent. Unfortunately, these particular surveys don’t even work by changing to Internet Explorer rendering within Firefox (something that is common practice to allow usage of sites that are not standards compliant). Longer term, continued invitations that are less easy to use may result in more people leaving the panel.
In conclusion, make your surveys sample as representative as possible, and don’t do anything in the invitation or survey to turn people off.
One last note on this subject. The problem invitations specifically state that the surveys don’t work with Firefox. Even if Firefox is the only excluded browser, it represents over 20% of the overall market as of Dec 2008 according to Net Applications. It probably doesn’t make sense to invest in development for older browsers, but as Safari (7.9%) and Chrome (1%) usage grows the challenges for survey developers are going to increase. Overall, browsers other than Internet Explorer are currently about one-third of total usage.
Idiosyncratically,
Mike Pritchard
5 Circles Research