The Dangers of Online Benchmarking Reports
Senior managers want hard and fast ways to contextualize the performance of their organization’s digital program. And benchmarking performance reports provide an all too easy and tidy metric to track success and compare to other organizations. Whether it’s the M+R eBenchmark Report, the Convio Benchmark Index, or others, the rise of the interest in data, optimization and benchmarking seems to have everyone from online organizers up to executive directors asking ‘What’s Your Open Rate?’
Don’t get me wrong. These reports have great utility for tracking overall trends. And a culture of testing and optimizing online tactics to boost response rates is vital. However, industry benchmark rates are a poor measure of the overall success of an online organizing program.
Why would it be a bad idea to stack up against the competition? The danger of benchmarking reports is conflating high response rates with a high quality campaign. Good rates must mean a good campaigners and vice versa, right? Wrong!
Take email for example. There are a multitude of factors that impact the open, click, action and unsubscribe rate of an email that go far beyond the quality of the content, staffer or overall program.
Other factors that have statistically significant impacts on the overall performance rates of an email can include:
– Audience targeting: are you sending an email to your full file, people who have previously taken action or some other segment such as issue activists who have opted into a specific type of communication
– Type of Action: you should expect, and benchmark, different performance rates based on the type of ask in the email (online petition vs offline activity vs donation etc) and hopefully find some ranges of response rates to group different asks into tiers of performance, a sample grouping is below
– Email Arcs: messaging people a series of email asks all under the same campaign in a ‘ladder of engagement’ has also proven to be a statistically significant factor in email performance
A more detailed matrix of email performance could look something like this:
So given the wildly varying rates depending on the type of action, audience and sequencing of email…does an overall open rate for a program really tell you anything? When you compare the hundreds or thousands of asks you will make to segments of your list over the course of a year, can it really neatly stack up against another organization’s to make a meaningful comparison into one single metric? I posit that there is very little information that is useful by comparing your organization’s online performance to others in this broad a fashion.
Digital campaigns should be focusing on maximizing overall engagement as efficiently as possible to reach your campaign goals, not on achieving the highest response rates. Too much emphasis on maintaining high rates can even discourage your program from making harder, higher value asks or reaching out to broader segments of your list.
Head on over to part two of this blog post for some tips on how to measure your online campaigns effectively.