Show Me The Numbers! [Why I never trust a calculation]

Marketers love data.

Data, data, data. Big data. Little data. Internal data. Platform date. Web site data. Sales data. Conversion data. You name it data.

Our fascination with data is bordering on the obsessive.

I’m all for making wise business decisions based on solid data, but show me the numbers!  What I’m categorically against is holding data, any data, up as some sort of false idol to be worshiped  at the “altar of numbers” simply because data involves math. And yet, I see it all the time.  Here’s why I never trust a calculation.

Show Me The Numbers

Making a decision on bad data, or worse, on bad interpretation of good data has the potential to harm your business far worse than making a decision on no data at all. Why? Because you have the data on your side – or at least you think you do – and the fact your decision has been “data validated” is likely to prevent you from seeing the truth of the matter. You may very well throw good money after bad in an attempt to prove the data right.

Marketers can (and do) make many types of mistakes when working with and making decision based on data. Today I’m here to encourage you to make one change in your data consumption habits.  I ask you to join me in never trusting a calculation. If you can’t see it, count it or measure it, then DO NOT TRUST IT! Validate!

For example, as a digital marketer and web strategist, I encounter “conversion rate”  ever day.  The conversion rate of a web site is the total number of web site visitors divided by the number of visitors who took action during their visit. Depending on our web site and how you define success, that action might be a purchase, a sign up for an email list, or any number of other desirable outcomes. It depends on your business and on your web site.

Whenever I see a conversion rate, my reaction is always the same. I want to see the underlying data.

Why?

Let’s look at just a few examples to illustrate the danger of looking at a calculation.

Show Me The Numbers - Chart

Clearly, site #2 has a “strong” conversion rate, relative to the other three web sites. But that doesn’t tell you much without knowing the underlying data related to number of visits, total number of purchases and sales related to those purchases.

Now, if you could just raise the $ per purchase, then you’d be in great shape, right? True, but that’s yet another calculated number and not raw data. It’s obviously calculated by dividing the total sales by the number of transactions.

Right about now, you’re probably scoffing, because this is so incredibly obvious. “Duh. I didn’t get my MBA to have a lawyer teach me basic math.” I can hear you saying it from here.  And yet, I bet you a dime to a dollar that some executive in your business is relying on a calculation or three in some business dashboard or another without paying too much attention to the real data underlying that calculation. I saw it regularly in my prior life as a corporate drone.

Conversion rate is a great thing improve. But, hypothetically, it’s possible for me to double your conversion rate by halving the number of visitors to your web (better targeting?) while keeping your total sales the same. Have I REALLY moved the dial?

Obviously, this isn’t the outcome you’re looking for if someone states that can double your conversion rate.

My advice is this: never trust a calculation. Hone your professional skeptic reflexes. Rely on the underlying data as a decision making mechanism first and get to the calculations second. Or at the very least, keep a wary eye on those calculations, and if something doesn’t pass your sniff test, delve deeper.

Show Me The Numbers!! Do you trust calculations? Are they crutches that lead businesses to inaccurate decisions and bad strategy? Join the debate, share your thoughts below.

Image Credit: Jorge Franganillo, Licensed via Creative Commons

Join the Conversation

9 comments
Sort: Newest | Oldest
Steve Birkett
Steve Birkett like.author.displayName 1 Like

I  agree, Sean! To a point... there's always a caveat ;-)

Understanding derivation is always key to providing meaningful numbers, but it's also true that too many deep dives into the underlying data can paralyze decision makers. As the person providing these numbers - both in the corporate world for clients of a large logistics firm and now for SMBs seeking to understand their online marketing efforts - I've seen both sides of the coin. I always encouraged a walk through of the data before steering meetings, but not everyone is interested in doing that due diligence. In that case it comes down to trusting those who provide the headline numbers, which can easily lead to the type of situation you  highlight in the end of your piece. If they have a vested interest in tweaking the metrics to show them in a good light, all the more so. 

On the other side, too much poking around in the raw data by those who should be focused on the bigger picture has sometimes led to micro-management and running in ever-decreasing circles, chasing that elusive perfect measure. I found that the happy medium was a comprehensive run through of the underlying dashboard data in one or two early sessions, giving the key people understanding of (and confidence in) the headline metrics presented at future reviews.   

The devil is definitely in the detail, just don't let him catch you loitering too long in his turf!

Sean McGinnis
Sean McGinnis like.author.displayName 1 Like

@Steve Birkett I see your point Steve, and I would agree. The devil, is indeed, in the details.

Measure what matters, but be clear about what does, in fact, matter. Do you have any tips for how to define what matters?

Steve Birkett
Steve Birkett

@Sean McGinnis As trite as it may sound, I think you begin at the end. Define the key objectives by words and outcomes first, then work backwards to create the metrics that best display progress towards those objectives. It's FAR easier to say than to do, and there will be frequent iterations and adjustments to the measurement. But the time spent will more than pay off in terms of understanding the number from the outset, rather than constantly turning the wheels with a metric that was ill conceived.

chieflemonhead
chieflemonhead like.author.displayName like.author.displayName 2 Like

Way to go, Sean!


So many rely on these metrics: conversion, total traffic, etc. And the bigger the number, the happier they are. But, as you so eloquently put it, the numbers in the background do matter. With the proper background numbers, you can, not only see these basic metrics, but you can start to truly garner insights and analysis by considering the value of each individual customer and the total profit generated on a program. (Yes @samfiorella, I did say "profit".)

I'd love to see more people develop in "insights" over data. It's a journey... :)


Thanks again Sean.

Judi

Sean McGinnis
Sean McGinnis like.author.displayName 1 Like

@chieflemonhead Sure thing Judy. It's something that's always bothered me. I think these shortcuts and dashboards have value, but they can easily mask a landmine as well if SOMEONE isn't looking at the underlying data and throwing red flags if something smalls funny. :)

samfiorella
samfiorella moderator like.author.displayName 1 Like

Great discussion starter Sean. Is it really the calculations that are the problem? Or how they're used?

Seems to me the real issue is the continuing need of marketers and other business "leaders" to demand quick-fix reports and analysis that they can make even quicker  decisions with. Frankly, the ame problem exists with social influence calculations by Klout, measurement by social media "Likes" and so on.  You can crunch numbers all you want, make them look sexy and meaningful .... but without context and in isolation, they're meaningless to strategic planning. 

Latest blog post: Show Me The Numbers – Chart

Sean McGinnis
Sean McGinnis

@samfiorella Agreed Sam. As we've discussed in the past, context is everything - in SEO, in influence and certainly in analytics. Numbers in a vacuum are just that. It's only by applying our professional judgment and experience that they become meaningful. I can give you a great example from personal experience.

When I launched a group blog many moons ago, we grew readership to over 150k unique visitors within about a month. Compared to other sites I was familiar with, that represented a very strong growth rate. But we capped out at that and were not able to continue growing. It turns out, there's not a whole lot of money to be made at that readership level, even though that sounds like a pretty strong number of unique visitors. 

Only after about 6-7 months did I have the ability to put those numbers into context  and consequently know that there wasn't a business model that warranted the attention I was giving that project.

Trackbacks

  1. […] post Show Me The Numbers! [Why I never trust a calculation] appeared first on Sensei […]