Have we lost sight of what effectiveness really means?

We recently published an article about System1's 'Twelve Ads of Christmas'. The creative effectiveness platform released its top 12 Christmas ads of 2023, based on viewers’ emotional response data captured via its Test Your Ad platform. To rank the ads that gained the maximum rating (5.9-Star) System1 used a secondary measure, Spike, which predicts short-term sales impact.

But the System1 report garnered a mixed response, with some questioning whether a focus on emotion-led creative effectiveness was distracting from the job at hand of driving business growth - though of course creatively effective ads have a better chance of doing exactly that. Are we bringing effectiveness down to a click-bait headline metric removed from the realities of ROI and the genuine impact that marketers need to achieve? Or should we embrace quick dipstick measures as a way of galvanising interest and broadening acceptance of what effectiveness can be?

We asked the industry to weigh in.

Jonathan Brown, Strategy Partner, McCann London

We’re very comfortable with the evidence-led approach to scoring the predictive long-term Aldi brand impact and short-term sales impact that System1 claims.

It’s a highly robust model that’s linked to the IPA effectiveness database. But more than that, having used this methodology for a number of years, and having seen Kevin’s seismic commercial payback we also trust the results and their predictive impact. For instance, our Sustained Success IPA paper showed that Kevin generated incremental revenue of £618 million and a Return on Marketing Investment of 241% over the course of 6 years.

System1 scores are a heuristic for creative effectiveness, but they are not a replacement for the forensic need for effectiveness modelling. I’d argue System1’s scores are a victim of their own simplicity. Reducing thousands of hours of work into a single-star rating is anathema to our industry. But what I love about them is that they provoke a discussion about what works and not just what we like as an industry.

Sandie Dilger, Chief Strategy Officer, TBWA

Any talk of effectiveness in our industry should be connected to commercial performance in order for us to gain any credibility as partners who care about our clients’ business..

The actual definition of effectiveness doesn’t make this distinction so things like interest and engagement could technically be labelled as effective but that makes the dangerous assumption that an advert only sets out to achieve those things and last time I checked, they don’t tend to get mentioned in an annual report. To hail anything that has appeared in the world a few days before as ‘effective’ undermines the credibility of the word, the industry, and our primary reason for being: ‘we sell or else’. Declaring who ‘won Christmas’ is most definitely something for the more frugal, data-driven people of January-land and most definitely not the preserves of the instant gratification, image-conscious people of December-ville.

Dom Randle, Director, Running Total (Total Media Manchester Office)

I must admit that when I see the headlines about the most effective ads or campaign then I do roll my eyes somewhat, especially when the headlines started in November! I think it is slightly misleading to position the report in this way because for me effectiveness should be talked about in driving against business outcomes over a longer period. So I would recommend some caution in the way things are presented. If you look at the latest top 12 ads, 6 of them are supermarkets, and so although it's great that we have a view of their strength, we can’t believe that all those businesses will meet their goals this Christmas and beyond, which is where effectiveness terminology should be used.

However, acknowledging that creative content holds the greatest influence on campaign effectiveness, System 1's approach essentially predicts this effectiveness. Testing creative content before allocating substantial budgets to media distribution can significantly enhance real effectiveness for numerous brand campaigns. However, it's crucial to clarify what exactly we're measuring to avoid associating effectiveness solely with short-term metrics or using it solely for single-variable comparisons.

Tom Sussman, Deputy Chief Strategy Officer, Leo Burnett

Ah, another day, another storm in ad-land’s already tempestuous tea cup.

The worry seems to be that copy-testing companies deeming advertising “effective” (prior to in-market evidence) might somehow soften our definition of true effectiveness and that, in turn, could lead to a softening of commercial rigour across the industry. So, obviously, it’s fun to have a bit of a barny about something – and why not, it’s Christmas! – but I think we can save our energy on this one.

In today’s climate, there just isn’t a CMO in town that would tolerate this sort of softening of commercial rigour from their agency. And, what about the children, I hear you cry? Won’t the next generation of ad-people be led astray by this sort of loosey-goosey blurring of terms?

Well, with the festive season rolling in, it only feels right to be generous and give them the benefit of the doubt. After all, the one thing the next generation knows better than anyone is that, when it comes to an effective campaign, it now takes more than just brilliant TV copy.

Jo Arden, chief strategy officer at Ogilvy UK

In the debate about best Christmas ads, it’s funny that the measure we’re having a pop at is the one based on what people in the real world think. Accepted, System 1 results are a prediction of effectiveness, not proof of it. But it is a prediction based on the responses of the people we’re hoping to win over, which is a better prediction than the smart but subjective views of most people reading this. Are we losing sight of effectiveness? Not on your nelly! Our appreciation for what counts as proper measurement has never been better (at least, that’s my own prediction for what’s to come in next year’s IPA Effectiveness Awards). The proclamations we’re seeing about the Christmas ads now are a useful pointer to what we’ll ultimately see in trading results and brand health. They’ve also got everyone - including creatives - talking about effectiveness. A small glass of strategic sherry to that!

Chris McKibbin – Group Head of Strategy Dentsu Creative

Yes, these headlines are click-baity and wildly reductive. They ignore knotty complicating factors and remove basic human agency. But if we’re worrying about semantics, we’re losing sight of the bigger battle. 

The System1 Christmas Campaign results make a case for the kind of work that we know – through experience, intuition, and cold-hard data – is likely to be effective. Work that’s bold, surprising, emotional, and story-led. Creative that gives the audience some credit and respects their time. Communication that aims for fame. 

So scoff at their generous definition of effectiveness all you like. But their message is not for you. If System1 is helping to embolden marketing’s collective creative ambitions, encouraging businesses to invest in brand, or inspiring briefs for work with a chance of being genuinely effective, I’m all for it.