224 Additional Minutes per
Headline Test: Epic A/B Testing for
Impact on Readership Trust, Not CTR

WARNING! This page has the potential to shatter illusions and uncover fallacies. Serious headline testing has become even more serious. Doing A/B tests on headlines to achieve higher click through rate is a completely wrong way to do it. CTR earned that way will become damaging (think bounce rate) and unsustainable really fast.

The legend goes that a famed digital publishing company once tried testing headlines for their impact on recirculation, which is the inverse of bounce rate, and article engagement rather than testing for CTR alone.

They never went back.

To receive a working approach that correlates CTR, recirculation, and average time on page in just the right proportions, so that real headline qualities can be assessed using a properly aligned metric - that may sound like fiction; but what that is in reality, is a dream come true for digital publishers.

Read on to find out:

  • How 342 additional article minutes and 2264 additional page views are, on average, generated per one experiment, which takes less than a minute to launch
  • What the fuss around clickless consumption is exactly about
  • Who has a working key to solve the headline-article performance connection puzzle
  • What are the main things to avoid as an editor and a creator of alternative headline testing experiments

A new spin on testing article headlines for online publishers is about connecting headlines to profits-related publishing metrics, not CTR. Forget testing for clickbait. The new approach to assessing headline qualities makes it possible to stimulate more far-reaching consumption behaviors, such as clicking on more of the website, staying on pages for longer and reading beyond headlines … attention … using only headlines and an intuitive overlay.

non ctr headline tester

Testing headlines just to test is wrong. Testing headlines for clicks alone is even more wrong. CTR is passé, and so is every click-centric model of content evaluation.

Countless researches into the real value clicks provide have proven times and times again that clicks are clicks; and content consumption does not require clicking. So, what is right then? The only correct way is testing headlines’ connection to engagement metrics, which CTR obviously isn’t.

Year 2018. Two Dutch researchers sit down with an impressive amount of people, one by one, at a desktop computer, and observe how every one from that group interacts with a website. They observe how individuals either click or don’t click article headlines. Questions are asked as to why some clicks are made, while others aren’t.

headline tester person and researcher

Curious things first.

The most common fallacy had been that a headline was ‘consumed’ only when it was clicked. It became apparent, however, that headline consumption practices do not need to involve clicks. People will even do so-called ‘clickless monitoring’ of headlines just to stay up to date with the news. Headline CTR, perceived headline qualities and the actual process of ‘headline consumption’ are farther apart than seems at first sight. To a great extent too.

“...clicks are a flawed instrument because a lack of clicking does not measure people's lack of interest in news... ” — research from Vrije University

Headline quality encompasses crucial aspects beyond CTR that contribute to its article consumption-related, hidden quality, which is very different from clickbait appeal and which should be assessed by all respectable media groups. There are on-page metrics that should complement, if not substitute, CTR in an evaluation concerning estimated quality and impact of a given headline.


Do you know editors who still test headlines for CTR? Tell them to stop!


A click is not needed for headlines to be consumed, so how does then CTR still occupy a central place in headline A/B testing?

It’s a legacy thing, that’s all.

Clicks have for a long time been majorly misplaced and overvalued in all of article headline split testing. Decades of clickbait-oriented website business models have surely had their impact. In a comic way, users now almost expect to be baited. 

Consumption of ad content and interaction with call to action elements are two other things that heavily depend on headlines.

”Headlines aren’t just reserved for homepages; [they] aren’t just reserved for product pages. They can be very effective even in checkout processes.” — A. McCraw, MECLABS Institute (source)

Let’s talk editions that rely on loyal readership that actually reads things. For such clients, relying on CTR, in a completely flawed way, has become the norm. And nobody knows it, because it has become such an industry standard.  Doing headline A/B testing to achieve impact on consumption and overall perceived quality is a small bit in a big puzzle of how headlines impact article consumption.


Driving conclusions about headline qualities from CTR alone is a critical mistake


A headline can be well crafted, can be adored by editors, can be wanted by users, and yet, if it sources no clicks, it will show as underperforming, provided CTR numbers are not there. Is that what online publishers really want? Surely not. The value CTR-centric tests provide in A/B headline testing is questionable at best. When a website is using A/B testing to measure clicks to impressions ratios on headlines and is driving assumptions on respective headline qualities solely from that, then that’s a critical mistake.

Headline Tester! Locate The Headlines-Revenues Touch Point!

The connection between headlines and revenues is not as indirect as it may seem.

A headline is a promise.

That’s how readers see it, and that’s how you should see it too.

When a visitor consumes a headline, relevant existing knowledge gets activated inside their head, mood gets set, attitude formed and article-related behavior pre-programmed for when, and if, there is a click.

Research has shown that readers who spend more time on a page are significantly more likely to make return visitors, so it is not surprising that headlines are increasingly being linked to websites’ revenues. Headline placement on a homepage and how headlines are written impact cumulative trust with readership, which means returning visitors, subscriptions, and revenues. So, assessing a headline from the perspective of those revenue-sourcing user activities, instead of clicks, is certainly vital.

“Metrics that measure a fuller range of user practices seem more suitable for capturing users' engagement with news content.” — G. Kormelink (source)

It is the headline that pre-programs individuals for later sharing, reading, and navigating deeper into a website. On-page article-related behaviors are for the very first time given form, direction and dynamic at headline consumption stage. A major study by the Poynter Institute, Eyetrack and University of Denver has shown that the gateway to capturing attention on the homepage is not graphics. It is bold headlines.

“By not making the flow of information on your site more attractive to what your customers are expecting, you are at risk of losing viewership to a competitor that learned more about your customers via A/B testing.” — C. Abisambra, CEO of Vortice Services (source)

Proper headlines can, and will, prime visitors into staying to read for longer and visiting more pages. In that way, alternative headline testing is a gateway to accumulating greater overall reader-perceived quality. Think again next time you’re being offered a headline tester tool to try out.

Cumulative Audience Loyalty and Reader Perceived Quality

Content consumers’ ways of engaging with web content have evolved majorly in the past few years. Content publishers - print or digital - are now realizing that they are ultimately working with a cumulative bank of reader trust and overall reader perceived quality. That is because the ultimate, final objective of every publishing business rests in deeply connecting to its core and extended readership; on a long term basis too.

To do that, the objective has to be on stickiness. Stickiness that arises from quality too. Yes, quality is an ambiguous term, and quality is, at times, extremely hard to assess. Yet, it is clearly better to orientate one’s efforts on the correct and hard rather than on the wrong and easy.

No article headline exists in a vacuum, and there is always a two-way relationship between a headline and an article it promotes. Every time reader’s expectations are met following a headline click, that adds to the cumulative trust bank. If expectations are not met, then that’s a failure and a subtraction from the bank.

It is important for websites to approve such headlines that, when coupled with respective articles, will ensure that content consumers’ expectations are met once a headline is clicked and article - read. If one of the two is lacking, reader’s disappointment and service failure follows. A headline should always rest upon such, and only such, article content that can appropriately fit and meet the expectations initially set out by its title.

Otherwise, loss of trust with readership is most certainly assured.

Do that enough times, and lost trust will accrue.


Unfulfilled promises made in headlines get accumulated over time. If the balance is negative, readership suffers


For a digital media company of any size, making promises that cannot be fulfilled later on is a foolish thing to do. Leftovers from unfulfilled promises made by headlines get accumulated over time, and if the balance is negative, readership suffers. Let readership suffer and watch how revenues plummet.

[HOW] Correct Way to Run Alternative Headline Testing


342 additional minutes and 2264 page views are produced on average by one single headline experiment


AB testing headlines for quality of clickbait, like many companies still do, is the surest road to failure, long term. One way to take article headline split testing up a notch was to construct and introduce a high-level metric, which output value would tell a publisher how profits-oriented a headline’s wording is. Counting time spent on page and monitoring further recirculation, and relating all that to CTR is just what field experiments have shown to work best. Naturally, that had to become a formula.

approach to ab testing headlines

Research into the evolution of users’ content consumption practices and an industry-wide ongoing departure from click-oriented models of doing business in publishing have equally confirmed that, in the future, to really gain control of headline performance means to gain access into post-click metrics. It may very well be the most ROI-efficient thing a publishing business can do these days, considering the breadth of impact properly structured headlines can have on a digital publishing business.

A standout A/B headline testing tool that produces accurate and, most importantly, reliable insight into how impactful a given headline is on reader perceived value can be found via link below. Its in-built logic brings together CTR, time spent reading, and recirculation in just the right proportions to deliver actionable insights and material for learning to writers and editors.

Interested? Try the functionality yourself for free.

Starting AB Testing Headlines in 1-2-3

Technology has certainly made A/B testing tools become more accessible to headline testers in recent years. First A/B testing tools required JavaScript and advanced Hypertext Markup Language from the user to get anywhere with testing. Times have changed and coding skills are no longer required. It’s no longer code that a headline tester works with but a graphical user interface (GUI).

Yet, issues with digitization efforts in the media field are well known. Editorial teams often times lack tech savviness and free time to really take on GUI solutions that answer questions editorial teams need and want answered.

So, what’s the workaround?

The answer is twofold.

The first part is to work with AB headline testing tools that have shallow learning curves. Only the very best article headline testing tools are made with end users and their preferences in mind. So, an article headline testing tool, which an editor, a writer, or any other headline tester can use to its fullest potential from the very start.

The second part is streamlining headline crafting with sights set on website revenues and not some abstract, already known to be virtually nonexistent, connection between CTR and headline quality. That connection is extremely dim, if existent at all. To streamline the process of AB headline testing for a media company means to empower writers and editors with an intuitive-interface tool that, due to its cost-effectiveness and ease of use, has the potential to shorten the AB headline testing cycle and thus bring massive return on investment. The bigger the website, the higher the ROI is going to be. Should you do it? Absolutely. Though in a way that doesn’t hurt writers’ creativity. It is the one thing that can bring most impact to a business reliant on content performance, and that is essentially any digital publishing company.

impactful ab testing headlines
  • Functionality for AB testing headlines that authors and editors just love
  • Not just homepages but also topic-specific website sections can be worked with
  • All past headline A/B testing efforts and respective experiment results can be viewed in the main dashboard for any custom time frame in the past
  • All past headline AB testing experiments are non-expiring. Forever.
  • Emphasis made on learning how to write headlines that influence editorial metrics that matter
ab headline testing tool screenshot
  • Widget-like overlay, so you don't have to log anywhere to check progress
  • Tests are launched without having to leave the homepage - from a widget, which is overlaid over the homepage
  • Widget gets brought up via a floating button, which are many - one for each headline
  • Real-time stats with no lag at all
headline tester screenshot
  • The back logic correlates metrics impact in just the right proportions and serves headlines, via split method, all by itself. No further action needed
  • Can add additional headlines to a test at any time
  • Starting, modifying and stopping article headline split testing is seamless and was designed with authors’ convenience in mind
  • Can stop/continue AB testing headlines through widget
  • All past tests can be viewed in the main dashboard

Improve Headline Writing by Learning From Past Tests

Creating masterful headlines is an art, and no analytical system will ever be able to substitute raw skill of an experienced writer. What can be done, though, is empowering skillful headline crafters with ready-to-use analytics that show how their headlines impact a business in the long term.

So, a tool that can be used for learning how to write better headlines as well as for achieving readership objectives.

  • A repository, where insight into historical performance of headlines can be generated
  • Additional page views as well as additional minutes earned for each test can be viewed
  • Tests can be viewed for a custom timeframe in the past

The Hidden A B Cs of AB Headline Testing

Although A/B headline testing may be an easy to understand to concept, it’s proper usage is tricky and certainly requires a seasoned specialist.

Headline Testing Best Practices for Editors:

  • Headlines that perform on the homepage will not necessarily perform on social media and vice versa. Traffic is different, so interaction with headlines is also going to be different
  • Relying on gut feel alone means signing up for a competitive disadvantage
  • A middle ground needs to be achieved between gut feel on one side and a plethora of relevant engagement metrics on the other, not CTR
  • When AB testing headlines, do not change a few words and expect that to be a good A/B headline testing design. Headlines must be significantly different for A/B testing to be most fruitful
  • Clickless consumption of headlines needs to be accounted for. Always.

The science of headline testing contains quite a number of pitfalls and knowledge of headline testing best practices is certainly vital. Once you begin article headline testing, you inevitably begin working with statistical concepts, and that’s where another bit of complexity comes in.

Of all professions, writers and editors are certainly not the most tech-savvy folks, so a tool needs to be intuitive and at the same time functional. Editors neither have the time to pick up complex tools, nor should they be doing something like that.

Catch Up With Chartbeat Headline Testing Now or Never!

While gut feel could get you far ten years ago, the abundance of good headline split testing engines on the market - like Chartbeat headline testing service, IO headline tester software, or some other headline split testing engine - makes that no longer true. When a website or a publishing business goes big, continued reliance on editors’ gut feel in wording and organization of headlines on a homepage is strange and risky. No amount of headline writing guides, gut feel or headline testing services will be ever able to substitute that same thing, but infused with a scientifically-informed approach to prioritizing one headline over another (think AB testing headlines). Editorial evaluation alone no longer cuts it. A scientifically informed layer of insight, such as headline ab testing, is commonly sought by those who are in the know.

 

Technological advancements have created a situation, whereby every single company reliant on content production is either already doing A/B headline testing, is using a third party headline tester service, or is in the process of discovering how to get AB testing headlines efficiently. Quite a few firms have turned to Chartbeat headline testing. Chartbeat is now competing with many other AB testing tools. One thing to take away from that is that not doing AB headline testing means allowing your competition to get a head start.

As simple as that.

An additional layer of insight onto editors’ efforts at headline crafting is a major improvement over the former ways of assessing whether a headline is going to be impactful, as viewed from a business point of view.

The market trend is such that those who haven’t already turned to article headline testing services are bound to catching up with the rest. The still-increasing ease of doing headline A/B testing has made it become a must for every respectable media firm priding itself in its loyal readership. Refusing to jump on the article headline testing services bandwagon implies a competitive disadvantage and a loss for content consumers and for editors alike.

Takeaways and Conclusions on Headline A/B Testing

It’s should no longer be a surprise to any headline tester that click-based evaluation is no longer deemed to be correct as an approach. Not in headline AB testing and hardly anywhere else. At the very least, CTR in AB headline testing, so not to be misleading, needs to be supplemented with engagement data. Optimally, that means mixing and correlation with other metrics - ones that are closer to long-term publishing revenue goals. For sites betting on stickiness and accumulated reader trust, that becomes the most vital thing ever.

Overall reader perceived quality and cumulative reader trust are the main attention points for attention-oriented business models in publishing. Making a consumer click on a pretty headline but making them disappointed with the article will lead to service failure in its purest form. For that not to happen, the testing loop for headline AB testing has to be shorter.

There are companies that are in the business of clicks and there are companies that are in the business of capturing attention. Both business models can and should delegate a dedicated headline tester to benefit from proper AB headline testing. After all, never before in the history of web publishing has it been as easy to use a headline tester tool, and by doing that, to bring as much cost-effectiveness and readership impact potential onto content production and editorial shop-floor operations.

+

What type of OS do you prefer?

Thank you for the information!
by IO Technologies