A/B Headlines Testing for Newsrooms

We dedicate most of attention to the headlines testing, as what is shown on the main page determines the overall user engagement.

A/B Headline Testing for Newsrooms

Previously, we discussed the general A/B testing questions and the use of split tests by publishers.

At IO, we pay more attention to headline testing, as what is shown on the main page determines the overall user engagement. Examples of The New York Times and BuzzFeed we shared in the first article are proofs of this fact. Headline is the first thing that needs thorough attention, as it is what readers pay attention to when they visit the main page (or an article’s page) and decide whether they should spend at least half a minute reading the text.

The Main Goals of Headline Testing Are To

  • Increase traffic to the main page. Remove underperforming articles.
  • Build a loyal audience and help authors avoid clickbait headlines.
  • Educate editors on how to write the best headlines in order to build loyal visitors.
  • Increase the quality of referral traffic, which comes to your main page from external links.
Five main goals of headline testing

The purpose of split testing is not just finding which headline was more attractive to readers; it is a source of further brainstorming and a way to understand your customers better.

Although comparing headlines seems like a simple solution, it brings enormous benefit to data-driven editorial teams, as their experiments become grounded and reasonable.

IO A/B Testing Solution

IO provides teams with insights on the effectiveness of their headlines. It is as simple as it can be. After configuring the A/B testing feature on a client’s website, the functionality appears on the Main Page widget, and the results of the testing can be found in the Editorial Dashboard > Headline Tests dashboard.

You may wonder how it works, so here is a simple explanation: IO JavaScript code changes the headlines for the articles of your choice on the main page (yes, it works for the main page only).

Headline testing widget on the main page

Here is how an A/B Test in progress looks like.

A/B headline testing in progress

You can configure everything just on your main page. Enter alternative headlines and monitor their success.

Different headline variants participating in the test

The Headline Testing tool determines the winner based on the measured results using best-practice algorithms.

IO headline testing solution shows the winning option in the end

You can also find all the information in the Dashboard. It shows both your current and previous tests:

Headline testing dashboard

What is Special About IO Solution:

  • Easy-to-use interface and full automation of IO solution allow authors to start A/B tests in just a couple of clicks. However, the manual above shows it accurately.
  • IO formula uses several variables to identify the best headline. It takes into account the CTR, the Recirculation rate and the time spent on a split test. The results you get are statistically correct and ready for implementation on other headlines.
  • The internal Dashboard highlights A/B testing benefits. This way, you can explore how many additional page views, clicks and time on site were generated by an experimental headline version. So, there will be a proof of why an experimental headline is better than the original one, or vice versa.
  • After your A/B tests are over, the data is kept in your system and you have instant access to it. Use this historical data for educative purposes. For example, show all team members what should be improved and why. As an alternative, publishers can create recommendations for editors based on machine learning. Journalists already use it widely and the perspectives are great. For instance, it is possible that the machine will recommend changing your headlines or parts of those headlines based on the analysis of tons of historical data.

How Long Does Headline Testing Last?

For each separate website, the duration of testing is calculated individually. It is based on the number of clicks per hour/day.

In case headlines from the top block are tested, the continuance is around 5-10 minutes. At the same time, IO A/B test code needs more than 50 clicks for results validation. If this is done another way, the test will last for an hour, and if there are no more than 50 clicks, it will be finished "without a winner".

Tips for Creating Headlines

Create different variants for your original headline, keeping them in mind. Experienced editors and marketers agree that headlines should surprise people, make them curious, answer their questions and provide them with how-to instructions, helping them solve the problems.

Here are several actionable tips on how to change your headlines to make them more engaging:

  1. Use numbers or statements.

    Well, even promises can work well. For example: "How to Create a Great Headline in 5 Minutes" or "Get 1000 Instagram Followers in 10 Days".

  2. Guides, how-to and manuals.

    Show your readers the helpfulness of your article: "Detailed Manual on Creating Catchy Headlines".

  3. Call to action.

    This works well for any component of digital content: headlines, articles, pictures, buttons, etc. "Find Out What Happened to Google Search Algorithm", "Tips on Becoming GDPR Compliant".

  4. Uniqueness and originality.

    For newsrooms, which are always competing with plenty of other digital newspapers, blogs and magazines, it’s crucial to post fresh and unique news no one has posted yet. Headlines should tell visitors what is unusual with the article to make them click on it. For instance, "Exclusive Interview with Apple CEO".

  5. Using positive and negative words.

    Both of these work well as long as they attract emotions. Use split tests to learn which tone your readers prefer.

Headline creating and testing


Every newsroom should utilize the headlines testing tool, as constant tests lead to improvement and a better understanding of what readers need and finally what they will pay for.

Use completely different variants and experiment with separate elements in your A/B tests; for instance, numbers or adjectives, or even the entire style. Make a hypothesis for every variation; you don’t have to change something just for the sake of experimentation.

Can numbers or re-phrased questions drive more traffic? Check it out with an A/B test.