Machine learning use cases in the publishing industry

Over the past 10 years, the field of artificial intelligence and in particular machine learning, one of its subfields, has developed considerably. The emergence of new techniques such as deep learning and the shared effort to build open source communities, frameworks and libraries have all contributed to advancing AI.

More recently, natural language processing (NLP), a subdomain devoted to understanding text content, has also benefited from new architectures such as transformers. These make it possible to build powerful language models and to reuse them easily for a wide range of purposes, such as translation, sentiment analysis and text classification.

It was therefore entirely logical for SPRYLAB to develop and offer innovative features that let publishers make use of these recent developments. That is why we started work on a set of APIs that form the basis of our Purple DS ACM offering (autonomous content management). We decided on recommender systems, article linking and topic modelling as the first use cases for building our ACM solution. Let us look at these use cases more closely.

‍

Use case 1: recommender system

As a digital publisher you constantly face the challenge of winning a loyal online audience in order to build a successful brand. The best way to address that is to deliver effective, personalised content that keeps the user engaged. Recommender systems (RecSys) improve the user experience, increase page views and improve site ranking.

RecSys are very popular and heavily researched. They can be implemented on their own or in combination with various approaches. The most popular are content-based and collaborative filtering.

‍

Content-based filtering

Content-based filtering is a technique that analyses what a particular person reads and then tries to prioritise similar content. This means recommended articles are matched to personal preferences on the basis of reading history. To be able to define a user's preference, we have to be able to identify what information (often called features in ML) can be used or derived from the articles readers are interested in. In our case, features such as category, subcategory or refined topics are available or could be derived by other NLP techniques.

‍

Collaborative filtering

Collaborative filtering is a technique that takes into account not only one person's article preferences but also the data of other readers with similar preferences. Compared with content-based filtering, the variety of content that can be recommended to the reader is much greater. Beyond that, there is no need to calculate the similarity between different articles.

‍

Ensemble learning

As with many machine learning systems, accuracy increases when we combine the results of different models. This technique is called ensemble learning. We implement a hybrid system of this kind in our RecSys.

In terms of implementation, it is extremely important to have the right pipeline for data ingestion and transformation in order to train our models. The volume of data involved in reader analytics is in fact quite significant. At SPRYLAB we chose a modern and flexible framework to carry out these tasks: Apache Beam. This gives us a single framework that can handle both stream and batch data pipelines with the same API. It is also independent of its execution platform, so you can use your own Spark cluster if you already have one. If you want to reduce your DevOps load, you can use an autoscaling service such as Google Dataflow.

To improve our performance and quality continuously, we are looking at various possible paths for improvement:

• Certain topics that are subject to seasonal interest can easily be captured and reflected in our model.
• A reader can be affected by a fixed strategy in terms of preferences. By using a suitable reinforcement technique, we may be able to find a better balance between exploration and exploitation. This technique consists of alternating between different strategies at the right frequency.

Use case 2: article linking

Further goals that all digital publishers share are retaining their audience and increasing visibility by improving search engine optimisation. This could be achieved by linking part of an article (usually a group of words in a sentence) either to another article or to an external website. Traditionally this task is carried out manually by the editor. The whole process is of course time-consuming and requires a lot of knowledge. Here too, machine learning can be used to support the editor with this task by automating the generation of potential link candidates.

‍

Let us break this down a little further and see what has to be done:

1. For each article we have to identify the text selection that can be used as a link.
2. For each text selection, a proper search has to be carried out that surfaces candidate articles to link to.
3. A re-ranking of these candidates has to be calculated.
4. Finally a filter is applied to the results that takes configuration parameters such as the number of links per paragraph and the confidence threshold into account.

Step 1: text selection

The first step is very difficult, because the number of selected texts very quickly suffers from a combinatorial explosion. To work around this, our first approach is to consider only a phrase (a group of nouns) coupled with the output of an NER (named entity recognition) algorithm. This reduces the number of searches considerably, but has the disadvantage of generating selections that can be too general. If a selection refers to "the EU", for example, we may get many search results that match the selection but not really the context. Fortunately we have a dataset of annotated selections that lets us train a model which uses POS tagging (part of speech) analysis to ensure a larger and more appropriate selection.

‍

Step 2: searching for linkable assets

Since we use Elasticsearch (ES) to index our content assets, the second step is currently the easiest. Its powerful engine delivers good results, which are refined further in the next step.

‍

Step 3: semantic similarity ranking

The third step takes into account, among other things, some predefined strategy parameters such as recency and similarity to the target article. The most interesting part comes from the similarity calculation. ES itself cannot perform well on semantic similarity, but the latest language models such as BERT can generate embeddings that take it into account. In the near future we will evaluate a way of using these embeddings to carry out an efficient vector similarity search, as described here.

‍

Step 4: final ranking

The last step is also fairly trivial. Here the final results are scored one more time against a set of additional parameters such as the number of links per paragraph, the confidence threshold, the number of candidate articles and so on. This helps reduce the number of links shown for validation.

Note that in our tool, if an editor is not happy with the automatic text selection, they can still find relevant articles automatically when selecting any piece of text themselves.

Use case 3: topic modelling

We have already briefly mentioned topic modelling as a mechanism for feature extraction that can be applied to the content filtering method when implementing a RecSys strategy. But its potential use of course goes far beyond that.

As a publisher you are always working to take your readers' interests into account and to spark their curiosity. To do that you have to pay particular attention to your editorial team and to the overall coverage of the topics you want to include.

Topics of interest can be divided into at least two categories:

1. Seasonal topics, which are recurring content that has to be addressed, such as events like Christmas or the summer holidays.
2. Hot topics, which are current or emerging subjects that are not normally covered but are connected to your content.

We therefore decided to focus on this kind of topic modelling, supporting editors in choosing the right topic at the right time and helping them plan their editorial calendar well in advance.

‍

Unsupervised learning

From a technical point of view, both topic modelling categories use a common underlying mechanism, namely keyword extraction. In our case we use well-known unsupervised learning techniques such as latent Dirichlet allocation (LDA) to extract keywords from our content databases, specifically the implementation in a popular library called Gensim. In summary, LDA makes it possible to extract keywords that can be interpreted as probability distributions of words.

Building on that, we are developing two main services: seasonal topic analysis and hot topic analysis.

For seasonal topic analysis, we can carry out a temporal analysis for a given list of topics that we have calculated beforehand. Google Trends gives us insight into each keyword that drives our topic selection.

For hot topic analysis we apply a strategy similar to the previous case. Here we use the Aylien news API to run a trend analysis on the news collected. We can then assess how relevant the news is to the topics the publisher mainly focuses on.

In both cases, once a topic has been selected, we offer additional functionality in our Purple DS article editor to assess easily how well the topic a user is meant to write about matches the target keywords for that topic

Conclusion

Although the second digital transformation with machine learning systems has only recently begun, we can see it being adopted at large publishers and gradually across the whole industry.

The features highlighted in this article represent only a fraction of what is possible. In NLP, constant progress and the emergence of even more powerful language models will shortly enable features such as text summarisation or even text generation for some simpler cases. Beyond that, another ML domain such as computer vision can easily be used in combination with NLP in order to further improve existing features, such as linking content to image providers, or to create new ones, such as automatic image or video annotation.