Segmenting Users With Activity Scoring

Following up from our article regarding meaningful activities last week, this post will cover a topic which is closely related: activity scoring and its segmentation use cases. Activity scoring is a segmentation technique that aims at assigning users with an activity (i.e. engagement) score over a specific timeframe. It is an extremely useful segmentation approach as it provides a rather intuitive way of segmenting your user base to create segments that will correlate with the key outcomes you want to predict such as future revenue, retention or subscription uptake and renewal.

A Simple Definition Of Activity Scoring

How do you get about creating an activity score you might be wondering? In its simplest way, an activity score will count all activities performed by a user over a specific timeframe. To start simple, we will count any action made by a user with the same weight (i.e. multiplier in the score). As a retult, the score for each user will simply be the count (or summing 1) for each activity the user has performed within the timeframe. For example, let’s assume that we are looking into the user base of a music app as we did in last week’s article. And that we are tracking the following activity list: app open, screen view, searching for a song, starting a song, ending a song, adding a song to a playlist and removing a song from a playlist.

Now let’s assume we have the following users who performed the following activities in the past month:

• User 1:
  • App open: 10
  • Screen view: 20
  • Searching for a song: 5
  • Starting a song: 6
  • Ending a song: 4
  • Adding a song to a playlist: 1
  • Removing song from a playlist: 0
• User 2:
  • App open: 100
  • Screen view: 60
  • Searching for a song: 8
  • Starting a song: 6
  • Ending a song: 4
  • Adding a song to a playlist: 0
  • Removing song from a playlist: 0
• User 3:
  • App open: 35
  • Screen view: 65
  • Searching for a song: 12
  • Starting a song: 20
  • Ending a song: 19
  • Adding a song to a playlist: 4
  • Removing song from a playlist: 1

As mentioned above, the activity scoring in its simplest form would be the count of these activities. So for the past month, the score of each user would be:

• User 1: 46 (= 10 + 20 + 5 + 6 + 4 + 1 + 0)
• User 2: 178 (= 100 + 60 + 8 + 6 + 4 + 0 + 0)
• User 3: 156 (= 35 + 65 + 12 + 20 + 19 + 4 + 1)

This scoring gives a ranking of engagement for the past month across your user base. In our case, if you look at this scoring directly user 2 is the most active user, followed by user 3 and then by user 1. However the most eagle-eyed amongst you will have likely noticed that this scoring might be a bit unreflective of “true” engagement. Why? Because even though user 3 is definitely most engaged than user 1, user 2 is probably not the most active user from a “meaningful” activity standpoint. Indeed, most of the activities done by user 2 which are driving its score up are app opens and screen views. But when looking at activities which are truer to using the product (e.g searching for a song, starting a song, ending a song, etc.) then user 2 is roughly on par with user 1 and far below the levels of activity of user 3. Yes, not activities are equal! This is what we discussed in last week’s post and I presume you can already see where we are going.

Tuning The Activity Score By Introducing Weights

In order to make the scoring carry more “signal” of activity and be more meaningful, we can tune the scoring by cherry picking which activities to include and adding weights to the activities selected. The first thing you might want to start with is some in-depth analysis into which activities are most likely to lead to some of your core KPIs as a business. Let’s say that our music app is in a growth phase and has for objective that users take or renew a subscription. We would then plot the impact of the user doing each of these activities on the conversion rate to taking or renewing a subscription. This could look as the following chart for taking a subscription being the conversion we look into. The orange line represents the percentage of users who took a subscription out of the users that did a specific activity. The black line represents the percentage of users who took a subscription out of the users that did not do a specific activity.

We can observe three main patterns:

1. There is a significant difference in conversion rate between the activities performed (in orange). Users that perform activities other than “app open” and “screen view” are between 1.5x and 6x more likely to convert. Meaningful activities start to appear.
2. The subscription uptake rate is the highest for playlist related activities, although the user base performing these is smaller.
3. There is very little uplift in conversion rate for users that did an app open or a screen view vs. the ones that didn’t. This indicates that opening the app or viewing a screen is not a strong signal of taking a subscription later on.

From this, we start understanding which activities carry more weight into giving us a signal that the user will perform further desirable actions. From this we could decide that we remove the “app open” and “screen view” from the scoring or give them an extremely low weight. For simplicity, we will remove these in our scoring. All other activities will then have a weight of 1 except for the playlist activities which will then have a weight of 4 (as they led to higher conversion rates). Our new activity score formula therefore becomes:

activity score = 1 * search song + 1 * start song + 1 * end song + 4 * add song playlist + 4 * remove song playlist

As a result, the new user scores are:

• User 1: 19 = (1 * 5 + 1 * 6 + 1 * 4 + 4 * 1 + 4 * 0)
• User 2: 18 = (1 * 8 + 1 * 6 + 1 * 4 + 4 * 0 + 4 * 0)
• User 3: 71 = (1 * 12 + 1 * 20 + 1 * 19 + 4 * 4 + 4 * 1)

We now observe a different pattern with this new score. user 3 is the most “active” user by far, user 1 is second closely followed by user 2. This new score ranking looks more meaningful, which we will then want to verify.

Activity Scoring Segmentation And Verification

In order to verify the activity scoring, we will want to start applying some level of segmentation using the activity score we produced and check the impact on the metric we are aiming to predict. In order to do so, we will then look at the distribution of all the user activity scores and separate these into buckets. The buckets do not necessarily need to be even in size (most digital products typically have right-skewed activity distributions meaning that the lowest activity segments have most users but there is a longer tail of user segments as activity increases) but you need to ensure you have enough users in each bucket to make the numbers robust.

Going back to our example, we will segment the user base with the following bands:

• No activity: 0 scores (you want this to be isolated as 0 is specific - i.e. no activity based on the score)
• Very Low: 1 to 20
• Low: 21 to 50
• Medium: 51 to 80
• High: 81 to 120
• Very High: 121 and above

This results in the following hypothetical distribution, against which we plot the subscription conversion rate uptake in order to visualise the relationship between the activity score we created and the metric we want to influence (i.e. taking a subscription). The chart shows the number of users and the conversion rate per activity score bucket as defined above.

Good news! We have a strong and positive relationship between the activity scoring and the subscription uptake conversion rate. In other words, the higher the activity score of a user, the more likely he / she will be to take a subscription!

This user segmentation can then be modelled into your data warehouse and refreshed on a daily basis for example. You can calculate activity scores based on different windows (e.g. last day, last week, last month, etc.). The activity score segmentation then provides a very good springboard for data activation. For example, you might want to tailor your CRM communications based on the activity score segment, use the activity score to identify users likely to churn or feed back the activity score to your marketing platforms as a signal of overall user quality.

We have seen how the activity scoring can provide you with an efficient segmentation of your user base and some first examples of how the segmentation can then be leveraged. Now, it’s your turn to play! How will you activate activity scoring?