Why is it so hard to capture complaints made through social media?

Last month, we published our quarterly statistics on train operating complaints and, despite the increase in the rate of complaints on the same time last year, the long term trend remains downwards. The question is why? Punctuality & reliability is the largest cause of complaints and with those measures deteriorating in the past year, one might expect complaints to rise but they have not. Could it be that passengers have come to accept current levels of punctuality as the norm and therefore feel less inclined to complain as it is a fruitless exercise? Or could it be the case that train operators are putting out more information via social media during times of disruption that cuts off the complaints at source? Passengers are then less inclined to complain as at least they understand why their train is being held up.

Talk of social media in relation to trains has become something of a hot topic. With all train operators now active on social media, albeit some more than others, one popular theory is that the complaints rate is only reducing in volume because passengers who may previously have complained via more traditional methods, are now taking to social media as a means of expressing their dissatisfaction.

We'd like to explain why the current methodology employed by ORR does not capture social media

Improving complaints consistency

Before we even seek to describe the current difficulties in recording social media complaints, it is worth bearing in mind how far we have come in recent years in terms of complaints reporting and how that compares with other comparable industries. ORR has done a lot of good work alongside the train operating companies to improve the consistency with how complaints are recorded and also in terms of what we publish, for example, ensuring all operators are applying the same processes in recording complaints and increasing the volume of categories reported so they can be directly compared to Passenger Focus' National Rail Passenger Survey categories. However, despite these improvements, there are still factors that will always impact on the statistics that we can do little about; for example, some TOCs are more proactive in terms of advertising to passengers how to complain and they operate vastly different types of services (e.g. long distance vs. commuter) but, in essence, they all record the same things on the same basis.

Learning from other industries

Furthermore, if you look at the telecoms industry, there is no requirement from Ofcom on providers to collect complaints through social media, although they can do so if they wish. If we were to go down this route, we would be capturing some of those additional complaints but we would lose that element of consistency that we have only recently started to achieve and one that, as statisticians, we hold quite dear.
In the energy sector, Ofgem does require complaints made to energy suppliers via social media to be logged as a complaint though they are not reported as a standalone item. ORR is hoping to work with some of the energy suppliers to see how they record this in practice.

As part of our research into social media complaints, we are seeking to liaise with organisations in similar industries to see if and how they record social media complaints.

Social media conundrum for rail statistics

Here are some of the emerging challenges we have identified in our work with the train operators on their approach to social media complaints ahead of a report we will publish later this year.

• Is the written word more powerful than the spoken word?  
  A complaint is currently defined as 'any expression of dissatisfaction'. If someone puts the following message onto social media 'Thanks TOC, I am late again' should this be classed as a complaint? It is an expression of dissatisfaction so, by the letter of the law, should be recorded as such. However, social media is more about the conversation and this kind of comment is something that could just as easily be muttered from one passenger to another without it ever being logged as a complaint.
• Shouldn't TOC engagement be a good thing?
  Some TOCs are more reactive than others on social media in responding to comments raised by customers. This then poses a question about customer behaviour – would a customer be more likely to complain via social media if they know it is likely to elicit a response? Conversely, if your TOC is not very active on social media, what is the point in expressing your dissatisfaction? In these instances, TOC A would receive far more 'complaints' than TOC B but surely the engagement with their customers should be welcomed. Our feeling is that publishing data on volume of social media complaints could introduce perverse incentives on TOCs.
• Can analytics pick up on the British love of sarcasm?
  A number of train operators record the sentiment of any messages they receive via social media; most broadly into positive and negative feedback though some do disaggregate further. Given this is not rolled out across the whole industry and some TOC reporting mechanisms would not allow for this to be done without expense, one potential avenue is to use data from an independent company who analyse social media activity. ORR has been to see some companies who offer this kind of service by looking for key words or phrases but how would it capture the British love of sarcasm? For example, "Good job [TOC], signs working as well as ever" – does that get classed as praise?

We are meeting with train operators to discuss these issues and more in late November and would very much welcome your thoughts.