Wednesday, 9 September 2009

An interesting week for misleading statistics

A few technology news headlines have attracted my attention this week that seem to be somewhat less than honest. Some of the ones that stood out included the 10% surge in BBC iPlayer traffic following its inclusion on the new PS3 Slim which in turn has been reported as bringing about a massive sales spike in Sony PS3 units following the slimmer redesign. The other noteworthy report involved the claim that 7 million of the 61 million UK residents is engaged in illegal file-sharing activity. Let's take a little look at those claims...

Sony's sales and why they might not be so impressive

There have been several reports that Sony is having a very successful PS3 Slim rollout. Popular games blog, Kotaku, noted that the redesigned PS3 sold more units in Japan in the first three days following the relaunch (150,252) than it did back in November 2006 when it was originally released. It sold 88,000 units in two days but there were issues with supply levels at launch, so not everyone who wanted one could get their hand on one.

The more interesting PS3 story came from VG247 courtesy of Chart-Track director, Dorian Bloch, who claimed that UK-based PS3 sales in financial week 36 were up over 999% on the week prior. This is a story which has snowballed and appeared in several other places. Great news for Sony. Or is it?

On the surface the 999% or 1000% sales boost seems impressive but this has to be seen in the harsh reality that very few people were likely to be buying the older PS3 model when the new slimmer model was about to be launched at a reduced price. It's not hard to see why a huge sales spike may occur when very few units were being sold in the week prior - but it makes for great headlines!

What is also unclear is exactly how many of these purchases were by new customers. Many existing PS3 owners have expressed an interest in trading their older machines in for the new model (the local GAME store where I live was accepting a trade-in deal of one 80gb PS3 plus two games and £60 cash for a new slimmer PS3).

This might not be so great for Sony after all. Sony are still producing the PS3 slim at a loss and require customers to purchase software in order to offset these losses. If they are selling lots of new units to existing customers who already own plenty software then they are unlikely to profit dramatically from these impressive sales figures. Also, Sony get no financial gain from the second hand hardware or software market. The more new customers the PS3 redesign attracts, the healthier Sony's bank balance.

It's understandable that Sony would want to view its new product launch as a success, given that the PS3 languishes in third place in the console hardware sales. It's also no surprise that Sony has announced a massive ad campaign to push its new hardware in the Christmas run in (with a campaign budget of £82 million!). Whether or not the PS3 proves to be the success that Sony needs it to be remains to be seen, especially since it experienced its first full year loss in fourteen years.

How many filesharers?

The other attention-grabbing (and more worrying) story that warrants a little exploration involves the claim that 7 million UK residents are file sharing criminals. This figure is one which has been around for some time now. The BBC ran a report back in May which cited the figure in a government-backed report. The report was issued by the Strategic Advisory Board for Intellectual Property (SABIP) and can be found on their site (.pdf here). Intellectual Property Minister David Lammy said the report put into context the impact illegal downloads had on copyright industries and the UK economy as a whole.

This figure of 7 million criminals is quite catching, seems precise and scientific and has helped perpetuate the notion of digital criminality amongst UK web users. It was also calculated that these file-sharers had access to £12 billion worth of free content. It has also helped keep file-sharing and its supposed criminality in the media spotlight in recent weeks - something Lord Mandelson has been using to great effect in his attempt to lobby for hard-hitting punitive measures against file-sharers.

Credit needs to be given to the BBC who took a second look at that figure of 7 million and decided to see if they could locate its origin, following an enquiry from the audience of the BBC Radio 4 show, "More or Less".

The figure may have been reported in a government document, but it's not a government figure. It transpired that the government commissioned a report from the CIBER research group at University College London, which contained the number. The CIBER report mentioned the figure four times. However, the figure actually came from yet another report from consultancy firm, Forrester. It doesn't end there. The Forrester report doesn't actually contain the 7 million figure despite the CIBER citation. The figure actually comes from a different piece of research called the Jupiter Industry Losses Project, which was an industry funded (BPI) attempt to gauge P2P use and the related losses produced by such internet use.

That's right. The official sounding figure cited by the government-backed report actually came from the British recording industry which has a vested interest in putting a figure on its losses at the hands of P2P users so that it can then lobby government to change policies and laws favourable to the industries interests. That explains why Lord Mandleson has been so anti-P2P in recent weeks...

It gets worse. The actual report was never published publicly and the industry declined to pass it over to the BBC but they managed to get in touch with Mark Mulligan, one of the report's authors, who revealed some interesting methodological assumptions. The report estimates that there are 6.7 million illegal file-sharers in the UK. It arrived at this figure when multiplying the total number of Internet users in the UK (estimated at 40 million by the report despite the UK government putting the figure at 33.9 million) against the percentage of the population engaged in file-sharing.

As for the estimate of the piracy percentage, that comes from a 2008 survey of 1,176 UK households. The survey actually found that 11.6 % of respondents admitted to using file-sharing software, but this figure was inflated to 16.3 % to account for "under-reporting". It's not quite clear how this figure was arrived at but it was based on the assumption that some respondents would lie about their P2P use, and Mulligan claims to it be 'based on evidence'.

The differences between these figures are staggering and produce vastly different results. If the lower figures are used instead it transpires that only 3.93 million UK residents are criminals, not 7 million.

And finally...

What does this all mean? Can we actually trust the figures used by goverenemtn and industry alike? As Nate Anderson (over at Ars Technica) puts it:
"The problem isn't that such calculations are done; they can serve as useful tools for industries and even for policymakers. But problems develop when the numbers are ripped from their original, provisional context by repetition and citation, eventually taking on the force of Fact. When such "facts" end up being used to make policy, the problems are compounded."
Statistics and facts are not always what they seem. Ben Goldacre, author of Bad Science and regular Guardian writer, puts it slightly differently:
"As far as I’m concerned, everything from this industry is false, until proven otherwise”
I'm inclined to agree, but there is a worrying point to these examples of misreported statistics and over-inflated figures. If they go unchecked, they become powerful rhetorical strategies that can gain momentum and if repeated enough times in the right places can take on the appearance of 'truth' from which it is only 'common sense' to act accordingly. I wonder how successful the new PS3 actually is and are shareholders seeing the benefit of increased sales? Does this equate to more inward investment in the technology at both the software and hardware levels, and does this bring benefits to the end user? Can we really believe the figures bandied around by the recording industry and what are the consequences if they are not contested? Who benefits from being disconnected from the internet on the back of wild exaggerations?

No comments:

Post a Comment