Release output from a price benchmarking exercise, particularly an international one, into the public domain, and it is a fairly sure bet that it will generate criticism from some camps about both the methodology and the accompanying results. After all, regardless of methodology, some providers and countries will appear more expensive than others, and while those that compare favourably will seldom complain, those that don't will often want to dispute the findings, and may even want to tweak the methodology so the output suits them better. At the end of the day, however, unsatisfactory results may simply be a result of high prices, or non-advantageous price structures.

That isn't to say it is wrong to challenge any benchmarking approach - after all, a benchmarking methodology can only really be considered robust if it can stand up to close inspection, and still be considered acceptable by a sufficient majority.

Almost all of Teligen's price-benchmarking services are based on OECD methodology. The reasoning behind this is clear. When Teligen began working with the OECD in 1995, to produce official OECD benchmarking output, the importance of working with a robust, credible approach was paramount. Central to the OECD approach is the concept of bottom-up baskets, which considers how consumers and businesses actually use telecoms services. The OECD and Teligen have together, created an approach where the methodology is developed through a comprehensive process that allows affected parties the opportunity to contribute and comment throughout.

Building representative baskets is no easy task, because, in reality, there is no 'typical' user. Usage patterns will vary from individual to individual, and will be heavily impacted by where they are in the world. For example, in the US, mobile pricing is often based on large 'buckets' of minutes, offered at a flat rate, in part to deal with the received party pays regime; post-paid plans proliferate, and prepaid plans are much less prevalent. On the other hand, in countries where there may be issues of affordability, prepaid plans will tend to dominate, and typical usage patterns will reflect this.

Another issue to take into consideration is how users actually select their tariffs. Users will often find themselves constrained by factors such as supplier loyalty, long term contracts that don't allow them to take advantage of the current best deal', or will simply suffer from inertia, meaning that they are not getting the best option for them - so how should these nuances be factored in?

In reality, these factors around selection only matter when users are already part way into a contract (with the exception of supplier loyalty). From a benchmarking perspective, it makes sense to assume that a consumer is buying a service from new, and seeks out the necessary information to enable them to make informed, rational decisions, as this creates as level a playing field as possible. The key is that the underlying system is able to choose from all available pricing plans - something which is central to Teligen's approach within its benchmarking services.

There are always levels of complexity that can be added around how users select tariffs, for example, by considering that consumers don't just buy the single lowest cost option, but use several different tariff plans simultaneously, as might be the case with mobile. However, when comparing across a number of countries internationally, such complexity may prove too difficult to incorporate in a sufficiently neutral way, and if not done properly, may undermine the credibility of any output. That is not to say that it can't be done, but rather, care needs to be taken, and the benefit of building in complexity needs to outweigh any adverse impact that this might have on the robustness of the methodology.

Currency conversion can also prove contentious for some. Cross country results have to be presented in a single currency. Irrespective of which currency is chosen, however, standard market rates ignore the fact that the purchasing power of a currency in one country may differ significantly from that in another - an effect we often see when we travel abroad. To deal with this, the use of purchasing power parities to 'normalize' prices is useful, particularly when measuring service costs as set by operators. Purchasing power parities (PPPs) are conversion factors that equalize the cost of goods and services across the countries that are being compared. Comparisons made using PPPs present a more accurate picture of how prices actually compare across countries as seen from a consumer point of view. Like any aspect of benchmarking, the use of PPPs may be contested, and the whole debate around how to normalize prices is one that is too broad to be covered within this paper. What is important, however, is that the issue of normalization is recognized and dealt with in a consistent and broadly accepted way.

A final point of contention around the use of baskets comes in the form of ARPU (average revenue per user). Critics of usage-based baskets often point to ARPU as a better measure of comparability across countries. Similar to the discussion around PPPs, this is a paper in its own right, however it is probably sufficient to say here that ARPU is a measure of the financial value of a user or traffic minute for each operator, and is based on the revenue and traffic reported by the operator for financial purposes. It does not necessarily reflect the prices experienced by end users. ARPU has a role of course, but one that rests squarely in the world of financial and market analysis. Baskets on the other hand, are specifically designed to measure the price level of an operator's cheapest offer, for a particular type of user, and are based on end user prices and end user behaviour.

At the end of the day, benchmarking is an art rather than a science, and like any art form, any methodology will have both its supporters and its critics. So long as it has been carefully constructed and tested, is regularly evaluated and has wide acceptance, until some magic wand is available that can suggest an alternative approach that creates a completely unbiased, fair, transparent and credible result, then our suggestion is to stick with the knitting. And in the meantime, don't set out to shoot the piano player, but maybe begin by taking a look at the underlying price levels and structures instead.