According to Cherian George in ‘Why RSF’s Press Freedom Index is flawed – and why it works’, the man argued that the organisation’s list of ranking by countries is methodologically and conceptually flawed.
First, he states that the survey lacks ‘inter-coder reliability’. He explains this means that RSF does not have a ‘common pool of rigorously trained assessors’ to rate on the perceived ranking. That is however an unjustified statement, at least on two grounds. First, according to the notes on the methodology, RSF states that the survey was handled out to ‘journalists, researchers, jurists and human rights activists’. Given that the respondents came from diverse fields who are most likely to have some first hand direct experience in the country that they were surveyed, their expertise must surely count for something. Moreover, contrary to George’s assertion, the questions were not based on ‘perceptions of the press’ but rather stating concrete cases of aspects of press freedom violations.
For example, in the first section, on ‘PHYSICAL ATTACKS, IMPRISONMENT AND DIRECT THREATS’, respondents were asked to answer
‘Yes’ or ‘No’ to questions such as whether there were any cases of journalists:
1. Being illegally detained (without an arrest warrant, for longer than the maximum period of police custody, without a court appearance etc)?
2. Being tortured or ill-treated?
3. Being kidnapped or disappearing?
4. Fleeing the country as a result of harassment?
During this period, were there (Yes/No):
5. Armed militias or secret organisations regularly targeting journalists?
6. Journalists who had to have bodyguards or use security measures (such wearing bulletproof jackets or using bulletproof vehicles) in the course of their work?
Out of the 40 questions in the survey, only 4 questions were posed to ask respondents to ‘rate’ their views on aspects of press freedom. Most of them were based on asking respondents to state very clear examples of press freedom violations. Therefore the argument that it is a ‘national perception of press freedom index’ or ‘national press freedom monitors’ perceptions of press freedom index’ is highly suspect.
Second, he argues that if respondents in Singapore were being asked where the country’s level of press freedom is, as compared to other countries, he estimates that Singapore would leapfrog to within the 50 to 100 range. Ironically, this is a clear contradiction of his first argument on inter-code reliability. By asking respondents in Singapore to rate other country’s press freedom (in relation to Singapore), he is assuming that these people are well-informed about the state of the press freedom in other countries. It would be interesting to question how he comes to have that assumption.
Third, he claims that the survey is ‘conceptually flawed’. In this case, RSF failed to disclose how they measure the different weights of indicators in their index. He makes a guess that ‘Singapore’s ranking must mean that RSF considers the Republic’s lack of alternative media and lawsuits against foreign media to be more serious than, say, torture and kidnapping of media workers and blocking of political websites’.
Short of having RSF publicly disclose how they reach their ranking, I have to concur that George’s guess is just a wild stab at the dark. Some rankings are clearly inconsistent with his evaluation. For example, Iraq ranks 145 (lower than Singapore) but yet sees 3 journalists being killed during the period. If his assertion is true (or at least have some validity), then Iraq would have ranked in a better position than Singapore.
Moreover, in his argument that there is no ‘objective’ measure between:
(a) killing a journalist to stop a single story? or (b) giving broadcast licences to only a small number of commercial broadcasters who pump out entertainment, leading to a long-term denial of the public’s need to be informed through news and current affairs?
he assumes that such indicators are at odds with each other since prioritising one indicator therefore means a relative de-emphasis on the other. While this appears to be true, it does not mean that a conceptual problem exists. In fact, the interpretation of the results of any survey conducted is dependent on many factors, including, how the organisation may award different (or sometimes the same) weights to different measures. The point is not that these indicators cannot have different weights as Cherian argues, but how to justify them. If we are to apply his benchmarks, then surveys including Singapore’s corruption ranking by the World Bank (rated very favourably) for example, would suffer from the same conceptual problem. The problem is therefore of bias but this is a problem that plagues every survey, not just the RSF press freedom index.
Given the intractable nature of the problem, Cherian argues that it is more advisable for organisations to publish ‘detailed reports on each country without attempting to make any international comparisons’. However, he concedes that despite the flaws, the rankings worked since it gained much media attention.
Again, this last criticism is unfounded since RSF does publish comments on each individual country pertaining to the media (albeit very short pieces. To date, the description of Singapore’s state of press freedom in 2009 is still a ‘work in progress’).
Certainly, every survey is bound to have its biases (including the RSF press freedom index). To however claim that the ranking is ‘flawed’, without explicitly knowing how the results are being derived, is a harsh judgement.
Readings From A Political Duo-ble 
“Certainly, every survey is bound to have its biases (including the RSF press freedom index). To however claim that the ranking is ‘flawed’, without explicitly knowing how the results are being derived, is a harsh judgement.”
Well, it cuts both ways, doesn’t it? If we (Cherian, you, and I) are not privy to the manner in which RSF derived its results as indicated in your post, then we can’t take its Index at face value too, don’t you think?
There are several flaws in your assessment of Cherian’s piece.
First you have obviously nitpicked at his arguments, without addressing his substantive points.
Consider the first bone you picked: On the issue of “inter-coder reliability”. His point is merely that there is no benchmark by which such “press freedom” can be analysed. This is simply because the evaluation forms are given out to, as you have pointed out, “journalists, researchers, jurists and human rights activists” of each country. What would have happened is that they probably ranked each question according to what they thought, not based on a proper benchmark across different countries.
As Cherian notes: “What we can conclude from Singapore’s low ranking is merely that whoever the respondents are for Singapore have a worse opinion of the country’s press freedom than 132 other countries’ respondents have of theirs.”
You defend your position by pointing out that just 4 out of 40 questions related to one’s perception of media freedom. Then the issue has to do with weights, which I will address shortly.
Secondly, his next point about making an assessment of press freedom relative to other countries is not contradictory. This is because it is based on the same assumption that he makes his first point. That there is no proper benchmark but simply what people perceive the state of the media to be.
For instance, I am asked what the state of the public transport is here. I say it stinks because it is too crowded and I can never get a seat. I rate it 3/10. But my world view changes when the question is rephrased this way: What do you think the state of the public transport is vis-a-vis New York, Hongkong and Japan.
I may not have first hand intimate knowledge of the rail system there but I’ve heard that it is as bad if not worse than the MRT here. I would then rate Singapore’s public transport 5/10, same with the other three countries.
The issue changes because the question changes.
This issue is made more complex given the fact that there are more than 150 countries to rate. Can a person rate a country on an issue as amorphous as press freedom relative to so many others? It is close to impossible. But what would be closer to the truth would be to have a panel
But this would be fixed if we had an objective benchmark that compares between countries. How many people are there in a train in the morning. How fast does it arrive. How often does the train breakdown.
That is a proper “objective” benchmark by which one can measure the performance of the rail system, instead of simply asking for one’s opinion of the system. It is the same point Cherian is making.
Second, you posit that if Cherian was wrong on his count of guessing that RSF has placed more importance on giving broadcast licences to only a small number of commercial broadcasters who pump out entertainment, over killing journalists, Iraq should be higher than Singapore.
I don’t see how you can make that judgement. First, I do not know how Iraq runs its media. But how do you not know whether it is equally repressed or more? Are the journalists there allowed to roam free?
I quote: “The point is not that these indicators cannot have different weights as Cherian argues, but how to justify them.” In most cases, surveys are indeed biased. The question is can a survey be biased to the point that it has become flawed?
Similarly, in the case of the example of Iraq and Singapore, how is it justified that a country which had 3 journalists killed and dozens more threatened is -only- three spots below Singapore. This has certainly to do with the weighting of each indicator. But even if we assume there are many other factors involved, how is it that a country which has journalists killed is only a few spots below Singapore’s? That does little to boost its credibility as an objective ranking system for press freedom.
Put in another way, it is simply too difficult to measure press freedom of different countries. As Cherian points out, and I agree, it is enough to highlight the sins of each country and leave the reader to decide the state of the press in each country. A ranking system works because it is sexy and attracts attention.
Lastly, I find your last point illogical. What does the manner in which results are being derived have to be do with one’s perception of whether it is flawed? If it is, it is. If a car does not drive well, it does not drive well. What does me declaring that it is a flawed car have to do with whether I know how the car system works. To me, it drives badly. Similarly, the RSF ranking is to me flawed. And it is as such.
RSF released the pdf documents on the methodology and the survey that was being distributed to participants on its website. We can therefore at least look at the questions being asked. We do not have privy to the sampling size nor whom they were addressed to specifically. This is however pretty standard in most polls to the best of my knowledge. But to therefore conclude that the survey is flawed would mean that we can’t trust any surveys from any organisation. Period. I’m not sure if that’s a constructive attitude towards looking at studies. I would think it is more insightful to analyse survey results (including its biases) rather than knock it down just because one doesn’t like the findings.
As for my nitpicking, isn’t that the purpose of critiquing? What’s the point of analysing something if one doesn’t study the fine details of the criticisms? As I’ve said, I think Cherian’s assessment is harsh.
I guess we can all agree to disagree.
I’m from Argentina. The list is seriously flawed. We are positioned at 47, but we should be close to 120…we COMPLETELY LACK freedom of speech, as a matter of fact Cristina Kirchner (president) controls most of the media, and now a law will make the government in charge available to censor whatever they like, plus a 70% ownership of television channels.
We also face lack of newspaper distribution because of a thug of truckers (lead by Moyano) send by the government to impede newspapers to be handed in to magazine stands.
Also there are communist genocidals from the seventies in the department of Human Rights and the congress…47 position? YEAH,RIGHT!