Validating Party Policy Placements.

IAN BUDGE [*]

Textual analyses of party and government programmes open up exciting possibilities for the investigation of policy and operationalization of theory. This Note focuses on the validity of the resulting estimates, particularly of the massive policy time-series assembled by the Manifesto Research Group (MRG) of the European Consortium. [1] These are important not only for the policy measurements they provide for fifty post-war democracies, but also from the point of view of validating other codings of texts, especially those deriving from computerized analyses. No other validating standard is available for any but a handful of post-war elections -- certainly none other that so unambiguously measures policy preferences as opposed to actual party behaviour and which itself has been so well established by extensive use.

Some doubts have been expressed about the extent to which the one-position saliency codes typically used by the MRG can really measure the kind of policy spaces assumed by classical theories of party competition and coalition formation. [2] The Note addresses this point by

(a) showing that they do (Figures 1-4 below).

(b) Examining 'saliency and valency' assumptions to see if they provide a reasonable theoretical underpinning for such spaces

The conclusion has to be that the MRG data provide a solid basis for measuring party policy and can therefore be used substantively, to provide reliable estimates for party policy in post-war democracies; and methodologically, as a general standard for validating other measures of party and government policy.

TEXTUAL ANALYSES OF PARTY ELECTION PROGRAMMES

Parties, preferences and policies are at the heart of rational choice theories of democracy. [3] Measuring policy positions and movements is crucial to the development of such theories. This accounts for the growing interest in estimating party policy positions. [4] Most recently, such estimates have taken the form of computerised word-counts of party manifestos -- an exciting development since it opens up the prospect of rapid and easy analysis of policy pronouncements in general, not just of party ones. This will facilitate more extended studies of strategic interaction between collective and individual actors in all sorts of political contexts.

Studying texts has the advantage that they are statements of policy made at a particular place and time by a specific person or organization. They therefore avoid the problems associated with expert judgements of party positions, which confuse preferences with the actual behaviour they are designed to explain, and are ambiguous about the time period involved, the criteria used to locate parties and what exactly they are locating. [5] The same might be said of electoral perceptions of where parties stand on policy. [6]

Given that non-textual measures have their limitations, the question arises of how text-based estimates -- more acceptable in principle -- are to be validated in detail? Word-counts can be made on very different bases and thus differ considerably in their results and the 'maps' they draw of policy movements by the same parties. [7] This creates the need for a generally acceptable, known standard against which counting procedures can be checked and if possible harmonized, before they are applied to previously unanalysed material. Only in this way can we accept these extended applications as trustworthy.

Party election programmes provide the best material on which to run such checks. This is partly because, as public documents issued for a mass audience, they are written in a very straightforward way. They aim at getting certain points over, clearly and simply. Repetition is their hallmark; making policy points involves highlighting them, repeating them in slightly varied form and coming back to them in a variety of contexts. Word counts should, therefore, work better here than on more complex arguments.

Election programmes also have a special standing as the only collective policy statement that parties as such ever make. No other source represents the combined views of the party as an organization. Constitutional provisions usually specify a series of formal processes through which the programme is composed and approved -- preparation by the leadership, discussion at various levels of the organization and endorsement by a representative gathering of the party. [8] If one wants to study party policy, and not the policies advocated by internal factions or individuals inside the party, one has to study the manifesto, platform or election programme.

The main methodological advantage of such programmes is that they have already been comprehensively studied, coded and analysed in a statistical form. Their sentences [9] have been counted manually into one or other of fifty-seven inductively-derived policy categories by a large team of scholars and assistants (the Manifesto Research Group) operating over the last twenty years (1979-99, and continuing). [10] Resulting datasets cover 634 political parties in fifty countries, from the first democratic election in the post-war period onwards. [11]

Clearly the MRG dataset is the only one which currently allows for comprehensive study of party and governmental policy positions. As textual analyses multiply, however, its main use may become that of validating other approaches. At the moment these are mostly computerized word counts (with some phrases thrown in). [12] While the unit of analysis of such counts (the word or phrase) differs from the (quasi) sentences of the MRG data, the basic idea of measuring the relative emphasis of the text on various policy areas is the same. If the counts are valid, therefore, they ought to produce broadly the same results - in particular similar spatial representations - as the earlier codings. [13]

The MRG data thus seem well placed to aid the development of policy-based measures and theories over the next decade. They have been challenged, however, for generally failing to assess whether a particular reference is positively or negatively viewed by a manifesto. [14] This criticism is directed at the basic MRG procedure of counting sentences into one-position policy categories, rather than into 'pro' and 'con' categories inside each issue area. Without explicit provision for party confrontation on different positions within each issue, the criticism runs, the coding simply measures the relative emphasis that parties put on the issue. Hence the MRG codes measure emphases on, or saliency of, policies rather than policy positions as such. They are consequently …