Experimentation in the Study of Political Science

BY AJISAFE, BLESSING OLUMUYIWA

                                                          Introduction

Every civilization of human existence always have underlying phenomena that mitigate, support, brace and sponsor life in their different forms and structures. These phenomena can be in form of physically observable elements like weathers, terrains etc, as well as non physical, yet observable elements like political ideologies, deviance etc. if natural scientists argue that development is all about energy consumption, social scientists can safely opine that we are developed if we are aware enough to shape our interpersonal conducts amidst the complexity of the modern society. To do this, it is pertinent that those underlying phenomena are unraveled and studied for our collective good.

For few centuries now, it is a serious intellectual business to study the ways these phenomena are studied. Social science also is not unparticipatory in this endeavour. Making causal relationship between variables has a pre-eminence in social science. Nature does not usually provide us with the observations we would need to make the precise causal comparisons that we seek

Social scientists have pursued two empirical strategies to overcome this conundrum:

Observational research and experimental research. Observational research involves a comparison

between people or entities subjected to different treatments (at least, in part, of their own

choosing).

This paper will not deal with these two strategies, but will attempt a handsome analysis on the preliminaries of experimental research design in social sciences, with good bias to political science. In other word, this paper will elucidate the core of experiment itself, a brief historical background, citing selected recent cases and the future of experimentation in political science.

                                             What is an Experiment?

In contrast to modes of research that address descriptive or interpretive questions, researchers design experiments to address causal questions. A causal question invites a comparison between two states of the world: one in which some sort of intervention is administered (a treated state; i.e., exposing a subject to a stimulus) and another in which it is not (an untreated state). The fundamental problem of causal inference arises because we cannot simultaneously observe a person or entity in its treated and untreated states (Holland 1986). An experiment therefore is a research design in which the researcher manipulates or varies an independent variable in a controlled setting in order to observe or measure the impact of the variation or manipulation on another variable called the dependent variable. Experiments are indeed the only means of ascertaining whether there is a causal relationship between two variables. The manipulation is also called treatment. The key distinguishing features of experiments are comparison, manipulation and control ( Nachmias & Nachmias, 1981).

Manipulation is the process whereby the researcher varies the level of the independent variable or administer a treatment on the experimental subjects in order to enable him observe the impact of the variation. The researcher increases or reduces the levels or intensity of a variable, changing the status of the variable. This is the key factor which separate experiments from any other designs.

Control in experiments means eliminating, reducing or isolating the impact of an extraneous independent variable whose specific impact we are not interested in at the present time. This is done so that we can specifically measure only the impact of the independent variable whose effect on the dependent variable we are interested  {ceteris paribus)

Comparison involves the researcher ascertaining whether there is a change in the dependent variable and whether the change is due to the treatment or due to some pure chance events. Every experiment implies some form of comparison (Agbonifoh &Yomere, 2007).

The Design

Why do we need experiments? We need experiments because they help to reduce the bias that can exist in less rigorous forms of observation. Experiments reduce the impact of bias by introducing standardized procedures, measures, and analyses. Important aspects of experimental design include standardization, randomization, between-subjects versus within-subject design, and experimental bias.

1. Standardization remains crucial in experimentation because it ensures that the same stimuli, procedures, responses, and variables are coded and analyzed. This reduces the likelihood that extraneous factors, of which the experimenter might not even be aware, could influence the results in decisive ways. Standardization requires that the same set of experimental procedures,

or experimental protocol, is administered in the same way to subjects across conditions; only the independent variable (or variables) of interest is manipulated. This process ensures that the data elicited from subjects are comparable and are not the result of some extraneous feature in the environment.

2. Randomization refers to the assignment of subjects to experimental conditions. Experimenters assign subjects randomly to ensure that no unrelated or spurious factors vary consistently within a given population and therefore bias the results. The idea is that background differences cancel each other out in the course of random assignment, since each individual is as likely to be placed in one condition as in another. Thus, no systematic differences in subjects can bias the results of the study. Many people unfamiliar with probability might think that random assignment is little more than alternately assigning Condition A or B as subjects walk in the door. However, the most elegant and pure way to ensure random assignment of subjects to conditions is to use a random number table to assign manipulations.

Experimental design can be between-subjects or within-subject. Typically, one experimental condition is compared to another experimental condition and then to a control condition. The control condition creates a baseline against which investigators compare the results of their manipulations. In between-subjects designs, different groups of subjects are randomly assigned to various experimental or control conditions. In within-subject designs, otherwise known as the A-B-A experimental design strategy, each person serves as his or her own control by participating across time in both control and treatment conditions. Subjects begin with a baseline (A), are then administered a treatment or manipulation condition (B), and are measured

again at baseline (A) once the treatment ends (Zimbardo&Gerrig 1996). The comparison of before and after measures on the variable of concern inform the investigator as to the impact, if any, of the treatment on the subject.

3. Placebo effects, in medicine, account for patients whose condition improves as a result of a fake treatment, such as a sugar pill. These effects can cause quite powerful changes in outcome based on an individual’s belief that the treatment will work. Control conditions are important in experimental verification precisely because they help determine the extent of placebo effects

in an experimental manipulation

.

4. Experimental bias. Although experiments seek to maximize experimenter control over the independent variables in a study, the experimental process itself can introduce potential sources of bias. Three important forms of experimental bias are expectancy effects, experimenter bias, and demand characteristics ( McDermott, 2002).

Experimentation and Social Sciences

History of experimentation in social science discipline can be dated to early time. In the modern history of the study however, all fingers usually point toward the popular Hawthorne experiment conducted between 1924 and 1933 in the US, led by Elton  Mayo. Mayo, a professor of Industrial Management at the Harvard Business School, led a landmark study of worker behaviour at Western Electric. This experiment saw the birth of the Human Relation school of thought in social science. Some of the independent variables that were manipulated were intensity of illumination, rest pause and pay. The dependent variable that was expected to vary with these independent variables was productivity. The study was designed to determine the relationship between lighting and productivity. It was initially sponsored by the National Research Council of the National Academy of Sciences. It was later supported by the Harvard Business School. After the illumination test, the relay assembly test and the bank wiring test followed.

The test challenged the prior assumptions about worker behaviour. Workers were not motivated solely by pay. The importance of individual worker attitudes on behaviour must be understood. Further, the role of the supervisor in determining productivity in determining productivity and morale was more clearly defined. Group work and behaviour were essential to organisational objectives and tied directly to efficiency and tied directly to efficiency and, thus, to corporate success. The most disturbing conclusion emphasized how little the researchers could determine about informal group behaviour and its role in industrial settings. Finally, the Hawthorne studies proved beyond certainty that there was a great deal more to be learned about human interaction in the workplace, and academic and industrial effort to understand these complex relationships.( Business Encyclopedia, 2014)

The Hawthorne experiment has been described most important social science experiment ever conducted in an industrial setting. This example however is to show that there is possibility for experimentation in social sciences and that this research design is not limited to the natural science. Other examples of experiments in social sciences are documented by literatures, but are obviously beyond the scope of this paper.

Experimentation in Political Science.

In his 1909 American Political Science Association presidential address, A. Lawrence

Lowell advised the fledgling discipline against following the model of the natural sciences: “We

are limited by the impossibility of experiment. Politics is an observational, not an experimental

science…” (Lowell 1910, in Drucker 2009). The lopsided ratio of observational to experimental studies in

political science, over the one hundred years since Lowell’s statement, arguably affirms his

assessment. The next hundred years are likely to be different. The number and influence of

experimental studies are growing rapidly as political scientists discover ways of using experimental techniques to illuminate political phenomena ( Drucker et al, 2009). The growing interest in experimentation reflects the increasing value that the discipline places on causal inference and empirically-guided theoretical refinement. Experiments facilitate causal inference through the transparency and content of their procedures, most notably the random assignment of observations (a.k.a., subjects or experimental participants) to treatment and control groups. Experiments also guide theoretical development by providing a means for pinpointing the effects of institutional rules, preference configurations, and other contextual factors that might be difficult to assess using other forms of inference. Most of all, experiments Sguide theory by providing stubborn facts – that is to say, reliable information about cause and effect that inspires and constrains theory. Experimentation in political science can be in form of laboratory studies, field experiment, field studies and even simulation studies. Usually, the crime of experimentation in politics has been seriously argued on the ground of internal and external validity. Using Campbell (1968) classification, internal validity threats are;

-          History

-          Intersession history

-          Maturation

-          Performance effects

-          Regression toward the mean

-          Subject self-selection

-          Mortality

-          Selection-maturation interaction

-          Unreliable measures 

External validity threats can include;

-          Unrepresentative subject population

-          Hawthorne effects

-          Professional subject

-          Spurious measures

-          Irrelevant measures

-          Testing interaction effects

These threats notwithstanding, it will be sterile to opine that experiments should be jettisoned in politics. The recent breakthrough in analysis and sophistication of study, has made this design to be ever promising.

In overview then, it can be observed that experiments published by established political scientists

reveal a total of 105 articles between 1926 and 2000. Only about 57 of them were published in political science journals. Many more strong articles written by political scientists, often in collaboration with economists, on political topics have been published in either psychology or economics journals. The list of 105 articles does not include those published in the now defunct Experimental Study of Politics. This journal was founded in 1971 because many believed that their experimental work had been unfairly rejected from the established political science journals (McConahay 1973); Experimental Study was created expressly to redress this difficulty. However, most of the articles published in this journal were not as experimentally sophisticated as those published in psychology journals (McConahay 1973), and the journal ceased publication in 1975.

The experimental study of politics has exploded in the past two decades. Part of that explosion takes the form of a dramatic increase in the number of published articles that use experiments. Perhaps less evident, and arguably more important, experimentalists are exploring topics that would have been unimaginable only a few years ago. Laboratory researchers have studied topics ranging from the effects of media exposure (Iyengar and Kinder 1987) to the conditions under which groups solve collective action problems (Ostrom et al. 1992), and, at times, have identified empirical anomalies that produced new theoretical insights (McKelvey and Palfrey 1992). Some survey experimenters have developed experimental techniques to measure prejudice (Kuklinski et al. 1997) and its effects on support for policies such as welfare or affirmative action (Sniderman and Piazza 1995), while others have explored the ways in which framing, information, and decision cues influence voters’ policy preferences and support for public officials (Druckman 2004; Tomz 2007). And while the initial wave of field experiments focused on the effects of campaign communications on turnout and voters’ preferences

(Eldersveld 1956; Gerber and Green 2000; Wantchekon 2003), researchers increasingly use field

experiments to study phenomena as varied as election fraud (Hyde 2009), representation (Butler

and Nickerson 2009), counterinsurgency (Lyall 2009), and interpersonal communication

(Nickerson 2008).

This overview revealed an increase in the number of published experiments over time. A single article was published in the 1920s, 5 in the 1950s, 7 in the 1970s, 42 in the 1980s, and 45 in the 1990s. (Note that the increase between the 1970s and the 1980s may have been at least partly due to the demise of the aforementioned Experimental Study of Economics journal in 1975.) Also interesting is the concentration of experiments in a small number of journals: Public Opinion Quarterly (POQ) and American Political Science Review (APSR) account for the most articles, with 21 each. These journals, with very few peripheral exceptions, were the only place that experiments were published in political science until the 1980s. Political Psychology has published 18 experimental articles since the 1980s. American Journal of Political Science (AJPS) has published 13 experiments and Journal of Politics has published 7. These five journals together account for 80 of the 105 experimental articles published by established political scientists in this period. Interestingly, the trend appears to be shifting, such that in the 1990s only 2 experimental articles were published in POQ and only 5 in APSR. AJPS captured the majority of the experimental articles published in the 1990s. Moreover, the remaining articles published in the 1990s appeared mostly in one of three additional journals [Political Behavior (PB); Journal of Risk and Uncertainty; and Games and Economic Behavior], only one of which (PB) is primarily a political science journal (McDermott, 2002). This time however, buck of the experiments rallied around voting behaviour, bargaining, games, international relations, media, leadership, legislation etc.

Generally speaking, political science has not widely explored the possibilities of experimental reseach, owning to the relatively few works over a relatively long period. We have underutilized this tool, unlike the behavioural economics and psychology. Where has our field gone methodologically, and why has this movement been away from experimentation?

There are at least four reasons why political science has not been as receptive to the use of experiments as other social sciences. First, methodology in political science has moved toward large-scale multiple regression work. There is nothing wrong with this; indeed, in behavioral economics, there is a robust subfield devoted to using experiments in concert with formal models and statistical analysis to generate, test, and develop original hypotheses. So although experimental and formal or statistical methods need not be contradictory, the topics studied using these methods tend to be orthogonal to each other. For example, whereas multiple regression tends to concentrate on large groups, experimental work often focuses on small numbers of individuals.

Second, an alternative movement in political science, which tends to eschew the large-scale regression work, focuses on cultural and social aspects of particular phenomena. Although constructivists and postmodern scholars pursue some of the same issues as social psychologists, such as status concerns and the evolution of norms, most cultural analysts remain disinclined to believe that experimental work can get at phenomena as complex and multidimensional as political institutions or cultural and social structures.

The third concern is more practical. Lack of experimental training in political science at the graduate level means that few students add experimental design to their arsenal of research tools as a matter of course. As a result, only especially motivated students will contact an experimenter in a psychology or economics department to learn the basics of experimental design and procedure. The best way to learn how to do experiments is to run experiments. When training and experience are difficult or unavailable, the concentration of experimentalists required to shift disciplinary culture and practice fails to emerge.

Last, many political scientists, unlike other social scientists, seem to believe that experimentalists expect experimental work to stand on its own, the way it does in physics, chemistry, or biology. Unlike in biology, where every aspect of a particular investigation can take place within a petri dish, most phenomena of interest to political scientists are complex and involve many different variables. Experiments can be used very effectively, as they have been in experimental economics, to provide a middle ground between theory and naturally occurring empirical data ( McDermott, 2002).

As earlier stated, political science has not really achieved quite spectacular in experimentation. This irregardless, recent studies have shown an improvement into this. Examples could be cited majorly from the West on recent studies. The next section will reference two selected cases in bargaining and deliberation.

Selected cases

 Fishkin’s Deliberate poll

The best known and arguably the most influential investigation of deliberation is Fishkin’s (1995) deliberative poll. In a deliberative poll, a probability sample of citizens is recruited and questioned about their policy views on a political issue. They are sent a balanced set of briefing materials prior to the deliberative event in order to spark some initial thinking about the issues. The representative sample is then brought to a single location for several days of intensive engagement, including small group discussion (with assignment to small groups usually done randomly), informal discussion among participants, a chance to question experts on the issue, and an opportunity to hear prominent politicians debate the issue. At the end of the event (and sometimes again several weeks or months afterward), the sample is asked again about their opinions, and researchers explore opinion change, which is presumed to be the result of the deliberative poll.

Natural Experiment

Chattopadhyay and Duflo's (2004) study of the quota system for women's political participation and the provision of public goods in India is such an example. The natural experiment was facilitated by the 73rd Amendment, which required that one-third of Village Council head positions be randomly reserved for women. Chattopadhyay and Duflo's evidence confirms that correcting unequal access to positions of representation leads to a decrease in unequal access to public goods. To begin with, the quota system was effective. In the two districts studied (West Bengal and Rajasthan), all positions of chief in local village councils (Gram Panchayats, henceforth GPs) reserved for women were, in fact, occupied by females. In turn, having a woman chief increased the involvement of women in GPs’ affairs in 738 West Bengal, but had no effect on women's participation in GPs in Rajasthan. Moreover, the increase in women's nominal representation translated into substantive representation. The study of the quota system shows that women invest more in goods that are relevant to the needs of local women: water and roads in West Bengal and water in Rajasthan. Conversely, they invest less in goods that are less relevant to the needs of women: nonformal education centers in West Bengal and roads in Rajasthan. The evidence from this study confirms that some classic claims of representative democracy, such as the relevance of rules and the identity of representatives, hold true.

These are just demonstrations to show that experimentations are not totally impossible in politics, internal and external validity irregardless.

Advantages and disadvantages of experiment

Advantages

1. Ability to derive causal inferences. “The major advantage of laboratory experiments is in itself ability to provide us with unambiguous evidence about causation” (Aronson & Carlsmith 1968). Because of the randomization of subjects and the control of the environment, experiments allow confidence regarding causal inferences about relationships among the variables of interest.

2. Experimental control. The experimenter has control over the recruitment, treatment, and measurement of subjects and variables.

3. Precise measurement. Experimenters design and implement the desired measures and ensure that they are administered consistently.

4. Ability to explore the details of process. Experiments offer the opportunity to explore phenomena of interest in great detail. Complex relationships can be broken down and investigated in smaller units in order to see which part of the process results in the differences of interest. In addition, experiments allow particular relationships to be explored in the presence or absence of other variables, so that the conditions under which certain relationships hold can be examined as well.

5. Relative economy. Although experiments may be more costly and time consuming than some research methodologies, such as formal modeling, they are certainly more economical than conducting large surveys or field experiments. Students are a relatively inexpensive and reliable subject pool, and a large number of them can be run in a semester. Experiments embedded in larger surveys (Sniderman et al. 1991, Kuklinski et al. 1997) may provide a more representative sample but would also require a great deal more time and money to administer.

Disadvantages

1. Artificial environment. Many experimental settings are artificially sterile and unrepresentative of the environments in which subjects might normally perform the behavior under study. There are at least two important aspects of this limitation. First, it might be impossible or unethical to create the desired situation within a laboratory. An experimenter could not study the effects of a life-threatening illness by causing such disease in a subject. Second, it may be very hard to simulate many phenomena of interest—an election, a war, an economic recession, and so on.

2. Unrepresentative subject pools. As noted above, subject pools may be unrepresentative of the populations of interest.

3. External validity. For political scientists, questions surrounding external validity pose the greatest concern with experimentation. What can experiments tell us about real-world political phenomena? Beyond the nature of the subject pool, this concern is at least twofold. First, in the laboratory it is difficult to replicate key conditions that operate on political actors in the real world. Subjects typically meet only for a short period and focus on a limited task. Even when money serves as a material incentive, subject engagement may be low. In the real world, actors have histories and shadows of the future with each other, they interact around many complex issues over long periods, and they have genuine strategic and material interests, goals, and incentives at stake. Can the results of a single-session experiment tell us anything about such a complicated world? Second, and related, many aspects of real-world complexity are difficult to simulate in the laboratory. Cultural norms, relationships of authority, and the multitask nature of the work itself might invalidate any results that emerge from an experiment that does not, or cannot, fully incorporate these features into the environment or manipulation (Walker 1976). In particular, subjects may behave one way in the relative freedom of an experiment, where there are no countervailing pressures acting on them, but quite another when acting within the constrained organizational or bureaucratic environments in which they work at their political jobs. Material and professional incentives can easily override more natural psychological or ethical concerns that might manifest themselves more readily in the unconstrained environment of the laboratory. Failure to mimic or incorporate these constraints into experiments, and difficulty in making these constraints realistic, might restrict the applicability of experimental results to the real political world. There are two important things to understand about external validity. First, external validity is only fully established through replication. Experiments testing the same model should be conducted on multiple populations using multiple methods in order to determine the external validity of any given realism created within the experiment than to the external trappings of similarity to real-world settings, which is referred to as mundane realism. As long at the experimental situation engages the subject in an authentic way, experimental realism has been constructed; under these circumstances, mundane realism may be nice but is hardly required to establish causality. Moreover, even if the experiment closely approximates real-world conditions, if its subjects fail to engage in an experimentally realistic way, subsequent findings are useless (McDermott, 2002).

4. Experimenter bias. Experimenter bias, including expectancy effects and demand characteristics, can limit the relevance, generability, or accuracy of certain experimental results.

                              Conclusion

Experimentation has been a long standing research tool for studies in many disciplines, even the pure science has benefitted immensely from its relevance. Unfortunately, experiments have been slower to acquire a dedicated following of practitioners in political science, mostly because of concerns about external validity. In many cases, this concern merely reflects a misunderstanding of the replication requirements necessary to establish external validity. This fatigue can rightly be addressed, not being discarding experimentation in political analysis, but by adopting multiple methods in analysis i.e combining experimentation with other methods to make inference. This will build more confidence to social/political research. With that, the years of Professor Donald Campbell’s toiling over experimentation will be deeply appreciated.

                                 References

Agbonifoh B, Yomere G 1999. “ Reseach Methodology in Management and Social Sciences”, Uniben Press, Benin.

Aronson E, Carlsmith M. 1968. Experimentation in social psychology. In The Handbook of Social Psychology, ed.GLindzey, E Aronson, Vol. 2. Reading, MA: Addison-Wesley. Rev. ed.

Butler, Daniel M., and David W. Nickerson. 2009. “How much does Constituency Opinion Affect Legislators’ Votes? Results from a Field Experiment.” Unpublished manuscript,Institution for Social and Policy Studies at Yale University.

Campbell DT. 1968. Quasi-experimental design. In International Encyclopedia of the Social Sciences, ed. DL Sills, Vol. 5. New York: Macmillan

Chattopadhyay, Raghabendra, and Esther Duflo. 2004. “Women as Policymakers: Evidence from a Randomized Policy Experiment in India.” Econometrica 72: 1409-43.

Druckman J, et al. (ed) 2009 “Cambrige Handbook of Experimental Political Science”, Cambrige.

Druckman, James N. 2004. “Political Preference Formation: Competition, Deliberation, and the (Ir)relevance of Framing Effects.” American Political Science Review 98: 671-86

 Fishkin, James S. 1995. The Voice of the People. New Haven, CT: Yale University Press.

Hyde, Susan D. 2009. The Causes and Consequences of Internationally Monitored Elections. Unpublished Book Manuscript, Yale University.

Iyengar S. 1987. Television news and citizens’ explanations of national affairs. Am. Polit. Sci. Rev. 81:815–32

Kuklinski J, Sniderman P, Knight K, Piazza T, Tetlock P, et al. 1997. Racial prejudice and attitudes toward affirmative action. Am.J. Polit. Sci. 41:402–19

Kuklinski J, Sniderman P, Knight K, Piazza T, Tetlock P, et al. 1997. Racial prejudice and attitudes toward affirmative action. Am. J. Polit. Sci. 41:402–19

Lowell, A. Lawrence. 1910. “The Physiology of Politics.” American Political Science Review 4: 1-15.

Lyall, Jason. 2009. “Does Indiscriminant Violence Incite Insurgent Attacks?” Journal of Conflict Resolution 53: 331-62

 McConahay J. 1973. Experimental research. In Handbook of Political Psychology, ed. JN Knutson, pp. 356–82. San Francisco: Jossey-Bass. 542 pp.

McDermott R. 2002. Experimentalmethods in Political Science. Annual Review of Pol Sc, 5, 31-61 

McKelvey R, Palfrey T. 1992. An experimental study of the centipede game. Econometrica 60:803–36

Nachmias D, Nachmias C 1981. “Reseach Methods in Social Sciences”, New York: St Martins Press

Nickerson, David W. 2008. “Is Voting Contagious?: Evidence from Two Field Experiments.” American Political Science Review 102: 49-57.

Ostrom, Elinor, James Walker, and Roy Gardner. 1992. “Covenants With and Without a Sword: Self-Governance is Possible.” American Political Science Review 86: 404-17.

Sniderman P, Piazza T, Tetlock P, Kendrick A. 1991. The new racism. Am J. Polit. Sci. 35:423–47

Sniderman, Paul M., and Thomas Piazza. 1995. The Scar of Race. Cambridge, MA: Harvard University Press.

Tomz, Michael. 2007. “Domestic Audience Costs in International Relations: An Experimental Approach.” International Organization 61: 821-40.

 Walker T. 1976. Microanalytic approaches to political decision-making. Am. Behav. Sci. 20:93–110

Zimbardo P, Gerrig R. 1996. Psychology and Life. New York: HarperCollins. 14th ed.