This case study aims to introduce students to the methods and the practice of documentary research, particularly in examining documentation pertaining to a national process (in this case the budget of a developing country) to identify innovations that could meaningfully address its procedural or institutional gaps. It does so while highlighting the fact that documentary research is a core element in much social research, and yet this approach is not given sufficient credit when compared with other qualitative research methods. Using documentation from the Republic of Fiji, including constitutional documents (current and prior) as well as previous budgets, the case shows how documentary research can be deployed toward solving extant practitioner problems. In this manner, it also visits the challenges that documentary research presents and delineates some ways in which these may be addressed.
By the end of this case, students should be able to
- Consider both the merits and demerits (limitations) of documentary research, and identify research questions that would benefit from using documentary research approaches
- Understand the factors that underpin documentary research and critically analyze documentation in ways that help to address a desired research question
- Draw conclusions from documentary research that allow for the positing of solutions to existing academic or practitioner problems
- Learn to situate their work within salient practitioner areas
Documentary research is a core element underlying a great deal of social research, and yet this approach is not given sufficient credit when compared with other qualitative research methods (McCullogh, 2004; Scott, 1990; Tosh, 2002). By contrast, questionnaires, surveys, case studies, and participant observation are treated as far more important in courses on research methods (Mason, 2017; Scott, 1990).
But the underestimation of documentary research is both a conceptual shortcoming and a deficiency in the education of research methods (see also May, 2011; Tosh, 2002), partly because of a trend over the past three decades toward researchers aiming to produce their own documentation and data through various other approaches and methods (McCullogh, 2004). Furthermore, when documentary research has been used, it has often been as a supplementary tool rather than as the main research method (Mogalakwe, 2006, p. 222).
In fact, documentary research may be as good at drawing lessons and answering research questions as other methods, and is often much more cost-effective and less time-consuming as well (May & Williams, 2002; Mogalakwe, 2006). To put it simply, “documents are all around us, they are inescapable” (McCullogh, 2004, p. 2). We can revisit a simple definition of documentary research to realize this point: It is the study of artifacts whose study has an inscribed text as its central feature (Mogalakwe, 2006; Scott, 1990).
By this account, documentation such as government publications, national constitutions, official records, newspaper reports, and budget papers, among many others, can hardly be ignored when scholars attempt to contextualize the foundations of their inquiry, not to mention when they attempt to muster evidence to support their arguments (see discussions in McCullogh, 2004; Scott, 1990).
For undergraduate students, therefore, it is important that they are made aware of the power of documentary research (Padgett, 2016; Tosh, 2002), both for the purposes of informing the background to their inquiry and for mobilizing evidence in the pursuit of their arguments.
They should also realize that documentary research is a very old research method, because well-known social scientists such as Karl Marx and Emile Durkheim drew extensively on them to gain important insights into the form and function of societies (May and Williams, 2012; Mogalakwe, 2006).
With that in mind, I must briefly discuss the background of my documentary research, which was titled “A Parliamentary Budget Office (PBO) in Fiji: Scope and Possibility,” published in the Australasian Parliamentary Review, a specialist regionally focused journal in the domain of legislative studies. The publication was the product of approximately 4 months of work, using methods and facing limitations, that shall be described below.
I had previously worked as a consultant for the World Bank Institute (WBI, now restructured), a think tank based in Washington that was part of the larger World Bank Group and which had an exemplary record of promoting governance reforms in developing countries, particularly those with large reserves of natural wealth (see Chohan, 2013). My work there advocated for the creation of an institution known as a Legislative Budget Office (LBO), which would be an office of economists attached to, but independent from, a country’s parliament (Chohan & Jacobs, 2016, 2017a).
The office would thereby present regular, nonpolitical analysis to parliamentarians about matters pertaining to the budget, and it would be of value to parliamentarians because they themselves generally would not possess such expertise, given that the budget is such a complex document with so many moving parts. Undergraduate readers should know this to recognize that there are sometimes interesting practitioner origins underpinning academic pursuits.
Emerging from the aforementioned practitioner problem, this subject then became the academic research question of a doctoral thesis that I began in 2015, after completing my consulting contract with the WBI. As part of that inquiry, I wanted to explore the possibilities of roles that LBOs can play in different budgeting contexts (Chohan, 2016, 2017).
While a significant part of that doctoral thesis dealt with roles in a theoretical sense, I also wanted to deploy several practitioner examples that would go beyond mere theorization. I wanted to explore unique budgeting contexts where LBOs could play new and heretofore unexplored roles, so as to address specific needs that were particular to a national or subnational context.
From this ambition, I came to see the unique context of Fiji, an independent republic in the South Pacific, as a distinct and peculiar budgetary context in which LBO could play a role that went beyond what would be expected of it in other countries. After all, as a country, Fiji’s political and social architecture is quite exemplary and can be characterized in many ways: post-colonial, archipelagic, multilingual, biracial, primary resource based, Commonwealth, and parliamentary, among others.
Furthermore, Fiji exerts a substantial economic, political, and intellectual influence within the community of Pacific Island nations due to its comparatively large size, including a population base of nearly 1 million citizens. It has also had a “lively” history since attaining independence in 1970, with various military coups and an ethnic exodus, while still keeping many institutional legacies from the pre-independence colonial era. This is particularly reflected in its parliamentary-style budget process.
Above all, Fiji has seen tremendous strides in reform in the present decade (2010 onward), with a new constitution that explicitly articulates a commitment to better governance and accountability. In fact, it goes so far as to include “governance,” “transparency,” and “accountability” as terms in the very first article of the constitution, as well as dedicating an entire chapter to the subject (Chapter 12 of the Constitution of the Republic of Fiji).
There is therefore an exceptional impetus for accountability reforms, include those involving the budget, that make Fiji the logical choice for developing a chapter that would mobilize documentary research to highlight unique roles for LBOs, and thus help me to address a pressing research question in both the academic and practitioner literatures.
I therefore sought to incorporate a chapter in my thesis that would deal with the scope and the possibility of creating an LBO in Fiji, and have that chapter published in a regionally focused parliamentary journal (this chapter will from now on be referred to as “The Fijian paper”; see also Chohan, 2016).
Undergraduate readers might choose to consider the logical progression found in the formulation of the above-mentioned paper, as they may often struggle with the identification of a research question to pursue. They may therefore choose to approach their research problems through a logical progression that clarifies their thinking and acts as a systematic framework for inquiry.
I must also note at this juncture that my work on this case was bolstered by three important advantages:
- a highly suitable and competent supervisor,
- significant practitioner background knowledge,
- low levels of distraction.
Undergraduate readers should know this because these elements are crucial to a successful research enterprise, as they help to overcome erstwhile obstacles that present themselves in frustrating manners. All of this background helps to elucidate the context in which powerful research questions are conceived and addressed, and it also facilitates the next portion of discussion on the deployment of documentary research.
The notion of “documentation” itself is so broad that it is perhaps best to consider some nuances when discussing the forms and functions of documents as they serve in documentary research.
As an example, there is an important distinction to be made between private documentation, which is intended for a more limited audience, and public documentation, which is intended for a more widespread dissemination (Hodder, 1994). There are different ethical, interpretative, and judgmental consequences that arise from a private/public distinction in documentation (McCulloch, 2004; Tosh, 2002). One example is that public documentation may be written in milder language, to be made for fit for public consumption (Tosh, 2002), while private documentation may paint a more unadulterated picture, and may be more forthcoming in its content, although this may not always be the case.
Another important distinction should be made between documents that are intended for researchers (and produced by them), versus documentation that is produced in the course of other activities and not the express intent of researcher attention. Most documentation falls into this category and is generally the product of the normal course of affairs in society (see McCullogh, 2004; Tosh, 2002). Many researchers argue that the latter form of documentation is far more useful for the purpose of study (see examples in McCullogh, 2004; Tosh, 2002).
Beyond this, it is also important for students to appreciate the digitization of documents. Even as recently as two decades ago, most of the crucial documentation that researchers would have required would only have been available offline and thus would have required much more tedium in parsing, sifting, studying, and analyzing (May, 2011; McCullogh, 2004; Scott, 1990). This has been made much easier thanks to the systematic digitization of historical documents through various mass-scale projects, whether by governments, libraries, or private foundations—that now allow researchers ready access to older documents (Scott, 1990).
Students should also be aware of the current trend in documentary research toward unlocking the “deeper meaning” of documents, which may involve social semiotic approaches that aim to drill down to the essence of language and word use (see detail in May, 2011; McCullogh, 2014). Usually, such an approach is not necessary for social research at the undergraduate level, but engaging with it may be especially insightful if it lies at the very heart of a particular research question (May & Williams, 2002; Mogalakwe, 2006).
Finally, it was argued since the earliest works on documentary research that the documentation produced by government and private agencies was the single most important source for researchers (see Scott, 1990; but also McCullogh, 2004). In my experience, this is indeed true, as nearly if not all of the documentation that my research has depended or drawn upon has emerged from governmental activity (see Chohan, 2016). For the Fijian paper in particular, there was more than enough government documentation to help paint a rich picture of the context in which budget reforms can be implemented.
It is necessary that students recognize the problems of access to documentation. Access is a twofold problem. On one hand, there are archival problems in finding documents pertaining to a certain place or time, whether due to poor collection, retrieval, inputting, or cataloging. Documents may become lost over time, although digitization has reduced this risk to a significant extent. In the case of my Fijian research, the government budgetary documentation was readily present, mostly digitized, and relatively easy to access.
On the other hand, there are problems relating to getting permission to see documents. Vast troves of vital documents are kept away from public view under auspices such as confidential, private, sensitive, or top secret. This is a proverbial tough nut to crack, and there are ethical problems in attempting to circumvent such restrictions, which students should avoid. In fact, students should ensure that they have all the necessary ethical clearances when proceeding with any research.
Even for documents that do not have such ominous labels as “confidential” or “top secret,” there may be hurdles in the form of authorities (“gatekeepers”) who will not allow documentation to be shared. In the course of my research for the Fijian paper, I was fortunate not to have the need for encounters with any such authorities.
In addition, the most important sources of documentary information for the Fijian paper emerged from publicly available documents, including the Constitution of Fiji, the remarks of Public Accounts Committee Members, reports of the Auditor General, and speeches of key officials (although it should be noted that “speeches” are not normally treated as “documents” for the definitional purpose of documentary research; see Mogalakwe, 2006). To put things in a historical context, I also read through the budget papers of previous regimes, as well as through records of colonial-era budgets, insofar as they were available.
In sifting through these sets of documents, I was able to glean a picture of what sort of accountability and governance had previously existed, and where such mechanisms had been met with success (or otherwise). The budget papers allowed for an examination of the degree to which budgeting rigor was implemented, but I was then also able to ascertain the gap between budget formulation and budget execution (i.e., between what government had said it would do with the budget, and what it actually managed to do).
Speeches and reports helped me to glean some sense of the underlying philosophy of budget allocation. They also gave me a keen sense of a peculiar problem: Fijian institutions were motivated to promote better governance, but had been working independently so, rather than in concert, and this weakened their overall effectiveness. Being aware of the budgeting culture in other post-colonial contexts, I was able to compare and contrast Fiji with other countries as well.
An important limitation in documentary research is that the process of documentation can mislead students into assuming a neutral character on the part of the document’s creators. In truth, critical analysis of a document’s underlying attitudes, biases, perspectives, and “subjectivities” more broadly is necessary for students to pursue documentary research more maturely (see McCullogh, 2004; Tosh, 2002). In Tosh’s (2002) words, “students need to be aware of the limits placed on historical knowledge by the character of the sources and the working methods of historians” (p. xix).
Another limitation in documentary research more generally is in establishing the authenticity, reliability, and representativeness of documentation (see Scott, 1990, or Mogalakwe, 2006). Authenticity refers to whether a document is genuine, as opposed to that which might be forged, tampered with, misrepresentative, or otherwise inauthentic. Reliability refers to whether the account provided in a document can be credibly relied upon and raises questions about biases, intentional, or otherwise (Scott, 1990). Representativeness refers to whether documented information can allow for making broader inferences about a population and is sufficiently representative of a population’s social experience.
For research that draws upon official sources such as government publications, this is not necessarily a significant concern in every instance, but in other cases, it can be quite a significant consideration. Students must therefore be aware of the issues around reliability, representativeness, and authenticity pertaining to the documents they select (more detailed treatment on these can be found in May, 2011; McCullogh, 2004; Scott, 1990; Taylor, Bogdan, & DeVault, 2015).
Students should also be mindful of dependency on a single source (Hodder, 1994; May, 2011; McCullogh, 2004; Mogalakwe, 2006; Scott, 1990; Taylor et al., 2015). Often, students may find that they have no choice but to rely on a single source, and must therefore be prepared to critically assess that single source as best they can. However, in instances where multiple sources are available, students must be careful not to discard the other sources, as these can provide alternate perspectives or richer detail that can bolster the student’s analysis.
In addition, students must be constantly aware of the wider context in which documentation is produced. This means they should not just be able to interpret what a document’s meaning is in a modern-day context but also try to situate its meaning as it would have been taken in the time and place in which it was produced (May, 2011; May & Williams, 2002; Mogalakwe, 2006; Scott, 1990; Tosh, 2002).
Once the requisite data are gathered, the important task of drawing lessons and gaining insights begins. Students must demonstrate three important attributes. First, they should analyze critically the information presented to them. Second, they should look for the gaps in the information being presented, insofar as the documentation raises more questions than it does answers. Third, as stated above, they should be aware of the broader context in which such documentation is being constructed.
In the case of the Fijian paper, a significant and original contribution emerged from situating a PBO not just in terms of a budget analysis body but also as a mechanism for coordinating accountability between existing structures of government (Chohan, 2016). By this account, the PBO would help to coordinate information and hold other institutions to the accountability cycle, with respect to budget work, when it situated itself between the Public Accounts Committee, the Auditor General, and the Accountability and Transparency Commission (Chohan, 2017).
This insight was invaluable to me because it represented an innovation in the roles that an LBO can play. It needn’t just be a mechanistic budget analysis body (see also Chohan & Jacobs, 2016) but can also serve to shore up the accountability efforts of existing institutions. I had noted through documentary research, particularly speeches by officials and reports from Fijian institutions, that key institutions were motivated to promote better governance, but had been working independently so, rather than in concert, and this weakened their overall effectiveness. So, in advancing the idea of an institution that helped the existing accountability mechanisms muster a better collective effort, I had found that there was an original contribution to be made to the (still scant) literature on LBOs.
Documentation is an ever-present feature of contemporary life (McCullogh, 2004, p. 2). I suspect that, as society grows in complexity, particularly in the era of “big data,” the volume of documented information can only increase, and exponentially so. At the same time, my observation has been that the methods of cataloging and archiving that information are improving, insofar as they are being made available with greater exactitude to an ever-larger group of people. So why shouldn’t documentary research remain an important research method?
As noted earlier, part of the trend in the social sciences has been for researchers to tend toward the production of their own data and documentation, rather than relying on existing material (McCullogh, 2004). That is fine, but it should not detract from the essential fact that we have much to learn from existing documentation. In addition, documentary research has held out in the social sciences but often manifests itself as a supplementary research method (Mogalakwe, 2006). To that point, undergraduate students should recognize that documentary research may often prove to be a more cost-effective method that still yields high explanatory power and is just as capable of making significant and original contributions to the literature.
In the context of my Fiji paper, a systematic study of the budgetary documentation in the country led to a twofold insight into the value of a PBO: on one hand as a normal budget analysis body and on the other hand as a mechanism for better coordination among other accountability institutions. Subsequent to its publication, the Fijian paper was shared with several Fijian government departments that would take an interest in budget reform and the implementation of a PBO.
The paper was also converted into an encyclopedic chapter in the Global Encyclopedia of Public Administration, Public Policy, and Governance. It just goes to show how a tried-and-tested method, documentary research, long having been a pillar of social research, can continue to add substantive value to academic inquiry.
By that measure, I encourage undergraduate students to deploy documentary research methods, whether as their main or supplementary research approach. So long as they are aware of the above-mentioned caveats, they are likely to find documentary research à propos and valuable to their academic pursuits.
- What might be some advantages of using documentary research methods?
- What are some of the caveats when dealing with documentary research?
- Why is it important to have a critical approach toward documentation used in research?
- When might it be important to lay emphasis on assurance about the authenticity and/or reliability of documents?
- How might you identify academic questions that would be suitable for documentary approaches?
- What does the Fijian example tell you about using documentary research approaches?