Document Analysis

Document analysis is required to inform all types of evaluation. However, even though document analysis is a common task, only a meager amount of literature has been written about this form of data collection in relation to evaluation.

The analysis of documents is useful to evaluators for a number of reasons:

• Analysis allows the gathering of new facts about a program, to understand why a program is the way it is.
• It is useful for determining the purpose or rationale of a program.
• It may help in determining the major stakeholders involved.
• It can assist in determining the history and other retrospective information about a program (and this may be the only way that such information is available).
• It can help the evaluator to see what data still need to be collected.

[Page 119]

The definition of document is wide ranging; a document may comprise existing or elicited material and be public or private. For example, one can use existing records, maps, plans, diagrams, minutes of meetings, correspondence, journals, biographies, annual reports, handbooks, guidelines, artifacts, articles, files, policies, legal documents, and prior evaluation reports. On the other hand, it may be appropriate to elicit new diaries, artifacts, records, submissions, and so on.

Most people associate “document” with textual material, but other formats can be useful, such as visual media (including existing or elicited audiotapes, photographs, films, and videotapes) and, most recently, electronic data. Consequently, evaluators are now using e-mail, multimedia, and Internet chat rooms and forums as new forms of documentation to be scrutinized or promoted for evaluation purposes.

The length of time it takes to find and analyze documents can vary from a few hours to (in the case of submissions) a few months. If time and financial resources are limited, documents may have to be sampled.

Advantages of carrying out document analysis include the following:

• The evaluator is provided with information about events prior to an evaluation.
• Document analysis may be superior to interviewing for collecting some kinds of retrospective data. For instance, dates may be more reliable from past records than from personal recall, and often those responsible for instigating a program may no longer be available.
• Documents appear more credible than other forms of data, as there is less evaluator bias in the production of such data.
• Document analysis can save time and money by providing information that would otherwise have to be discovered through original data collection and analysis.
• Documents may be available free or at little cost.
• Documents may be convenient to use.
• Documents are nonreactive.

Documents must, however, be used with caution. Problems can arise because documents can be written to make programs appear better than they are (as in the glossy brochure or company report) and so can be misleading; documents may not be clear or detailed enough; sections may be missing; inappropriate selections may have been made; there may be typographical errors, biases, or outright deception; the documents available may be quite unrepresentative; and there may be confidentiality issues. Other major problems can be associated with inaccurate record keeping, variable quality between program sites, material that is out of date, and time lag if agencies are behind with entries. Particular challenges for analysis also lie in comprehending how and why the documents were produced, determining the degree of accuracy, and understanding internal organizational texts.

...