Longitudinal research is central to understanding stability and change in psychological constructs. This case uses various definitions of stability and change to investigate these temporal aspects in a psychological construct called time perspective. To investigate stability and change, a number of quantitative approaches were adopted including latent growth modeling, correlational analysis, repeated analysis of variance, and the reliable change index. Longitudinal research presents significant methodological and design challenges such as creating fit between the theoretical framework adopted in the research with the chosen longitudinal design—timing and spacing of observations, measures, the unit of analysis, the sample size, and the statistical approaches selected to answer the research questions.
By the end of this case, students should be able to
- Understand some complexities of longitudinal research
- Appreciate that a longitudinal study must demonstrate the fit among the research questions, the study design, and the appropriate statistical approaches
- Understand the importance of collecting as much data as possible and the impact of the sampling design on statistical analysis
- Understand the importance of data screening in longitudinal research
Project Overview and Context
Importance of Stability and Change
My research examined stability and change in time perspective, which is defined as “the often-nonconscious process whereby the continual flows of personal and social experiences are assigned to temporal categories, or time frames, that help to give order, coherence, and meaning to those events” (Zimbardo & Boyd, 1999, p. 1271). The authors found five time perspectives: Future, Present-Fatalistic, Present-Hedonism, Past-Positive, and Past-Negative. These time perspectives are measured using the Zimbardo Time Perspective Index (ZTPI). Time perspective is considered as a disposition or stable individual difference, suggesting that time perspective demonstrates little evidence of change. Time perspectives are important to study because they are linked to a variety of health and educational attainment outcomes, and knowing when to intervene, bolster, or reduce the impact of a particular time perspective is central to generating new insights.
When I reviewed the time perspective literature, I was struck by the predominance of cross-sectional or use of a single measurement occasion design. Yet, this literature contained themes of stability and change despite the dearth of longitudinal research which requires a minimum of three measurement occasions (Chan, 1998) or as Judith Singer and John Willet propose, the more occasions, the better. More measurement occasions provide a basis for studying more complex descriptions of stability and change.
Linda Collins (2006) suggested that longitudinal research requires a theoretical framework to connect the research questions, study design, and measures. The theoretical framework I used to ground my research was created by personality development researchers. The framework helped me to connect different understandings of stability and change, develop research questions, and implement a research design.
Stability and change require longitudinal designs because they need the passage of time to emerge. Personality development researchers Brent Roberts, Dustin Wood, and Avshalom Caspi (2008) indicated that there are five ways of assessing stability and change:
- Rank order stability,
- Mean level change,
- Individual differences in change,
- Ipsative stability, and
- Structural stability.
This typology indicates that stability and change at the group level and individual level of analysis are independent. By using the framework, I could see that the time perspective literature over-emphasized rank order stability at the cost of other understandings, which created an incomplete picture of stability and change in time perspective. The framework allows researchers to counter the erroneous assumption that stability and change found at the group level applies to the individual level. I applied these definitions to time perspective to provide a broader understanding of stability and change.
Differential or rank order stability is assessed by administering the same questionnaire over two different time periods and correlating test scores to obtain a retest correlation. Rank order stability pertains to a group level. Mean level change involves comparing group means across measurement occasions and is distinct from rank order stability. Mean level change describes the extent to which the average amount of the construct changes over time. Ipsative stability presents an individual profile, perhaps a personality profile, which may or may not demonstrate variability over time and may provide evidence for a clinical intervention. Structural stability uses confirmatory factor analysis (CFA) to compare means over time by identifying sources of spurious change arising from parameters such as factor loadings and item intercepts. Tests of structural stability avoids comparing “apples with oranges.” Finally, individual differences in change describes increases or decreases in the trait by a person over a period of time.
The typology guides testing of change and stability in time perspective using the following research questions:
- Research Question 1: Do time perspectives demonstrate differential stability?
- Research Question 2: Do time perspectives demonstrate mean level stability?
- Research Question 3: Do time perspectives demonstrate interindividual differences in intraindividual change?
- Research Question 4: Do individuals demonstrate individual change in their time perspectives?
- Research Question 5: Does the time perspective index demonstrate structural stability?
- Research Question 6: Do individuals demonstrate ipsative stability and change?
Creating Methodological Fit
Amy Edmondson and Stacey McManus (2007) suggest that research designs must demonstrate methodological fit reflecting an internal consistency of elements within the research process. Table 1 outlines the decisions I made to ensure methodological fit in the project. To answer the research questions, I chose a prospective longitudinal panel design in which the same individuals were followed across a set of measurement occasions, the same data were collected, and relationships were examined at the individual and group levels of analysis (Menard, 2002; Taris, 2000). I planned to collect data across four occasions for a period of 1 year using equally spaced intervals of 3 months. I chose a 1-year duration because a longer period would increase my financial costs and would increase the risk of participant fatigue and subsequent attrition-loss of study participants during the research which would reduce the sample size needed for statistical analysis.
|Table 1. Criteria used to support choice design and analysis decisions.
Ground study in current research agenda of stability and change
Begin the study of long-term stability and change in time perspective by starting a new conversation in the field
Objectives of longitudinal research
Examine questions of stability and change in time perspective at group and individual level of analysis
To test various forms of stability and change in time perspective and to provide support for the coexistence of stability and change
Four equally spaced measures
Latent growth model requirements and the assumption that time perspective will change slowly
Repeated analysis of variance
Provide a more rigorous study of stability and change in time perspective to support the multifaceted view
Latent growth model
Reliable change index
Time perspectives may demonstrate both stability and change at the individual level of analysis
Change over the interval at group and individual levels of analysis
Implications for theory
Advancing a longitudinal research agenda into time perspectives.
Revising our understanding of stability and change in time perspective using a broader lens
Latent growth modeling (LGM) is a longitudinal structural equation model which combines a measurement model with regression analysis to investigate stability and change. With LGM, in a panel design, the same people are followed to make comparisons across different occasions to study change and stability. In addition, LGM requires a minimum of three measurement occasions to estimate model parameters (Byrne, Lam, & Fielding, 2008). Therefore, I chose four waves of data to examine the possibilities of curvilinear trajectories in the statistical analysis and provide a richer description of the data.
Time perspective was measured using a 56-item questionnaire—the Zimbardo Time Perspective Index (ZTPI). I chose the ZTPI because it has been used in more than 105 scientific papers, it has undergone extensive revisions by the scale developers, and it is a popular measure adopted in studies. As already noted, I planned to administer surveys to respondents across four measurement occasions.
Research Practicalities: The Research Context and Methods
Ethics, Consent, and Pilot Study
Prior to survey administration, I submitted the questionnaire and research questions to a Research Ethics Committee. To retain participant interest and mitigate against attrition, I decided to offer a €500 cash incentive in the form of a draw to occur at the end of the data collection phase. The Ethics Committee did not approve the incentive because it may encourage individuals to participate in the study who might not ordinarily do so. The incentive was subsequently withdrawn. Prior to administration, the survey was pretested to identify problematic questions.
Sampling and Consent
The population of interest were Irish youth workers and Boards of Management working across a network of training organizations. The research sites were selected because existing literature using the ZTPI relied disproportionally on university students, and a longitudinal field study using working adults provides an alternative sample to examine the behavior of time perspective. Moreover, I expected the adult sample to show greater levels of stabilities because time perspective is a personality construct which changes slowly over time. The research sites were chosen to maximize attitudinal diversity to time using geographical spread (Visser & Mirabile, 2004).
A sampling frame of 39 Irish training companies containing approximately 350 employees and 160 voluntary board members, which represented the population of the training companies, was selected to participate in the study. The research sites presented a sampling design challenge because individuals completing the survey were nested within the organizations so data followed a multilevel structure. This complication led me to choose a multistage sampling design, which involves the selection of primary level units at Stage 1 (organizations) followed by sampling of lower level units (individuals) at Stage 2 (Hox, 1995). Training organizations received a letter inviting staff members to participate in the study and to return signed consent forms to me, the researcher. In Stage 2, respondents who agreed to participate were then sent a questionnaire for completion. Individual participants were then given a unique identifier to track respondent returns.
Methods Chosen to Answer the Research Questions
First, scale averages were calculated and correlated across two measurement occasions to assess rank order stability. Repeated analysis of variance compared scale means and individual change and stability drew on the reliable change index (RCI) and LGM. The LGM generates population descriptions of change and stability using the mean, variance, and covariance of two model parameters: an intercept and a slope. The LGM also shows individual variation around the average slope and intercept values. LGM requires large samples to avoid model convergence problems and out of range solutions such as negative variances and correlations exceeding 1.00.
The RCI emerged from the psychotherapy literature and indicates whether meaningful individual change has occurred by gauging the actual change resulting from an intervention against what could be expected, given the measure’s reliability (Roberts et al., 2008). To test structural stability, I planned to use CFA to ensure that all the time perspective factors were present in my data.
Methods in Action: Decisions and Trade-Offs
The section presents the practical difficulties associated with carrying out the research and shows where departures occurred from the planned design. When my research design and data collection plan encountered the research sites, some aspects of the research became impractical and more complex. For example, respondents skipped survey questions, some refused to answer other questions, people lost surveys, and as the study moved forward, participants began to drop out and loose interest in participating in the study. Inevitably, I realized that collecting more data was uneconomical and impractical.
Complexities arose around sample size, choice of measures, method variance, panel conditioning, the chosen temporal design, attrition, and the study duration.
Choice of Measures and Sample Size
Despite choosing the most appropriate measure, respondents grew tired of completing a 56-item survey which created a missing data problem. At the data collection phase, my main concern was to maintain a sufficient sample size to conduct the analysis. To maintain participant interest, I offered to provide feedback on the completed surveys using a short newsletter. Some respondents were interested, but this activity produced additional work which did not guarantee full compliance even with participants who promptly returned surveys.
Screening the Data
When data were returned, I checked for outliers, missing observations, and distributional assumptions to ensure compliance with statistical approaches and sampling design. For example, missing data on a scale can negatively affect comparisons with the literature, and violations of distributional assumptions means that statistical models like the LGM and the repeated analysis of variance will produce larger standard errors. Screening identified other problematic respondent behaviors; for example, one participant tried to return a photocopy of a previous survey to avoid completing future surveys. Other considerations such as common method variance added further complexities to the statistical analysis which relied on maintaining a large sample.
Common method variance is described as variance that arises from the method by which data were collected rather than from the constructs under study (Podsakoff, MacKenzie, Lee, & Podsakoff, 2003). Method effects may arise because of a common measurement context, a common item context, or from item characteristics themselves. To manage method variance, I used temporal separation of measures to reduce the influence of responses, transient moods, and response styles across data collection periods as advocated by Aric Rindfleisch, Malter, Ganesan, and Moorman (2008).
Another challenge posed in longitudinal designs is panel conditioning or carryover effects whereby participants become accustomed to the same survey structure and ways of answering questions which may produce socially desirable responses. To ameliorate against panel conditioning, Jan Selmer and Corinna de Leon (2002) cite Scott Menard (1991), who suggests that the structure of surveys should be altered in longitudinal studies. I subsequently altered the survey for each round by slightly jumbling the survey questions which forced participants to read and answer the questions rather than habitually circling responses. While temporal separation of measures is important for dealing with panel conditioning, temporal separation of measures must also capture the topic of interest.
Collins (2006) details the importance of the temporal design which refers to the timing and spacing of observations and there was little guidance in the literature describing how slowly or quickly time perspective changed and this posed a significant challenge.
The temporal design raises questions about duration of the longitudinal study such that it is sufficient to capture the phenomenon of interest, and it attends to the interval between measurement occasions (Collins & Graham, 2002). This feature of longitudinal research is vital because it forces the researcher to explicate their theory of change, hypothesize the functional form of change—linear or curvilinear—and to map the data collection points to this trajectory. This is the ideal approach, but practically, there may be no prior guidance presented in the extant literature. This is a perennial problem for longitudinal researchers so, I adopted an exploratory position and used a 3-month interval because time perspective is a disposition and is expected to change very slowly over time.
Practical considerations such as a desire to avoid intrusiveness (Dillman, 2000) also shaped the choice of interval adopted in the study. The choice of time interval attempted to achieve balance between answering the research questions and minimizing the risk of attrition. Bombarding respondents with content-heavy surveys is likely to irritate them and dissuade them from further participation. At the research site, it became apparent that respondents were tiring of completing the survey at Round 3 due to several issues: changing work demands, the survey was taking too much time to complete, and the research context was complicated by the demands of an organizational change program which affected the willingness of respondents to complete a fourth survey. These insights were gained through my follow-up phone calls to remind respondents at Round 3 to return surveys.
My missing data process indicated that attrition between Time 1 and Time 3 was approximately 16% of the sample indicating an increase in attrition. Each round of survey administration costs approximately US$500 which was weighed against obtaining 25 to 30 respondents. Subsequently, I decided not to collect a fourth round of data which shortened the study duration to 9 months instead of 12.
The Impact of the Sampling Design on the Analysis
The statistical techniques, save LGM, used to answer the research questions assume simple random sampling which does not hold under a multistage sampling design. The task is to decide if variance in time perspective measures arose from clustering, in which case, a multilevel analysis is required. Failure to consider this structure can lead to underestimated standard errors. To assess the data set for a multilevel structure, I used the intraclass correlation (ICC) which measures the amount of variability that can be accounted for by clustering (Hox, 2010). I computed the ICCs which were small and Barbara Byrne (2012) suggested that ICCs close to zero indicate that modeling a multilevel structure in the data is unnecessary, so I decided to proceed using the intended quantitative techniques.
Data gathered from longitudinal designs are said to be time structured and there are a variety of ways in which time is coded in LGM (Little, Bovaird, & Slegers, 2006). Time was measured using the measurement occasion adopted in the study. For example, Time 1 refers to the first measurement occasion, Time 2 to the second, and Time 3 to the third occasion.
The approach I adopted to manage non-response was based on principles of survey design, respondent tracking, and adopting a missing data strategy. Figure 1 outlines the non-response and attrition management process I followed in the research and is based on guidance from Annabel Boys and colleagues (2003). Prior to survey administration, I provided respondents with advanced notice of the data collection schedule and 2 weeks advance notice of the survey’s arrival dates. Participants were given 3 weeks to return the survey; those who failed to return the survey after this time entered the non-response process and received two reminders followed by a phone call.
Figure 1. Managing non-response and missing data.
T1–T3 refers to the measurement occasions.
Missing Data Management
I contacted the participants who did not respond after three attempts and decided to allocate them to missing data processing. These respondents were not supplied with any further surveys; however, to ascertain their reasons for withdrawal from the study, I attempted to contact them again.
To manage data flow in a longitudinal design, I implemented a data management process, shown in Figure 1, whereby respondents who failed to return the survey after the final attempt to contact them were placed in the statistical strategy for dealing with missing data. The process also prompted follow-up calls and emails to respondents to mitigate against attrition and pursue the reasons for dropout. I classified missing data according to a missing data mechanism indicating whether data are missing completely at random (MCAR), missing at random (MAR), or missing not at random (MNAR). Little’s test of MCAR showed that data were missing completely at random, indicating that missing data arose because respondents randomly omitted responses, or respondent data were missing at random, or a respondent failed return a survey. To validate the MCAR finding, I decided to use double sampling to follow up non-respondents to understand why they dropped out. Individuals who left the study did so for various reasons such as retirement, job change, maternity leave, and career break. Missing data is regarded as ignorable if the mechanism that created the missing data is either random or the reasons for missing data are given in double sampling (McKnight, McKnight, Sidani, & Figueredo, 2007).
What Did Not Go to Plan?
It is advised in any longitudinal study to start with a large sample to offset the risk of attrition (Taris, 2000). Unfortunately, in organizational research, large longitudinal samples are an exception rather than the rule. Only a sample size of 130 individuals (N = 130) out of 300 to 380 materialized from 25 out of the 39 training organizations. The final sample size prevented tests of ipsative and structural stability, method variance, and further LGM analysis. Therefore, structural stability and measurement equivalence was assumed and not demonstrated. The CFA models produced out of range solutions and negative variances arising from the insufficient sample size. To conduct CFA, I considered alternative approaches such as shortening the measurement scales using the extant literature, but those approaches led to poor scale reliabilities.
Practical Lessons Learned
Make Choices Based on Assumption of Change
First, don’t wade into a research setting and expect to find changing constructs over time with sophisticated methods, validated measures, and an arbitrarily chosen start time and end time. Choosing the stage of life or a suitable developmental stage or an emerging phenomenon in the work place will inform when to collect data. For example, if exploring personality development, young adulthood is an appropriate stage in the life span to investigate change in personality type constructs. Beware the empty interval: collecting data during an interval when nothing is happening will cost time and money for little gain.
Thinking dynamically means considering how behavior unfolds over time during the observed time interval by using a trajectory rather than a variable or correlations. Correlations and variables do not provide a pictorial description of change and stability while a trajectory does. Dynamic thinking occurs at the start of the project and is driven by theoretical considerations. For example, organizations need committed individuals, and so describing how organizational commitments unfolds helps to answer questions such as, Does employee commitment increase over the first 6 months of employment? When does it plateau? and How long does the plateau last? These descriptions of employee commitment are useful for employee recruitment and retention.
Choose an Interval and Study Duration
The challenge is to choose the interval in which something is happening. The time interval and its granularity (i.e., minutes, hours, days, weeks, months, and years) will determine findings. To understand organizational commitment of newly hired people, subjects need to participate in your study when they are hired and followed for a period of time. Data collection occurs at intervals that capture the ebb and flow of commitment change. If there is no theory of change present, then data collection decisions are based on the researcher’s belief or informed view about how fast or slow the change happens. The researcher might decide to collect data at shorter intervals, such as a day, or every 2 hr if they think change is happening quickly, whereas they might want to consider longer intervals if the occurrence of change is slow, such as once every 2 months.
Establish a Missing Data Strategy
Longitudinal research incurs missing data because respondents skip survey items, do not return the survey, or drop out. A missing data strategy is necessary before data collection. Researchers should consider the following: respondent fatigue arising from long study durations and use of lengthy surveys, tracking respondents using a unique identifier, implementing a non-response process to follow up with participants, deciding on the missing data mechanisms, and the statistical approach to handle missing data.
Choose Measures and Sample Size Wisely
During the planning phase of a study, the researcher must consider the sample size needed to answer the research questions or hypothesis. Clearly, sound measures are also needed, but if structural equation modeling is part of the methods, then a small number of well-behaved measures is preferable. Deciding on measurement scales with large numbers of items, for example, 15 to 20 items per scale, will impose larger sample size requirements.
Using widely published, reliable, and valid measures may not help to discover individual- and group-level change because reliable measures are designed to be stable across measurement occasions. If possible, use measures which are sensitive to capturing change which ideally do not have a large number of questions. For example, see the study by Omar Solinger, van Olffen, Roe, and Hofmans (2013).
Select a site that can deliver the sample needed but ensure that the research setting is not planning any event that will negatively affect your data collection schedule and response rate. Making post hoc adjustments, arising from lower response rates, to reduce a scale’s size jeopardizes a scale’s validity because the version of the scale you have created has not be tested in a separate sample.
Longitudinal research is ideally suited to studying change and stability over time and represents a valuable pursuit. Data are collected over three occasions at a minimum and the more waves, the better. Dynamic thinking prompts consideration of temporal design and choice of measures that can detect dynamic phenomenon. The choice of interval and the time scale chosen will determine the research findings and its limitations.
Exercises and Discussion Questions
- Explain the importance of dynamic thinking to longitudinal research results.
- Explain the importance of structural stability to longitudinal research findings.
- Explain the importance of time in the longitudinal research design.
- Explain the importance of data screening in longitudinal research and comment on the limitations to data screening posed by sample size.
- Missing data can simply be ignored in longitudinal research as it has no bearing on the research outcomes. Would you agree with the statement; explain your answer?
Patrick J. Curran: http://curran.web.unc.edu/
Brent Roberts: http://www.psychology.illinois.edu/people/bwrobrts
Mplus website: https://www.statmodel.com/