Data Collection for Team Research: Assessing the Impact of Virtuality on Team Effectiveness

Why Teams?

If you have ever been part of an organization, you have probably been on a team or affected by a team. I am interested in studying teams1 because they are so pervasive and so important to organizations. Teams also have a big impact on the quality of our work life because they are used so widely. If we can understand what makes teams work and what keeps them from working, we have the potential to improve outcomes in organizations and the quality of work life.

Why Virtuality?

As technology and globalization have advanced rapidly, team work has changed. Conventional notions of teams meeting face to face and primarily communicating in person have been challenged by the reality that some teams never meet face to face and many teams use all sorts of mediums to communicate (e.g. in person, phone, email, instant messaging, text, video conferencing). These changes have raised important questions: What makes a team virtual? Does virtuality impact team effectiveness? My research project was designed to examine these issues.

The Usual Considerations, Plus A Few More

All of the research methods issues that must be taken into consideration when conducting a research project must also be taken into consideration when conducting research about teams, plus some additional concerns must be addressed. Researchers must make decisions about constructs, measures, models, samples, methods, and analysis, and for team researchers, these decisions may be a bit more complex. For example, when conducting empirical research, you typically need to confirm the independence of the measures, assess the reliability and validity of measures, and address power and sample size issues. When conducting team research, you must take these issues into consideration, plus determine the levels and units of analysis; ensure the congruence of constructs, measures, and data collection methods; acknowledge the lack of independence of some measures; and likely perform operations and tests to transform collected data into team data and to justify data aggregation or other operations. Thus, the cost and complexity of collecting data may be higher for team research.

Units of Analysis and Levels of Analysis

As with any research project, starting with a research question is critical. What is the purpose of the project? What phenomenon are you trying to understand better? What questions are you trying to answer? As noted, for my research project, I was interested in understanding virtuality and its impact on team effectiveness. Therefore, I had to define and assess virtuality for a team and then assess the relationship between virtuality and team effectiveness.

A critical question that team researchers must answer, early in the research design process, is what the unit or units of analysis will be for the research project (e.g. individual, team, and/or organization). Furthermore, researchers must decide on the number of levels of analysis (i.e. a single-level or multilevel study). If the individual is the unit of analysis, the researcher collects individual data, having individuals report on their experiences in a team.

If the team is the unit of analysis, the researcher may collect data via observation, key informant, team members, or some combination of these sources. If team members are the source, the researcher must get enough team members from each team to report on their collective team experience that the team can be the unit of analysis.

Both of these approaches would be a single-level study: one conducted at the individual level, the other at the team level. Another approach would be to conduct a multilevel study using both the individual and team as the units of analysis. The data collection implications are different depending on the unit(s) of analysis and the number of levels assessed. For my study, I chose the team as the unit of analysis and focused on a single level of analysis: the team.

Implications for Design and Data Collection

For a single-level study with the individual as the unit of analysis, there are no unique considerations for research design or data collection. However, this approach is used infrequently in team research because of its limitations. This approach provides a collection of individual perceptions about teams, but no insight into dynamics within or across teams or about team-level phenomenon (only individual perceptions about team-level phenomenon).

For a single-level study with the team as the unit of analysis, the study design I used, there are a number of unique research design and data collection considerations:

• the researcher must be sure to capture team-level phenomenon not just individual-level phenomenon.
• data should be collected via at least two different methods/sources.
  • typical sources include
  • observation,
  • key informant(s) (e.g. team managers, leaders, or customers),
  • team members (via personal interviews or surveys individually or by group discussion/consensus).
• if data are collected from team members, further operations and tests are typically required to translate individual responses into team variables (e.g. for shared properties, like team efficacy, individual responses may be aggregated and then tested for within-group agreement and between-group differences).
• the researcher must ensure that individual respondent records can be associated with the records of their team members and that any other data collected for a team can be associated with the appropriate team, while also maintaining anonymity and confidentiality for all respondents.

Thus, to conduct team-level analyses, data must be collected from and/or about team members, and the researcher must be able to isolate or identify the data for each team (i.e. know which respondent data record should be associated with which team). Furthermore, to avoid common method bias, data must be collected from some other source about the dependent variable of interest. The researcher must also be able to associate these data with the appropriate team. For multilevel research (e.g. both the individual and the team are the units of analysis), these same requirements apply.

For my research project, data had to be collected at the team level; the key constructs—virtuality and team effectiveness—are both team-level phenomenon. Furthermore, for the key dependent variable—team effectiveness—I needed an independent data source (i.e. other than team members). I decided to use team managers and/or customers as the independent source of information about team effectiveness because of their organizational roles and interaction with the team, they should know about the team's performance. Because I planned to collect data from individuals and then aggregate it to the team level, I knew that I would need to pay very close attention to the definition of constructs, the wording of questions, data collection, techniques for aggregation, and approaches for justifying data aggregation. I also needed to find a way to address one of the biggest data collection challenges associated with team research—how to ensure that the individual-level data collected from team members, managers, and customers can be associated with the correct team so that they can be aggregated to the team level. I designed a new data collection technique to accomplish this.

The Self-Select Network Approach to Team Data Collection

Because I did not have any funding source or special connection/relationship with a large organization, I realized that I needed to design a data collection protocol that would meet the requirements of the research project, be low cost, and have a low administrative burden for me, the researcher, and for participating organizations. My thinking was that if I could reduce the burden of participation to an organization, I could increase the probability that the organization would participate in my study. Also, based on my prior work experience in organizations, I realized that the process of identifying teams and their members is not only an administrative burden but also a less-than-perfect process. In organizations, team membership can be fluid, evolving, and fuzzy. Therefore, a lot of effort is required to generate lists of team members; these lists of team members are less than perfect, yet they become the foundation for the research and analysis.

To address these issues, I designed an innovative, respondent-centric alternative approach to data collection using email and a web-based survey. Instead of having administrators or managers identify teams and team members, I asked people, via email, to think about a team that they worked on and cared about, then create a team code name and password, share that information with their team via email, and ask them to complete the survey using that team name and password. I also asked them to invite customers and/or managers to complete a brief survey and use the team name and password so that I could connect their responses to the team. The advantage of this self-select network approach versus the conventional administrative coding approach is that the respondent defines which team(s) he or she is on and chooses which team is relevant or important enough to report on. Thus, this approach minimizes any biases, misinformation, or differences in perceptions introduced through the administrative coding process, and theoretically brings the researcher closer to the actual phenomenon or team as perceived by its members. This approach also places less of an administrative burden on the organization and the researcher, lowering the cost of the research and increasing the probability an organization will participate.

The respondent-centric or self-select network approach has some disadvantages. One potential disadvantage is the ambiguity of the process; the researcher and/or organizational leaders may be uncomfortable letting people within the organization define which teams they are on and decide what teams are important or meaningful. Respondents may be uncomfortable making such choices. Plus respondents have the added burden of creating the team code name and password and emailing their teammates and customers and/or managers. Respondents may be uncomfortable taking on this organizing work and/or resentful of the request to do it; both could lower response rates.

The Administrative Coding Method of Team Data Collection

In a typical single-level cross-sectional survey study with teams as the unit of analysis, a top-down, manager/administrator-driven process is typically used to design and implement data collection procedures. The researcher recruits organizations to participate in the study and then usually a point person, a manager or administrator, works with the researcher to identify study participants and arrange for distribution of questionnaires. Teams must be identified, team members must be defined, and some other independent source of information about the dependent variable must be obtained (e.g. feedback from team customers or managers). Then, to maintain confidentiality and anonymity for respondents, questionnaires must be coded and then distributed to the appropriate people (i.e. team A members get questionnaires coded to identify them as being on team A, team B members get questionnaires coded to identify then as being on team B, customers and managers for team A get questionnaires coded to associate them with team A). Once the individual respondent data are collected, the data must be transformed into team-level data. This approach requires a considerable amount of work to be done before any questionnaires can be distributed or data collected, making research about teams more costly and complicated.

Other Approaches to Team Data Collection

In addition to individual response surveys (administered via a self-select network or an administrative coding approach), there are other research and data collection approaches used to conduct research on teams. Qualitative methods, such as observations and personal interviews over time, may be used to collect data. See, for example, Connie Gersick's chapter in Doing Exemplary Research where she describes the research journey that led her to formulate the punctuated equilibrium model of team development.2

Other quantitative data collection methods used when conducting research about teams include using a team discussion approach and using key informants. The team discussion or consensus approach involves teams completing a questionnaire or survey as a group. A survey is given to each team, and they are asked to answer the questions via team discussion. This process may be facilitated or unfacilitated. The advantage of this approach to data collection is that there is no need to perform and justify operations on the data, such as aggregation; there is an answer to each question for each team that should represent the team's perspective. Potential disadvantages of this approach include the risk that team dynamics result in questions being answered in a way that does not truly reflect the team's thinking or situation and/or the risk that a single person on the team completes the survey without consulting the team. To address these limitations, the group discussion may be facilitated by a skilled moderator to ensure that a consensus emerges and the answers truly reflect the team's perspective. Some downsides of using a facilitator are cost, scheduling complexities, and the potential for introducing a bias into the process. Although both the individual response surveys and group discussion data collection methods have validity, the group discussion data collection approach is not widely used in the field. See Cristina Gibson and colleagues' study of different ways of collecting data from teams for a comparison of the construct validity and predictive validity of the individual aggregated and group discussion methods of data collection.

Key informants may be used to provide an independent source of information about team outcomes of interest. For example, the managers or administrators, who identify the teams and team members to be surveyed, often also provide information about the teams' performance or other outcomes of interest (e.g. customer satisfaction ratings by team).

To recap, in my research project, the research question called for a team unit of analysis, an assessment of the impact of virtuality on team effectiveness, not individual effectiveness. The number of levels of analysis could be a single level, with teams as the unit of analysis, or multilevel, with individuals and teams as the units of analysis. I decided to initially proceed with a single-level team analysis, primarily for pragmatic reasons (i.e. availability of analytical software packages and my experience with them). Therefore, I needed to collect information from team members about themselves and their team and get information about each team's effectiveness from a different or independent source (e.g. managers and/or customers). I designed and implemented a self-select network approach to data collection to reduce the administrative burden for the organization and allow teams to self-define their members, managers, and customers.

Alignment

Once the unit(s) of analysis and number of levels to be analyzed are determined based on the research question and theory, the researcher must select constructs, measures, and models that are congruent with these decisions for the data collection to be meaningful and usable. The researcher must determine what information is needed to address the research question, and then, each construct and its associated measure must be aligned theoretically and the data collection must be consistent with this alignment. For example, for a single-level study using individuals as the unit of analysis, the constructs, measures, data collection procedures, and analysis must be at the individual level. For a single-level study using teams as the unit of analysis, the constructs, measures, data collection procedures, and analysis must be at the team level. To further illustrate, if efficacy is a variable of interest, a study with individuals as the unit of analysis would use the self-efficacy construct, an individual-level construct, and use a measure of that construct that referred to the individual's beliefs about his or her capabilities to accomplish specific things (e.g. ‘your degree of confidence in your goal setting ability’). On the other hand, if teams were the focal unit of analysis in a study, then team efficacy would be the appropriate, team-level construct to use. And a team-level measure of this construct would be used to collect data using either the individual response or group discussion method of data collection (e.g. ‘when we set goals, we're sure we will achieve them’). Maintaining congruence with the level of analysis from theory to construct to measure through data collection allows researchers to generalize without committing reasoning errors (e.g. the ecological fallacy or the atomistic fallacy).

Because my research design focused on teams as the unit of analysis and only explored relationships at this level (single level), I needed to be sure that all the theory, constructs, measures, and analytic techniques used were appropriate for a single-level team analysis. For example, I hypothesized that team empowerment would moderate the relationship between geographic distance (a dimension of virtuality) and team effectiveness. So each construct and measure had to be theoretically justifiable and operationalizable at the team level (i.e. I used a measure of team empowerment, not a measure of individual empowerment).

Levels Models

For multilevel studies, researchers must also define the model or expected relationships between phenomenon at different levels of analysis. This ensures that the theory, constructs, and measures are appropriately aligned at each level and lay the foundation for effective data collection. David Chan outlines a typology of five models that describe different ways to think about the relationships between constructs in multilevel research. Katherine Klein and Steve Kozlowski describe three broad classes of models that may be used to design multilevel research.

Because my research project was a single-level study with teams as the unit of analysis, I used a single-level model. I had to make sure to use team-level theory, constructs, and measures when designing my research. I started by familiarizing myself with the theory and literature related to team effectiveness models and virtual team effectiveness models. This helped me identify key constructs, develop hypotheses about the relationships between constructs, and start to identify established team-level measures. Because I had made the mistake of using individual-level data to try to predict a team-level outcome in a prior research project and been sharply rebuked by a reviewer, I was very clear on the need to have alignment between theory, model, constructs, and measures in terms of level of analysis.

Variables

Another important consideration when designing research about teams and collecting data is the type of team variable under consideration. Katherine Klein and Steve Kozlowski, in their synthesis of the critical choices and issues researchers face when designing and conducting multilevel research, identify three types of team properties: global, shared, and configurational. Each requires different measurement and data collection approaches.

Paper or Electronic

Surveys may be distributed as paper documents or electronically using web-based platforms like SurveyMonkey or Qualtrics. The disadvantages of a paper approach include the cost of production and distribution; the environmental impacts of paper use; and the cost, time, and potential for errors associated with transferring respondent information from a paper survey to an electronic database for analysis. An advantage of a paper approach is that you do not necessarily need contact information (i.e. mailing address or email address); surveys can be administered to every physical mailbox or to everyone in a room or at a location. Online surveys, also referred to as Internet, electronic, or web-based surveys, have the advantages of being relatively low to no cost to produce and deliver; not requiring paper and thus being environmentally friendly; and having respondent data transferred directly, electronically, into an analyzable database. Some disadvantages of electronic surveys are concerns about data privacy and security, and the need for email addresses. For my research project, I distributed online surveys created using Qualtrics. I chose this medium for survey distribution for practical reasons. The cost was much lower, the logistics were simplified, there were no burdensome data entry requirements, and there were no risks of making mistakes during data transfer from the survey to a database. Having used a paper questionnaire on a previous research project, I was very aware of the pitfalls and eager to avoid them.

Timing and Sequencing

The researcher must decide when to collect data, how long to collect data, and how many times to communicate with prospective respondents. Generally, researchers want to avoid collecting data when response may be inhibited by other demands on prospective respondents' time. Therefore, cultural, social, industry, and organizational events that might interfere with respondent engagement should be taken into consideration and avoided. For example, in my study, the window between Thanksgiving and the end of the year was avoided for data collection because this is a time of year when people have many competing demands for their time and organizations may be short staffed due to holidays and vacations. Furthermore, each organization was consulted regarding the rhythms of their operations, and any particularly busy or stressful times (e.g. annual budgeting deadlines, annual reporting deadlines) were avoided for data collection.

The sequencing of data collection may depend on your research topic and survey design. If the topic is very engaging and the survey is very brief, less time and fewer reminders may be needed for effective data collection. If the topic is not that engaging and/or the survey takes more than a few minutes to complete, more time and more reminders may be needed. For my research project, the topic was moderately engaging and the survey was long (average completion time of about 20 min). Therefore, I used an initial announcement email, followed within 2–3 days by an invitation email, which was followed with two reminder emails (one 10–14 days after the initial invitation email, and one 10–14 days after the first reminder email).

Encouragement to Participate Versus Freedom to Decline

A researcher may have mixed motivations regarding communications with prospective respondents. From a career perspective, a researcher may feel pressure to be productive and deliver significant, publishable work. Therefore, the researcher typically wants to design the data collection process to be as successful and effective as possible, including encouraging participation in the study. The researcher also has professional and ethical obligations to conduct all research in conformance with the standards for human subject research. Therefore, the researcher must also ensure that respondents do not feel coerced, that they feel comfortable declining to participate. Thus, researchers must strike a balance between encouraging people to participate and making sure they feel free to decline. For my research project, every prospective respondent was told in the first paragraph of the invitation and in the introduction on the questionnaire that they did not have to participate and that they could discontinue participation at any time. Communications like this are usually required to comply with human subjects protection regulations. To encourage participation, without exerting undue influence, a few techniques were used:

• I briefly explained the reason or purpose of the study in the announcement and invitation emails (i.e. how it will help them, their organization, and the world).
• I offered each team a confidential, anonymous summary report.
• I had all the communications come from a leader in the organization, essentially conveying that this activity was authorized and approved by the organization.

Implementing a New Data Collection Method

In my case, I was introducing an innovative approach to data collection (i.e. the self-select network method vs the administrative coding method), which has both advantages and disadvantages. To be fair and to encourage organizational commitment, I offered each organization their choice of data collection procedure (administrative coding vs self-select network method). At each organization, I explained how each technique worked and reviewed the pros and cons of each approach. Each organization chose the self-select network method because it eliminated the need to identify teams and team members. From an organizational perspective, the challenges of finding someone who could identify teams and team members, who was willing to take the time to identify teams and team members, and who was willing to organize and coordinate this data collection effort posed significant obstacles to participation in the study. Essentially, management could agree that they wanted to participate in the study, but they did not want to or could not agree to invest resources in collecting the information needed to participate in the study using the administrative coding method. Consequently, they all opted to use the respondent-centric approach. Using this method, only one senior person or visible sponsor in the organization had to send emails to a master list of people in the organization, a resource to which each organization had ready access. I further simplified the process for the study sponsor by providing sample emails for distribution (an announcement email, an invitation email, and two reminder emails).

One concern associated with using this new data collection technique was sample representativeness. Did the self-select network approach to data collection result in a biased sample? To assess this issue, I compared the demographic and socioeconomic characteristics of the respondents from an organization with the characteristics of the organization as a whole and tested for any significant differences. For example, one organization gave me the gender and age distribution of their workforce. I used Chi-square tests to determine whether the frequency distribution of gender and age was significantly different between the workforce and respondents. No significant differences were found, so we were reassured that the sample was representative. Relatively high response rates also assuaged fears of non-representativeness.

Checklist for Data Collection Procedures When Conducting Research about Teams

• _______ Identify the unit or units of analysis
• _______ Determine the number of levels to be analyzed
• _______ Select an appropriate levels model
• _______ Align theory, model, constructs, and measures with units and levels of analysis
• _______ Ensure data can be associated with the appropriate team while maintaining anonymity and confidentiality
• _______ Identify independent source of data for dependent variable of interest (addresses common method bias)
• _______ Determine whether data collection procedures can be successfully implemented within budget and time constraints
• _______ Determine whether measures and data collection procedures are appropriate for each type of team variable (global, shared, configurational)