Survey Distribution Methods—The Pros and Cons of Using Social Media, Professional Organizations, and Email Distribution List: A Case Study of Leadership Research on Small Business Owners and Their Direct Reports

Abstract

Appropriate for graduate students exploring survey distribution methodology, this case study focuses on the practical considerations of using social media, professional associations, and/or direct email solicitations to distribute web surveys. Using information from a prior pilot study which describes survey distribution methods, audiences, challenges, and outcomes, case study participants are asked to identify the pros and cons of employing one or more of the survey distribution methods suitable for a leadership study of small business owners and their direct reports. A description is presented of the actual survey distribution option chosen, the survey response rate, and useable survey data rendered as well as practical lessons learned. This case study is fitting for those considering using a quantitative cross-section design using survey methodology including several instruments where individual business owners and their associates are the units of analysis.

Learning Outcomes

By the end of this case, students should be able to

  • Recognize web survey distribution methods which are appropriate to collect data from business subjects
  • Identify the pros and cons associated with the different ways by which survey subjects are identified and solicited
  • Determine ways to increase survey participation and completion
  • Describe how to ensure the survey data include approximately equal numbers of women and men business participants

A doctoral student in organizational leadership at a State University in the East, Cecilia planned to research small business leaders and those whom they lead. In preparing for this study, Cecilia had completed a prior pilot study where she used survey methodology to research the leadership strategies of small business owners. She concluded a pilot study using web surveys and was now preparing to conduct her doctoral research. Cecilia had determined her research question, completed the literature review, and gained approval for the research methodology, including the variables and surveys to be used and the participants to be studied. Her next step was to determine how she would do the following: gain access to the small business owners and their direct reports, ensure she had an equal number of women and men business owners, distribute the survey, and encourage survey participation. This case study focuses on the practical situations Cecelia encountered, namely, how to identify, attract, and encourage small business owners and their employees to complete leadership web surveys.

Background

The doctoral student’s research focused on determining the value authentic leadership can provide to modern organizations and especially those of small businesses and its effect on associates (Avolio & Gardner, 2005; Avolio, Gardner, Walumbwa, Luthans, & May, 2004; George, 2003; Walumbwa, Avolio, Gardner, Wernsing, & Peterson, 2008). Moreover, the research focused on determining whether gender and leader identity interference acted as antecedents to authentic leadership (Hmieleski, Cole, & Baron, 2012; Jensen & Luthans, 2006a, 2006b; Peus, Wesche, Streicher, Braun, & Frey, 2012; Randolph-Seng & Gardner, 2012a). This research sought to answer the question: to what extent does small business owners’ identity interference affect their authentic leadership behaviors (Sims, Gong, & Hughes, 2017)?

Research Design

This study used a quantitative cross-sectional design using survey methodology that included several instruments. Individual business owners and their employees were the units of analysis.

Subject Characteristics

The population of this sample consisted of women and men small business owners and one or more of the owners’ direct reports in selected states in the United States. The U.S. Small Business Administration’s definition of a small business was used, as “a small business concern as one that is independently owned and operated, is organized for profit, and is not dominant in its field” (SBA.gov., U.S. Small Business Administration).

Sampling Procedures

Prior to data collection, Cecelia determined a test of fit for a power of .80 for degrees of freedom of 100 that a minimum of 154 subjects were needed, based upon power analysis for hypothesis testing (MacCallum, Browne, & Sugawara, 1996). Since this was a gender survey, she set a target of 200 business owners, or 100 women and 100 men to ensure the resulting data set would be more than sufficient. Cecelia employed a non-probability convenience sampling and asked each business owner to identify two associates, and had a goal to net 200 business owners and 400 employee associates/contractors.

Data Collection

The participants were invited to complete a web survey from the commercial Question Pro Internet website. Cecelia used the following data collection process:

  • Sent email to certified small business to receive permission from associations to contact member businesses.
  • Received permission of businesses to participate in research.
  • Received approval from her university’s Institutional Review Board (IRB).
    • Created an online survey/dedicated website
    • Created demographics questions
    • Input content scales
  • Created an introduction and direct link to website.
  • Sent three email requests asking for participation in the survey, week 1, week 2, and week 3. Also, Cecelia and an assistant made follow-up calls to businesses where the owner had completed the survey but their direct reports had not, to encourage employee survey participation.
  • Gave survey participants the choice to be entered in a drawing to receive one of several US$25 gift cards.
  • Opened website for 3 months.
  • Closed website.
Survey Instrumentation

In the web-based survey, Cecelia included a section that collected demographic data as well as content scales. The scales were on authentic leadership, job satisfaction, identity interference gender, identity, work role identity, and job performance. Identity interference and gender identity survey items wording slightly differed depending upon whether the survey was to be completed by a woman or a man. Business owner gender was used as a moderator in this study.

Survey Administration

An informed consent form was obtained from her university’s IRB and provided electronically to the participants prior to the survey. In it, participants were asked to confirm their consent to complete this survey. All survey questions were replicated using the commercial Question Pro web tool. At the end of the survey, participants were thanked for completing the survey and given the choice to forward it to direct reports or other businesses. The surveys of participants who reached the end of the survey were considered complete, and those participants who dropped out before reaching the end of the survey were considered incomplete. Business owners who completed their surveys and who also had one or more of their direct reports complete the surveys were included in the study. Excluded from the data analysis were surveys where only the owner(s) or employee(s) responded.

Research Practicalities

Cecelia explored a variety of ways to determine who to include in her sample of entrepreneurial for-profit women and men business owners and their employees. Because she was interested in having an equal number of women and men business owners, Cecelia needed to ensure that there would be sufficient number of women owners in the sample.

Web Survey Distribution

As the literature suggests, though easy to administer, web surveys, due to the impersonal nature of an Internet inquiry, result in a lower response rate than mailed surveys (Fowler, 2009). Moreover, with little control over who sees and accesses the survey, researchers who use social media to recruit respondents cannot be sure they are reaching their target audience. Ways to improve Internet survey response rates include known sponsors, providing financial incentives, and repeated contact (Fowler, 2009).

Pilot Research Study Survey Distribution

To inform her dissertation, Cecelia had earlier conducted a smaller pilot research study, a convenience sample of women and men small business owners. Her target audience needed to consist of equal number of women and men. During the pilot study, she explored a variety of means by which she could solicit research participation from small business owners via social media and professional organizations. However, citing concerns about attracting individuals to complete the survey for the “wrong reasons,” for example, the potential of winning a drawing for a nominal gift card, Cecelia’s graduate advisers precluded her from employing gift cards to encourage survey participation, thus no incentives were used during the pilot study.

Cecilia began by exploring social media and contacted the owner of a LinkedIn group of over 1,500 small business owners. She asked permission to join the LinkedIn group and inquire of its members to complete the survey. Upon gaining approval of the LinkedIn group owner, she created a post which described the survey, appealed to target subjects to complete the survey, and provided the survey link. The social site manager suggested that Cecelia should be online at all times to answer queries and actively solicit participants. This proved difficult as Cecilia worked full-time and was a full-time graduate student. Cecilia also used her personal social media accounts in Facebook, Twitter, and LinkedIn to create a survey post, and asked for individuals of the target audience to complete the survey via a web link. Though the potential was great, Cecilia did not reach her pilot threshold needed of having at least 50% of the business owners survey respondents be women. Not reaching her pilot threshold, Cecilia decided to explore professional associations with women target subjects.

Through a snowball method and a Google search, Cecelia learned of several professional associations who focused on small business owners (Brewerton & Millward, 2001). She reached out to administrators of these organizations via email and by phone to determine if she could have access to their email list and/or ask them to distribute to their members the pilot survey on her behalf. A few organizations agreed to post her survey link on their website, but none would provide her with an email list of their members. Though more individuals completed the survey, she still did not have equal number of women and men business owners.

Cecelia decided to explore another option that might gain her more completed surveys by attending a multi-state conference of women business owners with a few hundred attendees. Cecelia took off from work, and paid lodging and conference registration fees to gain access to her target audience. She received permission by the local professional association to solicit participants during the plenary session. She participated in an evening networking session where she asked individual conference attendees to complete her survey. The next day, the author sat next to the conference registration table with a bowl of chocolates, a poster of her research project and paper research surveys. Because of the difficulty of providing computers to participants and wanting to ensure the participant had actually completed the survey, Cecelia provided paper surveys. As participants approached the registration table, the researcher asked women business owners to complete a survey on their leadership practices. At a nearby dedicated table, the researched had paper surveys, pencils and a box in which to place the completed surveys. Cecilia was able to get 44 women business owners to complete the paper surveys. The face-to-face appeal enabled the author to ensure she had as many women as men who completed the survey of small business owners. After the conference, Cecilia electronically entered the survey responses into the Question Pro website. This data entry process required considerable time and effort. To ensure her data entry was accurate, she asked colleagues to compare her e-surveys to the paper completed versions.

In summary, Cecilia’s pilot study was a convenience sample, close to 900 individuals accessed survey web link from social media solicitations and postings to professional organizations’ websites (Brewerton & Millward, 2001). Of those who started the survey, 155 dropped out before completing the survey, and 145 completed the web survey for a response rate of 14.9%. The author input 44 completed paper surveys from participants from the regional women business owners conference. In total, there were 189 completed surveys from the pilot study.

Current Research Web Study Distribution

Once the pilot study was concluded, Cecelia was now able to plan her dissertation research. Cecilia’s research methodology indicated that she needed to obtain 300 completed surveys, which would include 50 women and 50 men business owners and for every owner, to have one or more of their direct reports complete the survey which in summary would include 100 business owners and more than 100 employees of the business owners. This sampling approach is an example of a non-probability convenience sample (Brewerton & Millward, 2001).

In a plan to ensure she would have a large number of participants—both owners and employees, Cecelia approached national professional associations. One association required her to supply her university’s IRB approval form and would consider providing the survey link on their website but would not provide the author with the email addresses of their members. A second association would provide access to their members but required the author to provide the association with the raw data from the survey and agree to “co-own” all data and findings with the association. A third group suggested the author attend the national conference as a vendor, pay the national vendor fee, and solicit participants from a booth but the national conference was not for another 9 months, which would significantly impact her research timeline.

In discussing her data collection situation with a fellow doctoral student who studied small business owners, Cecilia learned that there was a U.S. Federal registry of small businesses which included a list of business names, descriptions, and email addresses. Here, organizations were certified as U.S. small business, minority business, woman business, and/or veteran business and listed by state. The Ohio registry, where the author lived, had approximately 1,000 businesses where as a large state like California had over 19,000 businesses. Cecilia needed to determine which of these survey data collection approaches—professional organizations, social media, and email lists—would be the most appropriate target survey audience.

Method in Action

Based on her pilot experience, Cecelia decided to forgo use of social media and professional associations and focus on the U.S. Federal registry of small businesses. E-mail addresses were obtained for small businesses from the U.S. Federal registry. All businesses with a valid email address were contacted. In a desire to increase survey participation and completion, research participants were given the choice to be entered in a weekly drawing to receive one of several US$25 gift cards to Panera. Three email requests were sent asking for survey participation in week 1, week 2, and week 3.

Each small business received an email with a survey link. The owners were asked to forward the survey link to two employees of their choice, and also had the ability to return the survey with the names and email addresses of the direct reports they would like Cecelia to survey. In these instances, Cecelia forwarded the survey link via email to the associates. In addition, because many small businesses owners direct the work of others in their organization but may choose not to have employees, those with whom the owner(s) provided income as reported on a 1099 U.S. Tax Form were also considered associates and were asked to participate in the study.

The study used QuestionPro, a commercial Internet survey provider. The website was opened for 3 months. In practice, Cecelia learned that when surveys were distributed in bulk, for example, 200–500 emails at a time, Internet providers, for example, Yahoo, Google, and so on, consider such emails as spam. After receiving error messages, Cecelia sent surveys in small batches of 100s. In addition, many organizations had firewalls preventing unknown emails addresses from reaching their mailboxes. Thus, many emails with the web survey were blocked. In addition, Cecelia received requests asking to be “unsubscribed” from the mailing.

Initially, Cecelia was going to solicit small business owners from only the state of Ohio, her home state. The U.S. Federal registry of Ohio had approximately 1,000 small business owners. However, based on prior returns rate of 5%–7%, generating responses from 50 to 70 small business owners would be much smaller than the minimum 100 owners sought. Therefore, the study was broadened to Maryland, which had close to 6,000 federally registered small businesses. A return rate of 5%–7% for Maryland would have theoretically netted 300–420 owners. Moreover, because the research was branded as being from one of the University of Maryland higher education institutions, it might be more likely those state businesses would agree to participate in a known state university’s research study. Thinking she would perform a random sample, one of every six organizations on the email list, the author started distributing the web surveys in weekly batches of 1,000. In practice, so few responses were received from Ohio and Maryland, all 7,000 registrants were sent web surveys. With still not enough web surveys returned, the author added California with their 19,000 registered small businesses to the study.

Practical Lessons Learned

The author initially planned to conduct a random sample. In practice due to the low number of responses, the study became a census of 26,000 U.S. federal registered businesses from the states of Ohio, Maryland, and California.

To further survey participation, Cecelia and an assistant reached out by email and phone to employees of owners who had completed the survey and had provided names of associates, but they were not able to entice many more employees to complete the survey. When we reached out by phone and could contact a named associate in an organization, some mentioned they were just too busy to answer a survey. Others agreed by phone to complete the survey but never did in practice. In talking with business owners, some were quite skeptical that the author was truly a doctoral student, and speculated she must be some other entity.

We did change the drawing based on feedback from some employees that they wanted the possibility of receiving a personal gift card versus relying upon their owner to share the gift card with their employees. However, the inducement of a gift card drawing and announcing the winners of the drawing, did not yield greater survey completion. In addition, some of the e-gift cards sent to winners were never accessed. Thus, in theory, gift card participation may be attractive but in practice did not seem to actually increase participation.

A review of survey responses resulted in further reductions in the study’s population. Individuals were deleted who did not complete one or more of the instruments, provided the same numeric values for all items, identified themselves as an associate but answered leader questions, and, identified themselves as one gender but answered the questions developed specifically for the other gender. As a result, the survey had 541 usable responses or just 2% of 26,000 who were sent emails. Reliability statistics were conducted on this broader population. Of this group, there was a subset of 155 individuals from 63 businesses where either one or more of the business’s owner’s and one or more of their associates completed the survey. The businesses’ owners were considered the leaders and their employees and/or contractors were considered associates. Besides reliability, all other statistics were based upon the matched groups of business leaders and associates that totaled 155 individuals.

Though there were many more usable responses, it was crucial that this study had an owner and at least one associate. The study yielded roughly half of the 300 target participants desired but met the minimum number of 157, actual, versus the 154, theorized from the initial power analysis for hypothesis testing.

Conclusion and Recommendations

This case study explored the different ways a doctoral student social science researcher could identify and solicit small business participants with an equal number of men and women small business owners. The options discussed included professional associations, social media, and U.S. registry of small businesses. Given the vast number of individuals and businesses who use social media, it is tempting for a researcher to consider using the Internet to solicit survey participants. In practice, due to firewalls, Internet protocols, and lack of personal relationship, the yield of participants from an Internet email list was quite low. Moreover, enticing survey participants through gift card drawings may increase the appeal of the survey but was not, on its own, found to be enough of an enticement to elicit research participation. The Internet and social media are great tools, but their use is not a research panacea. The wise researcher should ensure they have redundant ways to solicit survey participation of their target audience.

Student Discussion Questions

  • Due to Cecelia’s experience in her pilot research study in soliciting survey participants and learning of direct email list, which web survey distribution option—professional association, social media, or federal registry—would more likely yield the number of completed surveys from members of her target audience for her larger dissertation research study of business owners and their employees?
  • Based on the theoretical and achieved response rates, how many individuals would need to be included in the initial sample to yield 200 completed surveys?
  • How might Cecelia ensure that she had an equal number of women and men business owners’ responses?
  • What are the pros and cons of each survey distribution option identified?
  • For each web survey distribution option, how might Cecelia increase survey participation and completion?

Teaching Note: Case Study Discussion Questions and Responses

  • Due to Cecelia’s experience in her pilot research study in soliciting survey participants and learning of direct email list, which web survey distribution option—professional association, social media, or federal registry, would more likely yield the number of completed surveys from members of her target audience for this larger research study of business owners and their employees?

Sample Response: There is no right or wrong answer to this question. The survey distribution approach will be driven in part by the student’s timeline and resources.

Professional Associations: Based upon her pilot experience and research, attending a national professional association conference may likely yield the most responses from Cecelia’s target audience. Being personally present at the conference and establishing a rapport with individual members is more likely to generate a larger number of completed surveys with a smaller audience. However as a graduate student, Cecelia may not have the travel resources, conference registration fees, and months she may have to delay her data collection in order for her to attend the annual conference. Not to mention the back end of manually inputting the paper survey into the web tool and then chasing the associates of the business owners to gain her desired sample.

Relative to the professional association which asked her to share her raw data, because of IRB guidelines, Cecelia would not have been able to provide her data with an outside group, so Cecelia did not explore this option. Cecelia needed to work with a professional association which did not require co-ownership of her findings. Ideally, she would be well served if she could find a professional association that would provide her with their members’ email list, and agree to provide a cover letter which asked their members to complete her surveys.

For both the in person and email approach, having the organization introduce Cecelia and her research and encourage their members to participate in her study would have been ideal. Such an introduction is more likely to increase survey participation. Also for both options, Cecelia would need to expand her research timeline to follow-up with the direct reports of the owners since they were not members of the professional associations.

Social Media: Given enough time and online presence, social media could also have yielded Cecelia’s target audience. The advantage of social media is it is free and she could contact a plethora of online individuals directly and immediately. Due to the snowball nature of online responses, she would not be able to identify sufficient members of her target audience. So as not to be at the mercy of who happens to be online that day, it would behoove Cecelia to identify additional social media special interest groups who focused on small business owners, both men and women, and their direct reports. Cecelia chose NOT to go the social media route because she worked full-time and could not be online full-time over several weeks to encourage participation. If Cecelia had a longer lead time to establish a large social network of small businesses and had at the outset a significant social media presence and/or could be online all day, she might have been able to make the social media option work but she decided not to risk it.

Federal Registry: The use of federal registry was a bit of a gamble, since Cecelia had no prior experience with this email list or experience emailing individuals on mass using an Internet survey website. However, similar to the professional association approach, using a registry of small businesses who were certified women, minority and/or veteran business owners, she was more likely to have a gender diverse research sample. Unlike social media, every person she contacted from the registry would be a small business owner, thus she was theoretically more likely to reach her target sample. The use of the federal registry was free thus Cecelia incurred no cost to access the small business owners and she did not need to get the permission of anyone—the professional association, the group owners, and so on. She could also access the group immediately, without waiting for the next conference or the association to approve her request and post her survey. She did not need to share her data with anyone. Moreover, she could implement her survey immediately since she had access to email list. Thus, the registry saved Cecelia time and money. The cons of this approach are Cecelia was not known to the participants and because audience members were not familiar with her or her email address, she was likely to have a decreased survey response rate.

  • Based on the theoretical and achieved response rates, how many individuals would need to be included in the initial sample to yield 200 completed surveys?

Sample Response: Based on Cecelia’s pilot response rate of 14.9%, to generate 134 completed on line surveys Cecilia would to request 2,980 individuals to complete the survey.

  • How might Cecelia ensure that she had an equal number of women and men business owners’ responses?

Sample Response: Cecelia will need to target audience’s where there were an equal number of women and men business owners. If not feasible, then she should target women-focused business owner groups.

  • What are the pros and cons of each option identified?

Sample Response: See Response to Question 1.

  • For each web survey distribution option, how might Cecelia increase survey participation and completion?

Sample Response: See Response to Question 1.

Further Reading

Madariaga, L., Nussbaum, M., Burq, I., Marañón, F., Salazar, D., Maldonado, L., … & Naranjo, M. A. (2017). Online survey: A national study with school principals. Computers in Human Behavior, 74, 544.
McPeake, J., Bateson, M., O’NeillA. (2014). Electronic surveys: How to maximize success. Nurse Researcher, 21, 2426.
Petrovčič, A., Petrič, G., & Manfreda, K. L. (2016). The effect of email invitation elements on response rate in a web survey within an online community. Computers in Human Behavior, 56, 320329.

References

Avolio, B. J., & Gardner, W. L. (2005). Authentic leadership development: Getting to the root of positive forms of leadership. The Leadership Quarterly, 16, 315338.
Avolio, B. J., Gardner, W. L., Walumbwa, F. O., Luthans, F., & May, D. R. (2004). Unlocking the mask: A look at the process by which authentic leaders impact follower attitudes and behaviors. The Leadership Quarterly, 15, 801823.
Brewerton, P., & Millward, L. (2001). Organizational research methods. Thousand Oaks, CA: SAGE.
Fowler, F. J.Jr. (2009). Survey research methods. Thousand Oaks, CA: SAGE.
George, W. (2003). Authentic leadership: Rediscovering the secrets to creating lasting value. San Francisco, CA: Jossey-Bass.
Hmieleski, K. M., Cole, M. S., & Baron, R. A. (2012). Shared authentic leadership and new venture performance. Journal of Management, 38, 14761499.
Jensen, S. M., & Luthans, F. (2006a). Entrepreneurs as authentic leaders: Impact on employees’ attitudes. Leadership & Organization Development Journal, 27, 646666.
Jensen, S. M., & Luthans, F. (2006b). Relationship between entrepreneurs’ psychological capital and their authentic leadership. Journal of Managerial Issues, 18, 254273.
MacCallum, R. C., Browne, M. W., & Sugawara, H. M. (1996). Power analysis and determination of sample size for covariance structure modeling. Psychological Methods, 1, 130149.
Peus, C., Wesche, J. S., Streicher, B., Braun, S., & Frey, D. (2012). Authentic leadership: An empirical test of its antecedents, consequences, and mediating mechanisms. Journal of Business Ethics, 107, 331348.
Randolph-Seng, B., & Gardner, W. L. (2012). Validating measures of leader authenticity: Relationships between implicit/explicit self-esteem, situational cues, and leader authenticity. Journal of Leadership & Organizational Studies, 20, 214231.
SBA.Gov and U.S. Small Business Administration. (2012). What is SBA’s definition of a small business?Retrieved from http://www.sba.gov/content/what-sbas-definition-small-business-concern (accessed November 10, 2013).
Sims, C. M., Gong, T., & Hughes, C. (2017). Linking leader and gender identities to authentic leadership in small businesses. Gender in Management: An International Journal, 32, 318329. doi:10.1108/GM-06-2016-0121
Van Dierendonck, D., & Nuijten, I. (2011). The servant leadership survey: Development and validation of a multidimensional measure. Journal of Business and Psychology, 26, 249267.
Walumbwa, F. O., Avolio, B. J., Gardner, W. L., Wernsing, T. A., & Peterson, S. J. (2008). Authentic leadership: Development and validation of a theory-based measure. Journal of Management, 34, 89126.
locked icon

Sign in to access this content

Get a 30 day FREE TRIAL

  • Watch videos from a variety of sources bringing classroom topics to life
  • Read modern, diverse business cases
  • Explore hundreds of books and reference titles