Skip to main content
Search form
  • 00:10

    MARIO CALLEGARO: Hello, I'm Mario Callegaro.I'm a survey research scientist at Google, London[Mario Callegaro, Survey Research Scientist]I started my career with a BA in sociology.And then I was really excited about the methods,the question of design, and data collection methods.And I went to the University of Nebraskain Lincoln to do a master's and then aPh.D. in survey research.After that, I started working in a company called

  • 00:33

    MARIO CALLEGARO [continued]: Knowledge Networks.And then I was stolen by Google via LinkedIn, if you wish,which is true by the way.And now, since then, I still work at Google.I work on different projects, mostly survey related.I do a lot of consulting.I do work on our products, measuring our productsatisfaction, for example, and mostly using web surveys,

  • 00:56

    MARIO CALLEGARO [continued]: as you can imagine.I started my career in telephone interviews.I was running a small computer assisted telephoneinterviewing lab.And I was always passionate about data collection methods.So I moved from telephone to web and wrote a bookon online panels.And the latest book I wrote is actually with Sageand is called Web Survey Methodology, written

  • 01:19

    MARIO CALLEGARO [continued]: with two professors of the University of Ljubljanain Slovenia, who are actually the founder of a website, which is the best place, I guess,to find the latest and greatest about web surveys.[What are web surveys?]Web surveys are surveys conducted online,

  • 01:40

    MARIO CALLEGARO [continued]: where a respondent can answer an online questionnairewith different devices.Can be a laptop, a desktop, or smartphone,for example, which is actually becoming more and more common.So web surveys are actually a self-administered method.There is no interviewer and everythingis on the respondent, who is going through a questionnaireand answering, using any kind of device you can imagine,

  • 02:00

    MARIO CALLEGARO [continued]: just connected to the internet, of course.And so their responses are sent via the internetto a central server.[What are some typical applications of web surveys?]A typical application of web surveysare, for example, something we sayin our book, web surveys of highly covered population.

  • 02:20

    MARIO CALLEGARO [continued]: What does it mean?It means that people who are online.For example, think about students in a university.It's very common to do something like a class evaluation.So you have all the email of the students.You can send them an email.And they can do a survey.Or an employee satisfaction survey,where all the employees are reachable via an online survey.That's highly covered population.Or maybe the latest example is actually

  • 02:44

    MARIO CALLEGARO [continued]: an organization, an association of professional members,where you generally have a contact, which is an email.And you can send them a web survey.That's something very common.The most common case for a regular,say, Joe to do a web survey is a customer satisfaction survey.How many times do you go to a restaurant,to a hotel, wherever, even to your car shop, and youget a survey that asks you, are you

  • 03:06

    MARIO CALLEGARO [continued]: satisfied with the service and something along those nature.So those are the most common surveys that you can encounter.Then there are other cases, somethingwith say general population web surveys, whichare more difficult to do because not everybody is online.Then you have other cases where you can do a lot of experimentsonline.So psychologists use a lot of web surveys to do testing.

  • 03:29

    MARIO CALLEGARO [continued]: And finally, web surveys are very common in online panels.Online panels are panels where generally companies recruita population.They have contact information on this population.And generally you receive an email invitation.And with that you do a web survey.So those are very common examples, applications,of web surveys, which are becoming,

  • 03:50

    MARIO CALLEGARO [continued]: as you can imagine-- it cuts across different disciplines.Anybody can do a web survey.So it doesn't mean that you need to be a survey methodologistto do a web survey.But that's the very common examplesof application web surveys.[What are some of the advantages of doing a web survey?]There are many advantages of web surveys.The first one I think that people think about is cost.

  • 04:13

    MARIO CALLEGARO [continued]: The cost is actually the lowest cost you can get.For example, if you send-- think about a mail survey.You send 1,000 mail questionnaires.And the cost is proportional to how many you send.So it goes this way.You can send one email invitation or 1,000 emailinvitations, and the cost is flat.It's pretty much them same.So cost is something.Speed, very fast, in a few days you can get all your answers.

  • 04:36

    MARIO CALLEGARO [continued]: Another one, which we love is, we call ittime and geographic flexibility, whichmeans that anybody can do a web survey from whereverthey are in the world.You just need a server somewhere.It doesn't mean to be in your own county.It can be anywhere.And you can do a web survey.Very Important for cross-national surveys.So you program the questionnaire once.You do all your logic.And then you add the different translations.

  • 04:58

    MARIO CALLEGARO [continued]: And everybody can see the question exactlyin the same way.Multimedia, huge.You can show picture.You can show a video.You can have audio.And people can react to the stimuli.So it's very, very powerful, something you cannot doin a telephone interview.How can you do that, for example?Or maybe in a face-to-face you can do it,but it's definitely more expensive.

  • 05:19

    MARIO CALLEGARO [continued]: Other advantages is actually the pace.The respondent can answer at their own pace.There's no pressure.Especially in a telephone interviewit's very difficult to handle silences.So the respondent, they're feeling some pressureto answer.In a web survey, you do it at your own pace,which can be an advantage and a disadvantageas well because you can get sidetracked and do

  • 05:39

    MARIO CALLEGARO [continued]: something else on your computer, for example.But those are actually good advantages of web surveys.That's the reason, I think, why theyare very popular, especially the speed of data collection.It makes it very appealing to make quick decisions.Think about internationally, you canhave data collecting in different countriesat the same time.And in a matter of a couple of weeksyou get the answers there.

  • 05:60

    MARIO CALLEGARO [continued]: [What are the limitations of web surveys?]As with every data collection methods,there are, of course, some limitations.One is actually something we call digital divides.So not everybody is online.I have a few countries in the world where everybody'sbasically 99%, 100% online, like for example, The Netherlands.But even in US, for example, we are about 85% of people.

  • 06:22

    MARIO CALLEGARO [continued]: So we need to remember that not everybody is online.Number two is actually something,if you think about web surveys, you do it by yourself.So you need to be able to read.So there is some literacy level, which is in two fold.One is reading levels.And the second one is to be able to use a device.It can be a computer.It can be a desktop, a tablet, or a laptop.

  • 06:45

    MARIO CALLEGARO [continued]: So that's something to keep in mind.The last limitation is, unfortunately, youhave lower response rates than in other modes.And that's something which is working, of course,certain methodologies.A latest methodologies was just presentedat a conference just a few days ago,shows that when you do an experiment,a randomized experiment, and you have different modes

  • 07:08

    MARIO CALLEGARO [continued]: and you compare them to web, you getat least 11 to 12 percentage points lessin terms of response rates on average than other modeslike an interviewer administered mode, like a telephone,for example, or a face-to-face interview.[How can you achieve representativeness withyour web survey sample?]Depending on your sampling frame,

  • 07:29

    MARIO CALLEGARO [continued]: you can actually have representative surveys.Think about the same example we did before, the students.So you have all your students, you can reach them.That's your target population.That's your sampling frame.So you can have a representative sample.So if you have the frame in advance, that's your best case.The other case is when you don't have a sampling frame,but you create one on the spot.

  • 07:49

    MARIO CALLEGARO [continued]: And that's something that is used a lot on websites,where you ran a survey, and you randomly select cookies.So it's still a probability sample.You randomly select cookies with--and you pop up a survey invitation.So you are representing your visitors.And that's a very representative sample.You have very low response rate.You can have some no responses, [INAUDIBLE].The problem is when you don't have a sampling frame,

  • 08:10

    MARIO CALLEGARO [continued]: and you need to create it.Now you cannot create it on the spot.You don't have it in advance.At that point, you use, for example,online panels, which can be builtin a probability or non-probability way.And that's where everything gets a little bit more complicated.So we are still studying how to work on these problems.But that's actually the only way sometimes to do research,

  • 08:31

    MARIO CALLEGARO [continued]: especially if you need to reach a very niche population.You need to screen out a lot of people.So if you're looking for a very small population,the only option you have is to use an online panel, where youask some screening questions and actuallywhoever qualifies actually enters in your survey.[What are online panels?]

  • 08:54

    MARIO CALLEGARO [continued]: You cannot actually think about web surveys and not think aboutonline panels.So web surveys and online panels, they go together.Let's have a definition of online panels.Very commonly used since 15, 20 years ago.Online panels are actually a groupof people who agrees to take part in surveys.You can build an online panel in two ways.

  • 09:16

    MARIO CALLEGARO [continued]: One, we say in a probability-based way,where you randomly select a set of people.You invite them to join a panel.And you send them surveys, can be on a weekly basisor even more, or even less.And that's how you build a panel.You can also do it in a non-probability fashion, whichis the most common way just because it's cheaper

  • 09:37

    MARIO CALLEGARO [continued]: and that's the commercial sector is going there.Where you recruit people in very different ways,banner, online advertisement, all kindof ways to bring people in the panel.And then it's very similar, as I saidbefore, then you have a contact, let's say an email address.And you send them an email invitation.And that's how you can collect data quickly.Why are we actually building online panels?

  • 09:58

    MARIO CALLEGARO [continued]: The reason is to have a pool of peoplewho is ready to answer questions quickly.They have a relationship with you,so they trust that you are the panel company sending thema survey.And that's a good way to collect data in a fast pace.So in a few days you can get an answer.The other beauty, I would say, of online panels

  • 10:18

    MARIO CALLEGARO [continued]: is if they are big enough, you can find this very nichepopulation.You can screen the panel and say I need, I don't know,mothers with kids of this age.Think about doing that by telephone.It's going to be super expensive.Many times you cannot do it.So you can find where is more targeted population, whichis actually what the commercial sector wants.If you need to target and you need to sell something

  • 10:40

    MARIO CALLEGARO [continued]: to a very niche population, that'sthe only way you can do it.So it's a very, very important uses of online panels.[Are web surveys appropriate for sensitive questions?]Sometimes people ask can we ask sensitive questionsin web surveys.Actually, the answer is absolutely yes.In many research paper's, it's been

  • 11:00

    MARIO CALLEGARO [continued]: shown that self-administered mode dovery well in terms of answering sensitive questions.The reason is because there is no interviewer present.So you feel more at ease to revealsomething that is sensitive.So think about drug users.If you do an experiment, and you ask to estimate drug users.And you have an interviewer mode versusa self-administered mode.So web surveys, generally people report more

  • 11:23

    MARIO CALLEGARO [continued]: in a self-administered mode, whichthe idea is the more, the closer to a true value,which many times you can not estimate.So absolutely, sensitive questions can be done,and should be done in a self-administered mode.Therefore web surveys they do very well.[Why are survey response rates falling?]

  • 11:45

    MARIO CALLEGARO [continued]: Response rates are falling in every mode, so telephone, mail,face-to-face, and web of course.So that's one of the disadvantages,as we discussed before, the limitations, lowerresponse rate.We thought a lot about that in our book.And there are many reasons.One is, think about the mode of contactis not very strong like an email invitation sometime

  • 12:05

    MARIO CALLEGARO [continued]: or something on a website.So they're not-- it's different than having somebodycalling you or even an interviewerknocking at your door, or you receive a package in the mail.So the contact is very difficult.Then more and more people, the young generation now,they don't even have emails.You can not contact them over email.So it's just a contact problem.Some other issues, we are thinking maybe

  • 12:25

    MARIO CALLEGARO [continued]: it's difficult to demonstrate over-surveying.We get so many surveys now.The example of customer satisfaction,now everybody wants you feedback.And so maybe it is becoming-- maybewe are victim of our own success in a way.So we need to think more about whyour response rates are falling, doing more experiments.

  • 12:46

    MARIO CALLEGARO [continued]: Incentives generally work in increasing response rates.But it's definitely one of the problems we are facingand it's more of a societal problem,I guess than a methods problem.But we can work on it.There are examples of very high response rate,even if it is a web survey.However, in general, it's a little bit lowerthan other data collection methods.

  • 13:07

    MARIO CALLEGARO [continued]: [What challenges do falling response rates pose?]The challenge of response rates are two.The first one is actually to lookat the nonrespondents in comparison to the respondents.So you need to understand if thereis any difference between the people who don't replyand the people who reply.And you can do some nonresponse studies.There are many ways to study the nonrespondent.

  • 13:29

    MARIO CALLEGARO [continued]: The second challenge is actually a problem with numbers,especially if you start from a small population.Say your entire population is 1,000 people,and there's nothing else you can do.What do you do?If you get 100 people answering, and youstart doing data analysis, and you get 30 people per set,then you have a small number problem.So it's very difficult to manage those small numbers,

  • 13:51

    MARIO CALLEGARO [continued]: those small n-size.So that's the second challenge in nonresponse rates.[What challenges are posed by changes in technology?]One of the new challenges of web surveysis that we, as researchers, we lost control on which devicethe respondents are using.It can be a tablet.It can a smartphone.It can be a desktop or a laptop.

  • 14:11

    MARIO CALLEGARO [continued]: And it's very, very complicated also from a software pointof view, to make sure that that question that youshow on a screen is actually usable across different kindof device.That's really a big challenge that we are all facing.More and more people are actuallydoing surveys on a smartphone.I got data from different companies.Up to 50% of their respondents, especially in emerging markets,

  • 14:32

    MARIO CALLEGARO [continued]: markets we call mobile only, where most people, if theygo online, they actually go online with a smartphone,are actually answering the questionnaire via a smartphone.So think about the smartphone, very small screen size,like five inches maybe.So how actually we write questions firstthat can be done, can be actually answeredon a small screen.And also, how is the technology changing, adapting so, instead

  • 14:53

    MARIO CALLEGARO [continued]: of having a mouse, which is very precise--think about a mouse, how precise-- to a finger.And you can answer properly without making mistakesinadvertently.Or answering open-ended questions.How?You don't type very well on a smartphone.So that's a big challenge, a big industry.Not only the survey research community and market researchcommunity, we talk all over.You go to conferences.Everybody is working on that topic.

  • 15:15

    MARIO CALLEGARO [continued]: The good news is that cell phones are actuallygetting bigger.So if you think about-- think about 10 years ago,the screen was like two, three inches.And now it's getting bigger.But they'll never become 10.So I think we have to settle on five, six inches.That's your real estate.Think about that's your real estate,and that's how you can format the question.

  • 15:37

    MARIO CALLEGARO [continued]: And with that, we need to move forward and understandhow we can actually have people answering questionswith this lowest common denominator in a way.[Whose work has inspired you?]In writing our book, we were inspired by different readings,

  • 15:58

    MARIO CALLEGARO [continued]: different books.I think the book we got most inspired is a book came outin 2008 by Mick Couper.It's called Designing Effective Web Surveys.And it's a pioneering book.If you see, Mick spent so much time showing and making usunderstand how just changing a visual design or somethingcan make a difference in the way that people answer.

  • 16:20

    MARIO CALLEGARO [continued]: The idea is, the questionnaire communicates in two ways.One is actually the text of the question.And then the visual design is another huge communicationpiece.And understanding that is important because web surveysare actually communicating that way.You can not just focus only on the question,but the entire design, the way you show the question,

  • 16:43

    MARIO CALLEGARO [continued]: if you use images or not, they make a difference in just-- notthat people are cheating or they are not paying attention.It's just because we are communicatingin a different way.And people take everything into account.[What developments in the field are you most excited about?]There are exciting things you can do with technology.And we can collect data in very different ways.

  • 17:04

    MARIO CALLEGARO [continued]: So web surveys is just one piece.Think about big data or internet of things,where you can ask questions without actuallyasking questions, with like passive data collection.Wearables, very exciting, they are used a lotin the medical field, where instead of having people,for example, have a diary of how many hours they sleep,you can actually trace their sleep

  • 17:24

    MARIO CALLEGARO [continued]: patterns having a wearable.So that's very good.That's very exciting.We can ask less questions, maybe more important questionson a survey and then collect the other data in a different way.And also remember, surveys are notappropriate for every topic.So we need to understand, this is a research question.Which data set do I need?Is it a survey?Is it big data?Is it administrative data?

  • 17:45

    MARIO CALLEGARO [continued]: Is it something else.And I think that our job as a survey professionalis to understand the different sources, manage them, whichis becoming more complex.So we need that kind of different training.And then use the most appropriate data sourceto answer our research questions.

Video Info

Publisher: SAGE Publications Ltd

Publication Year: 2017

Video Type:Interview

Methods: Survey research, Internet research

Keywords: practices, strategies, and tools

Segment Info

Segment Num.: 1

Persons Discussed:

Events Discussed:



Mario Callegaro discusses web survey methods and the best ways to use them. Web surveys are conducted online so the respondent can answer without an interviewer present. Callegaro discusses the advantages and disadvantages of web surveys, online panels, and response rates.

Looks like you do not have access to this content.

Mario Callegaro Discusses Web Survey Methodology

Mario Callegaro discusses web survey methods and the best ways to use them. Web surveys are conducted online so the respondent can answer without an interviewer present. Callegaro discusses the advantages and disadvantages of web surveys, online panels, and response rates.

Copy and paste the following HTML into your website