Skip to main content
Search form
  • 00:04

    [Survey Design- Ipsos Public Affairs]

  • 00:11

    JULIA CLARK: So in terms of next steps,I think this is exciting.And I think actually, per our other agendaaround doing some interesting polls that Ipsospublishes on our own, this could be a really useful angle.So could you have a look at anythingwe've done in the past on the issuesaround judges and even around feminism, actually,because we want to, as far as possible,recreate trends that we've already captured?

  • 00:33

    JULIA CLARK [continued]: Hi.My name is Julia Clark.I work for Ipsos Public Affairs. [Julia Clark, Senior VicePresident, Ipsos Public Affairs] My title is Senior VicePresident, although colloquially I'm oftenreferred to as just "pollster" as well,since I run the political polling here at Ipsos.

  • 00:46

    SPEAKER 3: Hi, Julia.

  • 00:46

    JULIA CLARK: How are you?

  • 00:47

    SPEAKER 3: OK.And yourself?

  • 00:48

    JULIA CLARK: Good, good.Anything on today?

  • 00:50

    SPEAKER 3: Meeting Don at 11:00.

  • 00:51

    JULIA CLARK: All right.

  • 00:51

    SPEAKER 3: I've got Kaden coming.

  • 00:52

    JULIA CLARK: Sounds great.Thanks so much.Here at Ipsos, we do a lot of analysis,a lot of report writing, a lot of consultancyin some ways for our clients, in additionto doing the sort of core of our business,which is the really robust data capture.And it varies depending on the different teams.You know, here it Ipsos, some of them

  • 01:13

    JULIA CLARK [continued]: really focus more just on high-quality data capture.My group, Ipsos Public Affairs, doesa lot of the analytical work, a lot of the reporting workand almost consultative work a little bit with our clients.And that's really exciting, right?You get to not only capture data,but you get to sort of see what it meansand help clients understand it and see what

  • 01:33

    JULIA CLARK [continued]: they're going to do with it.So sometimes that means writing a report that'spublished about a research questionthat the client has, and you're helping them answer a questionor solve a problem.Sometimes we do work for communications or PRperspective.So the client's really interested in gettingsome good media coverage out of a poll,and so we have to design it in a way

  • 01:54

    JULIA CLARK [continued]: that it is very upstanding to external scrutiny.But we then help them push it out,because not only are they our client,but it's great for our brand, it's great for their brand,and we really work to publicize that data.What we're about to do is actually-- Elizaand I, one of my colleagues, are going to sit down and get readyfor a meeting we're going to have with one of our clients.

  • 02:14

    JULIA CLARK [continued]: We're working with the American Bar Associationabout some of the work they're doingon the ICC-- the International Criminal Court.It's a really fascinating piece of research,trying to figure out what the American public actuallythink about some of the issues faced by the ICC,the court itself.And we're going to sort of just talk a little bit about someof the questions we're thinking about designing for this wave's

  • 02:37

    JULIA CLARK [continued]: research.So there's a few hot-topic issuesthat they want to get into.And Eliza and I just need to sit down and come upwith the ideas for the questionnairethat we're going to take to the client for his review.So I'm thinking that what we do is,we pivot a little bit, because wewere going to ask about the situation with the government's

  • 02:58

    JULIA CLARK [continued]: declaration about ISIS committing genocide.But since this new ruling has come downwhere they have, for the first time,categorized rape as a war crime, Ithink the ABA may want to touch on that issue instead.So maybe we try to think of a question thatasks about the latest-- the latest ruling.

  • 03:18

    SPEAKER 1: Right.I think it would be really interesting, actually,to discuss whether or not rape shouldbe considered a war crime.

  • 03:24

    JULIA CLARK: If people think that, yeah.

  • 03:25

    SPEAKER 1: Because we talk about what's considered a war crime.Earlier on in the questionnaire, we discuss mass atrocities,things like this, and I think whether rapeshould be considered a war crime wouldbe an entirely different angle as a way to lookat war crimes in general.

  • 03:38

    JULIA CLARK: Survey design is actually fascinating,because survey design impacts fundamentallythe responses that you get.A bad survey design can take you down a pathof creating an answer somebody wants to hearthat maybe isn't as accurate.A good survey design can really sort of crack

  • 03:59

    JULIA CLARK [continued]: open Pandora's box.You can find answers to some of the most fundamental questions.And the design itself is so critical,because it guides the entire survey process.If you have a question you want to answer--what do people think about, I don't know, frogs?You can approach via two ways.

  • 04:20

    JULIA CLARK [continued]: You can approach it in a more qualitative way,where you're asking the whys.Why do people love frogs?What do they love about them?Or you can approach it from a more quantitative or numericperspective.How many people love frogs?How many people don't love frogs?It's a silly example.But what it does is sort of give you two different pathwaysthat you can take.And then along each of those paths--

  • 04:40

    JULIA CLARK [continued]: and I specialize more in quantitative workthan the qualitative, but I do do some of both.If we take the more numeric path,There's a lot of different ways to capture that data,and all of those things drive the design and methodology.Should we capture the information online?On the phone?Should we ask everyone or just people,

  • 04:60

    JULIA CLARK [continued]: in this example, who live near, you know, populations of frogs?So there's a lot of different ways to go about it,and the way that you design researchreally does dictate the types of outcomes that you get.And so good design and a sort of basisin true methodological fundamentalsis absolutely critical, but I would also

  • 05:23

    JULIA CLARK [continued]: caveat that by saying you have to be creative, too.You have to adapt to modern technology.You can't be so entrenched in traditional methodologicalvehicles that you're sort of missingthe forest for the trees, right, and that you'refailing to embrace online technologies or newer

  • 05:44

    JULIA CLARK [continued]: ways of capturing data or newer technologies--social, mobile, all these things-- which enable peopleto express their selves a little more clearly or a little morequickly or cleanly.And so I would say, sort of, whenI think about survey design and research design,I think about the fundamentals.I think about sampling and a highly representative sample.

  • 06:08

    JULIA CLARK [continued]: I think about robust design.I think about all of the things that I learned when I studiedand that I'm constantly learning day to day.But I also balance that with, whatare some of the newer techniques that have been evolving,often driven more by the commercial rather thanan academic environment, that allow us to capture datain new and different ways and that

  • 06:29

    JULIA CLARK [continued]: allow our design and our methodological approachto continually improve?It's a fine line to walk.You don't want to do something so new it's untested, sonew that we don't know what it's going to do,but you don't want to ignore those newer technologiesat the risk of getting a less optimal approach to data

  • 06:49

    JULIA CLARK [continued]: collection.So I think the question is, sort of,do we ask people if they think that rapeis or could be a war crime?Or do we ask people maybe if they agree or disagreewith the ruling by the ICC about rape being a war crime?I mean, that's sort of two anglesto approach the same question.I don't know.What do you think might work best?

  • 07:07

    SPEAKER 1: Well, I think that the first option-- should rapebe considered a war crime-- I think matches better with whatwe've asked about in the past and other waysthat we've framed the questions previously.And then, after we establish that,then we can go into how they feel about that specific rulingin that specific case.

  • 07:24

    JULIA CLARK: Yeah, and that actually-- Ilike that sort of trajectory a little better, because itdoesn't give too much information right away.We're sort of asking about it and then giving the example.So we're going to say, first of all, if they agree,that maybe sexual assault or rape specifically-- wehave to think about which word we use,but if rape is considered a war crime by the public,

  • 07:46

    JULIA CLARK [continued]: keeping in mind these are going to comeafter the series of questions about the ICC itself.So we do have a little bit of context.This isn't asked in isolation.For me, the survey-design part isone of the most fun and invigorating parts of my job,and it's actually one of the most prevalent parts of my job.Not everybody in my industry can say this,

  • 08:07

    JULIA CLARK [continued]: but I spend a lot of my time designing survey questions.Literally, I design questionnaires.It's a wonderful challenge, and itmakes you think very laterally about everything.I run our polling here at Ipsos, and wedo a lot of sort of daily-tracking polling.Our questionnaire is constantly changingto reflect issues and events of the day, both political

  • 08:28

    JULIA CLARK [continued]: and just sort of news-driven, international-event driven.And so I'm constantly being forced, willingly of course,to develop questions about complicated issues,economic considerations.Is the government going to shut down?What are the implications of, for example,a terrorist incident abroad?What are the implications of any type of social issue

  • 08:53

    JULIA CLARK [continued]: that could arise-- a Supreme Court decision, a hurricane?And they always have to be developed from scratch.But the trick with a questionnaire designis really to ensure that you're askingpeople questions they can understandand they can answer easily.You know, it's all very well and good to say,you know, how should we reduce the deficit?

  • 09:14

    JULIA CLARK [continued]: But the average American doesn't necessarily have those answersreadily at their fingertips.And why should they, of course?They're experts in many other thingsbut perhaps not debt reduction.And so you have to frame questionsin a way that provides them with neutral, unbiased, non-leadinginformation but that also allows them to have enough information

  • 09:36

    JULIA CLARK [continued]: to form an opinion.So you can't simply say, what's the answer to debt reduction?You have to say, let me tell you a little bitabout the US government's debt.Let me tell you what it comprises.Some of the things that people have suggestedwe do to reduce it are X, Y, and Z. Which of thesedo you like, or is there something else youthink would make sense?Now, that's of course a very long-winded question,

  • 09:57

    JULIA CLARK [continued]: and we would actually break that down into a series of questionsto make them individual, bite-sized, more digestible.But it's one of my most exciting challenges.You really have to be able to put yourselfin the shoes of the public or the shoes of any interested,you know, stakeholder or interested party whomay be answering this question.

  • 10:19

    JULIA CLARK [continued]: You know, if you're asking questions of lawyers,let's say, because you're doing a survey of lawyers,the terminology and the words you can useare going to be a little different,because there's a base sort of technologyor a sort of linguistic knowledgethat they have about their industry.So you can incorporate some of that language.You would never use that same language in a survey

  • 10:40

    JULIA CLARK [continued]: to the general public, where you have to make surethat everybody-- lower education, higher education,younger, older, engaged, less engaged, informed,less informed-- all of those peopleare able to equally answer the question that you'reasking in a way that's valid and legitimate.Is there anything else we're missing in terms of this angle?

  • 11:00

    JULIA CLARK [continued]: We're kind of addressing sexual crimes for the first timein this survey.Is there something else we shouldbe thinking about for the ICC?

  • 11:07

    SPEAKER 1: Well, something you brought up,which is that a lot of the reason that they've gottena lot of traction from this is the sort of feminist pressand seeing it as a major step for women.This case also was decided by an entirely female panelof judges, which would be really, reallyinteresting to talk about the fact that-- is itthe fact that there were three female judges?Does that make a difference?

  • 11:28

    SPEAKER 1 [continued]: Would it be different if it was three men?Would it be different if it was a mixed panel?

  • 11:32

    JULIA CLARK: And the prosecutor was female as well.Yeah.So this has a really strong sort of femaleor feminine or feminist componentto the whole situation.So in terms of next steps, I think this is exciting,and I think actually, per our other agenda around doingsome interesting polls that Ipsos publishes on our own,this could be a really useful angle.But we'll be designing some new stuff, too.

  • 11:53

    JULIA CLARK [continued]: So if you can pull anything that we've done in the past,we'll put it all in one big document,and I'll work on the questions we've just discussedand maybe we'll get them together and then sit downwith our client and see what he thinks about all this.But I think he's going to like it.

  • 12:06

    SPEAKER 1: I think so, too.

  • 12:08

    JULIA CLARK: Survey design categoricallyworks best as a collaborative enterprise.You need absolutely somebody who has done itso much that they can sort of make a sign-offand make the determination, and in most cases now, that's me,but collaborative design is always optimal.This is for a few reasons.First of all, not everybody is an expert on everything.

  • 12:31

    JULIA CLARK [continued]: So when I am designing questionnairesabout millennials' usage of social media,I need some of my millennials to participatein that survey-design process, because they'regoing to be closer to some of these issuesthan I am in some ways.But it's not just that.It's not just about subject-matter expertise.It's about ensuring that the questions you're designing

  • 12:52

    JULIA CLARK [continued]: are digestible and appealing and answerableto a huge range of audiences.And if we're talking about the general public,certainly, it's younger and olderand male and female and educated and lower-educated andhigher-income and lower-income and urbanand rural and all the dimensions wehave to consider when we're talkingabout America as a whole.If I sat there in isolation designing questions by myself,

  • 13:16

    JULIA CLARK [continued]: they'd be all right, because I've done this a lot,but they probably would lack a richness and a contextthat I gain from ensuring that it's a group of us designingquestions.And our team is great.We have different strengths and weaknesses.So some of us bring a real political acumen to the game,and some of us really focus much more on the social sciences

  • 13:39

    JULIA CLARK [continued]: aspect and the public opinion broader picture.And I have a psychological backgroundor a psych background, and so I'mable to really ensure that the language that we're usingis the most direct, the most simple, the leastnuanced, the least biased.We don't want words that are open to interpretationin a lot of different ways.We want questions that are clear and clean.

  • 14:02

    JULIA CLARK [continued]: And we do that by many iterations.One of us will take the first pass, send it to the group.A few others take a pass.Then we have a meeting.Then the client provides some input.Then we all take another pass.And each time we're making tweaks.You know, it's rarely a wholesale redesign,although that occasionally happens.But you need everyone looking at these questionsmultiple times to really get them in the best shape

  • 14:24

    JULIA CLARK [continued]: they can be.Hey, Mike.

  • 14:26

    SPEAKER 4: Hey.

  • 14:27

    JULIA CLARK: How are you?

  • 14:28

    SPEAKER 4: OK.How is it going?

  • 14:28

    JULIA CLARK: Good.Thanks.I was hoping I could pick your brain later on.I wanted to get your input on some of the questionnairesort of structuring issues we're having with the Reuters survey.Would that be all right?I know you have done that before for some of the other stuffwe've been working on, so it wouldbe great to pick your brains.

  • 14:42

    SPEAKER 4: Very good.

  • 14:42

    JULIA CLARK: Awesome.We're going to just meet in the conference room at noon.

  • 14:44

    SPEAKER 4: All right, see you then.

  • 14:45

    JULIA CLARK: OK.Thank you.Bad questionnaire design is really the baneof my professional existence.It is immensely frustrating.It is something we see, unfortunately, a lot,not so much in the professional realm of your top researchcompanies.There's a lot of control that goes into how

  • 15:07

    JULIA CLARK [continued]: people design questions.And of course, in the age of the internet,everybody's questions are available.So there's a lot of external scrutiny.Some of the places you'll see biased pollsor leading questions are-- often a media companyor another type of organization will put just a little surveyquestion on their website, and any of their readershipcan click in and ask it.That's what we call a convenience sample-- i.e.

  • 15:29

    JULIA CLARK [continued]: anybody who shows up at the website can participate.There's no sort of validity or robustnessor representivity to it.And often that's where you see a question that says sort of,how much do you like or dislike Donald Trump, right?But they wouldn't frame it that way.They'd say how much do you dislike Donald Trump, right?And that leads us down a path of assumingthat the dislike is there.

  • 15:50

    JULIA CLARK [continued]: Now, everybody's political opinion aside,that's the wrong way to design a question.If you say, how much do you dislike X,the premise is dislike.And so you're saying, I dislike them a lot,or I dislike them a little.But of course, what if you like the person, right?That's not controlled for within the context of the question.So biased question design is extremely problematic.

  • 16:12

    JULIA CLARK [continued]: It creates false data that's thenpushed into the public sphere in a way thatcan be very misleading.And it can be obvious, like the example I just gave--how much do you dislike X?But it can also be very insidious and much more subtle.And it speaks to the nature of the setupof the question itself.

  • 16:33

    JULIA CLARK [continued]: We can't just think about the question itselfwhen we think about a biased question design.We have to think about the context.If you ask a whole series of questionsabout-- let's use the ACA, or Obamacare as it's known--have you been impacted by Obamacare?Do you think you're paying too much for health care?Or do you think it's become worse?These are all leading questions.

  • 16:54

    JULIA CLARK [continued]: But if you ask a series of questions like thatand then you ask a question, do you approve or disapproveof the way Obama is handling health care in this country?You know, that's a leading series of questions.So not only is each question a bit biased,but you've led the respondent to a conclusion,because you've given some sort of negative, perhaps,information about the outcome or an attitude

  • 17:17

    JULIA CLARK [continued]: towards a politician, and then you ask that questionabout the politician.And that's leading and biased as well.Now, even worse is then if you onlypublish the responses to that one"do you approve or disapprove of Obama"question without indicating that the whole series of questionsyou asked in the lead-up are also leading.This is why Ipsos is a strong adherent to the Transparency

  • 17:39

    JULIA CLARK [continued]: Initiative, which is an initiativethat our association, the American Association of PublicOpinion Research, is an architect ofand Ipsos has signed on to.It's really about making sure the way you designthe questionnaire, the way you designthe entirety of the survey, as well as all the questions up

  • 17:60

    JULIA CLARK [continued]: to and including published questions are made public,are made available to anybody who wishes to see them,so they can assess for themselves question bias.What we're doing here now is, I've got some of the teamtogether, and what we're going to be doingis brainstorming some technical solutions to a problemwe're having with one of our questionnaires.It's really lengthy.

  • 18:20

    JULIA CLARK [continued]: It's starting to become cumbersome technically.And we're going to be brainstormingsome data-driven ideas to reformat it and rework itin a way that makes it much more usable.So I've got some of the team herewho bring some of that knowledge to the table,and we're just going to talk through those issues.We just wanted to do a quick brainstormabout this issue we're having with the Reuters questionnaire.

  • 18:41

    JULIA CLARK [continued]: I think it really just needs to work better for us.It's this huge, long Word document.It's cumbersome.It's difficult to search.It's difficult to match it up with dates and data.So we really need, I think, a good technical solution,and I think Katie and Mike, the two of you,have not only experience with the database side of thingsbut with the programming and sampling side, too.So I think the four of us together definitely

  • 19:01

    JULIA CLARK [continued]: can come up with something.So, Eliza, can you just explain to us a little bitthe current format, so we can start thinkingthrough some solutions?

  • 19:08

    SPEAKER 1: Absolutely.So, currently, the questionnaire is in a Word format.Due to the length of time that we've had this survey in fieldand due to programming restrictions,every question that we have ever askedis currently in one Word document,meaning that we are dealing with a questionnairethat, while it's only 15 minutes in length for the peopletaking it right now, it's actually over 240 pages long.

  • 19:31

    SPEAKER 1 [continued]: [LAUGHTER]So what we need to do is find a wayto make this questionnaire easily searchable,to have a way to decide when questions weremost recently live, because obviously sometimeswe rotate certain values questions in and out over time,and of course, how to do all of this while also satisfyingthe programming restrictions that we

  • 19:52

    SPEAKER 1 [continued]: need for the team that's actually makingthis accessible to everybody.

  • 19:57

    JULIA CLARK: Phone surveys and online surveysdiffer in terms of question design pretty substantially.And this is one of the biggest issues facing our industryright now, actually.It's a very substantive issue.It's very complicated.A lot of us are grappling with thisin sort of day-to-day real-life research problems right now.

  • 20:22

    JULIA CLARK [continued]: So the major difference between a survey designedfor the phone and a survey designed onlineis that one is administered by a phone interviewer.There is an actual person sitting at a phone bankon the phone sort of calling-- youknow, calling around saying, whatdo you think about X, Y, and Z?Whereas an online survey is in front of you on a screenor on a smartphone or tablet, and you're

  • 20:42

    JULIA CLARK [continued]: sort of clicking your responses.And that has immense implicationsfrom a design perspective.It seems maybe simple at the outset,but it's very fundamental, not onlybecause of what we call interviewer affectsor demand characteristics, which is that people may answera little differently if they're talking to a person,maybe because they want to seem a little smarteror a little more interested or they

  • 21:03

    JULIA CLARK [continued]: want to be a little more pleasantand say yes a little more often.So one of the factors we know that happenswhen we do online and phone comparisons is peopleare a little more negative online.They're more likely to say they're dissatisfiedwith things or disapprove of them,or they're more likely to say they're unhappy with something.It's not a huge factor, but it's notable, right?

  • 21:24

    JULIA CLARK [continued]: And that's across all companies and methodologies.Online you get a little more negativity, right?So that may be a factor related to interviewing.But there is a broader and more fundamentaleffect that we call mode effects.People simply answer questions differently.Again, this goes back to the psych principles,to some of the sort of fundamental psychology of not

  • 21:45

    JULIA CLARK [continued]: just questionnaire design but of how humans function.You read something on a paper, and you answer it one way.And you talk to someone on the phone,and you're maybe answering a little bit differentlyfor a huge host of reasons.A good example is the "don't know" response.Most survey questions have an option to say "I don't know."People choose not to respond.They don't want to give an answer.They just genuinely don't know.

  • 22:06

    JULIA CLARK [continued]: We don't provide that as a read-out response on the phone.So you say, are you satisfied or dissatisfied with X, Y, Z,on the phone, right?And they say, oh, I'm satisfied.And then the interviewer says, good,are you very or fairly satisfied?If the person says, 'I just don't know,' the interviewercan code that as a response option.But it's very rarely chosen, because most people simply

  • 22:29

    JULIA CLARK [continued]: give a response when asked.It's human nature.Online, you have a list of responses-- very satisfied,fairly satisfied, fairly dissatisfied,very dissatisfied, don't know.It's literally there in black and white in front of you.So we see the proportion of peoplesaying they don't know shoot up in online surveys, which

  • 22:49

    JULIA CLARK [continued]: makes a simple comparison between the two very difficult,because all of the responses have sort of been reallocatedto different categories a little if you have more peoplesaying they don't know.So we do at the moment have lots of clientswho have been doing tracking work online for many years.And in fact, we were doing our own political pollingon the phone for many years, as was all the pollsters

  • 23:10

    JULIA CLARK [continued]: in the United States.And making a transition online is not simple.You can't simply move your survey vehicle.You can't simply make the questions all the same.You can't simply balance your sampleto ensure you're accounting for the factthat not everyone is online.None of these things work.There is no perfect crosswalk to movea survey from phone to online and retain the tracking

  • 23:32

    JULIA CLARK [continued]: numbers.It just doesn't work.We manage to do it in other ways.We begin running the two in parallel.And then we slowly transition from one to the other,perhaps using some analytics to createa crosswalk between them.But you can't simply jump ship from one to the otherand expect the same thing, because mode affectsthe way the survey is administered,because of interviewer effects, because the person on the line

  • 23:55

    JULIA CLARK [continued]: means you respond differently to the click of the mouse.So lots and lots of differences.It's complicated, and it's reallya challenge for our industry.So what I'm thinking is that we needsomething that is searchable but also quite flexible.So maybe a database or a wiki.I don't know if Excel might be a good option.

  • 24:11

    SPEAKER 4: I think Excel might be a little too unpredictableand too easily corrupted.A database that has some safeguards built into it,probably work a lot better for you.

  • 24:21


  • 24:22

    SPEAKER 4: I think, if you're working with a document that'sso large and, from what Eliza has said,IIS does not want us to do away with it.

  • 24:30

    JULIA CLARK: Right.

  • 24:31

    SPEAKER 4: Let's work off of that as a baseand perhaps add tags to each questionthat we could put into a database thatwould attach dates and instructions and history.

  • 24:38

    JULIA CLARK: OK.So use the file we have now but addtags and some infrastructure around it?

  • 24:43

    SPEAKER 4: Or even just use the questions-- the questionnumbers themselves and have a tag associatedwith it that we could correlate and then usethat in the database.

  • 24:51

    JULIA CLARK: So we are going to startusing tags in the document itself, which willlink to a broader database.Now, we're going to have to includesome other variables around date in and out of field?

  • 25:04

    SPEAKER 1: Exactly.And I think that one other thing that weneed to be considerate of are so many questionsthat we trend over time.For instance, questions that we have after major attacks--we had one after the Boston Marathon bombings,and then we just modified that question for the Paris attacks,as well as the Brussels attacks.

  • 25:23

    JULIA CLARK: Opinion polls may affect the political process.There's evidence on both sides, actually.And if we look at things globally--and I worked for a long time in the UK doing political polling,and my company, Ipsos, is a global company,so naturally I look at these issues globally.Some countries have banned opinion polling in the lead-upto Election Day for a couple of days or even a week,

  • 25:47

    JULIA CLARK [continued]: because they believe that the impact of polling on voters'choices is significant.Some countries have absolutely no regulations around it.Here, there are really none, although exit-poll datais traditionally not published here in the USuntil the polls have closed for fear, presumably,of impacting voters' likelihood to actually show up and vote

  • 26:09

    JULIA CLARK [continued]: or not vote, if they think their candidate has alreadyeither won or lost.I would say that my personal opinion is that poll data isgood for democracy.It provides more rather than less information,and I don't think people go to the pollsbecause they've read a poll.

  • 26:29

    JULIA CLARK [continued]: They don't go to vote because they think their candidate isnecessarily winning or losing.They vote because, either, they believe voting is somethingthat they do or don't do.Or they've been galvanized by a campaign or a candidateto vote or not to vote.I don't believe that public-opinion pollsare driving them to the voting boothor keeping them away from it.

  • 26:49

    JULIA CLARK [continued]: But what I do think about public-opinion polling isthat it is a tool for democracy, because what it doesis, outside of elections, which happenat a presidential level every for years but, of course,for Congress and many other offices,let's say every two years-- outsideof those elections and a few interim elections,there are no opportunities for our political leadership

  • 27:13

    JULIA CLARK [continued]: to know what the public thinks on a large scale.And public-opinion polling offersthat-- a well-conducted, representative, and robustsample of Americans or of, you know, menor of women or of young people or of older people.That is a good way to understand and assess howAmericans feel on a subject.

  • 27:34

    JULIA CLARK [continued]: And in my mind, that feels very democratic,so that we know what the public think.They don't all agree on one thing,but we know what more rather than less of them think.

  • 27:42

    SPEAKER 4: I can look into creating a roll of structurefor tags and maybe have meta tags associated with it,and we can use that as a stepping stone.

  • 27:49

    JULIA CLARK: And that's great, because actually thatspeaks to the sort of group-tagging idea, the ideaof some meta tags that Eliza was describing,which will allow us to kind of pull themes as wellas actual question themselves.Wonderful.Well, this has been really useful, guys.If you could get some examples of whatwe've done in the past with that, that would be great.I'd to see what that looks like.And, Katie, the same for you for a wiki.That would be wonderful.

  • 28:09

    JULIA CLARK [continued]: And Eliza can get you the questionnaireas it exists now, so we can see where we're starting from.People who are interested in getting into survey designor doing even my job some day-- first of all, the thingthat I always tell people when I'mchatting with people about Ipsos or interviewing themis that, really, it's important to rememberwe're in client services.

  • 28:30

    JULIA CLARK [continued]: And my part-- not all of our companyis focused solely on client services, but most of it is.So while it is imperative that we have great research skillsand that we understand data and that we understanddesign principles, you know, this is a for-profit company.This is a commercial entity.And we are driven constantly by our clients

  • 28:51

    JULIA CLARK [continued]: and doing good work for them and working with them.And so I'm always either continually writing proposalsalongside delivering projects alongside meeting with clientsto help them understand the results of that data capture.And so the client-servicing aspect of itis something that some people who want to go into research

  • 29:11

    JULIA CLARK [continued]: are very interested in, and otherswho are very much more focused just on the methodologicaland data capture maybe haven't alwaysthought about that aspect of things.So I think pairing the both together,being able to work with and talk to and sell to clients pairedwith the technological and technical expertiseis really important to doing this job well.

  • 29:33

    JULIA CLARK [continued]: There are a lot of places you cango to research, to do research.Some of them have a more academic focus,less commercial, and some of themhave a much more commercial focus, like Ipsos does.I would say things are much more fast-pacedin the commercial environment.They are-- as I said before, they're very exciting.You're constantly challenged by a host of different clientsand projects.You're not doing one thing all the time.

  • 29:53

    JULIA CLARK [continued]: You're doing 12 things all the time.And so I would say that sort of balance between a client focusand between the methodological rigor are really important.I would also say that Ipsos is a placethat people who have a lot of ideas about how to improveand change things in our research worldwould be very welcome.

  • 30:15

    JULIA CLARK [continued]: It's an environment that's very permissive,I would say, of new ideas, challenging the normsin terms of research.We like people here who want to do different thingsand want to figure out different ways to capture dataand service clients and report dataand visualize data and use social mediaand all these types of things.

  • 30:38

    JULIA CLARK [continued]: And I like that about the company.I mean, there remains a very strong groundingin methodological rigor.We have reams of statisticians and datascientists and programmers with whom I work constantly day inand day out and who really form the backboneof the organization.But we also need people who are willing to push

  • 30:58

    JULIA CLARK [continued]: the envelope a little bit and try new things.Our newer competitive set are more online.They're the Googles and the SurveyMonkeysof the world, who are doing very interesting and exciting work.And so we need to be thinking about those typesof organizations as we begin to expand our capacities, too.Groups like mine within large companies who do, you know,

  • 31:21

    JULIA CLARK [continued]: commercial research but are working oftenwith government clients or clients whoare lobbyists with specific agendas,et cetera, there's a constant tension, really,between their objectives and retaininga really rigorous survey and methods design.And I think that's one of the most difficult thingsto grapple with in a position like mine,where you have a client who's very interested in either being

  • 31:44

    JULIA CLARK [continued]: able to make a certain claim or say something or hope that mostof the public will support their idea or their agenda,and that's what they want to see out of their researchthat they're paying you for, right?They're paying for this.And the task of the researcher or is to design the survey well

  • 32:06

    JULIA CLARK [continued]: but also simultaneously manage that client's expectationsand say, listen, we're going to design a survey.We understand what you're trying to ask.But the results may not be exactly what you want to hear.What are we're going to do if they're notwhat you want to hear?Because I am in a position of luxury.I don't have to do research that I think is poor quality.If a client is inflexible in that regard,

  • 32:28

    JULIA CLARK [continued]: they should find someone else to work with.Our premium and our reputation is staked on rigor.We cannot ever ask a biased question or a leading questionor a problematic design.These things have to be guarded against vigilantly,and we have a very large system internally set upto ensure those things are done constantly,especially if research is going to be published.

  • 32:49

    JULIA CLARK [continued]: And I would say balancing that sort of tension--and it's often very delicate with a client-- isone of the biggest challenges of my job, where I have to say,you might not like what you're paying for.What are you going to do in that case?Are there other aspects to this findingthat we can use that will tell partof the story you want to tell?Or does this create an opportunity for you

  • 33:09

    JULIA CLARK [continued]: in terms of changing your communicationsstrategy of education, right?Perhaps you need a campaign less about promoting somethingand more about communicating some sort of facet of educatingpeople about this issue, right?And it can be very difficult, because sometimes we'reworking not with the end client but with a PR companyor a communications firm.And so sort of managing the dynamics

  • 33:30

    JULIA CLARK [continued]: between those stakeholders is very challenging.It can be very rewarding, especially when clientsare willing to work with us, because they get such a betterproduct, such a better outcome whenthey are willing to flex to meet the needsof high-quality research.Things that I enjoy most about my jobare when I know that the research we've done

  • 33:51

    JULIA CLARK [continued]: is bulletproof and it completely stands upto an external scrutiny and it's awesome, frankly,and it tells a great story and, even better,if it tells the story the client wanted to hear.But if those things all come togetherand the client is happy with the findingsand I know the research is robust and wonderful,

  • 34:15

    JULIA CLARK [continued]: that's very satisfying.I would say that making my clients really,really delighted with the outcome of the researchthey're getting and the quality of the research they'regetting, when they feel they've learned somethingfrom the process and really gained some insight into eithertheir audience's mindset or the public's mindset,I mean, that's very rewarding.


Julia Clark describes her work as the senior vice president of Ipsos Public Affairs, a for-profit research agency that focuses on public opinion surveys. She explains the difference between telephone and online polling, emphasizes the client relationship, and discusses the importance of collaboration and innovation.

Looks like you do not have access to this content.

Survey Design: Ipsos Public Affairs

Julia Clark describes her work as the senior vice president of Ipsos Public Affairs, a for-profit research agency that focuses on public opinion surveys. She explains the difference between telephone and online polling, emphasizes the client relationship, and discusses the importance of collaboration and innovation.

Copy and paste the following HTML into your website