Cognitive Interviewing: NatCen Social Research

[Cognitive Interviewing: NatCen Social Research]

JOANNA D'ARDENNE: My name is Jo D'Ardenne,and I'm a research director at NatCen Social Research.And I work within NatCen's questionnaire developmentand testing hub, designing questions and testing thembefore they're used in large scale social surveys.NatCen is a not for profit social research organization.We carry out social surveys for a wide variety of people,

JOANNA D'ARDENNE [continued]: including government and third sector bodies.We carry out surveys on lots of different topics.And our data are used to inform national statistics.So we would use a variety of methodsto test our survey questions.Cognitive interviews are qualitative interviews,and we might do these with a small number of people.But we'd also want to pilot the questionnaire with a larger

JOANNA D'ARDENNE [continued]: group of people, to test the questionnaire length and alsothe processes about how we find people, the likely responserates.We might also look at the data whichhas been collected after the surveyif it's going to be a repeated survey, to see if there'sany anomalies in the data, if there's any questions whichhave a high item non-response rate,or which don't look quite right.So it is one step in multiple phases of question testing.

SUSIE PENNY: Would you like to come in and take a seat?Thank you very much for coming in today.My name is Susie Penny, and I workfor NatCen Social Research.And what I'd like to do today is test some questionsfor the European Social Survey of our client.

SUSIE PENNY [continued]: So I'd like you to use these cards as we aregoing through the questions.And I will just find the right pageand make a start of the questions.

JOANNA D'ARDENNE: A cognitive interviewis a qualitative interview, wherewe try to get depth information out of peopleabout particular topics.And cognitive interviews, it tendsto be finding out about thought process,what people think about when they'regiven a stimuli, in this case a survey question.We want to find out if people can verbalize their thought

JOANNA D'ARDENNE [continued]: processes and actually give a rationale for why they'reanswering the way they do.And that is useful from the researcher's point of view,to make sure our understanding of the questionand a respondent's understanding of the questionactually matches.

SUSIE PENNY: So let me just explainin a little more detail about how the interview will work.So I'm going to read you out the new survey questions,and I want you to tell me whatever comes into your mindas you are working out which answer to give.So we call this thinking aloud.And we found that it helps to have some practice beforehandat thinking aloud, so let me give you an example.

SUSIE PENNY [continued]: Let's say I ask a question, how manywindows are there in your home.So if I was thinking aloud when answering that question,I'd think, well, I walk into my hallway,and it's an internal hallway, so there aren'tany windows in the hallway.I walk into my bedroom, and thereare 1, 2 windows in my bedroom.

SUSIE PENNY [continued]: And I walk back out into the kitchen,and there's one window.And then I walk into my living room,there's actually four windows in there,so that's eight windows in total.So could I just get you to have a practice at doing that--

SURVEY RESPONDENT: OK.

SUSIE PENNY: --thinking aloud.So if I asked you how many windows there are in your home,how would you answer?

JOANNA D'ARDENNE: I think the main challengeof cognitive interviewing is getting peopleto actually verbalize their thought processes.So sometimes you get people who might be a little bit reticentto say what they're thinking, and they might be a little bitshy.In other cases, you might get people who don't necessarilytell you what they think of the survey questionbut what they think other people might think of the survey

JOANNA D'ARDENNE [continued]: question, so it's moving away from their own personal thoughtprocesses, just kind of theorizing and hypothesizingrather than thinking about what they themselves think.So a challenge is when you're doing the interviewsto make sure that you can try and elicitthese thought processes and this high quality verbal data.

SUSIE PENNY: The highlighted box at the top of this cardshows a number of energy sources.For each energy source, please tell mehow much you think the UK should use to generate electricityfor use in people's homes.Please choose your answer from the optionsat the bottom of this card.How about coal?

SURVEY RESPONDENT: I think for coal Iwould say a moderate amount, purely because I know it's notgreat for the environment but also Ithink it's probably the most efficient at the moment.So maybe-- I know obviously that energy demands aregoing to be high, so although it might not be desirable to use,

SURVEY RESPONDENT [continued]: it's still necessary at this point, I think.

SUSIE PENNY: OK.

JOANNA D'ARDENNE: Cognitive interviews are semi-structured.The test questions or things we're interested and testingalways have to be administered in exactly the same way theywould be in the actual survey, but the interviewer alsohas some freedom to ask additional qualitative probesto try and find out whether or not the questions are workingor not.Interviewers will be given a protocol, which

JOANNA D'ARDENNE [continued]: includes the wordings of the questionsand also some guidance about what types of probes to ask.But they also have the freedom to explore additional findingswith unprompted probes, if they need to.

SUSIE PENNY: And how easy or difficultwas it for you to answer these questions?

SURVEY RESPONDENT: It was quite easy to just sort of givemy opinion, but I wouldn't say itwas based on anything massively scientific about whatI know about the UK's energy needsand how much each source provides.I wouldn't be able to give an accurate description of howmuch energy they provide.

JOANNA D'ARDENNE: The main benefitsof cognitive interviewing are you get a real feel forwhether or not people are understandingthe words and concepts within your question as intended.The other benefits are you can explore sensitivities,so whether or not people are comfortable answeringthe questions.You can explore how people use a self-completion document,for example, whether or not they look at the instructions,

JOANNA D'ARDENNE [continued]: whether they understand how to navigate a form.So you're looking at specific issueswithin a questionnaire in quite a large level of depth.The other main advantage of cognitive interviewingis you are finding issues that wouldhave gone uncovered in other forms of testing,so for example, a pilot.Now in a pilot, people might answer a questionand seem perfectly happy, perfectly comfortable

JOANNA D'ARDENNE [continued]: and they look like they understand it,but actually if you'd asked them someprobing questions, their understanding of the question,it could have been completely different to what you meant.So it's good at detecting hidden problems,and kind of remedying them before the surveygoes to field.[Joint Analysis Meeting]

JOANNA D'ARDENNE [continued]: So we held a joint analysis meetingas part of the European Social Survey testing.This was attended by researchers from a range of countries.So for example, there were attendees from Poland,Norway, Spain, and Austria also present.We went through the questions one by one,found out what the response to these questionswere in the different countries where cognitive interviewing

JOANNA D'ARDENNE [continued]: had been conducted, and discussedpotential recommendations.So some of the questions we were testing this time aroundwere about attitudes towards the environment.So we were finding out whether or not people understoodclimate change in the different countries,and how they reacted to specific questionsthat we were asking on this topic area.

SPEAKER 1: This will be the start of the climate changemodule, probably.First item.And the question aims to understandwhether the respondent thinks the world's climate is changingor not, irrespective of the possible perceived causes.The question that was tested was youmay have heard the idea that the world's climate is changingdue to increases in the temperature over the past 100

SPEAKER 1 [continued]: years.What is your personal opinion of this?Do you think the world's climate is changing?Please choose your answer from this card.And the options were definitely changing, probably changing,probably not changing, definitely not changing.Perhaps we could start in Norway.

JOANNA D'ARDENNE: In the European Social Survey,we conducted cognitive interviewing first off to seeif the questions worked in the source language, whichis English.So do people understand the questions as intended?Can they retrieve the information?Can they give accurate answers?In terms of testing the European Social Survey questions,we're also looking at how well they work in translation.

JOANNA D'ARDENNE [continued]: So we conducted interviews in multiple countriesto see if once the items have been translatedthey are actually equivalent.So when people in the UK are answering,are they thinking about the same typesof things as someone in Spain or Norway, for example?

SPEAKER 1: If you were to sort of do a gloss back translation,what would it--

SPEAKER 2: So the first will be a majority of peoplewould absolutely want to limit their energy use.That's where completely--

SPEAKER 1: Cases.Yeah, so that's a serious translation error,in a sense that that's about motivationto do it rather than ability.

SPEAKER 2: And there's no [INAUDIBLE]would absolutely want to.And it sounds awkward, especiallythe second's would very like to.And there it is would like to.

SPEAKER 1: Well, I guess the problem'sthat it's not supposed to be about like to or wanting to.It's supposed to be could do, in a sense.So there's a translation error.

JOANNA D'ARDENNE: One of the thingswe look for in cross-national testingis whether or not the questions are portable.So by this we mean does the concept exist across allthe countries where the survey is going to take place,or is something so alien to certain cultures or countriesyou couldn't measure it, or at least not measure itin an equivalent way?

JOANNA D'ARDENNE [continued]: Fortunately for the questions we'retesting this time around the European Social Survey,we thought that all questions we tested were portableand could be asked in different countries.A different thing that we would lookfor when doing cross-national cognitive interviewingis more to do with the translation side of things.So is there a better word or phrasethat could be used than our first translation

JOANNA D'ARDENNE [continued]: that we tried?Are words really equivalent or are they not,in how we can go for the most equivalent word when we dointerviewing cross-nationally?

SPEAKER 1: I guess maybe there's two issues, one of whichis are the translations clear enough.That standing question needs to be checked maybe afterwards.The second one is actually, respondents don't reallywant to answer about ability to do it,because it's tricky until they really know.So they start to move on to willingness, or likelihood

SPEAKER 1 [continued]: of doing it, which is something they feelmore comfortable answering.So in a way, the respondents are kind of voting with their feet.They don't like the question, so they'resaying OK, I'm going to answer it about something slightlydifferent.Just to check in the source language in the UK,were people talking about-- does it feel they'retalking about willingness?

SPEAKER 3: I think they're just talkingabout the ability, and the fact that in Africa, theywouldn't have the ability to do that,because they don't have the resource-- morehaving the resources available to do that.

SPEAKER 1: So it's kind of worked in the source language,so it's either a case of it's too difficult to translate it,or the translations that were chosen were not optimal.Better ones could be found.It sounds like it's just too difficult to render itinto other languages.

JOANNA D'ARDENNE: For the most part,the questions on climate change worked well.One of the things we were investigatingwas whether or not the term climate change wasunderstood across countries.And actually, it was, so that was a very positive findingin establishing whether or not the questions would work.But there were some more minor issuesrelating to how the response options to some

JOANNA D'ARDENNE [continued]: of the questions were translated.So for example, some scales were about howable people felt to make a difference to climate change.And it turns out in translation this idea of abilitywas slightly different to translate,or difficult to translate.So those are a difficulty of establishing whether itmeant ability or willingness.

JOANNA D'ARDENNE [continued]: So we are now going to work with the different countryteams to think about a potential different translation,and a different way of making surethat we can measure efficacy and ability,rather than willingness.