Skip to main content
SAGE
Search form
  • 00:00

    [MUSIC PLAYING]

  • 00:09

    SANDRA BARRUECO: Today, I'll be discussingdesigning experiments, key points and lessonsfrom the field, Hi, my name is Dr. Sandra Barrueco,and I'm an Associate Professor of Psychologyat the Catholic University of Americawhere I direct two programs, the clinical psychologydoctoral program and Latin American and Latino studies.My work focuses on young immigrant children

  • 00:30

    SANDRA BARRUECO [continued]: and their families with research conducted both nationallyand locally.I taught the graduate research methods course for many yearsand have used a variety of research methodsfrom randomized clinical trials to observational approaches.Today, we'll be discussing four key points and four questions.What are key features of an experiment?

  • 00:51

    SANDRA BARRUECO [continued]: What should we consider when conducting experiments?What are lessons learned from designing experiments?And what are dos and don'ts for designing experiments?What are key features of an experiment?First is random assignment of participants.We do this to ensure equivalence to groups to the extent

  • 01:12

    SANDRA BARRUECO [continued]: possible.This helps us control selection bias.And we can randomized either individuals or groups.This is different from random sampling.We can also control conditions when we do random studies.Another key feature of designing experimentsis control of conditions.We not only randomized participants

  • 01:33

    SANDRA BARRUECO [continued]: to separate groups, but we can also randomlyassign individuals to conditions that we as researchers havefull and complete control over.What should we consider when conducting experimentsis another key point for us.There are human and sample considerationsthat are key and foremost.

  • 01:54

    SANDRA BARRUECO [continued]: For example, individuals and group interest.At time we create randomized studiesor experimental designs, and perhaps the participantsare not interested in being randomized to those conditions.They might not sign up.And so human volition is an important partof experimental design.

  • 02:14

    SANDRA BARRUECO [continued]: Another key feature is ethics and ethical behavior.What is the appropriate comparisonin an experimental design?For example, we might be interested in a new medicineor a new drug to treat a condition.What is there is already another one?Another one that has already beenproven to be beneficial to the individuals

  • 02:37

    SANDRA BARRUECO [continued]: with that condition.Is it ethical to randomly assign them to our new drugand then the comparison be that they take no drug,or is it more ethical approach to assign themto two different conditions, one taking the new drugs that wehave identified as a potential benefit compared to the one

  • 02:59

    SANDRA BARRUECO [continued]: right now that has been identifiedas a beneficial drug?So when you're considering your experimental design,consider the ethical approaches to the comparison groupas well.Another perspective for human sample considerationsis spillover effects between conditions.What if individuals can interact across the groups?

  • 03:19

    SANDRA BARRUECO [continued]: That is what if we randomly assigned individualsto the conditions and want to seewhether one program or one interventionis more effective than the other?However, what if after they receive those interventionsand they go home and before you administer the post analyzes,those individuals can actually interact with each other,speak with each other, and learn tips from each other?

  • 03:43

    SANDRA BARRUECO [continued]: And therefore, your experimental designnow has spillover effects and is notcompletely separated between the conditions.Another perspective with human and sample considerationsis differential dropout.What if one group decides to participateless over time and the initial equivalence no longer holds?Another consideration here is longitudinal drift

  • 04:05

    SANDRA BARRUECO [continued]: of the sample.Well, what-- who do the results apply to in the endif only half of the sample remainsafter two years of a study?The initial application of the resultsno longer apply to that initial sample.And actually, those experimental resultsapply to only those and other individuals

  • 04:26

    SANDRA BARRUECO [continued]: who are a like those that would remain in a studyafter two years.So as you're designing your study,you want to think about these human and sample considerationsas they relate to spillover effects,as they relate to differential dropout effects, as wellas ethical conditions for your study participants.We're also interested in thinkingabout scientific considerations.

  • 04:49

    SANDRA BARRUECO [continued]: For example, when you create an experimental designand you have two groups and you're interested in,let's say the effect of an intervention,at times we're interested in the effect of that interventionon individuals and therefore we randomly assigned individualsto two conditions.At other times, we are interested in the effectsof a condition on groups.

  • 05:11

    SANDRA BARRUECO [continued]: And actually, the randomized unitshould be the group and not the individual.This can be an error that is oftenseen in experiments and experimental designwhere researchers think that they can only randomly assignindividuals and not groups.But to the best extent possible if you'remaking group comparisons and you're

  • 05:32

    SANDRA BARRUECO [continued]: interested in making findings that relate to groups,be sure to randomly assign the groups.Another consideration from a scientific perspective is thatperhaps the unit that you're interested in cannot berandomly selected, for example, sex, height,family structure even.There are key questions in the social sciences

  • 05:55

    SANDRA BARRUECO [continued]: that we're very interested in along these dimensionsand others.And yet, we as experimenters cannot randomly assignindividuals to these conditions, because beyond ethics,it's just not possible.Yet there are many questions that we have in the field.And so experimental design is not necessarilyas applicable in the true sense of using randomized trials

  • 06:18

    SANDRA BARRUECO [continued]: to these types of conditions and other types of researchapproaches, such as quasi-experimental designs,And advanced scientific analyzes help us decipher and identifywhat the effects of things like, again, sex, height, and familystructure could be.Another scientific consideration are experimentsthat we might have just two levels in it,

  • 06:41

    SANDRA BARRUECO [continued]: and it might not allow us to see curvilinear relationships.For example, imagine that we're studying stress and the effectof stress on individuals.We might create an experiment wherewe have one group engage in an activity thatproduces low stress and other--another group that engages in activities

  • 07:02

    SANDRA BARRUECO [continued]: that produces high levels of stress,and we see a bit of a difference.But what's interesting is that we know through researchthat stress has a curvilinear effect on individuals.And low-- too low levels of stress weactually don't see individuals who necessarily havemuch production in their work.And if it's too high levels of stress,

  • 07:23

    SANDRA BARRUECO [continued]: if you think to yourself about timesthat you went through finals periods or verystressful periods of your time, you might not be as productive.We know that moderate levels of stressactually produce good levels of productivity individuals.So if we create a design and we create-- excuse meif we create an experimental design

  • 07:43

    SANDRA BARRUECO [continued]: and there are just two levels, we actuallymight not be able to identify an underlying relationship that'sactually curvilinear.So as you're designing your studies,think about whether the condition that you'reinterested in if this group has more of a linear feature to itor perhaps even a curvilinear feature to it.Another scientific consideration is

  • 08:04

    SANDRA BARRUECO [continued]: misinterpretation of results.What if the results of one outcomeare actually contingent on another?Let's say you were studying the effects of a languageintervention for children with disabilities.You randomized children to two groups.One receives the intervention, and then youstudy their vocabulary and then linguistic response time.

  • 08:26

    SANDRA BARRUECO [continued]: Let's say that an intervention effectwas found on linguistic time but not vocabulary.Was there really a program effect on linguistic responsetime, or is it a program effect of linguistic responsetime after completing a vocabulary measure?It is important to randomize conditions to individualsand also to randomize the measures to ensure

  • 08:48

    SANDRA BARRUECO [continued]: that there are not measurement order effects.A final scientific consideration that I will discussis whether early research has been conductedusing varied methodologies, such as correlational, observation,qualitative.What are key lessons learned from designing experiments?I conducted a five year intervention study

  • 09:08

    SANDRA BARRUECO [continued]: where one group of children and familiesreceived a language and literacy intervention.And it was a multi-state multi-systemic interventionwhere children were learning new vocabulary in their classrooms,new ways of discussing with their teachers.The parents were engaged in parent literacy courses.

  • 09:32

    SANDRA BARRUECO [continued]: There was learn-- there were lessons on English languagelearning and so forth.It was a five year study, as I mentioned, across two states.And in the beginning, it didn't seem like the interventionwas working.And in the first year, at the end of the first year,I went around and conducted focus groups and qualitativeinterviews with many of the key stakeholders that

  • 09:55

    SANDRA BARRUECO [continued]: were involved, the parents, the teachers, and the other staff.And what we learned through this workwas that the parents didn't seem to beusing the materials that we had boughtfor them to learn English.What we also noticed were that theyweren't going to the college classes

  • 10:16

    SANDRA BARRUECO [continued]: where we were providing free English language classes.And the question was why?And through the focus groups the qualitative studies,we found out it's not because theyweren't interested in furthering their own language learning.They were interested in learning new waysof engaging with their children around language.They were interested in checking out these backpacks

  • 10:37

    SANDRA BARRUECO [continued]: that we had made that had toys and manipulatives for themto use at home and engage and read with their children.What we learned is that they wanted to.They wanted to learn English.They wanted to use these materials.And there was one thing that was getting in their way,and that was their children interestingly.

  • 10:59

    SANDRA BARRUECO [continued]: They wanted to spend time with their children.Study was being conducted with migrant farmworkers who pick the fruits and vegetables that weeat every day.They work very long hours throughout the year,but particularly in summer.They're up at four or five in the morning.They're in the fields before it gets too hot.And they're picking and picking and picking our vegetables.

  • 11:19

    SANDRA BARRUECO [continued]: And at the end of the day, they want to see their children.They want to kiss them.They want to hug them.So the idea of going to an English language learning classat a community college was exciting,but would be a barrier for their contributionsto their own family and ability to hug their child.So what we did once we learn that was

  • 11:42

    SANDRA BARRUECO [continued]: we created English language classesfor the children and families and the childrenand the parents together, so that they couldsit down and learn together.And we made it more of a family event to learn English.And that actually helped the parentsfeel like they were moving along in their own English languagelearning, as well as helping their childrenand spending time with them.

  • 12:03

    SANDRA BARRUECO [continued]: Another important piece that came outof the qualitative research of focus groups and interviewsis that, as I mentioned before, theyweren't using these backpacks that had books and other toysin them.And we asked them why?Did it seem that they didn't reallywant to engage with their childrenaround these materials?Did they seem too foreign?

  • 12:24

    SANDRA BARRUECO [continued]: Were they too difficult?And it was related to these farm workers interestin serving in the community.We learned that they felt that if they use these materialsand brought them to their homes, which are very, very modest,they were concerned that in their children,they might rip up the books or dirty the materials.They were worried that they would take away

  • 12:45

    SANDRA BARRUECO [continued]: opportunities of language learning from other familiesin the community.So we learned to also explain that each of these materialscould be used to the best extent possible.They were to be used to read as much as possible, to playas much as possible, not to worrythat other families won't have a chance to engage with them.And what we learned from that research then

  • 13:07

    SANDRA BARRUECO [continued]: helped us strengthen the intervention.In the end, we strengthened the interventionbased on the qualitative research.We continued with the experimental designand found very strong effects with individuals in the sampleof about 200 to 300 families.In the end, we found positive impacts

  • 13:28

    SANDRA BARRUECO [continued]: on language, literacy, and socio-emotional developmentof children into such an extent that they came upto normative levels.And so therefore, my lesson learned from this five yearstudy and experiment is to couple itwith strong designs, such as qualitative features.

  • 13:50

    SANDRA BARRUECO [continued]: What would we have done differentlywith hindsight for this particular study?I would have conducted the focus groupsand qualitative research sooner, sayat three or six months than waiting a whole year laterinto the study.What are dos and don'ts for designing experiments?First, don't design an experiment only thinkingabout what you want to study.

  • 14:12

    SANDRA BARRUECO [continued]: Do consider how you will study it through strong measuresand creating conditions that are appropriate for the study.Second, don't assume that randomized experimentsare the best and the only methodology.Do realize that randomized experiments cannot be usedto answer many important questions,

  • 14:34

    SANDRA BARRUECO [continued]: and that other methods along with statistical analyzes havebeen developed to answer these in the most scientificallysound manner possible.In conclusion, experimental designis a critical tool within research.When conducted correctly under good conditions,they can provide scientifically valid and rigorous results.In developing them, it was important to address

  • 14:55

    SANDRA BARRUECO [continued]: many of the considerations describedand to examine what role they play in the full repertoireof research design approaches.Remember that best practice in experimental designincludes two key dimensions.First and foremost, human and sample considerations,including attending to ethical concerns.Second, scientific considerations

  • 15:16

    SANDRA BARRUECO [continued]: including identifying the variable that you are primarilyinterested in studying and deciding if and how itcan be isolated and independently examinedin studies.I'll leave you with two final reflective questions.What are the advantages and disadvantagesof use in experimental design, and what are the first stepsthat you will take when starting to design your own experiments?

Video Info

Publisher: SAGE Publications Ltd

Publication Year: 2018

Video Type:Tutorial

Methods: Experimental design, Quasi-experimental designs

Keywords: educational research related topics; random allocation; Second language acquisition; work-family balance

Segment Info

Segment Num.: 1

Persons Discussed:

Events Discussed:

Keywords:

Abstract

Dr. Sandra Barrueco examines the considerations and issues involved in effective experiment design. Barrueco illustrates with examples from her own research.

Looks like you do not have access to this content.

Designing Experiments: Skills for Success

Dr. Sandra Barrueco examines the considerations and issues involved in effective experiment design. Barrueco illustrates with examples from her own research.

Copy and paste the following HTML into your website