Skip to main content
SAGE
Search form
  • 00:00

    [MUSIC PLAYING]

  • 00:10

    NIEKY VAN VEGGEL: Hello.My name is Nieky van Veggel.I'm a Senior Lecturer in Bioveterinary Scienceat Writtle University College Chelmsford, UK.My main interest lies in evidence based practicein animal health and in higher education.And in addition to that, I have a main interestin helping students and student support in numeracy

  • 00:32

    NIEKY VAN VEGGEL [continued]: and transitioning to higher education.This video is designed to explain to you howyou can combine a number of different types of information,in this particular case three sorts of information.And in this case, we're mainly focusingon the combination of demographic data, questionnaire

  • 00:53

    NIEKY VAN VEGGEL [continued]: data, and focus group data to evaluatean educational intervention in a cohort of students.The plan is that at the end of this video,you'll have an idea of where to startand if you need any further information on whereto find that information.

  • 01:14

    NIEKY VAN VEGGEL [continued]: For this particular film, we're going to focus on a projectthat I did in 2012 where we focusedon an educational intervention on a cohort of highereducation animal science students.These students were on the enrolls on BSE honorsand foundation degree and science courses

  • 01:36

    NIEKY VAN VEGGEL [continued]: in animal science and animal management.Students on this particular research projectwere recruited indirectly at the start of every first yearintake in the Animal Science courses.We give these students a numeracy diagnostic testto see where they are with their math skills.

  • 01:57

    NIEKY VAN VEGGEL [continued]: And I mainly myself mark this test on the scale of 0 to 10and every student scoring 4 out of 10gets the advice to undertake a series of math tuitiontutorials to help them get back up to scratch with their maths.We're going to be using three different types of data.

  • 02:20

    NIEKY VAN VEGGEL [continued]: As I mentioned before, we've got a questionnaire data.We've got demographic data.We've got focus groups.And in addition, we've also lookedat student performance in diagnostic tests,but also in their formal modules that are part of that course.

  • 02:40

    NIEKY VAN VEGGEL [continued]: In order to investigate student confidencein maths and numeracy, I set up an online questionnaireand I e-mailed a direct link of this questionnaireto all students in the first year cohort asking them to takepart if they wished to do so.And this was both for students who did and did not

  • 03:00

    NIEKY VAN VEGGEL [continued]: undertake the tutorials.Students were obviously informed of the nature of this study.They were promised confidentialityso that we were at least able to get honest opinions.And students were also informed of my role in the researchprojects.And they were therefore so explained

  • 03:22

    NIEKY VAN VEGGEL [continued]: that if they had any further issues or questionsor any concerns, to come and see meas I was one of their tutors.The first section of the questionnaireconsists of a number of demographic questionsabout their background, their previous math skills,and what kind of courses they didbefore they came to Writtle to study Animal Science, their age

  • 03:43

    NIEKY VAN VEGGEL [continued]: and various other questions.After which you moved on to the more confidence questionsand asking students to give themselvesschool confidence score of 0 to 10and a number of questions regardingtheir experience with the tutorials and how they got onand how they felt that they did and thesewere assessed on a scale of one to five,

  • 04:05

    NIEKY VAN VEGGEL [continued]: and so essentially Likert Scale.The section on confluence was retrospective and self-reportedwhich has a weakness or few criticisms, which you'llsee later on, where students rated themselveson the skill of 0 to 10, and 0 being not confident at all,10 being very, very confident.

  • 04:25

    NIEKY VAN VEGGEL [continued]: And was supplemented by a number of Likert questionson a scale of 1 to 5 regarding their experiencewith the tutorial system.And this was finished off with a number of open ended questionswhere students were asked to give ustheir honest opinion of the tutorial systemand whether or not they found it useful.

  • 04:49

    NIEKY VAN VEGGEL [continued]: In addition to the survey, we alsoheld a number of focus groups where students wereinvited to come and take part.They did receive a monetary incentivein the form of Amazon vouchers because theyare students after all.And in these focus group discussions,

  • 05:10

    NIEKY VAN VEGGEL [continued]: we try to explore the trends from the questionnairea little bit further.Obviously, we couldn't directly link the outcomesof the questionnaire to the outcomes of the focus groups,because the questionnaire was anonymous.But the focus groups are also open to all students,whether or not they took part in the questionnaire or not

  • 05:34

    NIEKY VAN VEGGEL [continued]: and whether they took part in the tutorials or not.And it mainly explored reasons as to why studentswould and would not take part in tutorials, what it wouldtake to get them to take part.And also, for the students that did take part,we tried to delve live with deeper into their experiences.

  • 05:55

    NIEKY VAN VEGGEL [continued]: [Student Performance]Finally, we also analyzed the student performancein both the diagnostic test and in a formal curriculum module.For the students who did the tutorials,they received a diagnostic test at the endof the set of tutorials, whereas, all students received

  • 06:17

    NIEKY VAN VEGGEL [continued]: a diagnostic test at the start of the academic year, whichallowed us, for the students who took the tutorials, to actuallymake a direct before and after analysis.Because that part was not anonymous.We knew which students were taking part in the tutorials.And we could actually see a very crude effect

  • 06:38

    NIEKY VAN VEGGEL [continued]: of that tutorial system.In addition to that, we also looked at the effectthat the tutorials had on the actual performancein a formal numeracy module.And because we knew which students took placein the tutorial set, we also could makedirect links between students who did and did not take part,

  • 07:02

    NIEKY VAN VEGGEL [continued]: and whether taking part in the tutorialhad an effect on the formal curriculum,and whether or not there was a differencein students who did or did not take partin the actual mark or grade that they got in that module.The diagnostic tests which we usedto evaluate the tutorial program contained12 questions, which were very basic numeracy questions, where

  • 07:23

    NIEKY VAN VEGGEL [continued]: students were asked to add, take away, multiply,divide, do some square roots and some very simplelinear algebra, all of this whichwas to be done without a calculator.And this was basically just to see how confident they were.And the level of this diagnostic test was roughly at GCSE level.

  • 07:46

    NIEKY VAN VEGGEL [continued]: So all students, in theory, should be able to do it.The post-tutorial diagnostic test, if you will,was delivered by the math tutor who was an independent tutorand was not part of the course team for the students.Whereas the pre-tutorial diagnostic test

  • 08:07

    NIEKY VAN VEGGEL [continued]: was done in week one of the formal module and wasdelivered by me at the end of a timetabled sessionthat all students were taking.[Data Analysis]A total of 30 students took the tutorial sessions.And after the diagnostic tests were taken,

  • 08:30

    NIEKY VAN VEGGEL [continued]: I analyzed the data.And I sought to find correlations between attendingtutorials and performance in modules, and in addition, alsohow many tutorials the students attendedversus their performance and whether therewas a critical number of tutorialsa students should attend before their performance was affected.

  • 08:55

    NIEKY VAN VEGGEL [continued]: After all the data was collected,we analyzed it using Microsoft Excel and IBM SPSS 19.The self-reported confidence levels and the demographic datawere analyzed using bivariate, mainlyFisher's exact and Chi-squared.Whereas, the diagnostic performance

  • 09:18

    NIEKY VAN VEGGEL [continued]: was measured using a t-test before and after.And the other performance data was measuredand was analyzed using a matched-pairs t-test.Mainly because British students cancome into these projects having various experiences with math.

  • 09:38

    NIEKY VAN VEGGEL [continued]: It's important to make sure that the two groups that you analyzeare actually matched for details such as previous mathsexposure, age, previous courses, and the likes.Because otherwise, you'll introduce some skewnessinto your analysis.As I mentioned earlier, this was research

  • 09:60

    NIEKY VAN VEGGEL [continued]: done on a self-selected sample.It was not a probability sample.And although there are some drawbacks with that--main issue is you can't always be surethat your sample is actually representativeof the population that you're trying to investigate--in this particular case, the sampleconsisted of nearly half the population.

  • 10:20

    NIEKY VAN VEGGEL [continued]: And we statistically showed using Chi-square analysisthat the actual contribution of the various aspectsof the sample were relevant to the population, the samplerepresented the population nicely,and that all subgroups within the population

  • 10:41

    NIEKY VAN VEGGEL [continued]: were pretty much evenly represented in the sample.Lastly, with the focus groups, weheld a number of focus groups with up to 10 studentsper group.And these focus groups were recorded and afterwardstranscribed by a research assistant, whichwas an undergraduate student themselves

  • 11:02

    NIEKY VAN VEGGEL [continued]: who was not in the same cohort.And after transcription, we used the text, or the transcriptionreports, to find themes in the discussions.And we grouped these themes to cometo what turned out to be four themesand motivators for students that either door do not take part in these programs.

  • 11:25

    NIEKY VAN VEGGEL [continued]: [Practical Considerations]When I started the project, first, wehad to come up with a questionnaireand an overall research design.The questionnaire design was quite critical.And it took me a number of attempts to get it right,particularly with relation to the clarity of the questions.

  • 11:48

    NIEKY VAN VEGGEL [continued]: In questionnaire-type research, itis essential that your questions are clearand that they can only be interpreted in one way.So I used a number of pilot rounds, if you will.In this case, I think I used about three pilot roundsto make sure that the questions were understandable.I piloted it with first-year undergraduate students, but not

  • 12:10

    NIEKY VAN VEGGEL [continued]: the students that were potentiallythe target of the questionnaire, justto make sure that they wouldn't see the questions beforehand.And in this case, because it was goingto be an online questionnaire, there's alsothe choice of platform to deliver your questionnaire.initially, I was playing with the thought

  • 12:30

    NIEKY VAN VEGGEL [continued]: of using SurveyMonkey.But once I ran and set up the questionnaire on there,I was a little bit disappointed with how muchyou could actually do with the free version.And although this was a funded project,the funding didn't extend to gettingpaid to use SurveyMonkey.And this was when I realized that the use of Google Forms

  • 12:51

    NIEKY VAN VEGGEL [continued]: is actually incredibly powerful, because you got complete designand technological control over your questionnaire.You can put images and videos in to show.You can design it how you want.All question types are there.And the beauty of it is that all of your datais recorded, in this case, a Google spreadsheet, which

  • 13:13

    NIEKY VAN VEGGEL [continued]: is exportable as a Microsoft Excel spreadsheet.So it also saves a lot of transcription work.Because there is some published evidence out therethat says that entry profiles can predict outcomes of highereducation courses, I chose to take some demographic datafrom the institutional databases and correlate it

  • 13:35

    NIEKY VAN VEGGEL [continued]: with these diagnostic tests and with the student results.And in this case, because both the diagnostic testand the actual performance in a module was not anonymous,I could take the data from the institutional databasesfrom data that is routinely collected from students.So think, for example, about gender, age,

  • 13:58

    NIEKY VAN VEGGEL [continued]: previous qualifications, entry qualifications,what kind of exposure to maths students have had in the past,because we could deduce that from the previousqualifications, and, for example,where students were from.Because this was an online questionnaire,we e-mailed the link to the questionnaireto the entire cohort of students.

  • 14:21

    NIEKY VAN VEGGEL [continued]: And because return rates for online questionnairescan be notoriously low, we did remind the studentson a regular basis to complete the questionnaireif they had not done so.And this was both done by me and some colleagues.In the end, this meant that although studentswere quite often reminded of the questionnaire, this, in turn,

  • 14:46

    NIEKY VAN VEGGEL [continued]: led to a 45% response rate for this particular project, whichwas quite high.And because we were working on a fairly small population,it was essential that we call as many students to fill outthis questionnaire as possible.As I mentioned earlier, for the analysis of performance datain this project, we used a matched-pairs analysis,

  • 15:09

    NIEKY VAN VEGGEL [continued]: mainly because we needed to be unable to makea fair comparison between student groups.The matched-pairs analysis itselfis not particularly complicated.However, the specific detail of itgoes beyond the scope of this video,and I would refer you to various sources that are available,such as the SAGE Research Methods collection

  • 15:30

    NIEKY VAN VEGGEL [continued]: or Biomeasurement by Dawn Hawkins.The main reason for the approach that Iused for this particular project was not necessarilybecause this was the best way to do things.It was mainly a pragmatic solutionto doing a student-focused research projectin a busy academic year where I hadto mold the research around my own timetable,

  • 15:52

    NIEKY VAN VEGGEL [continued]: but also the student timetable and the student availability.For example, the student focus groupscouldn't take place until after all the tutorial slotshad taken place.And because this was a weekly tutorial program,it took 12 weeks for everything to finish.To then have questionnaires to go to the students

  • 16:14

    NIEKY VAN VEGGEL [continued]: and focus group discussions to finish everything off with,there was only one real way of doing it,which is the way we've actually used for this project.[Lessons Learned]Although this project went fairly smoothly,and we had very little issues, therewere a couple of small things that, if were

  • 16:36

    NIEKY VAN VEGGEL [continued]: to have to do this project again,I would pay a bit more attention to and therefore probablycost myself a little bit less frustration.So therefore, a top tip and five top tipsfor people who are looking to do a project like this.Tip one-- use pilot studies.Pilot studies are incredibly useful to fine-tune

  • 16:57

    NIEKY VAN VEGGEL [continued]: your questions and to make sure that everybody understandsyour questions and they can be interpreted in only one way.I found out that what makes perfect sense in my headquite often does not make perfect sense to students.And therefore, not doing pilot studiesor not doing them properly means that the actual outcomeof your research is going to be pretty useless.

  • 17:20

    NIEKY VAN VEGGEL [continued]: Tip two-- choose a suitable platform for your questionnairedelivery.In this case, we used online questionnaires,because it was convenient.I could have chosen telephone questionnaires or paperquestionnaires.But even within the realm of online questionnaires,there are so many options that it'sgoing to be important to choose something that works for you.

  • 17:42

    NIEKY VAN VEGGEL [continued]: There is no ideal platform every platformhas positives and negatives.And it makes sense to first look at whatever'savailable in your institution.And if you haven't got anything available,try to find a good, preferably free option.And I can highly recommend Google Forms.

  • 18:02

    NIEKY VAN VEGGEL [continued]: Tip three-- response rates.In order to make sure you get enough responses to havesomething meaningful to say about your population,it's important to remind your cohortson quite regular intervals to do your survey.My approach was to use both email remindersand verbal reminders at the start

  • 18:24

    NIEKY VAN VEGGEL [continued]: or at the end of lectures.And in this case, I did that for about three weeks--not because three weeks is ideal, but because three weekswas convenient for this research.And after three weeks, unfortunately,the semester ended, and the studentswere not there anymore.And therefore you lose any type of control or influence

  • 18:45

    NIEKY VAN VEGGEL [continued]: you have over those students, because e-mails are easilyignored.Tip four-- focus groups.Focus groups are very useful, but they're also very intense.There can be a lot of things happening at the same time.People can be talking at the same time.So it's important that when you chairor when you coach focus groups that you keep control overwhat happens.

  • 19:07

    NIEKY VAN VEGGEL [continued]: Try to record them-- either voice,or if your participants allow it, try the video record them.Because it can be quite difficult to allocatecertain pieces of text to certain members of the focusgroup.And transcription can be quite difficultif people talk over or through each other.And as a practical thing, make sure

  • 19:29

    NIEKY VAN VEGGEL [continued]: that the software that your recording device usesis accessible and that there are no things that you'llfind out afterwards.So for example, for this particular project,we used a voice recorder which happenedto have proprietary software thatwasn't available on Macintosh.And therefore we needed a Windows computer

  • 19:49

    NIEKY VAN VEGGEL [continued]: to transcribe it to make it available to us,which although not a problem, does cause delays.Tip five-- on your enrollment dataor your institutional data, it's a very rich and useful sourceof data on your student subject and on your cohort.However, when you ask for this data

  • 20:09

    NIEKY VAN VEGGEL [continued]: from the people that manage it, makesure you ask them specifically for things that you are after.And tell them in which format you want the data.Retrospectively, I was probably not entirely clearon how I wanted the data.And although I received the data in a very good formatand in a spreadsheet, I had quite a lot of work

  • 20:29

    NIEKY VAN VEGGEL [continued]: to make sure that the data was in a formatthat I could actually use easily and that could be importedinto statistical software packages,just because that was possibly not entirely clearto the staff that were extracting the datafrom the system for me.[Conclusion]Hopefully, you now have an idea of how

  • 20:51

    NIEKY VAN VEGGEL [continued]: to undertake similar projects where you combinequantitative data, qualitative data,and institutional and demographic datato evaluate research projects and evaluate interventions.There are, of course, many, many ways of doing similar projects.And each you can use has its own drawbacks

  • 21:14

    NIEKY VAN VEGGEL [continued]: and its own positives.But at least it's a start.And hopefully you will now also havean idea of where to find some moreinformation on these approaches shouldyou wish to find out more.[MUSIC PLAYING]

Video Info

Publisher: SAGE Publications Ltd

Publication Year: 2018

Video Type:Interview

Methods: Mixed methods, Demography, Questionnaire design, Focus groups, Evaluation

Keywords: diagnostic testing in educational settings; educational assessment; incentive systems; lessons (education); performance appraisal; practices, strategies, and tools ... Show More

Segment Info

Segment Num.: 1

Persons Discussed:

Events Discussed:

Keywords:

Abstract

Nieky van Veggel discusses his research using mixed methods. Veggel explains his research design, findings, and the lessons learned throughout his research.

Looks like you do not have access to this content.

Using Mixed Methods to Evaluate an Educational Intervention in a Cohort of Students

Nieky van Veggel discusses his research using mixed methods. Veggel explains his research design, findings, and the lessons learned throughout his research.

Copy and paste the following HTML into your website