An Introduction to Using Twitter Data for Social Research.

[MUSIC PLAYING]

ZACHARY STEINERT-THRELKELD: I'm Zachary Steinert-Threlkeld.I'm a Professor of Public Policy at the UCLA LuskinSchool of Public Affairs.My PhD is in political science, and I use big datato study protest mobilization.And today, I'm going to talk about how you can use Twitterdata in your own research.

ZACHARY STEINERT-THRELKELD [continued]: I knew, when I was starting my dissertation,that I wanted to do something computational.I wanted to develop these skills.And so, that meant I knew I was goingto use Twitter or Facebook.I happen to be--I have an advisor who probably could have set me upwith Facebook, but Facebook would have veto power,because it's their private data.

ZACHARY STEINERT-THRELKELD [continued]: There are free public versions of it,but it's very limited because most people have their Facebooksettings private.So it was Facebook or Twitter, and Twitterwas public and free.Twitter is best for breaking news--

ZACHARY STEINERT-THRELKELD [continued]: things that people are talking about.So if you've seen something on television or a newspaper,you can study it in Twitter.Basically, anything that people are talking aboutis going to be on Twitter if it isin a country that's rich and with a lot of people. right?Twitter, even though it has about 300 million users,

ZACHARY STEINERT-THRELKELD [continued]: is not Facebook, it's not Instagram,and it's blocked in China.So even though you can almost study anything peopleare talking about you're a bit limited geographically.To gather and analyze Twitter data,you need some programming proficiency,

ZACHARY STEINERT-THRELKELD [continued]: and really R or Python knowledge would be best.Because you have to connect to what's called an ApplicationProgramming Interface and tell Twitter what you want,and then, Twitter gives it back to you.And it's going to give it back to youin this format that is JSON--JavaScript Serialized Object Notation.

ZACHARY STEINERT-THRELKELD [continued]: And so, it's actually very easy to work with.But most people with a social science backgroundare used to what's called comma separated values, solike an Excel worksheet.And this doesn't look like that.So you have to learn how to convert this JSON to somethingthat looks like an Excel worksheet.And so, once you've done that, then youcan use the statistical skills and knowledge that you already

ZACHARY STEINERT-THRELKELD [continued]: have, to manipulate the data.It turns out that most of data scienceis data cleaning and management.So most of your work is going to be getting the data,and then deciding what you want to keep.Maybe Twitter cut off some words.Or maybe when you converted it to CSV you lost some things,

ZACHARY STEINERT-THRELKELD [continued]: and so you need to add those back in,or you need to aggregate things.Most of the work of data science is getting to the stepwhere you can then do your statistical analysis.If you were looking to use Twitter data,

ZACHARY STEINERT-THRELKELD [continued]: the main thing to keep in mind isthat learning how to work with Twitter data, learning howto program, is a skill.And it's a difficult skill, especiallyif you have no experience, and it'sgoing to be to be frustrating.But if it's not frustrating, it means you're not developing.Right?So you're going to spend hours trying to figure something

ZACHARY STEINERT-THRELKELD [continued]: out, not understand why Twitter is not giving you data,or not understanding why the data you're gettingisn't what you thought you were going to get.But that's part of the process.Part of the frustration is that academics are justnow starting to understand that programming is its own skill.So I got my PhD in 2016, started doing all this 2012, 2013,

ZACHARY STEINERT-THRELKELD [continued]: and the kind of mentality was that it'slike learning multiplication, or it'slike learning how to write.The programming is something you haveto do to do the really interesting stuff.And that has started to change.People now understand that programming and data cleaningis not just background stuff that you have to do.It's own skill, like doing advance statistical work,

ZACHARY STEINERT-THRELKELD [continued]: or doing game theory.And so, there's a bit more recognition nowfor the value of learning those skills,so you don't have to constantly justify yourself.But that also means that you should notexpect it to be like as simple as opening Excel, or clickinga button, or something like that.

ZACHARY STEINERT-THRELKELD [continued]: It's difficult. Silicon Valley is huge for a reason.There's a reason there's a lot of demand for programmersand not very many programmers.When I was in that and teaching myself to program,the only way you can really keep going--and this is something my advisors at the time told me,and I didn't really appreciate it--

ZACHARY STEINERT-THRELKELD [continued]: was if you have some internal motivation.Right?You have to care about the project,or you have to really enjoy the logical thinking,the logical rigor that programming requires.Because if you're doing it because it's justcool to do that, or you think you're going to get a job,you're quickly going to get frustrated and not do it well.

ZACHARY STEINERT-THRELKELD [continued]: So you shouldn't say, oh, I want to program because of x.You should say, I want x.I have this research question, and I need these kind of data,and I need to program to get it.And then the programming is the last step.It's what will motivate you to keep with it whenit's really frustrating.

ZACHARY STEINERT-THRELKELD [continued]: I've learned that most people don'tcare about politics, actually.Yeah.Which is very humbling.It's something to keep in mind.So my dissertation was about the Arab Spring.So I was looking at tweets from a monthbefore things started, though a whole yearwhen things were going on.And Tunisia is having this revolution, and people in Egypt

ZACHARY STEINERT-THRELKELD [continued]: are talking about the traffic, or the weather,or Justin Bieber, I believe, had a concert in December 2010,in Cairo.This is what I think from reading these tweets.So I mean, once there's a revolutionary protest,yes people are talking about that, of course.But a couple of days before, not so much.A week or two after, they're backto talking about everyday things.

ZACHARY STEINERT-THRELKELD [continued]: The real fun of using Twitter and other social media datasetsis that it's like eavesdropping on everyone who'swalking by you on the street.And it turns out that people are just living their life.So to keep in mind when you're analyzing Twitter data,you're analyzing how people tweet,is that even though you see things kind of disconnected,

ZACHARY STEINERT-THRELKELD [continued]: because you're getting a sample of tweetsthat Twitter has provided, and we don't reallyknow how Twitter provides it onceyou tell Twitter some of the variables that you want.So it looks very disconnected to you, the researcher.But people are in their bubbles.People are going to search out one or two subjectareas they want to follow.

ZACHARY STEINERT-THRELKELD [continued]: It could be news.It could be celebrities.It could be sports.And as the researcher, the only wayyou would know that is if you then looked at who they followand classified those accounts.Which you can do, but that's a lot of extra workafter you've already acquired the tweets.So people's experience on Twitter,and really on social media, and really in life in general,

ZACHARY STEINERT-THRELKELD [continued]: is not going to be the aggregate of the datathat you're looking at.It's going to be it's going to bethe aggregate of a very narrow and deep part of what you'relooking at.And as the researcher, starting with just a sample of tweets,you can't really get at that.