Skip to main content
Search form
  • 00:01

    [MUSIC PLAYING][Understanding Protests Using Geolocated Images& Convolutional Neural Networks]

  • 00:10

    ZACHARY STEINERT-THRELKELD: I'm Zachary Steinert-Threlkeld.I'm a Professor of Public Policy at the Universityof California Los Angeles Luskin School of Public Affairs.[Zachary Steinert Threlkeld, PhD, Assistant Professor,Public Policy, UCLA Luskin School of Public Affairs]I use big data, primarily Twitter,to study protest mobilization.[How did you start using Twitter data in your own research?]

  • 00:31

    ZACHARY STEINERT-THRELKELD [continued]: My current research is using geolocated imagesto understand factors affecting protestthat we wouldn't be able to understandwithout geolocated images.So those are primarily actually the size of protests,which is very hard to determine with traditional methods,the amount of violence, as well as

  • 00:51

    ZACHARY STEINERT-THRELKELD [continued]: whether violence is from protesters or stateauthorities, as well as demographic informationof individual protesters.And we can code all that using machine learningalgorithms based on images shared from protests.I became interested in that because my earlier work wasusing tweets, using texts to understandsome features of protest, and you

  • 01:15

    ZACHARY STEINERT-THRELKELD [continued]: can do some interesting stuff with that,but you don't really know anything about the protesters.And to know things about the protest,you have to rely on newspapers, which academicshave done for decades and are very limitedand very known ways.And what I saw was that a lot of the researchwas focused on getting more data from newspapers,

  • 01:38

    ZACHARY STEINERT-THRELKELD [continued]: but you're still limited by newspapers.It's kind of like if you're tryingto get more coal from a mine, but you're still mining coalat the end of the day.And so it occurred to me that imageswould circumvent a lot of these problemsthat we were having with newspapers.There's a substantive protest part,and there's an application of the methodology out

  • 02:00

    ZACHARY STEINERT-THRELKELD [continued]: for protests, as well.We haven't been able to measure size, violence,and demographics of protesters in essentially real time.And in doing that, we, first off, can actuallybuild models that take into account these things that webelieve that we think we know actually matter, right?So people study whether women or children or the educated

  • 02:23

    ZACHARY STEINERT-THRELKELD [continued]: are more likely to protest.They study, well, how does the state usingviolence affect whether protests becomes larger or smaller?But it's always just that one thing separately.And so you get a lot of different results,and it's quite possible that you get the different resultsbecause you're not studying how they operate at the same time.So we can look at a protest, instead

  • 02:43

    ZACHARY STEINERT-THRELKELD [continued]: of having to run out and do a surveyand just get age and gender or something like that,or study a protest and just try to documentwhat the police are doing.We can do it all at once.And since we're using big data and machine learning,we can do it really wherever there is a protest.So the ability to attach much more accuratelymeasure variables that affect protest size

  • 03:06

    ZACHARY STEINERT-THRELKELD [continued]: is the fundamental advance.There's then a professional merit to that.It should publish well and get cited.There's certainly applications thatcould be used, as well, for protesters or for governmentauthorities.But one apolitical application would be using our technology

  • 03:31

    ZACHARY STEINERT-THRELKELD [continued]: as a filter.So we look at all our images and we say,OK, well, is this one very likely to be a protest,kind of likely, not so likely?Also, is it very likely to have violence, not violence?We get a rating.And so you can imagine, if you'rea news organization with large photo archives or a wire

  • 03:51

    ZACHARY STEINERT-THRELKELD [continued]: service, you have more photos than you know what to do with.And right now they just use tags.So someone has to manually say this is protest,this is violence, or something like that.But you could use these neural networksthat we've developed to say, well,give me an image where the protest score is 0.8 or above,or violence 0.2 or below.So it would make it a lot easier for journalists or editors

  • 04:14

    ZACHARY STEINERT-THRELKELD [continued]: to identify content to add to their newspapersor content to package for clients.[What data do you use in the research?]The images from Twitter are uploaded by individuals.They theoretically could be uploaded by news organizations,

  • 04:35

    ZACHARY STEINERT-THRELKELD [continued]: as well, but they overwhelmingly are notuploaded by news organizations because theyhave geocoordinates.They have latitude and longitude coordinates,and organizations don't really do that.And so that both gives us high confidencethat it's individuals just documentingwhat's happening at the protest, and it'show we then know where the protest is happening.

  • 04:58

    ZACHARY STEINERT-THRELKELD [continued]: So we actually analyze all imagesthat are shared to see if it's a protest.And then if it is a protest image,we know where it's happened because we have GPS coordinatesthat go with the tweet.[How do you collect and analyze image data?]We collect the data in real time from Twitter.

  • 05:20

    ZACHARY STEINERT-THRELKELD [continued]: I've been collecting it for years just to have.We collect it using Amazon Web Serviceand then store it in a server in my office.And then we have a computer read the tweets.We write a program to open all the tweets.If it has an image, we download the image.And then we apply a convolutional neural network

  • 05:41

    ZACHARY STEINERT-THRELKELD [continued]: to each image.So we had to customize the convolutional neural network.That's just a model.It's a statistical thing that gives you an output.So we first gave it 100,000 imagesthat we got from Google Image searches.We searched for protests using wordslike "protest," "Black Lives Matter," "Women's March."

  • 06:02

    ZACHARY STEINERT-THRELKELD [continued]: And so we got about 10,000 images that Google returned,and we got about 90,000 hard negatives,so things that look like protists, but that, as a human,we know are not protest.And so we built the convolutional neural networkto identify protest images.So then, once we had an image in our Twitter data set,we applied this protest classifier to the images.

  • 06:23

    ZACHARY STEINERT-THRELKELD [continued]: So then we got--for where the project is right now, we got about 125,000highly likely protest images.And so we took about 40,000 of thoseand gave them to people on Amazon Mechanical Turk.So these are contract workers.You pay them by an image.So you pay them cents per task.

  • 06:47

    ZACHARY STEINERT-THRELKELD [continued]: And we asked them, does the image contain a variablethat we're interested in?So we said, does it contain police?Does this image document fire?Does it document violence?And so then we did--there were 12 variables for each of those images.We then got 12 yes-no responses from the Mechanical Turkers.And so then we used that to build

  • 07:07

    ZACHARY STEINERT-THRELKELD [continued]: a second convolutional neural network that would determine,oh, there are police in this image, or there are women,or there is violence.And so we built the second convolutional neural networkand applied that to all of the 125,000 highly likely protestimages.

  • 07:27

    ZACHARY STEINERT-THRELKELD [continued]: The convolutional neural networksis the real innovation.That's, like, the meat of the advanceis being able to have a computer analyze images reliablybecause, previously, you're just using humans,which both is very slow and unreliable because we gettired or cranky or whatever.

  • 07:48

    ZACHARY STEINERT-THRELKELD [continued]: Like, we're humans.So the computer lets you just analyze these all at once.And so the real innovation is the convolutional neuralnetworks applied to images shared on Twitter.But you could do it for any image.Twitter was the easy data set.But you could get an image from Instagram.

  • 08:08

    ZACHARY STEINERT-THRELKELD [continued]: I've taken images at places I've been.You can feed it all to this model, now, that it's trained.We typed in crowd-related terms.So we typed in--so this is it the Google Image search.We typed in "concert," "rally," maybe, like,"sports" or "sporting arena."So we basically wanted to get large crowds of people

  • 08:28

    ZACHARY STEINERT-THRELKELD [continued]: that were not protest.So it could be that Google Image did returnto us things that were actually protestand they had just mislabeled.But based on the output from that first stage neuralnetwork looking at what it said is a protest or not a protest,it seems like both the true positives

  • 08:48

    ZACHARY STEINERT-THRELKELD [continued]: and the true negatives that we got from Googleseem to have been pretty good.There are always going to be false positives whenyou're having computers code something, and we've actually--we've decided on a precision level of 85%.And so what that means is once we trained our modeland we said, OK, well, it says this is police,

  • 09:09

    ZACHARY STEINERT-THRELKELD [continued]: this is not police, police, police, and so on,we give those back to humans.So we went back to Mechanical Turk and we asked them,does it contain police, does it contain fire?And so we could see when the computer thinksthere are one of these variables, whatis the threshold?What is the probability the computer gives?And we set the threshold such that when

  • 09:32

    ZACHARY STEINERT-THRELKELD [continued]: we go from the probability to yes or no,we'll know that 85% of the yeses are true positives.So then we'll only have 15% false positives.So that's, like, the art part of it.You could set it at 90%, 80%.It kind of depends on how much data you want,but also how much noise.

  • 09:52

    ZACHARY STEINERT-THRELKELD [continued]: [Were there any challenges in collecting and analyzingthe data?]It turns out there are very few protest images.So it's about 0.05% of all tweets have protest images.10% of tweets have images, based on our sample,but I suspect that's true for tweets we didn't look at.

  • 10:16

    ZACHARY STEINERT-THRELKELD [continued]: So 0.05% means out of 2,000 tweets,one contains a protest image.So it sounds really cool to say, oh,we're getting millions of tweets per day.We have billions of tweets.That's all true.But we actually have very little protest data.We have a lot of protest images.We end up coming up with about 30,000.But we had to look at 55 million tweets to get those 30,000.

  • 10:37

    ZACHARY STEINERT-THRELKELD [continued]: So it's data mining.I mean, it's really the equivalentof going underground to find the diamonds or the coalsor whatever.That was surprising.I thought-- we knew we were looking at protests periods,and so I thought we would have many more protests imagesthan we had.So we originally went to look at 14 protests,

  • 10:59

    ZACHARY STEINERT-THRELKELD [continued]: and of those, we could only use four.Only four had enough data.They all had protest images, but sometimes it'dbe one image every two days or two images a day.There were only four protests wherewe had 10 images per day, which we decided was enough.So the challenge is just getting enough data,getting enough relevant data, and circumventing

  • 11:22

    ZACHARY STEINERT-THRELKELD [continued]: it is the next part of the project.So to circumvent it we have essentially two strategies.One is to make the classifier better.So I told you 85% precision.We have very low recall, about 30%.And so what that means is, in our training data,we know, yes, protest image, no, not protest image.

  • 11:42

    ZACHARY STEINERT-THRELKELD [continued]: And it turns out that with the threshold we set,of the images that are protest, we only keep 30%.So we need to train the neural network betterto get that 30% as close to 100%.And so that's going to be the easiestway to go from maybe 28,000 images to 50,000 or 100,000.And then once we have trained a better network,

  • 12:05

    ZACHARY STEINERT-THRELKELD [continued]: we also just want to collect more images.And there are a multitude of waysto collect more images both on Twitter and outside of Twitter.[What advice would you give students interested in thiskind of research?]For a student interested in analyzing images--it doesn't have to be protest images, really just analyzing

  • 12:27

    ZACHARY STEINERT-THRELKELD [continued]: images--it's going to be a student who's interested in codingand programming and kind of, like, machinelearning, computational social science, big data more broadly.And really the best thing you can do is learn Python.The best skill you can have is comfort with Pythonbecause anything that comes out of the computerscience or the electrical engineering community

  • 12:49

    ZACHARY STEINERT-THRELKELD [continued]: is going to be in Python first.There might be the really hard core,like C++ or some other obscure language,but they're always trying to promote themselves,and at this point that means Python.So there's almost always going to be a Python package to use.Sometimes there will be R. There's nevergoing to be Stata.So it should be Python, then R, then Stata.

  • 13:11

    ZACHARY STEINERT-THRELKELD [continued]: R is great, and if you need to learn Rto work with collaborators, that's fine.But there's almost nothing R can do that Python can't.And on top of that, Python is just easier to write.It's a little bit easier to read.R is basically from the statistics community,and the community of computer scientists--

  • 13:32

    ZACHARY STEINERT-THRELKELD [continued]: there are more programmers than there are statisticians.So on a lot of levels, it's just a lot easier to use Python.You need to get comfortable using a good text editor.Some people say you need to use an integrated developmentenvironment, an IDE.And those are fine if you're doingreally complicated programming.

  • 13:53

    ZACHARY STEINERT-THRELKELD [continued]: Most of the time, though, that's goingto be overkill for the sort of stuffthat social scientists use.So a good text editor like Sublime Text--you could talk about Emacs or vim,but Sublime Text is really good.I know there is some in the Microsoft world.I think Notepad is one that people like.TextWrangler people like.But a good text editor that is lightweight,

  • 14:16

    ZACHARY STEINERT-THRELKELD [continued]: it's going to help you complete code.It's going to suggest things.It's going to have packages that give yousome more functionality.It's going to make you a much more productive coder.[Further Reading, Won, D., Steinert Threlkeld,Z. C. & Joo, J. (2017).Protest Activity Detection and Perceived Violence Estimationfrom Social Media Images.In Proceedings of the 2017 ACM on Multimedia Conference (pp.786 794.) ACM.][MUSIC PLAYING]


Zachary Steinert-Threlkeld, PhD, Assistant Professor of Public Policy at the UCLA Luskin School of Public Affairs, discusses using geo-located images & convolutional neural networks to understand protests, including using Twitter data in research, collecting and analyzing image data and its challenges, and advice for students interested in this type of research.

Video Info

Publication Info

SAGE Publications Ltd
Publication Year:
SAGE Research Methods Video: Data Science, Big Data Analytics, and Digital Methods
Publication Place:
London, United Kingdom
SAGE Original Production Type:
SAGE Case Studies
Copyright Statement:
(c) SAGE Publications Ltd., 2019


Zachary Steinert-Threlkeld

Segment Info


Segment Num: 1


Segment Start Time:

Segment End Time:


Things Discussed

Organizations Discussed:

Events Discussed:

Places Discussed:

Persons Discussed:

Methods Map

Convolutional neural networks

Convolutional neural networks are a type of deep neural network that utilize multiple levels of input to process complex data. They are often used to process and classify visual data.
Convolutional neural networks
Understanding Protests using Geolocated Images and Convolutional Neural Networks

Zachary Steinert-Threlkeld, PhD, Assistant Professor of Public Policy at the UCLA Luskin School of Public Affairs, discusses using geo-located images & convolutional neural networks to understand protests, including using Twitter data in research, collecting and analyzing image data and its challenges, and advice for students interested in this type of research.

Copy and paste the following HTML into your website