• ## Summary

Search form
• 00:02

SELMA AVDIC: Hello, everyone.Thank you so much for joining us today.My name is Selma Avdic with the American Marketing Association.And I want to welcome you all to our member only webcast.This is titled "Artificial Intelligence for Marketing:Getting Started."So before we get started, I just wantedto cover a few important housekeeping items.

• 00:25

SELMA AVDIC [continued]: So this webcast is being recorded and will be madeavailable to you soon, so please look out for thaton ama.org/webcast.We also encourage everyone to continue this conversationon Twitter.And you can do so by referencing #amamow.

• 00:45

SELMA AVDIC [continued]: And if you have any technical or content-related questionstoday, please feel free to ask them at any time.You can use our chat box that's locatedto the left of your screen.So with that, I'm very pleased to introduce our speakerfor today.We have Jim Sterne with us, and heis the board chair at the Municipal Analytics

• 01:07

SELMA AVDIC [continued]: Association.So just a little bit about our speaker before Iturn it over to him.So he has written 12 books on internet advertising,marketing, and customer service, including Devil's DataDictionary, and his latest, Artificial Intelligencefor Marketing: Practical Applications.

• 01:30

SELMA AVDIC [continued]: He has also produced the eMetrics Summit since 2002and was also named one of the 50 most influential peoplein digital marketing by Revolution,which is the UK's premier interactive marketing magazine,as well as one of the top 25 hot speakers by the National

• 01:50

SELMA AVDIC [continued]: Speakers Association.So I'm very pleased to have him with us.And I will now turn it over to Jim to get us started.

• 02:01

JIM STERNE: Thank you very much, Selma.It is a real pleasure to be here.I'm delighted to participate, and I'mglad that you all have logged in for this.My job for the last 20 years or sohas been to look over the horizon and see what's coming.In '95, I waved the flag and waved my hands and said,

• 02:22

JIM STERNE [continued]: hey, we're going to be marketing on the internet.Take a look.Back in 2002, gee, this web analytics and digital analyticsstuff is really valuable.And that started the conference, and the conference audiencestarted the association.So I've had pretty good luck so far finding

• 02:42

JIM STERNE [continued]: things that are important to marketing people.That brings us to 2017, where artificial intelligenceis happening.Now, this is an area that is rife for misunderstandingand lack of clarity.And I'm sure you heard of these top four or five

• 03:06

JIM STERNE [continued]: different terms, but the rest of them, no.And because you're marketing people, you're not going to,and that's OK.So my job is to help explain how this is important to youas a marketing person.I am not here to teach you how to do data science.

• 03:29

JIM STERNE [continued]: This is not about the specific tools.This is not about the cool startups.It is specifically how do you keep your job.What is it that we need to know as marketing peopleto make this all happen?And essentially, I want to give you the framework.

• 03:49

JIM STERNE [continued]: If somebody at a dinner party brings upartificial intelligence, I want youto know what they're talking aboutand be able to join the conversation.If you get stuck in an elevator with your boss's boss,and they say, what are we doing about machine learning,you're going to be comfortable instead of breaking outin a cold sweat.That's my job today.

• 04:12

JIM STERNE [continued]: We begin with the vision that everybodyhas about artificial intelligence, whichis when Skynet becomes self-aware and takesover the world and starts killing people.That is what is referred to in the businessas strong AI, which pretends to act like a human and then,

• 04:33

JIM STERNE [continued]: of course, become self-aware and kills everybody.We're actually thinking and worrying about weak AI,which is a very unfortunate term.But what it means is something that is specifically designedto perform a specific function.It is the production version of all of this technology.And it's where we're going to get real value out

• 04:54

JIM STERNE [continued]: of this stuff.So it's not about the science fiction.It's about what can we actually get done.Computers start with programming languages.It is very linear and very specificof do this and that and that and that, and if this, then that,

• 05:15

JIM STERNE [continued]: and if I put the semicolon in the wrong place,it all falls over and spits up an error.That's what we've all gotten used to.And we tried to build expert systems, rules-based systemsthat if somebody clicks on this more than that,take this action.If somebody opens or doesn't open their email,

• 05:36

JIM STERNE [continued]: take this action.And it requires lots of thinking in advance.It takes lots of agreement about who our target audience is,and what we want to do, and how we are goingto influence their behavior.And it takes millions of lines of code to make that happen.The next step over is the mathematical model,

• 05:56

JIM STERNE [continued]: which is not nearly as interesting as it sounds.It's what we all do with Excel.I have a mathematical model of my budget at home.How much can I spend on groceries this month?How much do I need to save in order to buy a car next year?That's a very simple model.It's how we play what if games.

• 06:17

JIM STERNE [continued]: Then we move on to statistical models.Now, this is predictive analytics.This is the stuff where I do need somebody with a mathbackground to build the models for me that predict the future.And I am going to create a model,and run it, and run it against current data

• 06:38

JIM STERNE [continued]: that the model hasn't seen yet to seeif the model imitates reality.And this is my--where my favorite quote from George Box comes in.All models are wrong.Some models are useful.So it's a model.It's not the real thing.It's like a map.

• 06:59

JIM STERNE [continued]: The map is not the territory.But if I can build a model that kind of imitates the future,and I believe it, and I take action, and it works for me,that model is useful.That's great.But there's constantly a human in the loop tweakingthe math to pretend that we know what'sgoing to happen in the future.Artificial intelligence is a different animal.

• 07:21

JIM STERNE [continued]: Its job is to figure out how to build the model,and decide which model is better,and if we allow it to, to take action on that.So here's what I want to talk about today.And it's sort of-- if you think of this as a word cloud,you'll see that we're going to spend a whole lot of timeon machine learning and just a little bit

• 07:42

JIM STERNE [continued]: of a hat tip toward the others.What's important to understand is that artificial intelligenceis a catch-all phrase for a number of things--natural language processing, computer vision,robots, and oh, by the way, machine learning.Well, machine learning is what's most valuableto us in marketing.

• 08:02

JIM STERNE [continued]: Natural language processing is a complex problem to solveand is very useful for capturing what happens in a call centeror analyzing emails or doing sentimentanalysis in social media.But it's a very specialized form, and it is rules-based.

• 08:23

JIM STERNE [continued]: You start with a bag of words, and yousay these words mean these things,these phrases mean these things, and youteach the machine over time, Whichis necessary because human communication is so awkward.To call something sick is very context-specific.The movie was sick.

• 08:43

JIM STERNE [continued]: I really enjoyed it.The movie made me sick.Not so much.Vacuum cleaners suck is funny because itmeans two different things.The computer has trouble.The picture is Bill Gates's favorite phraseabout natural language processing,which is it's really easy for humans to recognize speech.

• 09:06

JIM STERNE [continued]: It's hard for computers to wreck a nice beach.So this gets really complicated.And where it gets actually impossibleis down in the lower left.That is a seven-word sentence thathas seven distinctly different meanings dependingon information that is not included in the text.

• 09:28

JIM STERNE [continued]: I never said she stole my money, I never said she stole,I never said she stole, I never said she,I never said she stole my money, oh,I never said she stole my money, and of course,I never said she stole my money are all completely differentmeanings that a computer cannot determine by looking at text.

• 09:48

JIM STERNE [continued]: Maybe by sentiment, by listening,it can figure it out.And that's the job that they're trying to accomplish.The visual side is also somethingof value to marketing people as more people areposting their pictures online.Sometimes they include our logo.Sometimes we want to get a picture.

• 10:08

JIM STERNE [continued]: We want to see what people's experiencesaround our brand in the store or at the location.And a computer looks at this pictureand tries to make sense of it.A human goes, oh, it's a flying carpet.No, wait, no.It's the shadow of a flag and a trick of the eye.OK, fine.So a computer learns by giving it tens of thousands

• 10:30

JIM STERNE [continued]: of pictures of cats and saying, go look at the rest of thisand figure out what's a cat.We do not program it has two eyes, it has two ears,it generally looks like this.Instead we say, these are cats.And it says, oh, like this?No.That's a tiger.Like this?No.That's a adorable little dog, but it's a dog.It's not a cat.

• 10:51

JIM STERNE [continued]: And over time, the machine learns.And then we get to the [INAUDIBLE],,and the conversational side of things.And this is where the machine is imitating humans.It ain't sentient.It ain't smart.But it pretends to understand complex concepts.

• 11:11

JIM STERNE [continued]: It pretends to understand emotion so that whentrigger words happen--I'm really upset at your company--the response is, I'm sorry to hear that.Tell me more.And this is one of these things that in its infancy.And we know it's in its infancy where if you go to a bot event,

• 11:32

JIM STERNE [continued]: they hand out bags that include what do we want?Chat bots.When do we want them?Sorry, I didn't understand your request,which is the experience today.It's still new enough that it's tough workto make it work, which is why the second cartoon--what do we want, bots?When do we want it?No, really, we don't want bots.It's just-- if you've try working with Siri or Google

• 11:56

JIM STERNE [continued]: at home, they're aggravating more than anything else.But there are points.There are people who are building bots--again, for very specific purposes--that are useful.So x.ai is a personal assistant.It schedules meetings for you through email.It's not an app.

• 12:17

JIM STERNE [continued]: You don't need a special website.You send an email, and you copy Amy at x.ai.And Amy pops up and says, happy to get somethingon Greg's calendar.Does Tuesday at 11:00 work, or maybe March 3rd at 4:00 PM?And here's where Greg's office is.And it's very conversational.Now, this company has spent three years

• 12:39

JIM STERNE [continued]: and has raised $30 million.Why on earth would you spend that much timeand that much money on scheduling meetings?Because the amount of time that people do this by handis significant.And it's well worth the$49 a monthto have a machine do it for you.

• 12:60

JIM STERNE [continued]: So there is a point where bots are valuable.They're useful.And then, of course, there are robots which actually do stuffout in the world.And that-- personal opinion--self-driving cars can't happen fast enough.They will save lives.They'll get me there on time.I can do work while I'm traveling, all of it.

• 13:21

JIM STERNE [continued]: But you have to train it.Now, you have this combination of telling itwhat the rules of the road are.This is what a stop sign looks like.Here's how you stop at a stop sign.But then there are those things that you haven't thoughtto tell it about that it has to experience and ask you, well,now what do I do.So the pedestrian on a skateboard walking a dog

• 13:45

JIM STERNE [continued]: is not something in the database.So a human looks at that in the blink of an eye.You know how fast that kid's going,and where he's likely to end up next,and will he cross the street way before youget to the intersection.That happens in less than a blink of an eye.But the computer has got to have some basic rules

• 14:07

JIM STERNE [continued]: that it works on.And again, it can be taught over time.Now, the one in the middle, the Boston Dynamics dog,if you will--that's doing a combination.We teach it that when you see something that looks like this,that's called a door, and you can go through it.And it sees something that looks like a door,and it bunks its head into the wall and says,

• 14:29

JIM STERNE [continued]: oh, I thought that was a door.Maybe it's not.How is that different than what I thought was a door?And it learns.The same is true for the anthropomorphic robotsthat are supposed to entertain, interact.They are imitating emotional intelligence.They can learn.They can learn to recognize faces.

• 14:52

JIM STERNE [continued]: They can learn that different people in the housespeak in different ways.But it's learning and teaching.Now, the learning part-- this is wherethis technology gets really interesting and valuable.The learning part of machine learningis three basic critical important elements.This is overall what it's all about.

• 15:14

JIM STERNE [continued]: Number one-- you get a big pile of datato the machine and a given outcome.You tell the machine that you haveall of this information on prospective customersand actual customers.And the outcome that you want is to getthem to click through, consume content, and purchase

• 15:36

JIM STERNE [continued]: the product.So the machine is going to look at that pile of dataand say, well, based on what you've already collected,it seems these bits of informationare more important than others.Then it decides what to do about it.It says, well, given the fact that these bits of information

• 15:56

JIM STERNE [continued]: are more predictive, if the weighting on these thingsis this way, I'm going to act that way.And if it's that way, I'm going to act this way.And it can suggest a course of action,or you can empower it to take action.But the third bit is where the magic comes in.It can change its mind.It says, this information is the most important.

• 16:17

JIM STERNE [continued]: And based on the weighting that I've determined,this is what the outcome should be.And it looks at the results of that outcomeand can change its mind.So I said that this is about--the book is Practical Applications.So let's get very specific about how this works.Number one-- I am going to detect.

• 16:38

JIM STERNE [continued]: So here's a bunch of information about prospective buyersout there.What time of day are they looking at their phone?How old are they?What's their income range?Where do they live?How much education do they have, and what are they doing?Now, if I'm launching a new product called a Tesla car,

• 16:60

JIM STERNE [continued]: and nobody's ever seen one before,then the machine is going to figure out that age, income,location, and education are the most predictive of somebodybeing interested in spending that much moneyon an electric car.But if I'm selling shoes, it kind of doesn't matter.

• 17:21

JIM STERNE [continued]: Everybody is going to buy shoes.Income doesn't matter, how much education.It doesn't matter whether you're living on the coastsor in the middle of the country.Everybody wear shoes.So behavior is clearly the most predictive ofwhether or not you're going to be interested in this stuff.The machine figures this out instead of humans doing it,which means it is not logic.

• 17:44

JIM STERNE [continued]: It's not emotion.It's mathematics.And it clusters people together to say they aremore likely to be interested.Then it decides.It creates rules.It says, well, I think behavior is the most important.And what I have determined is that if an individual clickson an ad, looks at three pages, opens an email,

• 18:05

JIM STERNE [continued]: and says, yes, I'm interested in discounts,then, given the goal that we want them to buy something,sending them an email for 50% offwould be the best course of action.And then it looks at the results.And the result says, well, actually,if they look at five pages and open email twice,

• 18:25

JIM STERNE [continued]: then we can send them a coupon for only 10%.Oh, wait a minute.Why did the percentage go down?Because we said, ah, we've got a secondary goal.Not only do we want them to buy, but wewant people to buy at the highest margin possible.So now the goal has been set to improve margin as wellas improve sales.

• 18:45

JIM STERNE [continued]: And it says, well, if margin is important,then let's give them only 10% off, because thatwill move the needle.That will make them purchase, and we haven't given awaytoo much money.This is not sitting down and logicallythinking through things according to a business plan.This is looking at the data and making that decision.

• 19:08

JIM STERNE [continued]: To make this stuff practical, we need to look at why we measure.And over on the left, we've got the four basic reasonsthat business--the four goals that business has--make more, spend less, make people happy,and get things done more efficiently.

• 19:29

JIM STERNE [continued]: That's it.If your company is actively doing somethingthat does not serve one of these four things,you've got a problem.And then we go to the secondary, which is the marketinggoals on the right.We need to let people know we have something on offer.We want them to like what they see.We want them to talk to their friends about it.

• 19:49

JIM STERNE [continued]: We want to get them to engage with the brand.We want them to buy it, and oh, by the way,tell everybody how wonderful it is.So this is the why AI is important,why analytics is important, and whymarketing-- what is it that marketing is trying to achieve.It's the goals.And you always start with the goals, or you get lost quickly.

• 20:13

JIM STERNE [continued]: Now we take a quick look at how we measure.Way back in the old days, we took clipboards outto shopping malls, and we stopped peopleand asked them questions.Are you familiar with the brand?Do you recognize any of the brand attributes?Do you feel good about the brand?How do you feel about the competition?And then we chart that out over timeto see if we are gaining market share

• 20:35

JIM STERNE [continued]: or losing market share or awareness.In fact, I'm old enough to remember random digitdialing, which is using the telephoneand literally calling people at random and saying, hey,we're interested in getting your opinion about Oreo cookies.That doesn't happen anymore, thankfully.Then we moved to mixed modeling that says,

• 20:57

JIM STERNE [continued]: I have a bucket of money, and I'mgoing to put it in different categories.And I'm going to spend this much on TV, and this much on print,and this much on billboards outside.And that's my baseline after three to six months.And then I move things around a little bit.I take money out of this categoryand put it into that category.And oh, my sales went up.Lift went up by 10%.

• 21:19

JIM STERNE [continued]: Awesome.Let's try to improve it even more.And the next time, oh, it went down by 8%.OK.Let's go back and then tweak it a different way over and overand over again, with anywhere from a month to a quarter,sometimes year over year.So this is very slow optimization.Then team econometrics.

• 21:39

JIM STERNE [continued]: We're good at building models.Let's build this giant model thatincludes all of this gross economic data,including housing starts and unemployment,and yes, even the weather, and whateverelse we can fit into this model to predict what might happen.And again, like any predictive model, you build the model.

• 22:00

JIM STERNE [continued]: You run it against the next month's dataand see if it predicted what actually happened.And if so, OK, well, let's use it to--let's trust it to predict the month from now.Then along came this little thing called the internet.And with cookies and JavaScript, log files, oh, my goodness,

• 22:22

JIM STERNE [continued]: we got behavior.We could see intent.It wasn't just they said they liked it.It wasn't just wildly predicted we like it.No, no.We could see that they saw the ad, they clicked on it,they stuck around on the website,they downloaded the app, they interacted with the app,they bought something, they told their friends, and just

• 22:43

JIM STERNE [continued]: huge amounts of information.And that was just web analytics.We've also got-- the diagram here is the ad network.You go to the NFL's home page, and your cookiebounces all over the internet.And the fact that you were interested in the NFLis shared among all of those redistributors.

• 23:06

JIM STERNE [continued]: So much information available about behavior,but also offline stuff.So for instance, the aggregators who collect them--of all of these, I've just picked out Acxiomas an example--1,500 attributes about each of half a billion people.An attribute might be where you went to school.

• 23:26

JIM STERNE [continued]: It might be what your credit card limit is.It might be what magazines you subscribe to.And out of that, they create 3,000 propensity models.How likely are you to buy brown shoes next week?And they've connected up to 90% of their peopleto social profiles.So that's available for purchase.Then there's location information,

• 23:48

JIM STERNE [continued]: which is wildly communicative about who you areand what you do.I can watch where you're going and see that you are--for the last month, you spent an hourand a half at the Lamborghini dealershipand another half an hour at the Ferrari dealership.

• 24:08

JIM STERNE [continued]: And I can make some guesses about how much money you'regoing to spend.On the other hand, I also see that you eat fast foodbreakfast, lunch, and dinner.So I have some serious worries about your health.That's all very revealing.And as long as we're talking about health,we're in the world of Fitbit, right?So I can see-- are you awake?Are you asleep?

• 24:29

JIM STERNE [continued]: How is your heart rate?I can measure your gait.Are you limping?Are you weaving down the highway because you'vehad a couple of too many at the bar?We're now getting into an area where the data is adding up.And yes, back to offline--I can tell through your loyalty club card

• 24:49

JIM STERNE [continued]: whether you're eating healthy or you buy a lot of junk food.I can see what your alcohol consumption is.There's a pharmacy in the back, so yes, Iknow what prescriptions you're buying.I know how many condoms you're getting.If you're feeling a little nervous at this point,you should.There's a point at which it movesfrom being informative to being creepy,

• 25:09

JIM STERNE [continued]: and we need to be very careful.And anybody who is selling into Europeneeds to be hyper-careful about how the European laws arecoming together.But just for an overall feeling of how much informationis available to create a model out of,I put together this taxonomy of identity, history,

• 25:31

JIM STERNE [continued]: proclivities, possessions, activities, and beliefs.In other words, who are you by identity?This is your IP address.This is your home phone number.This is the fingerprint of the specific deviceyou're working with.What have you done in the past?You've got a degree.You've published a paper.You have a criminal record.

• 25:52

JIM STERNE [continued]: You were in a divorce, which is a public documents.All of your finances at that point in time are available.What do you like?What are your preferences?What political party?What do you like to do for fun?What news feed do you subscribe to?What brands do you prefer?What do you own, so what do what do you have?

• 26:14

JIM STERNE [continued]: Home, car?What kind of clothes do you wear?Do you collect stamps?And then there's the behavior, actual keystrokes and gestures,pinch and zoom and swiping left and right.What are you buying?What are you watching on TV?Where do you like to go for fun?And then what is your opinion?How do you feel about things?What political party?

• 26:35

JIM STERNE [continued]: How engaged are you?So some of this just comes for free.I mean, some of it you give to me when you buy stuff.You just hand it to me.Some of it I observed by watching what you post onlineor in public records.Some of it I calculate--time on site, customer lifetime value.And then some of it, I create a statistical model to suggest,

• 26:57

JIM STERNE [continued]: oh, I have an idea for the propensitythat I think you're going to buy something at a high value.This works as long as I understand the trustworthinessof all of these elements.So as a marketing person without AI,I have to know that social sentiment is not terriblyvalid, but retail sales--

• 27:19

JIM STERNE [continued]: yeah, that's kind of black and white.What you did on my website is pretty clear.But now we're going to get the machine to help us,and we have to, because we have reached that point of whatwe lovingly refer to as big data.It's more than we can actually manage.It's more than we can think about.So we bring the machine in to wrangle, manage, and comprehend

• 27:42

JIM STERNE [continued]: this for us.We bring the machine in to decide which of these bitsare valuable and what they can ignore that willsave some processing time.What should the weighting be?Is day part more important than income?Is that more important than behavior?And based on these weightings, we

• 28:03

JIM STERNE [continued]: guess this is what we should do in orderto achieve a given outcome for a specific goal,and then change minds.So what are machines good at?Well, as a marketer, the first thing I do is segmentation.I'll do this by hand.What is gender?

• 28:27

JIM STERNE [continued]: Easy.Or location-- easy.And divide up.Let's take women between 18 and 34who had a certain amount of schooling, who havetwo kids, who drive an SUV.And at the time you get down to the leafat the end of these branches, youget a pretty good target of somebody to go after.

• 28:47

JIM STERNE [continued]: But it takes a lot of meetings to decidewho is your target audience, whatmessage do you want to send.This is something the machine is brilliant at.And again, it's not going to depend on logic.It's just going to say, well, these peopleare grouped together.These people are like each other.And it can then start playing with different messaging.

• 29:09

JIM STERNE [continued]: It can-- the human--true story-- selling motorcycle insurance-- clearly,you have a better chance of selling to males than females.But the big aha moment came when somebody in the industrysaid, you know, if they store their motorcycle in the garage,chances are much likely that it's a midlife crisis

• 29:31

JIM STERNE [continued]: motorcycle, and it belongs to an accountant,rather than if they store it outside,and it belongs to a college student or a drug dealer.So yeah, we would like to know if you have it inside.But then if we start breaking it up by age and by credit score,and by where you live, et cetera, et cetera,suddenly there's too many variables to deal with.The machine is brilliant at this.

• 29:52

JIM STERNE [continued]: The machine can do segmentation and classificationin a heartbeat.The machine can also do testing like nobody.We're going to do an AB test.Buy one, get one free, two for the price of one.It's the same result, but one will poll betterthan the other.But when you start linking these together,

• 30:14

JIM STERNE [continued]: when you start having a series of behaviors,by the time you just get out to four different branches,you've got 32 different outcomes.The machine can go through all of themin a heartbeat to find which one is goingto give you the best lift.So whether it's clustering or testing,machines are great at that.They have a little trouble with creativity.

• 30:37

JIM STERNE [continued]: Today's special-- buy one beer for the price of twoand receive a second beer absolutely free.The machine is not going to come up with that.Creativity is currently science fiction.Now, remember, science fiction is anythingthat artificial intelligence can't do yet.So I'm not discounting that will ever happen, but right now, no.

• 30:59

JIM STERNE [continued]: If you're a creative person, you're safe.So where's the line?What do you give to the machine, and what do you do?So the machine is good at patternmatching, seeing if there's an outlier,figuring out what it can ignore, back-propagatingits calculations.But when it gets into the creative side,

• 31:20

JIM STERNE [continued]: there are things that are outside the scopeof the machine.For instance, what if we're tryingto solve the wrong problem?What if really, what we want is a different goal?And what additional data might be helpful?Now, the machine is very good at lookingat a pile of information and figuring outwhat's valuable in there.

• 31:40

JIM STERNE [continued]: But it only knows its whole universeis that pile of information that you pointed it to.What other information might be usefulthe human is responsible for.And then just the wacky, creative, and collaborativebrainstorming kind of stuff that humans dois why you are still going to have a job for a long time--

• 32:04

JIM STERNE [continued]: because you know the domain, the specialty domain of marketing,which, oh, my god, is wildly confusing.I think about people coming out of school todaywith marketing degrees, and the depth of knowledge that theyneed to have, and the breadth of knowledge

• 32:25

JIM STERNE [continued]: that they need to have to just keep everything going together.It's a huge amount of information.This is where, instead of it being man versus machine,we've got to look at man and machine.Computers are fast and accurate and stupid.Humans are slow, inaccurate, and brilliant.

• 32:47

JIM STERNE [continued]: Let's put them together.OK?So I want to go through some stepsof what you should do in dealing with artificial intelligence.Number one-- be the person who says, oh, Iwant to work on that project.The Right Stuff is the movie that they came upwith the phrase pushing the outside of the envelope.

• 33:08

JIM STERNE [continued]: Be the one to bring it in.Be the one to get on the team, to join that meeting,to have an opinion positively supportingartificial intelligence.And during every step of the way,you are going to use your common sense because the machinecan't.The machine will do exactly what it's

• 33:28

JIM STERNE [continued]: told and only what it's told.And a human looks at this and goes,yeah, no, that is clearly not going to work.We know this is not going to work.And the human-- the common sense plus creativitymeans that you can help keep the machine awayfrom the local maximum.This is a standard statistical trope

• 33:50

JIM STERNE [continued]: that you want to go higher.And so every step you take, if it goes higher,you're going in the right direction.If the next step is going lower, then youneed to change direction until the next step is higher.And eventually, every step is lower.Therefore, you're on top.Great.Except you're on the top of the hill,

• 34:11

JIM STERNE [continued]: and you wanted to be on top of a mountain.Humans have a way of seeing it from a bigger picture.So you can build a great new building,or you can build an insanely great new building.The machine will do exactly what it's told.The human can think outside the box.

• 34:37

JIM STERNE [continued]: There's also a desire, this need--sorry-- this need to know that it's not justabout raising revenue or lowering costs.Increasing customer satisfaction is important,but then there's also this other element.There is no formula, there is no calculationthat says that it's a good idea to bring the airplane back

• 35:00

JIM STERNE [continued]: to the gate, take the woman off the airplane,tell her that her son is in a coma,put her on a different plane, and send her home for free.That's not a good business decision,but it's the right thing to do.So humans are involved.That's important.And five-- somebody has to teach the machine, right?

• 35:24

JIM STERNE [continued]: This is where you are going to step forward and say,I understand marketing well enough.I understand the goals well enough that I can process.I can participate.I can be the one who's going to look to see which technologiesare most promising, what we should be automating,what needs more education.

• 35:45

JIM STERNE [continued]: This is a fascinating philosophical problem,is the architects' biases have been embedded in the system.This is what's also known as the white male syndrome wherethere's a bunch of white guys sitting in a lab creating an AIsystem that says, hey, let's have it recognize people.

• 36:10

JIM STERNE [continued]: And OK, let's take all of our photographs and put it in.The machine learns from all of these photographs.And guess what?The photographs are all about white guysbecause that's their knowledge.That's what they know.The biggest deal, the biggest important thingis the smell test.And that's where domain knowledge is critical.That's where it really becomes important that you

• 36:32

JIM STERNE [continued]: know what marketing is about, because youare going to be the one administering the smell test.It all starts with being able to clearly identifythe outcomes, the goals.So the paperclip is another one of those common storiesthat you hear when people talk about artificial intelligence.

• 36:52

JIM STERNE [continued]: You build a factory that makes paperclips,and you make it in an AI factory.And the factory figures out how to improve the supply chain,how to package the paperclips better, howto buy raw materials better.It figures out that it can spend time

• 37:15

JIM STERNE [continued]: learning how to learn to improve its learning capability.It figures out that it needs more money to expand,so it invests in the stock market.And then it figures out how to manipulate the stock market.And then eventually, it cracks theoretical physicsand learns how to turn every atom on the planet

• 37:36

JIM STERNE [continued]: into a paperclip.This was not a well-defined goal.The same thing with revenue.If you ask the machine to improve revenue--and that's the goal, make more money--it will take mere seconds for it to decide the bestway to raise revenue is to sell $10 bills for$5.

• 37:60

JIM STERNE [continued]: Revenue will go through the roof.Not quite the goal we were after.We have to keep in mind that yes, wewant to raise revenue and lower costsand improve customer satisfactionand be more efficient while we're driving these marketinggoals.So the smell test is where the machines

• 38:21

JIM STERNE [continued]: says online buying goes up when the weather is bad.And everybody in the room looks at the machine and goes,yeah, we knew that.And the machine is right.Online buying goes up when it snows.This is true.And we've already built that into all of our models,

• 38:42

JIM STERNE [continued]: and we've already decided that we'regoing to watch the weather in orderto determine when we sell sunscreenand when we sell umbrellas.And we're used to that.OK, great.Ice cream causes drowning.Oh, wow.That could be serious.Except the humans in the room go, no, actually,that's related to something completely different.It's not causal, and it's just not helpful.

• 39:05

JIM STERNE [continued]: Oh, well, shoes cause headaches.This is something that's proven.And in fact, I know this personally.Every time I wake up in the morning wearing my shoes,I have a horrible headache.Well, it's not the actually the shoes that's the problem.The headache is caused by something else.But it is true that Nicolas Cage is a monster,and I have graphic proof.

• 39:25

JIM STERNE [continued]: The number of people who drowned by falling into a swimming poolcorrelates surprisingly well with filmsNicolas Cage has appeared into.This and many other spurious correlationscan be found at the website called Spurious Correlations.This guy has taken public data and mapped it

• 39:48

JIM STERNE [continued]: against each other until it finds correlations and thenpublishes these outrageous things.And how people die is one of those databases that'savailable, and it correlates with all kindsof strange stuff, like suicide rates and margin consumption.It's just-- it's wildly entertaining.I recommend Spurious Correlations to prove.

• 40:10

JIM STERNE [continued]: So what do we do instead?We know the computer can't figure out causation,but it does figure out correlation.So there is a correlation between weatherand online buying.Is it true, interesting, and useful?Sure.Well, it's not so interesting, but it's useful.There is a correlation between ice cream and drowningand between shoes and headaches and between Nicolas

• 40:32

JIM STERNE [continued]: Cage and drowning, but they aren't really interesting.They aren't really useful.If instead the machine says, thereis a correlation between sending this messageto this type of customer after they have viewed

• 40:52

JIM STERNE [continued]: these products and an increase in cart value, if that's true,well, it's testable, it's useful, it's valuable.And it's exactly the sort of thingthat machine learning is perfect for because itcan look at hundreds and thousands of variables,and figure out that this is the right message

• 41:15

JIM STERNE [continued]: to send at the right time to the right personfor the right reason, and get a 50% lift in shopping cartvalue.So the smell test is where we need humans to be involved.So clearly identify the desired outcome, use the smell test.And here's the other bit the machine cannot do--figure out which datasets are useful.

• 41:38

JIM STERNE [continued]: Which data within the dataset is valuable-- yes,that is what the machine does.But which data sets are useful--oh, that's something a human needs to do.Over on the left, we have a tool that--it's enough data to do exactly the right thing.And over on the right, we have a toolthat's actually just way too much and way too expensive

• 42:00

JIM STERNE [continued]: and too difficult. So in the middle,there is this magic amount of information or amountof datasets.What's useful and that is where your knowledge of marketingis necessary.That is where you are able to say, hey, machine,you should look at this other thing.Now, how do you do that?Well, you have to know your data.

• 42:24

JIM STERNE [continued]: Let me walk through a marketer's knowledge of datawith an analogy of water, because today,data is falling out of the sky at us.Our job is to collect that data and put it togetherin these puddles that we can thenuse for a specific purpose, so customer relationshipmanagement, salesforce automation, usability,

• 42:46

JIM STERNE [continued]: customer satisfaction, et cetera, et cetera.We are collecting these data, and we'reputting them in these silos.And they're good for using for specific purposesas long as we keep them fresh.Because as soon as this data get dirty or old or untrustworthy,we have a serious problem.And it has to be something that is

• 43:07

JIM STERNE [continued]: attended to from the beginning.It's so important that our-- the first US chief data scientistput it emphatically that if you're not thinking about howto keep your data clean from the very beginning,you're in trouble.I guarantee it.From the start, it is--data that's streaming towards us and from way upstream,

• 43:30

JIM STERNE [continued]: it's got to be clean by the time it gets to us, because we'renot just satisfied putting it in pools anymore.We are watching it as it streams by in almost real time.And we're using those streams to power the marketingoptimization work that we do, whether it'smultivariate testing or email customization or recommendation

• 43:53

JIM STERNE [continued]: engines.That's great for specific, but thenwe want to accumulate all of those streamsinto the giant data lake--OK, the data warehouse, the big Hadoop cluster in the skyso that we can do deep diving.Only now we're going to get the machineto do the diving for us.So you need to be really on top of your game

• 44:16

JIM STERNE [continued]: when it comes to data.If you're a creative person, if you're out there doing mediabuying, if you are out there just trying to move the needle,you really need to get to know data.Now, you don't have to know data processing.You don't have to know statistics.You don't have to know analytics.But you have to know where the data comes from.

• 44:37

JIM STERNE [continued]: A data dictionary is like a glossary.It says customer ID number.OK, well, who creates that?Where does that come from?How many different systems use that customer ID number?And if there is one element or one attribute thatdescribes whether this person is a foodie or a sports person

• 45:01

JIM STERNE [continued]: or whatever they might be, you have to know how that data--who entered that information into that machine.You need to understand all of those variables.And a lot of them are going to be not raw data,but derived data.So we have raw data.We did some calculation pre-processing modifications

• 45:24

JIM STERNE [continued]: to it, like normalization, like the abilityto say that in this system, the week starts on Sunday,and in that system, it starts on Monday.And then we do some calculations,and then we do post-processing.Absolutely everything here is context,and it's crucial for you to understandwhere the data comes from.

• 45:44

JIM STERNE [continued]: You have to know that it's reliable, valid, clean.And it has to be so that it can becorrelate-able by the machine.And as a marketing person, you are responsible.This is where the smell test comes in,because you have to look at the resultand say, yep, that's what I told the machine to do.And darn it, I forgot to tell it that you

• 46:09

JIM STERNE [continued]: got to put the lock over on the other side,or you can't get the key to--yeah, it's not going to work.Humans can just go, yeah, that's not going to work.So what should you do?You should bring AI into the company.You should be the proponent.Number one-- don't boil the ocean.Start small.And Tom Davenport, who most recently

• 46:32

JIM STERNE [continued]: wrote humans only need apply, saysstart with those little, repetitive tasks.So that's why scheduling meetings is valuable.Is there a knowledge bottleneck where everything goes,and then you've got to wait for it to come out the other end?Let's automate some of that.Is the knowledge too expensive to provide everywhere,

• 46:54

JIM STERNE [continued]: or is it just too much data, and we need the machine,or the decisions have to be consistently high quality?Where can we go in and make little changes that will help?Where can we free up time rather than worryingabout replacing people?We're going to make it easier for you to schedule meetings.

• 47:16

JIM STERNE [continued]: That will free up some of your time.The computer is good for sorting, ranking,pattern matching, needle in a haystack, so the stuffthat is mind-numbingly boring but absolutely crucial,like lead scoring, or figuring outwho should answer which emails, or programmatic advertising.

• 47:37

JIM STERNE [continued]: That's great for the machine to do.Don't boil the ocean.Try to make it all--don't try to do it all at once.Find those little tasks.But then you're going to have to understand getting alongwith the data scientists.So your job is to clearly outline the goal, whatproblems do you face.

• 47:58

JIM STERNE [continued]: Exactly what does success look like?What data do you have available?What other data might be interestingthat they can go explore?And then answer those questions that are--you start a machine working on this stuff,it's going to come up with all kinds of strange results.And there you have to be the smell test person

• 48:19

JIM STERNE [continued]: and explain stuff.But you know you're working with a really good data scientistwhen they start drilling down on the domain side.They ask you about, tell me exactly whatsuccess looks like.How is this different from another outcome?Can we talk to the salespeople if you're in a B2B environment

• 48:41

JIM STERNE [continued]: and about how they work with actual clients?What about date parts or seasonality, or whatis the competition up to?The more domain questions that a data scientistasks, the better--you're in better hands.Most of all, when somebody throwsAI or machine learning at you, justremember these three things.

• 49:02

JIM STERNE [continued]: It can look at a pile of informationand figure out what's the most important information.It can weight those factors and say, well, because it's rainingand there's a slowdown on the freewayand our competition launched a nuke and thisand this, therefore, the recommendation is to do x.

• 49:24

JIM STERNE [continued]: And then the biggest deal of all isit can look at the results of x and come back and say,all right, I'm going to change my opinion about whatis the most important informationand how much I should weight each one.And for all of this, the crucial bit is to remember the goals.Whatever the machine is supposed to do clearly,you remember the goal.You keep that top of mind.

• 49:45

JIM STERNE [continued]: And that's what will make you a valuable marketing person.You'll understand the machine and whatit's trying to accomplish, but you will understandthe marketing domain.You will be the human expert in the roomthat the machine can learn from.With that, I am happy to open it up to Q&A

• 50:08

JIM STERNE [continued]: and find out if I have helped or just confused everybody.Selma, do we have any questions?

• 50:16

SELMA AVDIC: Yes, we do.Thank you so much, Jim.All right.So I'll start with a question from Mark.And he writes, I work in the digital marketing staffingindustry, and I'd like to hear your insight on howthis is going to affect the job of a recruiter and the needsof a company that needs to hire talent with AI skill sets.

• 50:41

JIM STERNE: That's a wonderful question, Mark.And in fact, there is a whole area of workforce analytics.I've even seen a conference called Predictive AnalyticsWorld of Workforce where we're trying to use the mathto screen the resume, to figure outwhat are the most important factors for hiring.

• 51:02

JIM STERNE [continued]: And of course, the skill set comes into it hugely.However, the human is always involved because in recruiting,we know that it is that who you know rather than what you know.It's asking that important question at the right time.Well, have you considered changing jobs,

• 51:23

JIM STERNE [continued]: or do you know anybody else who?So there's things that humans will always be better at.And then, of course, there's the issueof when you're bringing somebody on board,do they-- if they're a data scientist, boy, theyneed to know their stuff.Because this is new.It's been changing fast, and it's complicated.

• 51:43

JIM STERNE [continued]: When you're hiring for a digital analyst for a marketinganalysts who can work with data scientists,oh, well, that's a different skill set.It's a lot more about marketing domain knowledgeand comfortable with you.I mean, you're-- so you're talking to somebody who is oldenough to remember when word processors first came

• 52:05

JIM STERNE [continued]: into business.And when I went to my boss and saidI wanted a word processor so I could be more efficient,and I was told that, no, no, no, only secretaries had keyboards,I had to go out and buy my own Macintosh and printerand make that work for me.Marketing people who are comfortable with technology,

• 52:25

JIM STERNE [continued]: who are enthused about technology--those are the people we want to hire first.So just as I've described the marketing domainas being the specialty, Mark, your specialty is recruiting.And you understand the whole domain of recruiting.Be the one to bring the AI into the roomto help you figure out which marketing people have the best

• 52:49

JIM STERNE [continued]: skills to bring aboard.

• 52:53

SELMA AVDIC: All right.Wonderful.Thank you, Jim.Next question for you is coming from William,and he wants to know what are your thoughts about softwarevendors saying they have machine learning builtinto their solutions.

• 53:10

JIM STERNE: Several things.First of all, yay!Second of all, skepticism.Third of all, show me the lift.Show me how that's helping me and how I can use it.Salesforce has Einstein, and IBM has Watson,and everybody is using machine learningin some way or another.

• 53:31

JIM STERNE [continued]: But show me where it's significantlydifferent from doing it the good old way.And the second part of my answer iswe are seeing a huge development or growth in startups thatare folks that are saying, yeah, I'mgoing to I'm going to tackle email marketing using machine

• 53:51

JIM STERNE [continued]: learning, and I'm going to learn across all of my clients.And so if you do email marketing,if you let my machine choose your targetand choose the message, you'll get a better lift.And the answer is, great.Show me the money.Prove it.I want to see it, which is to say,definitely this is something that

• 54:12

JIM STERNE [continued]: is being built by the big guys and built by the startups.So you understand how-- you have to understand how it works.You do not have to build it yourself.

• 54:23

SELMA AVDIC: All right.Wonderful.And I've seen a couple of audience membersask this question.So the question is, how soon do Ineed to get started with this?

• 54:38

JIM STERNE: That would be yesterday.This is-- it's exactly the same question thatwas asked in the early '90s is, well,I don't really need to have a website, do I?Well, yes, you do.Well, nobody really knows how to do it yet.Yeah.That's why you need to have one--so you can start learning, so youcan be right on top of things as they happen instead of waiting.

• 55:03

JIM STERNE [continued]: And everybody else is going to eat your lunch.So start now.Get your toes-- dip your toes in the waterand start making the mistakes that experimentationwill reveal so that you can ramp up better than the competition.

• 55:25

SELMA AVDIC: All right.Wonderful.And I think we will end it with one more question becauseof time.But as you can see on the screen,Jim has graciously provided his contact information.So feel free to connect with him if youwant to discuss this further.

• 55:43

• 55:45

SELMA AVDIC: Yes.And our last question is coming from Mara.And she wants to know, is there an AI ecosystem graphicthat gives us insight into the different typesof AI-related firms that exist today?

• 56:01

JIM STERNE: So there's that classic roomescape of all of the logos.And the AI version of that has a coupleof hundred different companies.So that's findable.Every day, I google artificial intelligence marketing,

• 56:23

JIM STERNE [continued]: and hit the Images link, and come upwith another graph that explains it in a different way.I find all that to be helpful.So can I point you to one URL?No, I'm afraid not.But I had just yesterday made a listof four different graphs I need to go study and learn morefrom.So good question.

• 56:43

JIM STERNE [continued]: Thank you.

• 56:46

SELMA AVDIC: All right.Wonderful.Well, thank you so much, Jim, for taking some timeto share your insight.This is a very fascinating presentation.I want to remind our AMA members again,you can view an archive of this webcast by visitingama.org/webcast.I have a feeling that a lot of people

• 57:07

SELMA AVDIC [continued]: will be going back and reviewing your notes.I also want to thank Ready Talk for providing uswith a web conferencing platform for today's webcast.And last and not least, I want to thank our audience.Thank you so much for being our membersand for taking some time to participate and engage.

• 57:27

SELMA AVDIC [continued]: We hope that you enjoyed it, and wehope to see you for our next member only webcast next month.Thank you again to everyone, and Ihope everyone has a wonderful rest of your day.And Thank you, Jim.

• 57:42

JIM STERNE: Thank you.

### Video Info

Publisher: American Marketing Association

Publication Year: 2017

Video Type:Lecture

### Segment Info

Segment Num.: 1

Persons Discussed:

Events Discussed:

Keywords:

## Abstract

Jim Sterne, author and Board Chair of the Digital Analytics Association, discusses and provides a framework for understanding ways marketing and artificial intelligence can partner to produce tangible results.