CROUD - Avinash Kaushik (CSO). Part 2. How to use creative testing to 'win before you spend'. The importance of trust but verify; how to measure effectiveness; how to best use social media analytics.

The FutureView podcast is published in conjunction with InsightPlatforms.com. This episode is sponsored by Human Made Machine - optimising media ROI through high quality creative testing - find out more at HumanMadeMachine.com

Delighted to publish part 2 of the interview with Avinash Kaushik. For those who haven't listened to part 1 (yet), Avinash is author of two best selling books, hugely popular blogger, former analytics evangelist at Google (among many other roles during his 16 years there) and now CSO at Croud.  

In part 2 we get into:

- Using creative testing to win before you spend
-  Matching testing to in-market performance
-   Balancing creativity & data analytics
-   Croud - what comes next
-   Looking for the next generation of leaders
-   Strong opinions, loosely held...does the mantra work
-   Avinash's best and worse characteristics....from someone who knows him very very well!

Book recommendations:

YUVAL NOAH HARARI - SAPIENS
LIU CIXIN - THE THREE BODY PROBLEM

All episodes available at https://www.insightplatforms.com/podcasts/

Follow FutureView on Twitter at https://twitter.com/FutureView7

Transcript

Henry Piney: 0:08

Welcome to Futureview and part two of the interview with the brilliant Avinash Kaushik. In this 30 minutes, Avinash explains more of what he's doing at Crowd, what he's learned about leadership in multiple organizations and the work he pioneered at Google, and that's where we pick up the conversation. Avinash explains how he developed systems to refactor marketing spend in-flight and then moved on to building creative testing programs to win before you spend. I hope you enjoy the rest of the interview.

Avinash Kaushik: 0:35

So in-flight optimization became a very big deal. So you use machine learning, sophisticated analysis, automation, to make decisions in-flight, figure out while we are spending money that some of it is not going to work out and stop it or change it. So we became so good at it that about 30, 40 percent of the budget we could actually refactor while we were spending it. Now remember, for most companies they would have only realized after all the money spent that this didn't work, or we could figure out while we were spending money. When I presented that to our CFO she said well, win while you spend is great, but win before you spend is better.

Henry Piney: 1:14

Well, this is what I was just about to ask, actually, in terms of how something like creative pretesting plays into that process.

Avinash Kaushik: 1:23

That's exactly right. So we did two things in parallel. One is we started to build a lot of predictive models that could analyze media plans and predict likelihood to succeed before we spend money. So, for example, we would take the media plans recommended by our offline and online agencies There were two different ones. We could put it into a predictive algorithm and he would say 30 percent likely he had success. Okay, go back. We did this thing for 60 percent, whatever, But very quickly.

Avinash Kaushik: 1:49

One of the early lessons I learned and this is a lesson in humility I had spent so much time and energy in my career focusing on all other things that I'm trying to be embarrassed to admit that I had personally underestimated, until like seven years ago, the powerful impact of creative on driving overall success. So when I tried to figure out what could we do to win before we spend, I sort of ended up identifying the team, ended up identifying creative as an opportunity. I'm like, okay, if creative is such a big opportunity, then what is win before? you tend, Well, pretest it, Don't put it out there because the CMO likes it, or I like it, or the creative director likes it.

Avinash Kaushik: 2:31

So we started working with Kantar And then, soon enough. We ran into Al and Sabrina, who I adore, and we started a relationship with them And then sort of Kantar did some of our offline stuff. We shifted all the digital to HMM and had and HMM continues to be a key provider to Google. But through my tenure there, the fourish years, we became more and more reliant on HMM because the thing we were able to show is not that creative pretesting is a good signal to get before you spend money. But I instituted this practice of trust but verify So okay, the testing says yes, this will do great. How often does that also do great in the real world?

Henry Piney: 3:20

And how do you track that? Sorry to butt in, but because I think that's something that again is quite siloed, i think, kind of historically in terms of creative pretesting, which is very well embedded across lots of sectors, But I think research companies in particular have really really struggled to match that against in market performance. How do you go about doing that?

Avinash Kaushik: 3:42

So our approach. So there's also a psychological challenge I had to overcome because, let's say, I create a test of five ads and I go back to the super powerful, influential creative director, who is a lot of power, and I tell her or him hey, four of your five creatives failed. The first thing I was told is to fuck off.

Henry Piney: 4:09

Well, yeah, I've been there.

Avinash Kaushik: 4:12

I am empathetic to the point of view and this is okay. Fine, you quant person, how could you put a bunch of people in a room and test my thing and think that that's like, how dare you trust my genius? So I had to put a trust but verify program to solve both of these challenges the people challenge, but also, how good is our methodology problem, because basically it's me questioning my own effectiveness or, in this case, my colleagues who did the testing. So what we did is very simple, henry we would take a bunch of creatives that failed, we put them in market and we put most of our money, of course, behind past creatives and then we would measure real world results. What happened? And it turns out and I'm going to give you fake directional numbers, because obviously that belongs to Google But one of the things we found is that, if we said yes, our colleagues at HMM said this piece of creative will has passed creative pretesting.

Avinash Kaushik: 5:13

It also delivered above average results for brand marketing in the real world about two thirds of the time. I'm making up a number. Yeah, that's great. What? 70 percent of time It also succeeds. It's fantastic.

Avinash Kaushik: 5:31

But the other one was even more cool, henry, if the creative pretesting says the creative will fail in the real world. It actually also failed in the real world at roughly 80 percent of the time. Say it was like really good at finding crap. On the yes side, it wasn't perfect, but remember, before you had zero knowledge. Now you have 70 percent, and 80 percent to deal with much higher countries. Now does this mean that we only relied on CanPower and HMM for your pretesting?

Avinash Kaushik: 6:09

Oh, what we ended up doing as a company is to say roughly 85 percent of the budget will be spent on the signals we get from the creative pretesting program, when, before you spend, because we have trust but verify, we'll take 15 percent of the budget and keep it loose so that our creative teams can still imagine amazing, crazy things. Doesn't matter what the creative pretesting says. Let's just try a bit of things. Let's try a bit of formats. Let's try. We've always used babies in the ads. You know what? Buy, buy babies. Let's use trees, go crazy. I think the secret was both of these things trust but verify but also giving our creative colleagues that freedom to have 15 percent of the budget. Use it however you want to use it, and that innovation that fostered meant that the results that Google was able to demonstrate in the near term using pretesting, and long term, using that 15 percent innovation built a competitive advantage for our company.

Henry Piney: 7:10

Yeah, I can very much see that. I think that's great advice in so many different levels around how you present it to creatives. I love the spend before you win idea. The other analogy I've sometimes been given by other people is if you were to jump out of a plane, you'd like to test the parachute before you do it.

Henry Piney: 7:31

But just diving into some of the technicalities now, i'd also like to spend a bit of time on crowd at you as well, to understand what you're doing there. How would you actually match it up, though? Because generally, creative pretesting will focus on awareness, unedited awareness, intent, consideration, brand metrics, all those types of things. Are you matching against actual sales, or is it view through rates, or what are you measuring the effectiveness?

Avinash Kaushik: 7:59

Yes, so this varies for different companies. Some companies I work at they will work with at Crowd or Google before, which is Google's clients. They would spend 80 percent of the budget on brands, 20 percent performance. Sometimes it flips. So there's different variations in how much how people spend their marketing budget. So my answer to that question depends a little bit on how much you're spending on brand, because brand is the harder one to accommodate for and tie back to sales in the short term. So we typically take two strategies. Again, at Crowd we'll build a custom thing for depending on the client's budget and media strategy. But what we do is we say, in the short term for brand marketing and short term will be 100 days, not every day because you need for purchase intent We'll say over a period of 100 days. We have models that can tie brands impact on performance And because we're also measuring brand's impact on brand, the influence of the creative flows through that And so we're able to isolate that. Ah, we spend 60 percent of the budget of past creative, 40 on failed creatives. Now let's measure impact on brand plus sales. Or we spent 10 percent on past, 90 on failed. So actually we were able to model these things really well over that 100-day period.

Avinash Kaushik: 9:33

For metrics like purchase intent, where the impact horizon is shorter. Now you go back to consideration. The impact is over nine months or so, in which case we rely, or you go to unedited brand awareness, my favorite. The impact is over nine to 18 months. So the impact horizon influences what tactic you want to measure.

Avinash Kaushik: 9:55

For the latter two, we use our mixed models to identify brand's impact on sales. So if you're working with Crowd, you're working with any good company out there. They will be able to use their mixed models, especially if they use machine learning, to identify brand's impact and performance actually tied back to sales. So if you, since we work closely with Al and Sabrina, we would go back to Al and Sabrina and say, ah, here's how the creative is influencing the outcomes on brand, unedited brand awareness three points of lift, two points of lift, whatever, and then, over a period of six to 12 months, we're able to then tie that back to the number of phones sold, number of new accounts opened or whatever, and we can take two steps back and say, ah, and, by the way, tv did this, newspapers did this, email did this.

Avinash Kaushik: 10:51

All you have to do is sort of stretch the duration of the analytics because you need to collect more signals. If we were starting ads on Google, the impact horizon is very short. You spend money the next day. It works or not. So you use that. What do you call that? The accordion brand is fully accordion a little more so you can get a solid signal back.

Henry Piney: 11:12

Yeah, thank you. So it kind of boils down in some ways to a couple of things. First of all. I mean, firstly, you need to be very clear on what you're trying to measure on a timeframe And secondly, having patience. That's it. If it's around brand marketing, i'm assuming in terms of the detailed around survey, though. So if it's things like unaided awareness, that then you're running tracking studies and that that's like a thing Again, it's another survey based methodology. Would you use social media or another data forms as a proxy?

Avinash Kaushik: 11:42

Not for campaign impact. There are two challenges with using quote unquote social listening tools. The problem is that for most companies this might not be for all companies. So for kindly cosmetics, maybe what I'm saying doesn't apply, but for if you take outliers like kindly cosmetics out because the Kardashian brand runs on social. So let's just set that aside for one moment. For most companies, the social audience is skewed and unrepresentative of the company's actual consumer base. So if you're bringing the social listening as a proxy for campaign, the best you can say is for the subset of this biased audience. This is the results, but it is not a good way to understand the holistic impact. So that's one big problem. And then the other one is that it is very hard to see quick impacts for things in social that you're trying to do over the long term, because the reach, unless you're buying a lot of paid media on social, the reach of the social channels, is so tiny. So the organic reach on Facebook now I believe they said a study said is 0.7%. So almost nobody's seeing your campaign unless you pay Facebook. By the way, i don't begrudge them, it's their business. So that's the other thing that biases your results.

Avinash Kaushik: 13:18

What we do use social listening for is for two interesting and wonderful use cases. The first one is you should use social listening, use some NLP algorithms to parse it all out at scale to identify leading issues. Social people love complaining about products and services and companies. So I'm like, oh, that's okay, people love complaining. Is there a way that we can scrape all this data on a daily or weekly basis so we can find leading edge problems with our company? We can find patterns in the data.

Avinash Kaushik: 13:56

Oh, 100,000 people suddenly complained about our seats. What the hell? We didn't know anything about what's going on. So it's a really good leading indicator of challenges the company's having. Big patterns don't respond to individuals. That. The second really good use case, henry, closer to our heart, is that sometimes in some of our campaigns all the creative fails. We have no idea how to come up with new things that would resonate with the research and all the way we really don't know. So I've had like the best experience using social as a way to seed creative. Give influencers the power to say, hey, you built creative for us instead of our creative team.

Henry Piney: 14:39

That's interesting.

Avinash Kaushik: 14:40

Yeah, let's see what you come up with. Use their ideas and then their data to figure out like, oh, we hadn't thought of this creative concept or we hadn't thought of that. So social becomes a very rapid way to experiment with creative ideas, using an outside force for good rather than our own company to let new ideas come in. So both YouTube and Facebook and Instagram all three of them we've used to unlock creative imagination using people outside our companies. So those are two really great uses of social using it to see if what the unneeded brand awareness was is an incomplete signal at best.

Henry Piney: 15:23

Yes, as you say. I mean, on social, you tend to get the lovers or the haters are very vocal, aren't they? But I love that idea of using influencers or the social world as kind of like a vast pool of creatives. I guess TikTok have done that for a large extent. I mean, they put an enormous number of creatives And some of it's not great, but some of it is.

Avinash Kaushik: 15:47

She bites into bullying.

Avinash Kaushik: 15:49

I agree. Honestly, I think that the strategy of using influence to parrot the company's words and terms and creative concepts is such a waste of money and time. If you're going to spend money on influencers, give them the freedom to reimagine concepts of creative. Let them come up with stuff. Tiktok has been particularly good at it, as you said, because the formats that work on Tiktok are so dramatically different than YouTube or television or Facebook. What 30-second videos don't work on Tiktok? 30-second ads don't work on Tiktok. By the way, if you're good at it, you can pull off that on YouTube with 15 seconds or 6 seconds. Tiktok's a completely different landscape. So I found it a really good way to challenge the company's current creative lockbox into which they may be trapped, And Tiktok is a way to challenge ourselves to think even more imaginatively. So it's a really great point, Henry.

Henry Piney: 16:53

So, avinash, i'm conscious we're coming up on time. There's all sorts of things I'd love to ask you about, particularly AI, but that may have to be a separate podcast, if you'd be up for doing that. So, crowd, you've just described in really compelling terms some of the things that you're doing, but could you answer a couple of things? What does a chief strategy officer do Which I've, by the way? I've asked some other chief strategy officers. it'd be interesting to compare notes And what does crowd do and where is crowd going next?

Avinash Kaushik: 17:20

Yes, so crowd is a full service media agency for our clients in the Middle East, uk and the US. We manage their media spend to drive outsized marketing results. So everything from a Pratt here and how we can use billboards with local geo-targeted promotions and changing billboard ads in London to deliver higher-foot traffic to Pratt, to how do we help compare the market, build a sustainable competitive advantage on paid and organic search So in the spectrum is everything else in there. So we do that. For some of our clients we also do lead a strategic analytics practice. So for our one of our large clients in the US, they spend hundreds of millions of dollars in marketing and we're helping put together a strategic analytics platform end-to-end people process tools, kpis, dashboards but we don't manage their media. So it's just this sophisticated, bleeding edge analytics practice. So those are two big areas of focus In terms of my role.

Avinash Kaushik: 18:49

I lead product analytics, m&a and strategy at Crowd. So with my title, my big role is around building amazing strategy for the company What market should be in, what products should be offered, what should be our investment in AI versus people. One of my internal mantras is I come to work every day to create unhealthy profits for our customers. That's the reason I'm showing up, so translate that into real things. So big part of that and, of course, working with our investors, which is Lloyd's Bank they are the investors in Crowd, so figure out how to manage the goals and expectations they have set for us, our internal leadership and clients. That's a big chunk of my role. The other one, of course, building a bleeding edge analytics practice. It's sort of my natural home. But also what I'm excited about Crowd is to bring the other side of my work at Google and build amazing products. So we're building a bunch of tools. We have a number of products already, but to basically turbocharge that. That's where I'm spending my time And I also clean conference room tables and vacuum the floors.

Henry Piney: 20:08

I was just about to ask how do you fit all this in? The first three things were probably enough that you mentioned. It must be very hard to remain hands-on though, especially when you're dealing with potential investor relations and helping to form strategy and looking at M&A and that type of thing.

Avinash Kaushik: 20:25

It definitely is. So not watching a lot of TV helps. No, i think that one of the things that has been very instrumental in my career over the last, especially in the last decade and a half, is focus my strengths on the think and the strategy and how do you innovate? Those are things that I spend most of my time And then figure out how I can identify incredible leaders who can go out and inspire and execute. And so I always say, like there's never a moment in my day, no matter where I am, what I'm doing, i'm not looking at a person and saying, is this the next leader?

Avinash Kaushik: 21:06

Because I'm looking to figure out how we can elevate people, challenge them, make them uncomfortably excited And I'm lucky that we have so many of them at Crowd where I can just give them things, and they are so fantastic at doing them, so they are extraordinary in that they help share that burden. And then I can focus on these other things. But you know, henry, i wanna be honest. There's some days I'm cycling back to my apartment in Camden and I'm like, oh my God, i'm so tired.

Henry Piney: 21:38

I bet you are. I bet you are. I love some of the juxtapositions as well that you present. I like being uncomfortably excited. I think it's a great one. I am conscious of time, though, and so I'm just gonna do a couple of quick final questions to wrap up, possibly from all at personal level. So what have you changed your mind about recently, avinash? Is there something that you used to believe was true but that you now no longer believe is true?

Avinash Kaushik: 22:04

Yeah, oh, i have a user manual I write for people to get to know me better when they join the team, and one of the item, number six, says I have strong opinions loosely held. So I have lots of things I've been wrong about because I have strong opinions on things, but they're loosely held And I could give you like so many examples. They apply to people, i'll be honest. Sometimes I'm like, oh, this person maybe isn't doing such a great job at this area And that sort of challenge them to do something. And I'm like, no, i was wrong. They sort of supply to that.

Henry Piney: 22:38

Do you feel that one is not Sorry, Srinivas. Do you feel that works, though, in terms of? I've heard that mantra a lot, particularly within the tech industry. Do you believe it works in terms of managing people, though? Because I think the danger of it is? you know, I've heard senior people say strong opinions loosely held and challenge me, but people don't want to challenge the CEO or the CSO.

Avinash Kaushik: 23:01

You don't know. This is true. You have to figure out how to walk the talk. One of the leaders I worked with. They said to me oh, one of the things, avanash, i wanted to do when you join our team, this is in my previous roles. He said challenge me, give me feedback, because I'm constantly asking him feedback. Nobody gives me feedback. I said don't worry, i don't have a problem with that. And then I started to give her feedback And she was shocked. She was not used to getting it. I'm like well, people don't give you feedback because they don't perceive that you're open to hearing it. So you have to figure out how to create a culture where it is okay.

Avinash Kaushik: 23:41

So I'll give you one very simple example. I've created a new modern analytics maturity model. I wrote it on my flight to London the other day. I write on planes And the first thing I did is I sent it to the key leaders in the team and said criticize this. And they happily ripped it apart. So I'm modeling the thought that I am inviting and open. And, by the way, they gave me criticism. I used that to change my thinking and that became the final version. I'm giving you just one small talking about.

Henry Piney: 24:19

I love that. I love that Because the conventional way most people do it is what do you think? and then people feel obliged to say, oh, this is really good, and then, with a couple of caveated maybes, but a two word instruction, criticize this. Yes, it's very clear, you've asked them to do it and they're going to do it.

Avinash Kaushik: 24:35

That's exactly right. So there's many, many things you can do to create an environment. Where I have one of my other practices is every company I've been at. We all gather the team together for a day, We put the best work on the screen and we collectively critique it, and I call them the naked sessions, in the sense that everything stays in the room. We're going to be very open and transparent And, henry, the first piece of work that goes up is mine. I stand up first. Usually I have the biggest title in the room, So my work goes up for us. Everybody sitting there critiques it, and then we go on to the other person And again, it's little things that you can do, where you can have strong opinions loosely held, but if you're not open as a leader to walking the talk, then all you hear are platitudes And underneath it, that's not what people think about you.

Henry Piney: 25:30

Interesting. It's also and again could be the subject of a different podcast. I think it's an area where it's been quite commonly observed that there are differences between US culture, particularly Silicon Valley culture, and European culture around this as well, but various reasons. I think the US arguably has a more individualistic approach and has a tendency to put senior people, often the CEO, on a pedestal, and whereas the European approach is perceived, to some extent, some of the Asian cultures has been more collectivist, of building from the ground up. But anyway, i digress.

Avinash Kaushik: 26:06

I don't know. I agree with you. I have six months that I've been spending time in London and meeting the CEOs and folks here. There's definitely a different approach to Silicon Valley, and Silicon Valley titles don't matter. Honestly, it really doesn't. At Google, i was completely free to send Eric Schmidt an email saying hey, this thing he said in all hands, i disagree with it. And he would reply just extraordinary.

Henry Piney: 26:35

Yeah, it's fantastic.

Avinash Kaushik: 26:37

But here and again in Japan you can't even say anything in Japan because it's been a long time working in Japan. So you do have to make cultural accommodations and adapt around it. So when I do a keynote in Tokyo I take a subtle different approach to language than when I'm in Sydney or Netherlands. I love doing tinos in the Netherlands. I can just say whatever I want.

Henry Piney: 27:03

Yeah, that's the polar opposite. That's the European exception of renowned funding for non-Libreland.

Avinash Kaushik: 27:10

So we have to adapt. We have to adapt our language and things. But if you want to create a competitive advantage for your company, you have to figure out as a leader where are the new ideas going to come from? Who's going to challenge convention? How do you break down silos? It doesn't matter if you're in Germany or in South Africa or you're in Brazil or America. You may execute that in different cultural ways. But if you don't have a spigot that opens up and lets in new ideas, then you become one more company that slowly dies over time.

Henry Piney: 27:42

Yeah, very much so, aminesh. Again, i'm conscious we are over time. You've been very, very generous. Final question for you. This is a personal one. Tell me, if it's too cheeky, what would your partner, what would your wife say? your best and worst characteristics are No good.

Avinash Kaushik: 27:59

I'm happy to answer it. I think she will say my worst characteristic is that I'm very impatient with her sometimes, and I think she's right. I think she's right. She's just an extraordinary woman, so competent and brilliant and smart and beautiful, and sometimes she'll do something and I'm like what? This is it, and it's mostly because my standards for her are so high that I sometimes behave in impatient ways with her, so that that's actually a true criticism. I think the best thing she told me the other day is she's amazed at how completely self-motivated I am without any extrinsic requirements, in the sense that I can push myself super hard without needing somebody else to. Or she can't believe that I spend so much time writing a newsletter every week. It takes a lot of time every single week to put new ideas into the world, and then I do that for no personal benefits. So she remarks sometimes sometimes positively on the ability to be self-motivated, although she spends more of my time pointing out my flaws. Keeps me honest, for which I am grateful.

Henry Piney: 29:21

Yes, it's very true, isn't it, that we often hold higher standards possibly unfairly high standards for those who we love the most or just reach the most, who we have the highest opinion of. I said it was the final question, but just very quickly Any favorite recent books or pieces of media that you would recommend I know it's not TV shows, yeah.

Avinash Kaushik: 29:46

Yeah, i actually went back and read both of Yohal Harari's books recently all over again It's with all the things that I say Red Homo Deus and Sapiens all over again. And I think in context of the development of the last two years, 18 months, with AI, it was good to go back and read the two books again. They shot out the history of humanity and all the changes in all kinds of things over time and let it spark a set of new ideas about the moment that we are in again. So I recommend if people have not read them. Of course they are fantastic, fantastic books. Yohal has an amazing way of compressing an extraordinary amount of complexity into something very simple. But especially with an eye towards the present moment in AI, i found it sparked new reflections in my mind. So I was glad I sort of went back and read them all over again.

Avinash Kaushik: 30:49

And one more, if I have fellow sci-fi fans I'm a huge sci-fi nerd And the best sci-fi books of all time I think are the Three Body Problem by Chik Singh Leo. It's a set of three books And I think they're the best ever written. I just adore them, even though I'm reading them in English and my wife was Chinese I can't get the depth and the nuance of the original Chinese. But I also recommend the Three Body Problem three books.

Henry Piney: 31:24

Both sets of recommendations are fantastic. I have read them. The trouble is I then try to explain the Three Body Problem to a friend of mine who's a very good mathematician And I just totally butchered it. She was like but anyway, that's a different subject in itself. Avinash, thank you so much. It's been so much fun talking to you, incredibly illuminating, And hopefully I might be able to get you back on at some point and we can dive into all sorts of other issues.

Avinash Kaushik: 31:52

No, i'd love to Thank you so much for the opportunity, henry, it was wonderful speaking with you.

Henry Piney: 31:59

It was wonderful, as always, to talk to Avinash. Out of the many great interviewees I've been lucky enough to have on so far, i think he's probably the king of the pithy phrase, with the unique ability to encapsulate big and sometimes quite complicated ideas. Part one he was very, very well received, so if he's kind enough to put up with me again, i'll aim to get Avinash back on soon. There's a lot we didn't get onto, in particular, the parameters by which he recommends you think about AI. Thanks again to Insight Platforms for their support and to HumanMadeMachine for sponsoring this episode. To explore how creative testing can directly impact your ROI, visit humanmademachinecom.