Do we really need humans for insight? Will synthetic data grow or destroy the insights market? What on earth is synthetic data? Check out the latest FutureView interviews with the brilliant Mike Stevens of Insight Platforms to get a great overview of one the most important new areas that will affect everyone and anyone involved with insight and data analytics.
Among other areas we cover:
- The evolution of Insight Platforms
- The importance of adding business context to insights
- Characteristics of insights companies who fail (and succeed)
- Definitions of synthetic data
- Prospective limitations of synthetic data
- Conversational/ augmented surveys
- The impact of LLM based analysis on the insights sector
All episodes available at https://www.insightplatforms.com/podcasts/
Follow FutureView on Twitter at https://twitter.com/FutureView7
Do we really need humans for insight? Will synthetic data grow or destroy the insights market? What on earth is synthetic data? Check out the latest FutureView interviews with the brilliant Mike Stevens of Insight Platforms to get a great overview of one the most important new areas that will affect everyone and anyone involved with insight and data analytics.
Among other areas we cover:
- The evolution of Insight Platforms
- The importance of adding business context to insights
- Characteristics of insights companies who fail (and succeed)
- Definitions of synthetic data
- Prospective limitations of synthetic data
- Conversational/ augmented surveys
- The impact of LLM based analysis on the insights sector
All episodes available at https://www.insightplatforms.com/podcasts/
Follow FutureView on Twitter at https://twitter.com/FutureView7
And what that's doing is actually giving people who actually would never have done any research before. What they would have done is make it up or make assumptions or you know, just relied on someone's opinion. They're actually using that synthetic model to inform decision making in a way that there would have been able to before. So I think it's likely to enhance or grow the total show of decisions that are supported with data, even if that data is generated from a language model rather than primary research.
Speaker 1:Now if you want succinct but very knowledgeable summary on research, tech trends, potentialities and also challenges, there's one person you should really be talking to, my Stevens Mike is the founder and editor in size platforms, before which he ran vision critical in the UK. You should do some unusual other jobs and manage to survive a bit of Saturday night violence as youth. You'll find out what I mean. But perhaps more pertinent, lee, what I wanted someone to give me the rundown on the latest in synthetic data. I couldn't think of anyone better than Mike, as he describes it, since that data can mean multiple things, but in simple terms, within the context of the inside space, we can think of it as artificially generated data that can stand in for data you might otherwise generate by primary means. So surveys, qualitative interviews and other well known techniques is a big subject and one that's going to become increasingly important in terms of replacing or, possibly, more likely, supplementing other methodologies. So I'm going to interview with Mike. So, mike, firstly, thanks so much for joining today. Really great to see you again.
Speaker 2:Good to see you. Thanks for having me on the on the podcast not at all.
Speaker 1:I'm gonna pick your brain and also some interesting areas, as you know. Before we get onto those, though, could we do the traditional icebreaker and what's one thing most people wouldn't know about you. They might find surprising, so something they wouldn't necessarily be able to find out just through a quick web search, or something like that.
Speaker 2:Well, I once worked as a bin man, as a refuse collector. Not many people would know that, but it was. It was a one day job and they didn't ask me back. I thought I did quite well, but they didn't ask me about what's. What's interesting, I guess I actually grew up in the Middle East and in Africa, which a lot of people don't really know, and I went to boarding school which is a consequence of that, so Not really your typical boarding school type. People usually surprised when I tell them that. But now, growing up in the, you know, in the Middle East, in the 80s, in Africa, it was, you know, real. I hope no, it was, I think probably instilled a bit of an international perspective. I Didn't really listen to news that wasn't on the BBC World Service until I was maybe about 12 years old. So, yeah, definitely helped broaden perspectives.
Speaker 1:I think, yeah, both of those are really, really interesting. I mean, I have some. Glad you didn't remain as a bin man. I think the insights world would have lost a great resource and a great mind. Where did you go to boarding school? Was it in Africa, or was it it was in? It was in the glorious city of Wakefield, which is about 10 miles south of Leeds.
Speaker 2:For those, of you who International listeners and was at one point Renowned as the most violent city in the UK for Saturday nights out, which I hope is no longer the case. But yeah, we're certainly a lively environment.
Speaker 1:Well, I'm sure you didn't contribute to any of that, but I remember Now the focus of the interview. Moving on from that is really about two key areas that potentially interlinked synthetic data and conversational survey software. I also want to get a bit of personal background as well, so let's not start at the beginning, because that's way too obvious. Let's start with where you are now and insight platforms. So why did you set it up and what the goal?
Speaker 2:Well, I have to work backwards a little bit.
Speaker 2:I suppose the inside platforms has been live for about five years now and I set it up because I was working as an independent consultant and advisor Anybody who's done that knows that it's hard work and I was looking for another angle to the business that could be a bit more stable, predictable, if I'm honest.
Speaker 2:I was looking for something I thought might be easy, passive revenue. So I thought if I, if I create a directory website for this growing area of research tech, there'll be, you know, lots of affiliate click revenue that will just land in a bank account automatically. And you know, of course you have to work incredibly hard to get easy money and so it doesn't work like that. So I needed to generate, build an audience, generate content to attract an audience, and the different formats of content I guess blog articles, ebooks, you know webinars, virtual events that's all really grown From that, trying to connect the two sides of this industry the kind of research, digital, really, research providers, tools and data providers with the audience that they want to connect with. So that's what it's evolved into.
Speaker 1:And it's evolved into really an ongoing resource, as far as I've been able to see, and we were talking about it just beforehand. So on one side it's a directory, which is great, but there are also a lot of training resources on there as well.
Speaker 2:Yeah, there are. I mean, I think for I guess I think of it in two halves it's really it's a marketplace between providers of tools and knowledge and an audience of people who want to use those tools or gain some knowledge. People are coming to the site. I mean it's, you know it's it's about 30,000 people a month are coming to the site to take a training course, to watch videos, to see demos of software, to join in webinars, read stuff, download stuff. So there's a lot of information resources. But the directory of tools is probably one of the biggest, highest traffic areas because there's nearly 1500 companies that are listed in there. It's a very diffuse collection of, you know, market research, consumer insights, analytics, hanl data, ux research. There's all sorts of different categories in there and I think that's probably why people are coming there to discover new stuff.
Speaker 1:Yeah, that makes a lot of sense. As I said, it's a great resource. How does it intersect with the consultancy work I mean, do you get companies coming to you and going? Mike, this looks great. So much information in here, but I can't really be bothered to read it all myself, so will you tell me what I should be looking at?
Speaker 2:They do and I'll tell them that I charge for that and they normally go away quiet. But, you know, for mates obviously like you, then, you know, happy to give a bit of direction. Now I do about probably 20% of my time now is consultancy advisory and it tends to fall into three different areas. So advice for teams or agencies who are looking to change. So it's how do they become more, you know, digitally focused? What's their tech stack? How do they structure the kind of skills that they need?
Speaker 2:It's sometimes for companies who are looking for go-to-market strategy so for a product launch, for entering a new market and, you know, need help with mapping out the landscape and the planning around that. Or it's for in-house teams, often appraising what do they have and what should they have going forward. So that might be. Please review all this stuff we're spending money on now and tell us what we should be doing differently. Or we know that we need a solution for X. Can you help us to narrow down the list of companies we should be speaking to and, you know, tell us who's likely to fit our needs? So those are the kind of the three buckets.
Speaker 1:I will. We might diverge from the schedule, mike, and this is kind of getting into the weeds that, but when I have got involved with businesses that are looking at creating data lakes and that type of thing and they've got quite structured data in other forms or areas of their business, they look at survey data quite often and they go what a bloody mess. How do we integrate that? Is that quite common?
Speaker 2:Yes, I mean funny. You should say that actually because in we hosted an event last week and there's a nonprofit initiative to tidy up that mess that you're describing. So it's creating consistent language for interchanging data in the survey industry. It's APIs for the technical people how you actually get data to flow between platforms. Historically, it's been very vulcanized and it's been a mess, and different providers have all had their own systems for doing things.
Speaker 2:And that means that if you're an agency or if you're a brand. If you're, you know, if you're buying all of this different survey data from all the different companies, you can't actually integrate that, interchange it, you can't move things between platforms. So that's a solution. Well, that's a huge challenge that a lot of people in the industry just don't perceive, because they don't have your kind of perspective across data analytics and the other types of data that organizations are trying to integrate. So I think it's a big challenge, but there is an opportunity for you know, for addressing that.
Speaker 1:Sorry, what's the name of the initiative again?
Speaker 2:So the initiative is called TSAPI and it's being run by. Matthew Gibbs and Tim Brandwood are taking the leads on that and you can actually watch a demo. There's a 10 minute demo of it on the Insight Platforms website, which begins with live action on the Clifton Suspension Bridge in Bristol. So never thought you'd see a software demo that started with live action. Then that's your one.
Speaker 1:Good stuff. I look forward to seeing that. Now, going back to the beginning, as promised we would do, how did you get into this world and could you give us maybe kind of a little bit of a whistle stop tour and anything you learned along the way?
Speaker 2:Yeah, okay. So after my one day as a bin man, I was a student, you know. I graduated. I didn't study anything particularly useful for this industry. Well, no, that's a lie. Actually my degree was in French and German literature, so I think it did give me some practical skills in terms of languages, but it gave me some fundamental skills about decoding, interpreting, understanding to be able to apply that. So I guess I'm doing my education at disservice. But I was looking for a part-time job or, you know a job, without knowing what I wanted to do. I started working for a tiny B2B agency in Blackheath that had a small telephone unit doing interviews, with telephone interviews across Europe in the languages that I could speak, with incredibly niche audiences. So design engineers, production engineers in companies that manufactured locomotive engines they might be for coal mines, they might be for other things Talking to them about their needs for cooling systems and fans that would be built into their engines. So very very niche.
Speaker 1:I'm cracking up at this point so I'm getting this vision. It's very niche, but you've done French and German literature, so what you're sort of trying to weave more the air and go to the conversation.
Speaker 2:Well, I didn't get so many opportunities for that but it was let's say it was a crash course in learning niche vocabulary for locomotive engine design, anyway. And then I actually joined my first research agency after working around the industry for about 10 years and went to join Research International, which was a fantastic organization, but, my God, what an eye opener for how not to do things efficiently. At the time I'm doing the greatest service. Now I can't always be careful what not to say, but at the time I'd worked for these small boutique consulting firms. You'd be getting data. You'd be working with it the night before and pitching up and making, let's say, confident assertions off the back of the data you've got.
Speaker 2:In this organization there were reams of departments printing out stacks of tables of data, other people would be responsible and nobody would actually make a commercial recommendation off the back of it. So it was for the first few months. I was like what the hell have I joined? But it was. In some ways we had an agency with an agency working for the agency's biggest client so Vodafone at the time.
Speaker 2:You remember, telecoms was huge in the early 2000s, so we actually built some very different ways of doing things using technology. We pioneered a lot of methods that then got adopted elsewhere in the business, and then I left about. Where are we now 2010,? Something like that. I joined a Canadian software company called Vision Critical, which was a pioneer in the kind of communities panels, space building, lots of private audiences for retailers, media companies to be able to do research very quickly and get in-depth insights from their own customers or audience members, and that was alternately infuriating and enjoyable and rewarding. But ultimately I ran out of steam with that, with a kind of mini midlife crisis, not really knowing what I wanted to do next, expecting someone to come and offer me an amazing job, and when they didn't, I had to start, you know, scrounging around for consulting and advisory business, which is really where this part of the story starts.
Speaker 1:Yeah, good. Well, thank you, my dad, that is a really good whistle-stop tour. The point that you made about RI and then later Cantar and this isn't about Cantar, by the way at all, but it does seem that that is still there's still preponderance, or how can I put it. The industry sometimes seems to be kind of caught between two stools. Still around that around, we're all about the data. We only give data-based recommendations that are statistically significant, and so on and so on. We've got how many pieces of evidence to back them up? And that is our job. We are going to report that data and we'll give you very, very logical recommendations but which sometimes aren't that interesting. They almost seem a little bit obvious and they don't necessarily take into account broad a business kind of context. Is that fair, would you say, do you?
Speaker 2:think that it's totally fair. I think it's an issue that afflicts the majority of advisory businesses in the research and insight space, which is, you ask just to go and investigate this. Here's the data that we got. Here's what the data is telling you and for an awful lot of people it makes them tear their hair out and they say, well, what about XYZ? You've got no context of our organization, you've got no competitor landscape, you've got no real advice or opinion in there, and I think you know the evidence in our opinions with evidence I think are lacking.
Speaker 2:I think there's a there is a mindset that I hadn't encountered because I'd worked in consulting for first 10 years of my career.
Speaker 2:When I came into the research industry, I was surprised at the reticence Incredibly bright people, very respectable, and you've only got to see the way that people get crucified when holding is off to understand where a lot of that legacy lies. You know, particularly In organizations that are buying research for consumer innovation, advertising. You've got big opinions in marketing and commercial teams who will shout at researchers if you know, if numbers are a little bit off. So we built this culture of Methodical Safeness and I think that is A little bit dangerous for where the industry needs to go next. I'm not saying we should get us and any of that respect for methodology and data and quality and any of those things but there is a challenge in the personality and culture of people who need to be leading from the front and advising and saying this is what we do it. There are some great exceptions to that. There are some really very strong consulting, that advisory led businesses in the industry, but they're the exception. You know it's not really the broader culture still of the insights business.
Speaker 1:So you have a great perspective in that you've got this repository information. You work with a lot of different companies, so maybe this is an unfair question, but, based on what you've seen, one of the common denominators of companies that Succeed in the inside space at the moment, what characteristics do they imbue?
Speaker 2:Can I answer the question a different way? To start with, can I tell you what the the three characteristics of companies that are doomed to fail, or the bad habits that I see, before going on to the good stuff, because yeah, yeah, you do see a lot of bad habits.
Speaker 2:You know you don't. You don't necessarily see all three of these things in the same place, but you know, I think I've seen a lot. This is not unique to the inside space. This is probably more about our current culture of Vc led funding for technology promises. So you know, when you see a company that believes that getting its funding around is really its goal, you know that believes that it's been successful because it's found investment, you know that in itself is not the goal. The goal is to build something and generate value for customers. But I think we've we've managed to create this lens through which an awful lot of startups and founders see, you know, the funding around as the badge and that they've achieved this amount arrived if they've been funded, and that's problematic for me.
Speaker 2:I think the second thing flows from that is, I worked for a company that made the fundamental error of Pivoting its energy towards pleasing its investors away from doing the right thing for customers, and you would think those two things should be aligned. You know, if you're gonna Deliver value for investors, you doing it right for customers. It's not the case at all. What it means is you start focusing your energy on Jerry mandering, the way that numbers are presented, you start Telling customers that you know the thing that they've been doing. They can't do.
Speaker 2:You hire the types of people that look right from an investment, an investor or advisory perspective and don't actually work for, for customers. So that's a challenge, I think. And the third thing that makes me really suspicious accompanies is when they say Everything that's been done in this industry historically has been crap and here is the new way and this is the only way, and you know, it's almost like a kind of evangelical, you know, make research great again, kind of mission, and it's. It's a very, very suspicious of those people, those companies that are, you know, with the revolution and everything else is terrible. So those kind of three characteristics, I think Our big red warning lights to me.
Speaker 1:I think they're very fair, mike. I mean, the third one is probably related to the first one as well, in the sense that we're all interrelated, but yeah, particularly because, in order to get your funding, you've got a pitch for a big market. So I can, so I want to claim that you're transforming it. The second one's interesting too, in that how do you think companies can then balance this desire to create a product, if they want to create a product long do we still pleasing customers? And but not pleasing customers in the sense that you're constantly changing your products and providing custom deliverables, or changing, necessarily, your, your data structure. I think that's the thing a lot of businesses wrestle with in this space.
Speaker 2:Yeah, I think I'm gonna do. You know, I think it does come back to the thing you said, which is, in order to Secure the funding that you want, you actually have to create a story pitch which is bigger and more dramatic and faster to value than actually the markets gonna support so you already Sold.
Speaker 2:You sold to the devil by the time the check is landed because you've made promises that you could never keep. So Investors are always gonna be like hang on. You said you know that this was gonna happen by then. You know, understandably, they start to put pressure in this. So you've not got the right sales Profile, you've not got the right pitch. You know all of those things. So I think it's it's very bound up in the nature of you know expectations around funding and the pace at which Value is gonna be delivered and the overall scale of the opportunity. Because you know, when I think about the number of startups, the number of Funding pitches and promises that are being made, this industry just isn't big enough to support all of them, no matter how big. You know you look at the top down. Any which way you cut it, there just isn't that kind of opportunity. So they start to define all these adjacent market spaces, lose focus on the real customer. You know it becomes a bit kind of self defeating.
Speaker 1:Yeah, very much so I'm hopefully I'm not involved with those businesses. We try to encourage them not to take that particular attitude. My unconscious of time slightly and I'd love to just pick your brain and go back to your past history. Yeah, I didn't tell you what the good things were, we didn't we didn't actually talk about the three. Let's hear about that stuff too probably.
Speaker 2:They're probably inverted, I. So you know there's something fundamental actually about you know build something that people want, you know. So there's a. You know product market fit, whatever it is, build something useful. You know there's a. There's a kind of fundamental component of that which might be A new way doing things. It might actually just be a better way, things you've been doing already. That's a kind of foundation thing, obviously, but then I think there's a.
Speaker 2:There's a lot of value attached to doing the basics right, you know. So, providing Continuous sort of service, providing the right level of skill to support technology. On top of it, helping Customers, you know showing up and just doing the boring stuff every day, and I think you know it's companies that. The third thing for me is about aligning value with customers. So you need to. You do need to hire the right people. You need to get people who are going to speak the language of the customers. You need to Blend services and expertise with technology and product in the right kind of mix. You know some of this is not very Sass by ball, you know, but accept that. You know you're not always the answer at knowledge when your product doesn't actually do the thing that customers are trying to do, don't you know how are a ramp against the square hole and say you know we're not the right solution for that, and I think you know those are. Those are some of the attributes that I see in successful companies in this space.
Speaker 1:Yeah, again, they make a lot of sense. To the third area that you're describing as well. I'm involved with one company it's a German company, not commission the inside space, but when we were looking at their model, they have a good foundation for the business in terms of consultancy and they're trying to create like a SaaS platform around more like the SME type of space. It's more of a DTC type of area, but one of the key things that we've identified is we want to keep the consultancy business and not because it keeps you close to your clients and it can almost act like a paid R&D engine room for you, so you could see what's working, you can refine it with your clients and then you can take the pieces that are most relevant out of that and potentially then put them into the SaaS platform.
Speaker 2:Yeah, I think it's the, I think they. If you were to step back and say what is the fundamental tension for any provider in this industry, any company that's delivering value to buyers, that fundamental tension is the trade off between expertise and product. How much of your offer do you productize through technology, through data, through defined solutions? How much of it do you provide as professional services, consultancy, support? It sounds like a simple trade off but actually it's not, because those things can smear and blur together across a line and it's a very difficult thing to say we're going to premiumize this bit of consulting and advisory or we're actually just going to bundle it in with the product and we're going to have it as a component of customer success. Where you draw those lines and how much you value the different components is a really difficult thing because you know, we know investors are all looking for scalable product, investors all want high touch services and expertise. So that's a real tension.
Speaker 1:Yeah, yeah, an interesting tension. So, getting onto this question of synthetic data one of the issues we're going to talk about so I've tried to educate myself, I've looked it up, I've read some of the things on the Insight platforms, which are good, but so many different people have described synthetic data in a in different ways. What is synthetic data?
Speaker 2:to the extent that it is relevant to the insights world, yeah, I think your experience of different definitions and different understandings of it is because there are different ways of cutting this. So you know, synthetic data is a monolithic term but underneath it actually there's lots of different ways of achieving it. In the simplest possible terms, it is artificially generated data that can stand in for data you might otherwise collect through primary means. So an interview, a qualitative discussion, surveys, that type of thing. So in a chat GPT world, if you were to say to chat GPT, please answer the following questions as if you were this persona, and this persona is a single parent with two children, struggling on a low income, trying to achieve career aspirations, that type of thing. And then you know the large language model will try to answer with that persona. That is the most. That is one of the simplest ways you could get it to say please answer. As if you're suffering from hay fever, please answer. So you know you can generate these personas, you can instruct the language model to. You know, adopt these personas. Now there are many, many more sophisticated ways of doing this and there are ways of doing it at scale, and it's not all large language model based. So there is a company called Synthetic Users that does this in a more of a qualitative persona generation way for UX research. So you'll brief it you'll get you know 10, 12, 15 personas with whom you can have qualitative dialogue about your topic, your challenge, your product. At the other end of the scale, there are companies that are generating digital twin audiences that reflect, you know, at the scale of you know thousands or tens of thousands that reflect attributes that you might find in buyers of particular products. You know where you can then go and ask and you'll get survey type responses from them.
Speaker 2:And then, beyond research and insights, synthetic data is used for all sorts of things that they kind of you know million case level, to be able to stress, to security systems or to, you know, to simulate, you know how people might flow through an e-commerce site, that type of thing. But in research and insights terms, generally large language models are generating, you know, synthetic personas, either at the qualitative scale or at the quant scale, in order for people to interact with them as if they were real respondents. There are exceptions to that, though, because Yabble, which is a real pioneer in the space New Zealand company, actually has a model that isn't LLM originated. The data is culled from all sorts of other third party sources. So it might be, you know, trends, search data. The actual sources are kind of a secret source, so to speak, but they'll then put a language model interface so that you have a conversational interaction with the data that's been collected, so you can see there's. You know, there's different ways of generating synthetic data and working with it.
Speaker 1:Got it, yeah, so thank you. That's a great description. I actually finally began to understand at least 80% of it, I hope. Anyway. One of the questions, though if you look, not the use case for Yabble, how they're doing it, but they're synthetic users or synthetic audiences. So you are putting them through a user experience or you're trying to duplicate a cohort of audience members, but how do you, how does the information then update? How does the how does the dialogue remain dynamic? Because it sounds to me like in that case you're constantly then referring back to the same LLM and, admittedly, learning from it and deepening the knowledge, but it's almost like the source material is large but relatively constant.
Speaker 2:So I guess it depends on your brief.
Speaker 2:So you know, yes, most of the large language models are trained at a point in time and are not necessarily updating a lot of that stuff you know in.
Speaker 2:Certainly not in real time, but not as frequently as you might expect. For you know, let's say, a tracking study or something like that, but in a lot of cases it doesn't really matter because the type of input that you want is hypothesis building or testing, response to a particular set of, you know, product ideas or communications concepts, that type of thing. So it's really about synthesizing all of that knowledge that's built into the language model that, even if the training data only goes up to a few months ago, it's not, it doesn't require recency in the same way. Now, that is different from the Yabba model that I described, is much more recency focused and the language models are starting to build in much more, you know, longitudinal time series approaches to collecting data over time and refreshing the training data more regularly. But it's a you know it's probably not something that's going to happen in that way for another six or 12 months.
Speaker 1:Got it. Now certain companies go back and validating the synthetic audiences or models against human respondents.
Speaker 2:They're absolutely are, and for people whose primary business is supplying survey panels and data, they should be a little bit anxious about this, if that's where you know the presumptions of future revenues are, you know are coming from, because there are some companies, big companies, brands, who you know can't really name, but who are extensively validating for different use cases, different audiences, different business questions. How do these things compare? So you know, there's a lot of validation work going on and I have to say they're coming out. So, to be honest, for a lot of our needs, it's either good enough or it's kind of spookily close. You know, these things are, I suspect, going to really transform the way in which we work with data over the next few years.
Speaker 2:Now the question is is it going to grow the overall pie or is it going to substitute primary research data?
Speaker 2:I mean, I'm in the bigger pie camp by a long, long way, because I saw a demo yesterday and I don't know if this is really public yet, so I won't speak about it, but it's a.
Speaker 2:It's a B2B platform which is entirely large language model based, and a B2B company can go there and say give me, you know, 50, 100 personas, characters of customers who might be buyers? How did they break down? What are the segments? What are the ways in which we could sell to them? What's the relative market shares versus competitors for our brand? And the stuff is really good enough, and what that's doing is actually giving people who actually would never have done any research before what they would have done is make it up or make assumptions or, you know, just relied on someone's opinion. They're actually using that synthetic model to inform decision making in a way that they wouldn't have been able to before. So I think it's likely to enhance or grow the total share of decisions that are supported with data, even if that data is generated from a language model rather than primary research.
Speaker 1:And I guess hopefully you'd also use it as a foundation. I guess I would say this, but to then go find the synthetic model has got me 90% of the way there. I now know the two or three concepts that I believe are most likely to succeed. I haven't had to do my traditional quantum research to get to that point, but now I'll do something that's much more focused on those concepts.
Speaker 2:Yeah, I think that's likely to be. Certainly in the short term, people are going to have a lot of questions about going is this real? Can I trust it? Do I need to validate it? That's going to play out all over at some point, in the same way that 20 years ago people were like we can't rely on surveys that have been completed on the internet because they're weird, the data's not right. Now that's not an argument you hear in many cases. In some it should be, because the data's actually wonky, but people are relying on data where we know there's broadly, has validity and, in some cases, has serious quality problems will end up in the same place. We'll have broadly trusting, valid data coming out of LLMs for synthetic and we'll have some edge cases where people never should have used that to drive a decision because the data's terrible. That's just the reality of working with data.
Speaker 1:Where do you think the limitations are likely to lie? So flipping around, say, the 90% or good enough argument. Are you seeing consistencies around those 10% of cases, if I'm going to use that figure where LLMs or sorry, synthetic data approaches struggle?
Speaker 2:Yeah, I think that the recency and the time series, you know longitudinal stuff, is an issue right now, but I think it's going to come. You know the speed with which the models are improving. Real-time access to public data. You know, if you've got an LLM that's updating daily with you know search trend data, it's got all of that stuff that's kind of feeding in then that helps to remove that as a barrier. The thing that broadly, what I worry about is the sort of banal output risk. So overreliance on some of these things generating knowledge or data that is ever more towards a central tendency and the eradication of interesting outliers, you know. So I've seen maybe a dozen of these tools I've seen in the past. I've seen some of these new outputs. I've yet to seen anything that surprised me or made me laugh or you know I thought was controversial, you know so it's the risk of banal yeah, okay, that makes sense.
Speaker 2:Kind of outputs, rather than anything that's you know. That's, I think, will be where maybe some of the innovation opportunity lies is going to be in generating data that actually do have light bulb opportunities that are not just about you know okay, that makes sense. Type responses.
Speaker 1:Which is potentially a very nice segue onto the conversational survey, your conversation as well, in terms of how you might do that. So if we jump onto that, mike, I mean, then you're in an environment whereby you're actually talking to real respondents, but you're assisting the interviewing process. As I understand it, is that a fair summary?
Speaker 2:Yeah, it is. It's a kind of augmented survey, the way that it's being implemented right now mostly. So I've seen more startups, actually, you know, list their company on insight platforms in this category in the last little while than I have. You know, in most of the categories and I think there's a lot are coming out this from a user and product research perspective saying, hey, you know, you can automate your user interviews and the way the language model works is the conversational AI will ask a question that's pre scripted and then the AI will interpret the response and a little dig deeper so it'll probe on the response. So there's a company that's pioneered this model called Inca, and they have actually, you know, they've got an API solution that now integrates with lots and lots of other survey platforms, so you can embed this approach in whatever survey tool you're using, which will be a conversational thing. So it'll be like you know why did you like that particular car? And it's like well, you know it's because it's got, you know, nice red paint and it's what is it about the color red that particularly appeals to you? You know it's. It's not. You know, it just helps to surface stuff beyond the obvious. Now, what that means is you get richer, longer answers, you get debt that you don't always get from the kind of, you know, totally prescripted surveys. You actually get more participant involvement and satisfaction with the survey and it gives you, you know, a sort of richer perspective from that open-ended data without really adding a big load of cost To me, if you can sort of imagine a series of steps about where this evolves.
Speaker 2:You know this is basecamp, you know so for AI mediated research conversations. This is this is really step number one. You know step two you start to get into much more flexibility around survey design. So, instead of needing to pre-script and put things in boxes and grids, you know you've got much more flexibility. But then, when you start to introduce language models and personalities, what you're doing is really blurring the distinction between what's a survey and what's an open discussion, what's a qualitative discussion. Then you start to introduce things like, you know personality into chatbots, you maybe give them into you know video avatars, so that you've got much more interactive. What you start to have is the opportunity to do very wide, you know large scale Open discussion that's much more empathetic, that is much more focused on the actual Opinions of the user and participant. That is about trying to pre box everybody into a, you know a pre defined structure. The researchers put into a survey.
Speaker 1:I mean, is there a virtual circle, if I'm use that phrase in that I can see one of the objections to it from an analytical perspective, as you got Potentially quite a lot of on relatively unstructured data because it's been more fluid. However, I imagine you can then stop plugging it into various ml or a I Infused analytical systems that are helping you speed up your categorization enormously.
Speaker 2:Exactly so. No, and these things are becoming. You know there are some innovators in the space, so there's a couple of start ups and products Focus specifically on qualitative understanding and certain qualitative feedback. So what's happened historically is, you know we've been able to train text analytics and machine learning Unstructured data that's been very wide and shallow so you might have a thousand cases with twenty word answers. That's relatively easy to then go and code and categorize and structure. What large language models of down is is enable that for narrow and deep. So you may have ten interviews that maybe you know. Three thousand, five thousand words back and forth between interview, lots of contacts needed. We've got some very good tools now to be able to analyze and understand that kind of depth. So when you scale that, you have these, you know these big contacts windows now in the large language models that can start to generate qualities of insights at depth, you know and in the future much more scale.
Speaker 1:It is interesting, as you start to piece it together, to see how these different pieces of tech and potentially can a code here together. Might you just go back on some of the area that we're talking about before around this idea of regressing to the mean or, yeah, you know, synthetic data or l? Lm, so what if they might be kind of regressing to the mean? I think some of the areas that I've read in the past is on the academic research around this have suggested that, due to some of the factors that you're talking about, this type of analysis maybe more applicable for certain categories as opposed to others. So, as an example, no brands yes, makes a lot of sense that a lot of information out there, but if you've got a new product launch around a of an unknown brand or spin off from a well known brand, then potentially it's this type of approach is less effective. Have you seen any evidence of that or would you agree?
Speaker 2:I think it. I guess it depends really on what the research question is. If you try to understand, you know, like your response, then you know, to a new product or new concept that's that's not been seen before, you know, then you're right. Anything that is Radically different, that doesn't have a frame of reference, is going to challenge the sort of model. But to be honest, that's always been the issue with traditional research, which is people don't really understand the radically new. You know they need to be kind of educated. They've got to have the heuristics to place it in the context of things they know about already. So I don't think that's necessarily gonna be a bigger challenge for synthetic than it is for traditional research. Notes, that thing that I am gonna miss quote you know these things. That is awesome. Steve jobs know. You know you can expect people to be able to tell you what they want. You know you can expect people to be able to give you a sensible view of a kind of radical innovation.
Speaker 1:Yeah, that's a great point. I think about it. It's very much an inhibiting factor for traditional research techniques as well. The nintendo we, I've heard, uses an example. What you put it through research, you're gonna call it the nintendo. We was like that's ridiculous and you get that response from traditional research techniques as well, as I'm sure from that's. It approaches, but a lot of it depends on the creator, yeah, would you say. The other direction I've heard around as well is that it's potentially easier for a I infused approaches To predict failure I what isn't going to work as opposed to what is going to work, because this relate to something you just mentioned about that question, like inspiration, that quite often the things that really work in a market, particularly if then you Product launches, all those ones that are a little bit different, they're not regressing to the mean, they just do something different. Is that a comment.
Speaker 2:Is this a? Is this the risk of synthetic data? Is this the thing? That research also does, more than Is? It you know, does it? Does it kill radical ideas?
Speaker 1:because I'd, I'd say, particularly risk of synthetic data as opposed to I think there is a risk inherent with traditional research as well. However, at least within traditional research, you've had that more open ended kind of qualitative forum, but I see it as more of an issue around the synthetic world or the a I predicted world, possibly might be right, I don't.
Speaker 2:I don't have a strong view, but I think it sounds. It sounds sensible.
Speaker 1:So, mike, thank you, I've taken up so much of your time is usually picking your brain. I may go back with further questions to, but if it's right, I want to just move on to a quick fire round, no problem. Okay, slightly cheeky question, but it's one of my favourites. So what would your partner say? Your best and worst characteristics?
Speaker 2:Best. I think she would say that I'm loyal, I'm generally quite kind and I don't take myself too seriously. I think she'd say that my worst. I can't admit when I'm wrong. I never finish anything and I often bang my head. So I walk into things and bang my head a lot. I'm Tell them I look when people see me on zoom and often just seem to miss cover doors, that type of thing.
Speaker 1:Well, the last one really isn't your fault. Well, you could say I should pay more attention, as she often does if you could be CEO of any company, which would it be and why?
Speaker 2:You know they say you should never meet your heroes, and I think you should never work for companies you love because you're bound to end up feeling disappointed. So companies that I love right now are why is transfer wise? Just because I think they're awesome and they're fantastic business and Zapier, which is just an incredible innovation I know there's lots of me to is, but it's basically it. You know no code, automation, integration. I think that About sixty dollars, hundred dollar the month, it saves me the equivalent of a whole full time person. So those two are amazing businesses I'd love to work for. Honestly, I'd like to run sterling Cooper, but you know that's just a little madman, aside for people who know what I'm talking about.
Speaker 1:But yeah, yeah, I do know what you're talking about. What would you do with sterling Cooper now?
Speaker 2:Oh god, I just have a lunch and whiskey. Would know, I mean you know, would be smoking all the time. It'd be amazing.
Speaker 1:Yeah, you thought you'd forget about, like the synthetic data and all the rest of it, you do you just rely on personality, creative ideas and supreme levels of confidence. Yeah, and john ham's amazing hair, so maybe related. Final couple questions what do you know now that you wish you'd known twenty years ago?
Speaker 2:The running your business isn't terrifying. You know, I was never really, I was one of my own thing, but it was never quite the right time and I was never quite ready to take that gamble with a big mortgage and, you know, never had a lot of capital to fall back on. So no, I would say, if you think you fancy it, do it. There's no going back, you know, once you there, so it's, it's great.
Speaker 1:Great advice. And what's your favorite? Almost impactful book or recent book could be a bit of media. Doesn't have to be a book.
Speaker 2:Honestly, I'm a little bit obsessed with the state of the world right now, so I'm putting on the return of history, which is so martin. Six months is next. Bbc journalist has written Fantastic account of the historical context for what's going on between Russia and Ukraine right now and the complexity, the depth, the sorts of decision making and the errors, on all sides you know, that have led to this and also just the you know the kind of the knowledge of leaders. It's a. It's a cracking read, a very, very well researched and put together.
Speaker 1:Brilliant. Thanks so much. Like I should let you get on with your day, but it's been an absolute pleasure.
Speaker 2:Thanks, henry. Thanks for having a lot of pleasure.
Speaker 1:Huge thank you to my for letting me pick his braid as well. As is always acute analysis of the industry as a whole. Lots more to come in future episodes including interviews with some of the specialist companies in the synthetic space, the new subject of behavioral learning models with element human influence. Economy with wayla insights family who just completed new funding round, client side perspective from sky and the latest on the next gen of credit testing and measurement with human made machine. Thanks again to Mike and insight platforms for their support and to you for listening. See you next time.