Anecdotal Market Research for the Start-Up

Innovators often begin their process of innovation with inspiration from some experience they have had or observed. Keen observation can be very powerful for identifying opportunities, but it's also easy to go wrong and draw bad conclusions from faulty logic. I had this experience myself this week when my phone went wrong not once, but twice.

By way of background, let me begin my story by saying that I use Windows 7 and 8, have an Android tablet and Samsung S3 smartphone, and was very happy with my previous smartphone, an iPhone 4.

But being happy with a technology does not necessarily mean that it was reliable. I went through three iPhone 4s in my first six month of ownership because for some, never explained reason, two of my phones decided not to permit phone calls unless I also had 3G data turned on as well. Support people speculated it was a cell tower software problem, but my third phone did eventually work and be reliable. Now you may conclude that I might be upset with my carrrier AT&T or Apple, but I was not. I thought the service of the Apple store was superb and there was never any hesitation about replacing the phone on which I had purchased an extended warranty. And I am guessing that AT&T deserved some credit for fixing the cell tower.

This week my Samsung phone suddenly took upon itself to become erratic in terms of charging. After being charged all night, it only showed 7% power. And the problem showed up with different batteries, cables and chargers. Searching the Internet, I found this was not an uncommon problem. The local AT&T store acknowledged the problem and referred me to their maintenance center, where pleasant and friendly technicians handled me well. I ended up with a new phone, which promptly exhibited a pattern of switching itself off for no good reason, something I had also read about on the Internet. Returning the following day, AT&T acknowledged the problem and gave me my third Samsung Galaxy 3. It's annoying wasting several hours, but I have to say that AT&T's service was excellent and very friendly.

Because I write about technology, I took the opportunity to ask other customers waiting, what phones they had and what kind of problems they were having.

The first clear observation was that they all seemed to have Samsung phones. One person had a one week old Samsung Note 2 and all his phone calls were going to voice mail so he was getting no calls. A second person had a Samsung Galaxy S4 and the screen was unresponsive. A third person also had a Samsung Galaxy S4 and his screen had a defective column of pixels that only showed up in some screen color backgrounds.

So, what conclusions should I draw? I first caught myself thinking that Samsung seems to have a lot of quality problems, but then the error of my thinking occurred to me. Most people who have an iPhone will take it back to the local Apple store if there is one nearby. So any conclusions about failure rates will be biased for non-Apple phones in any area like Silicon Valley with multiple Apple stores.

Second, it's worth remembering that Samsung is on a tear and having huge success with its smart phones and phablets like the Galaxy Note. And given that all electronic devices have failure rates, the more you sell, the more units will come back with problems. It's just a percentage issue.

So, here is a lovely example of where anecdotal research is likely misleading without additional confirmation. There is data out there on the failure rate of phones, e.g. http://blog.fixya.com/pr/feb2013/smartphone-manufacturer-report.html and studies that purport to assess breakability for example, e.g. warranty vendor SquareTrade.com, http://squaretrading.wordpress.com/2013/05/08/how-breakable-is-the-new-samsung-galaxy-s4/ but that old stalwart of product ratings, Consumer Reports does not provide any useful information on reliability.

But it's worth pointing out that hidden assumptions about observational research need to be checked at every turn. This blog posting has said nice things about my interactions with both the AT&T store where I bought my phone and the service/support store. Consumer Reports is less favorable about AT&T. Its ratings are available at http://www.consumerreports.org/cro/electronics-computers/phones-mobile-devices/cell-phones-services/cell-phone-stores-ratings/ratings-overview.htm for subscribers. JD Powers in contrast, rates AT&T highest in overall satisfaction, e.g. http://www.pcmag.com/article2/0,2817,2422645,00.asp

So, what are my take-ways.

First, it's easy to leap to conclusions without realizing your logic may be faulty.

Second, observations should be backstopped with additional data. Small samples are pretty much meaningless.

Third, in fast changing markets, market research can get old very quickly. Past performance of a competitor is not necessarily predictive of their rate of improvement.

Fourth, customer perceptions of reliability are often affected by content that may or may not be accurate, significant or relevant. For example, one of the five odd reasons I switched from Apple to Samsung was a series of reports about paint chipping from iPhone 5s. It was not the key reason, but it did contribute to my decision.

Why Wargaming is Useful. The Bankruptcy of Detroit.

One of the major impacts of inexpensive computers is the ability to simulate and game business situations. If you have ever played, SimCity, an early success in the consumer gaming business, you may well have attempted to manage some of the cities that came predesigned in the game. Detroit was one of those cities. I have to admit I was never able to figure out a way to save the SimCity version of Detroit. But playing SimCity certainly honed my interest in and my understanding of urban planning.

Over the years, I have designed and delivered a number of simulations. My largest and most successful was a simulation designed to teach telecom executives about how to compete in a converging world, where fixed line, mobile, broadband data, broadband wireless and video were overlapping. An unusual wrinkle in the design was the idea that in a large corporation, it's not enough to compete with external competitors, you also need to cooperate with internal stakeholders. And just as importantly, you need to understand the different leads and lags that individual functions in the business operate under.

Surprisingly, even managers with long experience in an industry will often underestimate the impact of planning cycle differences. And it's very common for managers to underestimate the range of outcomes possible.

This need to push managers to consider wider extremes is one of the reasons that scenario analysis is talked about so much. But there is theological split in the world of scenario creation and use. Some scenario consultant are quite adamant that scenarios should be stories not numerical models. These professionals are reacting against the phenomenon that spreadsheet sensitivity analysis is sometimes confused with scenario analysis. The use scenario analysis to drag resistant managers kicking and screaming into considering the unthinkable, the Black Swan, and the fat tailed distributions or risk.

But if that battle has been won and managers now understand the vocabulary of  and the differences between sensitivity analysis and scenario analysis, then there is a next generation modeling opportunity, i.e. to translate the stories of a scenario into the impact upon a business.

The impact upon a business may like the wargame reveal not just the sensitivity changes under different scenarios, they may also imply more complex differences in states, e.g.

- what portfolio of IT projects works best over multiple different scenarios
- investments that need to be made before a contingency plan kicks in (as in "We needed to have done that yesterday or last year.")
- business investments that can be thought of as options on a capability, customer group, market or R&D.

In effect, the availability of low cost computing, modeling software and scenarios gives companies the ability to maintain an enterprise and market model as well as providing a vocabulary of stories about possible futures.

New Reviews of Innovation Zeitgeist by James Lau, Founder of Movesandshakes.com, and Vint Cerf, Google's Chief Internet Evangelist

Innovation Zeitgeist Reviewer Comments

Latest reviewer comments on Innovation Zeitgeist includes new comments from Australian James Lau, CEO and founder of the new startup, movesandshakes.com and Vint Cerf, VP and Chief Internet Evangelist at Google. Vint is sometimes referred to as the father of the Internet.

"Innovation Zeitgeist is a marvelous creation.  I love what Alistair Davidson has done.  I enjoy most the narrative style (feels more like a weekend read) and yet it is packed with facts, ideas, authority. There is more content here than an entire MBA course.  It's an MBA on digital" - James Lau, Founder movesandshakes.com

“Innovation Zeitgeist is thoughtful, pragmatic and comprehensive in its analysis of digital product and service opportunities. The business advice it offers is understandable, even to an engineer like me!” Vint Cerf, VP and Chief Internet Evangelist, Google

"Alistair Davidson has written an ambitious, encompassing, and down-to-earth treatise on the current digital transformation.  It’s the best description yet of the digital transformation engulfing us and what we should do about it, written in plain language and illustrated with his own photographs and questions for the reader after every chapter. An iconic, original contribution to the literature on digital transformation that is hard to put down. There is good advice and commonsense written on every page.  Must reading."  Stanley Abraham, Professor of Strategy and Entrepreneurship (Emeritus), Cal Poly Pomona University, author of “Strategic Planning: A Practical Guide for Competitive Success”.

"Innovation Zeitgeist is essential reading for any venture capitalist or startup. It forces the reader to consider what they are doing and what they must continue to do that is different than their many domestic or international competitors, a critical issue for successful startups and for high growth companies." Norm Fogelsong, General Partner, Institutional Venture Partners

“Alistair Davidson has produced an insightful analysis of innovation in the digital era.  This book is a must read for executives seeking survival and success in today's challenging competitive arena.” Robert Allio, founding editor of Strategy & Leadership magazine, former Dean of Rensselaer School of Management. Author of “Seven Faces of Leadership”.

 “A.T. Kearney's clients are all facing the issue of their business being transformed by digital technology.  Innovation Zeitgeist is a great resource for helping our Digital Business Forum clients understand how important it is to actively scope and manage the changes to their business, organizational structure, acquisition strategy, business model and required innovation approaches.” Michael Roemer, Partner and International Co-head of A.T. Kearney Digital Business Forum

“Mr. Davidson has a knack for bringing the thirty-thousand foot practice of strategy down to earth. His discussion of strategic considerations leads directly to an actionable planning structure. Strategy, then, can form a tactical plan. The benefit of the book is twofold: food for thought and a framework for action.” Bruce Rosebrugh, President, VPQ Scientific

"In Innovation Zeitgeist, Alistair zeroes in on many of the key issues we see our market research service-industry clients facing - rapidly changing customer expectations, the need for rapid service introductions, more competition, a need to rethink business models and the types of innovation pursued." Jim Hollingsworth, VP Finance and Security, Pacific Consulting Group

“Bravo! Innovation Zeitgeist offers deep yet pragmatic ideas to address today’s business challenges. Alistair Davidson’s depiction of trends dramatically altering the competitive landscape such as technology overabundance and customer attention scarcity hits the nail on the head.”  Adrian C. Ott, author, “The 24-Hour Customer”; CEO, Exponential Edge Inc.