Anecdotal Market Research for the Start-Up

Innovators often begin their process of innovation with inspiration from some experience they have had or observed. Keen observation can be very powerful for identifying opportunities, but it's also easy to go wrong and draw bad conclusions from faulty logic. I had this experience myself this week when my phone went wrong not once, but twice.

By way of background, let me begin my story by saying that I use Windows 7 and 8, have an Android tablet and Samsung S3 smartphone, and was very happy with my previous smartphone, an iPhone 4.

But being happy with a technology does not necessarily mean that it was reliable. I went through three iPhone 4s in my first six month of ownership because for some, never explained reason, two of my phones decided not to permit phone calls unless I also had 3G data turned on as well. Support people speculated it was a cell tower software problem, but my third phone did eventually work and be reliable. Now you may conclude that I might be upset with my carrrier AT&T or Apple, but I was not. I thought the service of the Apple store was superb and there was never any hesitation about replacing the phone on which I had purchased an extended warranty. And I am guessing that AT&T deserved some credit for fixing the cell tower.

This week my Samsung phone suddenly took upon itself to become erratic in terms of charging. After being charged all night, it only showed 7% power. And the problem showed up with different batteries, cables and chargers. Searching the Internet, I found this was not an uncommon problem. The local AT&T store acknowledged the problem and referred me to their maintenance center, where pleasant and friendly technicians handled me well. I ended up with a new phone, which promptly exhibited a pattern of switching itself off for no good reason, something I had also read about on the Internet. Returning the following day, AT&T acknowledged the problem and gave me my third Samsung Galaxy 3. It's annoying wasting several hours, but I have to say that AT&T's service was excellent and very friendly.

Because I write about technology, I took the opportunity to ask other customers waiting, what phones they had and what kind of problems they were having.

The first clear observation was that they all seemed to have Samsung phones. One person had a one week old Samsung Note 2 and all his phone calls were going to voice mail so he was getting no calls. A second person had a Samsung Galaxy S4 and the screen was unresponsive. A third person also had a Samsung Galaxy S4 and his screen had a defective column of pixels that only showed up in some screen color backgrounds.

So, what conclusions should I draw? I first caught myself thinking that Samsung seems to have a lot of quality problems, but then the error of my thinking occurred to me. Most people who have an iPhone will take it back to the local Apple store if there is one nearby. So any conclusions about failure rates will be biased for non-Apple phones in any area like Silicon Valley with multiple Apple stores.

Second, it's worth remembering that Samsung is on a tear and having huge success with its smart phones and phablets like the Galaxy Note. And given that all electronic devices have failure rates, the more you sell, the more units will come back with problems. It's just a percentage issue.

So, here is a lovely example of where anecdotal research is likely misleading without additional confirmation. There is data out there on the failure rate of phones, e.g. http://blog.fixya.com/pr/feb2013/smartphone-manufacturer-report.html and studies that purport to assess breakability for example, e.g. warranty vendor SquareTrade.com, http://squaretrading.wordpress.com/2013/05/08/how-breakable-is-the-new-samsung-galaxy-s4/ but that old stalwart of product ratings, Consumer Reports does not provide any useful information on reliability.

But it's worth pointing out that hidden assumptions about observational research need to be checked at every turn. This blog posting has said nice things about my interactions with both the AT&T store where I bought my phone and the service/support store. Consumer Reports is less favorable about AT&T. Its ratings are available at http://www.consumerreports.org/cro/electronics-computers/phones-mobile-devices/cell-phones-services/cell-phone-stores-ratings/ratings-overview.htm for subscribers. JD Powers in contrast, rates AT&T highest in overall satisfaction, e.g. http://www.pcmag.com/article2/0,2817,2422645,00.asp

So, what are my take-ways.

First, it's easy to leap to conclusions without realizing your logic may be faulty.

Second, observations should be backstopped with additional data. Small samples are pretty much meaningless.

Third, in fast changing markets, market research can get old very quickly. Past performance of a competitor is not necessarily predictive of their rate of improvement.

Fourth, customer perceptions of reliability are often affected by content that may or may not be accurate, significant or relevant. For example, one of the five odd reasons I switched from Apple to Samsung was a series of reports about paint chipping from iPhone 5s. It was not the key reason, but it did contribute to my decision.