Generally, an interviewee is inclined to agree to pay for a new feature that hasn’t yet been provided, even for something they actually wouldn’t use. This is not an instance of misrepresentation on the interviewee’s part, for there’s a gap between what people think and what they do, which is just the idea that has been known amongst economists since the 1930s and product managers keep falling prey to over and over again.

This idea can be found everywhere in everyday life. For example, a person may say they need more customization, and yet never look through settings. Another person may state they will find the procedure comfortable, only to send three tickets for assistance to help achieve it. Third, a person may say that dark mode is a must and turn it on once without ever returning to it. None of this makes these people dishonest. They’re reporting a genuine belief about themselves that their own later behavior doesn’t back up, because predicting someone’s own future behavior is not easy, even when answering in a sincere way.

Most mistakes in instrumentation happen before it even starts, because teams track whatever is easy to track instead of metrics that actually relate to decisions. Things like page views and session numbers, as well as “usage,” defined as a single click, are examples of vanity metrics, whose increased values might give you the impression of progress without revealing any useful insights about whether users gained value. Decision-level data looks different. Such data is based on a specific question that someone in the room is asking: did the cohort that was exposed to a new onboarding process experience the activation phase faster and did they remain active after week 4?

This doesn’t mean that interviewing people is useless, and failing to realize this is a failure in itself. Behavioral information tells us what has happened, but not why it has happened, or what users would have done in a situation that has not yet been designed. Interviewing helps us understand why something happened and gain information about areas with no data. The mistake is to equate preferences reported by five interviewees with patterns observed by a larger audience as if the two reflected the same measurement.

A light aspect in assessing the two approaches: behavioral data prevails on anything already measurable by the product because it only shows what a person has done, not what he/she thinks he/she would do. Qualitative signal wins in cases where the data does not provide any visibility on a newly identified challenge, need, or the reason for the drop that is confirmed by numbers but cannot be explained. The failure case is using either one of the approaches alone: the team relying only on the dashboard ends up with the product that is optimized, but not innovative; the team relying on the interviews only ends up with the product that is impressive in theory, but once the data comes in, it shows that the product does not work.