How far can you rely on a concept test: the generalizability of testing over occasions

In practice, product managers have to assume consumer evaluations of concepts generalise from the time (and research environment) of concept testing to the time (and market environment) of market introduction. However, little is known about the temporal stability or generalisability of the results of concept testing over occasions. Rarely have concept-testing studies incorporated testing of the same concepts on the same respondents on more than one occasion. This research investigates the importance of occasions as a source of error variance in estimates of the generalisability of concept test scores for both minor and major innovations within the context of Generalisability theory. The study collected concept evaluations of ten innovations from members of an online panel on three occasions, approximately a month apart. The results show that the three-way interaction among subjects, concepts and occasions is a substantial contributor to variation in concept testing of both major and minor innovations, with the contribution for major innovations even more substantial than for minor innovations. Moreover, failure to recognize occasions as an explicit source of variance in the generalisability analyses will lead managers to overestimate the generalisability of their decision studies. However, the impact of neglecting occasions varies by purpose of measurement and associated object of measurement. This research provides insight about how well concept testing can generalise over occasions. Concept test evaluations provided on an initial exposure are more favourable than will be received on any later occasions, and apparent differences in consumer evaluations of a particular concept in an initial test do not provide a generalisable basis for identifying which consumers will respond most favourably to it on a later occasion. For concept testing to be used for targeting or segmentation, more occasions will need to be sampled.