How far can you rely on a concept test: the generalizability of testing over occasions

Ling PENG, Adam FINN

Research output: Journal PublicationsJournal Article (refereed)Researchpeer-review

5 Citations (Scopus)

Abstract

In practice, product managers have to assume consumer evaluations of concepts generalise from the time (and research environment) of concept testing to the time (and market environment) of market introduction. However, little is known about the temporal stability or generalisability of the results of concept testing over occasions. Rarely have concept-testing studies incorporated testing of the same concepts on the same respondents on more than one occasion. This research investigates the importance of occasions as a source of error variance in estimates of the generalisability of concept test scores for both minor and major innovations within the context of Generalisability theory. The study collected concept evaluations of ten innovations from members of an online panel on three occasions, approximately a month apart. The results show that the three-way interaction among subjects, concepts and occasions is a substantial contributor to variation in concept testing of both major and minor innovations, with the contribution for major innovations even more substantial than for minor innovations. Moreover, failure to recognize occasions as an explicit source of variance in the generalisability analyses will lead managers to overestimate the generalisability of their decision studies. However, the impact of neglecting occasions varies by purpose of measurement and associated object of measurement. This research provides insight about how well concept testing can generalise over occasions. Concept test evaluations provided on an initial exposure are more favourable than will be received on any later occasions, and apparent differences in consumer evaluations of a particular concept in an initial test do not provide a generalisable basis for identifying which consumers will respond most favourably to it on a later occasion. For concept testing to be used for targeting or segmentation, more occasions will need to be sampled.
Original languageEnglish
Pages (from-to)363-382
Number of pages20
JournalInternational Journal of Market Research
Volume52
Issue number3
DOIs
Publication statusPublished - 1 Jan 2010

Fingerprint

Testing
Generalizability
Innovation
Evaluation
Managers
Consumer evaluation
Targeting
Market environment
Segmentation
Interaction
Test scores

Cite this

@article{2d51cff01cf640f79af9d676ee31846e,
title = "How far can you rely on a concept test: the generalizability of testing over occasions",
abstract = "In practice, product managers have to assume consumer evaluations of concepts generalise from the time (and research environment) of concept testing to the time (and market environment) of market introduction. However, little is known about the temporal stability or generalisability of the results of concept testing over occasions. Rarely have concept-testing studies incorporated testing of the same concepts on the same respondents on more than one occasion. This research investigates the importance of occasions as a source of error variance in estimates of the generalisability of concept test scores for both minor and major innovations within the context of Generalisability theory. The study collected concept evaluations of ten innovations from members of an online panel on three occasions, approximately a month apart. The results show that the three-way interaction among subjects, concepts and occasions is a substantial contributor to variation in concept testing of both major and minor innovations, with the contribution for major innovations even more substantial than for minor innovations. Moreover, failure to recognize occasions as an explicit source of variance in the generalisability analyses will lead managers to overestimate the generalisability of their decision studies. However, the impact of neglecting occasions varies by purpose of measurement and associated object of measurement. This research provides insight about how well concept testing can generalise over occasions. Concept test evaluations provided on an initial exposure are more favourable than will be received on any later occasions, and apparent differences in consumer evaluations of a particular concept in an initial test do not provide a generalisable basis for identifying which consumers will respond most favourably to it on a later occasion. For concept testing to be used for targeting or segmentation, more occasions will need to be sampled.",
author = "Ling PENG and Adam FINN",
year = "2010",
month = "1",
day = "1",
doi = "10.2501/S1470785309201314",
language = "English",
volume = "52",
pages = "363--382",
journal = "International Journal of Market Research",
issn = "1470-7853",
publisher = "Market Research Society",
number = "3",

}

How far can you rely on a concept test: the generalizability of testing over occasions. / PENG, Ling; FINN, Adam.

In: International Journal of Market Research, Vol. 52, No. 3, 01.01.2010, p. 363-382.

Research output: Journal PublicationsJournal Article (refereed)Researchpeer-review

TY - JOUR

T1 - How far can you rely on a concept test: the generalizability of testing over occasions

AU - PENG, Ling

AU - FINN, Adam

PY - 2010/1/1

Y1 - 2010/1/1

N2 - In practice, product managers have to assume consumer evaluations of concepts generalise from the time (and research environment) of concept testing to the time (and market environment) of market introduction. However, little is known about the temporal stability or generalisability of the results of concept testing over occasions. Rarely have concept-testing studies incorporated testing of the same concepts on the same respondents on more than one occasion. This research investigates the importance of occasions as a source of error variance in estimates of the generalisability of concept test scores for both minor and major innovations within the context of Generalisability theory. The study collected concept evaluations of ten innovations from members of an online panel on three occasions, approximately a month apart. The results show that the three-way interaction among subjects, concepts and occasions is a substantial contributor to variation in concept testing of both major and minor innovations, with the contribution for major innovations even more substantial than for minor innovations. Moreover, failure to recognize occasions as an explicit source of variance in the generalisability analyses will lead managers to overestimate the generalisability of their decision studies. However, the impact of neglecting occasions varies by purpose of measurement and associated object of measurement. This research provides insight about how well concept testing can generalise over occasions. Concept test evaluations provided on an initial exposure are more favourable than will be received on any later occasions, and apparent differences in consumer evaluations of a particular concept in an initial test do not provide a generalisable basis for identifying which consumers will respond most favourably to it on a later occasion. For concept testing to be used for targeting or segmentation, more occasions will need to be sampled.

AB - In practice, product managers have to assume consumer evaluations of concepts generalise from the time (and research environment) of concept testing to the time (and market environment) of market introduction. However, little is known about the temporal stability or generalisability of the results of concept testing over occasions. Rarely have concept-testing studies incorporated testing of the same concepts on the same respondents on more than one occasion. This research investigates the importance of occasions as a source of error variance in estimates of the generalisability of concept test scores for both minor and major innovations within the context of Generalisability theory. The study collected concept evaluations of ten innovations from members of an online panel on three occasions, approximately a month apart. The results show that the three-way interaction among subjects, concepts and occasions is a substantial contributor to variation in concept testing of both major and minor innovations, with the contribution for major innovations even more substantial than for minor innovations. Moreover, failure to recognize occasions as an explicit source of variance in the generalisability analyses will lead managers to overestimate the generalisability of their decision studies. However, the impact of neglecting occasions varies by purpose of measurement and associated object of measurement. This research provides insight about how well concept testing can generalise over occasions. Concept test evaluations provided on an initial exposure are more favourable than will be received on any later occasions, and apparent differences in consumer evaluations of a particular concept in an initial test do not provide a generalisable basis for identifying which consumers will respond most favourably to it on a later occasion. For concept testing to be used for targeting or segmentation, more occasions will need to be sampled.

UR - http://commons.ln.edu.hk/sw_master/3782

U2 - 10.2501/S1470785309201314

DO - 10.2501/S1470785309201314

M3 - Journal Article (refereed)

VL - 52

SP - 363

EP - 382

JO - International Journal of Market Research

JF - International Journal of Market Research

SN - 1470-7853

IS - 3

ER -