Whose crystal ball to choose? Individual differences in the generalizability of concept testing

Ling PENG, Adam FINN

Research output: Journal PublicationsJournal Article (refereed)Researchpeer-review

11 Citations (Scopus)

Abstract

The product development literature has identified several individual characteristics that could influence how subjects respond to new products in concept tests. Few of these characteristics have been thoroughly investigated. The purpose of this research is to examine whether a number of personality traits (1) do influence concept evaluation scores and (2) can be used to identify respondents who provide substantially higher-quality data in concept testing and whether the answers to these questions change for major versus minor innovations. The data quality of the concept testing data is defined using the generalizability theory, which provides a decision-specific G-coefficient. Higher quality means a G-coefficient closer to 1 for a particular managerial decision. A Web-based study to concept test 10 appliance innovations on multiple occasions was conducted among 105 panelists from the Institute for Online Consumer Studies (IOCS). During the concept testing, respondents' innovativeness, change-seeking tendency, and propensity to exert cognitive effort were also measured. The results showed that the respondent characteristics influence the mean evaluation of the concepts and the psychometric quality of the concept testing data: (1) there is a significant linear relationship between concept scores and all of the innovativeness scales and change-seeking measures; (2) the effect of innovativeness on concept testing outcomes is even more substantial for major innovations than for minor innovations; (3) the study provides evidence that the quality of concept testing data provided by respondents varies substantially with their innovativeness, whereas the differences are more modest when scaling just minor innovations; (5) there are also strong effects on data quality for the Need to Evaluate scale used to capture cognitive effort characteristics; and (6) there is little effect of segmenting on social desirability on data quality. Managerially, the current results indicate that a product manager wanting to concept test a pool of appliance concepts can benefit from screening for the respondents who will provide higher-quality concept testing data. For example, respondents who are high on domain-specific innovativeness provide the highest-quality concept testing data for both minor and major innovations. The effects of traits are stronger for major innovations, supporting the claim that subject selection is a more critical issue in concept testing of major innovations. Product managers can improve the quality of their concept testing data without an increase in cost by screening the subjects they use in concept testing.
Original languageEnglish
Pages (from-to)690-704
Number of pages15
JournalJournal of Product Innovation Management
Volume27
Issue number5
DOIs
Publication statusPublished - 1 Sep 2010

Fingerprint

Crystals
Innovation
Testing
Screening
Managers
Individual differences
Generalizability
Product development
Innovativeness
Costs
Data quality

Cite this

@article{4f32f4f5dca9485f9309741604cc0285,
title = "Whose crystal ball to choose? Individual differences in the generalizability of concept testing",
abstract = "The product development literature has identified several individual characteristics that could influence how subjects respond to new products in concept tests. Few of these characteristics have been thoroughly investigated. The purpose of this research is to examine whether a number of personality traits (1) do influence concept evaluation scores and (2) can be used to identify respondents who provide substantially higher-quality data in concept testing and whether the answers to these questions change for major versus minor innovations. The data quality of the concept testing data is defined using the generalizability theory, which provides a decision-specific G-coefficient. Higher quality means a G-coefficient closer to 1 for a particular managerial decision. A Web-based study to concept test 10 appliance innovations on multiple occasions was conducted among 105 panelists from the Institute for Online Consumer Studies (IOCS). During the concept testing, respondents' innovativeness, change-seeking tendency, and propensity to exert cognitive effort were also measured. The results showed that the respondent characteristics influence the mean evaluation of the concepts and the psychometric quality of the concept testing data: (1) there is a significant linear relationship between concept scores and all of the innovativeness scales and change-seeking measures; (2) the effect of innovativeness on concept testing outcomes is even more substantial for major innovations than for minor innovations; (3) the study provides evidence that the quality of concept testing data provided by respondents varies substantially with their innovativeness, whereas the differences are more modest when scaling just minor innovations; (5) there are also strong effects on data quality for the Need to Evaluate scale used to capture cognitive effort characteristics; and (6) there is little effect of segmenting on social desirability on data quality. Managerially, the current results indicate that a product manager wanting to concept test a pool of appliance concepts can benefit from screening for the respondents who will provide higher-quality concept testing data. For example, respondents who are high on domain-specific innovativeness provide the highest-quality concept testing data for both minor and major innovations. The effects of traits are stronger for major innovations, supporting the claim that subject selection is a more critical issue in concept testing of major innovations. Product managers can improve the quality of their concept testing data without an increase in cost by screening the subjects they use in concept testing.",
author = "Ling PENG and Adam FINN",
year = "2010",
month = "9",
day = "1",
doi = "10.1111/j.1540-5885.2010.00745.x",
language = "English",
volume = "27",
pages = "690--704",
journal = "Journal of Product Innovation Management",
issn = "0737-6782",
publisher = "Wiley-Blackwell Publishing Ltd",
number = "5",

}

Whose crystal ball to choose? Individual differences in the generalizability of concept testing. / PENG, Ling; FINN, Adam.

In: Journal of Product Innovation Management, Vol. 27, No. 5, 01.09.2010, p. 690-704.

Research output: Journal PublicationsJournal Article (refereed)Researchpeer-review

TY - JOUR

T1 - Whose crystal ball to choose? Individual differences in the generalizability of concept testing

AU - PENG, Ling

AU - FINN, Adam

PY - 2010/9/1

Y1 - 2010/9/1

N2 - The product development literature has identified several individual characteristics that could influence how subjects respond to new products in concept tests. Few of these characteristics have been thoroughly investigated. The purpose of this research is to examine whether a number of personality traits (1) do influence concept evaluation scores and (2) can be used to identify respondents who provide substantially higher-quality data in concept testing and whether the answers to these questions change for major versus minor innovations. The data quality of the concept testing data is defined using the generalizability theory, which provides a decision-specific G-coefficient. Higher quality means a G-coefficient closer to 1 for a particular managerial decision. A Web-based study to concept test 10 appliance innovations on multiple occasions was conducted among 105 panelists from the Institute for Online Consumer Studies (IOCS). During the concept testing, respondents' innovativeness, change-seeking tendency, and propensity to exert cognitive effort were also measured. The results showed that the respondent characteristics influence the mean evaluation of the concepts and the psychometric quality of the concept testing data: (1) there is a significant linear relationship between concept scores and all of the innovativeness scales and change-seeking measures; (2) the effect of innovativeness on concept testing outcomes is even more substantial for major innovations than for minor innovations; (3) the study provides evidence that the quality of concept testing data provided by respondents varies substantially with their innovativeness, whereas the differences are more modest when scaling just minor innovations; (5) there are also strong effects on data quality for the Need to Evaluate scale used to capture cognitive effort characteristics; and (6) there is little effect of segmenting on social desirability on data quality. Managerially, the current results indicate that a product manager wanting to concept test a pool of appliance concepts can benefit from screening for the respondents who will provide higher-quality concept testing data. For example, respondents who are high on domain-specific innovativeness provide the highest-quality concept testing data for both minor and major innovations. The effects of traits are stronger for major innovations, supporting the claim that subject selection is a more critical issue in concept testing of major innovations. Product managers can improve the quality of their concept testing data without an increase in cost by screening the subjects they use in concept testing.

AB - The product development literature has identified several individual characteristics that could influence how subjects respond to new products in concept tests. Few of these characteristics have been thoroughly investigated. The purpose of this research is to examine whether a number of personality traits (1) do influence concept evaluation scores and (2) can be used to identify respondents who provide substantially higher-quality data in concept testing and whether the answers to these questions change for major versus minor innovations. The data quality of the concept testing data is defined using the generalizability theory, which provides a decision-specific G-coefficient. Higher quality means a G-coefficient closer to 1 for a particular managerial decision. A Web-based study to concept test 10 appliance innovations on multiple occasions was conducted among 105 panelists from the Institute for Online Consumer Studies (IOCS). During the concept testing, respondents' innovativeness, change-seeking tendency, and propensity to exert cognitive effort were also measured. The results showed that the respondent characteristics influence the mean evaluation of the concepts and the psychometric quality of the concept testing data: (1) there is a significant linear relationship between concept scores and all of the innovativeness scales and change-seeking measures; (2) the effect of innovativeness on concept testing outcomes is even more substantial for major innovations than for minor innovations; (3) the study provides evidence that the quality of concept testing data provided by respondents varies substantially with their innovativeness, whereas the differences are more modest when scaling just minor innovations; (5) there are also strong effects on data quality for the Need to Evaluate scale used to capture cognitive effort characteristics; and (6) there is little effect of segmenting on social desirability on data quality. Managerially, the current results indicate that a product manager wanting to concept test a pool of appliance concepts can benefit from screening for the respondents who will provide higher-quality concept testing data. For example, respondents who are high on domain-specific innovativeness provide the highest-quality concept testing data for both minor and major innovations. The effects of traits are stronger for major innovations, supporting the claim that subject selection is a more critical issue in concept testing of major innovations. Product managers can improve the quality of their concept testing data without an increase in cost by screening the subjects they use in concept testing.

UR - http://commons.ln.edu.hk/sw_master/619

U2 - 10.1111/j.1540-5885.2010.00745.x

DO - 10.1111/j.1540-5885.2010.00745.x

M3 - Journal Article (refereed)

VL - 27

SP - 690

EP - 704

JO - Journal of Product Innovation Management

JF - Journal of Product Innovation Management

SN - 0737-6782

IS - 5

ER -