Scale Construction and Development

December 10-15, 2018 at Nord university, Bodø

ECTS credits: 7,5
Level of course: Ph.D. course
Type of course: Elective for students in business or other behavioral and social  sciences. 
Course responsible persons: Trond Bliksvær and Tor Korneliussen, Nord University 
Language: English. 
Faculty: Professor Tenko Raykov, Michigan State University,
Costs: The course is free. 
Course evaluation: An online survey will be implemented asking for feedback on the course, including how it can be improved.

Course content 
Business, marketing, banking, economics as well as social and behavioral research frequently employs measuring instruments consisting of multiple components, such as questions, items, problems or tasks, which are administered to sampled respondents from studied populations of interest.  These instruments, often known alternatively as scales, questionnaires, surveys, self-reports, inventories, subscales, tests, testlets or test-batteries, aim to provide multiple converging pieces of information about underlying latent constructs (traits, latent variables, latent dimensions) which typically represent theoretical concepts of main relevance and concern in these and related sciences.

The development of high-quality scales is a multi-stage process that is typically referred to as scale construction and development (SCD).  This process can become in applications rather complicated, need not follow specific prescriptions – with such not being generally applicable – and may involve phases where the researcher needs to address key issues using what may be seen as an iterative process of improving initial or tentative versions of the instrument.  Major criteria for quality of multiple-component measuring instruments (scales, or psychometric scales) are reliability and validity, as well as the examination of the latent structure underlying putative or preliminary scale versions that is of special relevance in efforts aimed at quantifying and particularly enhancing the quality of an instrument under construction and development.

The goal of this course is to provide the course participants with thorough knowledge about and skills needed for the application of various approaches to SCD aimed at enhancing the quality of psychometric scales to be used with respondent populations of research interests.  As a main vehicle of SCD, the popular latent variable modeling (LVM) methodology is employed throughout the course.  At the software level, numerous uses are made of the increasingly widely circulated LVM software Mplus.  (Knowledge of the software is beneficial but not needed for the course attendees.  The package will be introduced, in the detail needed, during the course, and its applications for SCD purposes will be thoroughly discussed, as will the command files needed, ensuing results, and associated outputs.

The course “Scale Construction and Development" aims to introduce and thoroughly discuss statistical methods that are useful for any researcher involved in multi-component measuring instrument construction, development, and revision.  The course will also provide knowledge and skills needed in the process of evaluating existing psychometric scales, questionnaires, surveys, self-reports, or inventories when these are considered for application in a given empirical study.  The course covers classical linear factor analysis and its extensions for categorical items, classical test theory and models based on it that are widely used in the process of SCD, as well as sophisticated and more recent extensions of these frameworks to address complicated situations that business, marketing, banking, economics, behavioral, and social scientists are confronted with in empirical research.  Substantial emphasis is placed on the concepts of reliability, validity, and latent structure underlying a psychometric scale.  The statistical package Mplus is used throughout.  (The demo version of Mplus that is freely available suffices for essentially all examples used.) 

Learning outcomes: 
By taking this course, the student will acquire

-- knowledge
of a set of statistical methods that are useful in their research involving SCD;
of the relevance of routine examination of the latent structure of measuring instruments under consideration, as well as of their reliability and validity;
of statistical approaches to handle missing values and possible nesting (clustering, hierarchical) effects in the context of SCD;
of the bases of the frameworks of classical test theory and factor analysis that are essential for SCD, and how to use them for the aims of SCD;
Have a good understanding of the forefront of knowledge about scale construction and development, including current debates and state-of-the-art;
Have a understanding of the theoretical foundation and methodological challenges of scale construction and scale development;

-- skills
to examine the latent structure of an initial, putative, or tentative measuring instrument;
to interpret the associated statistical results and express these verbally; 
to enhance validity and reliability and where desirable and feasible obtain unidimensional subscales evaluating (possibly multiple) single latent constructs of theoretical and empirical importance in a given subject-matter domain;
to be able to construct and develop a scale related to their own Ph.D. work;

-- general competence
to assess the applicability of the statistical methods important to the student's research across various empirical settings related to SCD;
to understand the usefulness of the quantitative approach to research as relevant for SCD;
to become sufficiently familiar with the highly popular and widely circulated statistical package Mplus and its use for the aims of SCD;
have increased their ability to communicate (in writing and orally) problems, analyses, and results related to scale construction to colleagues, including contributing in academic debates in conferences.

Course prerequisites:  
Admitted to a PhD program or have the qualifications to be admitted to a PhD program. Some basic statistical knowledge is required, for example as obtained in an introductory statistics course, including in particular knowledge of basic statistics such as mean, variance, standard deviation, correlation, and linear regression.  Familiarity with Mplus is not a prerequisite. 

Course Outline (Main Topics Covered):

1. Factor analysis: A modeling basis of measuring scale construction and development (SCD): 
- Latent variables as theoretical concepts of relevance in empirical business, marketing, banking, behavioral and social research;
- The classical common factor model;
- The distinction between exploratory and confirmatory factor analysis, and its importance in SCD.

2. Classical test theory: A theoretically and practically useful framework for revising psychometric scales:
- The classical test theory (CTT) decomposition of observed score;
- Reliability of measuring instruments;
- Validity of psychometric scales.

3. An introduction to latent variable modeling and the highly popular LVM software Mplus:
- What latent variable modeling (LVM) is, and why it is of special relevance in SCD;
- The LVM software Mplus – A brief introduction to its syntax.

4. Measuring instrument development with (approximately) continuous scale components:
- Examining underlying structure, unidumensionality, and its importance in SCD;
- Assessing the possibility of developing subscales from a multidimensional measuring instrument.

5. Scale development with categorical items: 
- Latent structure examination;
- SCD with missing data, violations of missing at random, and nesting effects.

6. Construction of initial measuring instrument version – categorical and continuous items or components:
- Selection from an available item pool;
- Evaluation of inter-item and item-total (adjusted item-total) interrelationships.

7. Scale revision for enhancing psychometric quality.  Reliability and validity and how to enhance them:
- Coefficient alpha and what it is and what it is not;
- Revisions for reliability improvement;
- Validity improvement due to revision. 

8. Essential unidimensionality of multiple component measuring instruments:
- What essential unidimensionality is and how to quantify it;
- The bi-factor model and its relevance in examining essential unidimensionality of measuring instruments.

9. Exploratory analyses for general structure scales (multidimensional scales):
- Measuring multiple underlying constructs, and reliability of multidimensional scales;
- Second order measuring instruments and why they are important in empirical research.

10. Practical issues in scale construction and development: 
- What can go wrong in SCD and how to fix it;
- Validity, reliability, and finite mixtures.

11. Scale construction and development with data from nationally representative studies (large-scale/complex sampling design studies): 
- What are nationally representative studies and why they are becoming increasingly relevant in empirical research;
- Psychometrics with nationally representative studies.

12. Optimal shortening of psychometric scales:
- Why it is important to be in a position to shorten a scale;
- Optimal shortening based on the concept of maximal reliability.

13. Extensions, limitations, and conclusion:  
- What to watch out for when trying to generalize findings from SCD analyses and      modeling efforts;
- When we can trust SCD analyses and modeling;
- Finite mixture modeling as a main avenue for future SCD research;
- Conclusions.

Mode of delivery
Face-to-face lectures.

Organization and learning activities
This is an intensive course of one week with individual study required prior to and after the course. 

Exam and evaluation:
Participation in lectures, and the use and application in a course assignment of statistical methods covered in the course, written up as a paper.  Paper graded: pass / non pass.

Raykov, T., & Marcoulides, G. A. (2011).  Introduction to psychometric theory.  New York: Taylor & Francis.
In addition, copies of the below published research articles by the instructor and colleagues will be provided to the attendees of this course, along with the data files used throughout it.  Furthermore, thorough lecture notes containing all presented slides during the course will be made available to the course participants.

Readings list:
The following texts support this course:
Raykov, T., & Marcoulides, G. A. (2011).  Introduction to Psychometric Theory.  New York: Taylor & Francis.  (Textbook of this course).

McDonald, R. P. (1999).  Test theory. A unified approach.  Mahwah, NJ: Lawrence Erlbaum and Associates. 

DeVellis, R. F. (2015).  Scale construction and development.  Theory and applications.  Thousand Oaks, CA: Sage. 

Furr, R. M. (2017).  Psychometrics. An introduction.  Thousand Oaks, CA: Sage.

Price, L. R. (2014).  Psychometric Methods. Theory into practice.  New York: Guilford.

Raykov, T. (1997). Scale Reliability, Cronbach's Coefficient Alpha, and Violations of Essential Tau-Equivalence for Fixed Congeneric Components.  Multivariate Behavioral Research, 32, 329-354.

Raykov, T. (2008).  “Alpha if Item Deleted”: A Note on Loss of Criterion Validity in Scale Development If Maximizing Coefficient Alpha.  British Journal of Mathematical and Statistical Psychology, 61, 275-285.

Raykov, T. (2007).  Reliability If Deleted, Not “Alpha If Deleted”: Evaluation of Scale Reliability Following Component Deletion.  British Journal of Mathematical and Statistical Psychology, 60, 201-216.

Raykov, T. (2012).  Scale Construction and Development Using Structural Equation Modeling.  In R. Hoyle (Ed.), Handbook of Structural Equation Modeling (pp. 472-492).  New York: Guilford Press.

Raykov, T., Marcoulides, G. A. (2012).  Evaluation of Validity and Reliability of Hierarchical Scales.  Structural Equation Modeling, 19, 495-508.

Raykov, T., & Pohl, S. (2013).  On Studying Common Factor Variance in Multiple Component Measuring Instruments.  Educational and Psychological Measurement, 73, 191-209.

Raykov, T., & Pohl, S. (2013).  Essential Unidimensionality in Multiple Component Measuring Instruments: A Correlation Decomposition Approach.  Educational and Psychological Measurement, 73, 581-600.

Raykov, T., & Traynor, A. (2016).  Evaluation of Scale Reliability in Complex Sampling Designs.  Structural Equation Modeling, 23, 270-277.

Raykov, T., Rodenberg, C. N., & Narayanan, A. (2015).  Optimal Shortening of Psychometric Scales.  Structural Equation Modeling, 22, 227-235.

Raykov, T., & Marcoulides, G. A. (2016).  Scale Reliability Evaluation Under Multiple Assumption Violations.  Structural Equation Modeling, 23, 302-313.

Deadline: November 22