Executive Summary for Reliability Study of the MAPP™ Career Test (Motivational Appraisal of Personal Potential)

Test-retest reliability of the Motivational Appraisal of Personal Potential (MAPP) instrument was established by a scientific study completed in July, 1997. While .50 is an acceptable reliability coefficient, MAPP exceeds standards with larger numbers showing consistency over time. The median test-retest correlation for the worker trait outcomes was .95, and for job ratings was .90.

These high correlations for 72 outcomes in nine categories were based on the weighted outcomes that reflected individual characteristics. Worker trait outcomes and job ratings make up the information that workers, employers, and career consultants use when working with MAPP in the world of work. The nine categories of worker trait outcomes and job ratings are: aptitude for the job; capacity for mathematical applications; capacity for usage of language; interest in job contents; relationship to data; relationship to people; relationship to things; relationship to reasoning; and temperament for the job. The median test-retest correlation for item responses was .71. This high correlation was based on people tending to give the same answers to 213 choices on 71 questions each time the test was taken. These responses are the basis for all the interpretations given to workers and employers.

The study used a sample of 32 people who were employed full time. They took the MAPP test twice, with an interval of approximately 9 months between tests (279.5 median number of days, ranging from 152 to 496 days). The gender composition of the sample was 12 women (37.5%) and 20 men (62.5%). The overall median age was 49, with ages ranging from 26 to 67 years. Those in the study sample had diverse occupations representing a full range of professional work. Evidence for the stability of MAPP is based on the fact that the time interval between tests contained major life changes for 11 (34.38%) of the participants in the study.

Reliability Study of the MAPP™ Career Test

A research study to establish test-restest reliability of the MAPP instrument was completed in July, 1997. This type of reliability was chosen for the study because an important feature of MAPP is that it be stable over time. Stability over time allows the test taker to use the results confidently for pursuing vocational choices as well as developing personal characteristics that would upgrade job qualifications. For employers, stability over time provides confidence that the test results can be used to carefully select candidates for job positions after the test is on file. Moreover, employers can use the test results with certainty for employee development in the workplace over time after the test has been taken.

Correlations between First and Second Tests

While .50 is an acceptable reliability coefficient for true score variability and measurement error, the MAPP instrument exceeds standards with larger numbers as explained below, thus showing great consistency over time.

Reliability of MAPP was established in three different ways using the test-retest method. First, the worker trait outcomes based on the responses to the test items were compared. Second, the ratings for jobs based on the responses to the test items were compared. Third, answers to the test items were used to determine how people answered the questions from one time to the next. Each of these ways of establishing reliability is discussed in the following paragraphs.

The median correlation for the worker trait outcomes based on the first and second responses to MAPP items was .95. The individual correlations for those included in the sample ranged from .66 to .99. The calculations were based on 72 outcomes in nine categories. The interpretation of this correlation about the differences in worker trait outcomes is that 95% reflects true variability, while only 5% of the differences can be attributed to measurement error. A perfect reliability would be a coefficient of 1.0, showing 100% true score consistency, and 0% measurement error. Total lack of reliability would be a coefficient of .0, showing 0% true score consistency, and 100% measurement error.

The median correlation for the job ratings based on the first and second responses to MAPP items was .90. The individual correlations for those included in the sample ranged from .57 to .98. The calculations were based on 72 ratings in nine categories. The interpretation of this correlation is the same as for the first correlation. In other words, 90% reflects true variability in the differences in job ratings, while only 10% is due to measurement error.

The median correlation between first and second responses to MAPP items was .71. The individual correlations for the 32 people in the sample ranged from .52 to .92. The calculations were based on responses to MAPP's 213 choices for 71 items. The interpretation of the correlation is the same as for the other two. In other words, 71% reflects true score variability in the differences in responses, while only 29% is due to measurement error.

The understanding of the differences among correlations is based on the structure of MAPP. The first and second correlations are quite similar because they are both referring to the outcomes of MAPP and reflect the consistency of the test to describe individuals. The first correlation (.95) is based on the worker trait outcomes that weights the responses to the test items for 72 different trait descriptions. The values of the worker trait outcomes can vary from 0 to 100. The second correlation (.90) is based on job ratings that directly correspond to the worker trait outcomes, but the ratings only have possible values ranging from 1 through 5. Thus, the smaller range of possible values accounts for the reduction in the correlation from .95 to .90.

The third correlation (.71) is based on item responses and reflects the consistency of the person taking the test to answer the same way at both times when taking the test. The other two correlations are higher because they refer to the interpretation of item responses through a weighting process that reflects characteristics of the individuals taking the test. This third correlation pertains to repeating the same responses to the 213 choices for the 71 items each time the test was taken.

All three types of information are valuable for the research study to establish reliability. However, the first two correlations are especially pertinent, because the worker trait outcomes and job ratings make up the information that workers, employers, and career consultants use when working with MAPP in the world of work. The nine categories of worker trait outcomes and job ratings are: aptitude for the job; capacity for mathematical applications; capacity for usage of language; interest in job contents; relationship to data; relationship to people; relationship to things; relationship to reasoning; and temperament for the job. The third correlation is also critical because the responses that people make to individual items form the basis for all of the interpretations of the test.

Sample for the Test-Retest Study

The study used a sample of 32 people who were employed full time. They took the MAPP test twice, with an interval of approximately 9 months between tests (279.5 median number of days, ranging from 152 to 496 days). The gender composition of the sample was 12 women (37.5%) and 20 men (62.5%). The overall median age was 49, with ages ranging from 26 to 67 years. This range indicates that the ages typically found in workers has been represented in the sample. In addition, this broad range was found for both men (from 26 to 67 years) and women (from 26 to 54). Table 1 shows the data by individual person for the study of reliability.

Those in the study sample had diverse occupations representing a full range of professional work. They were selected because most of them were involved in some way with the test developers, as employees, distributors, employees of customers, or acquaintances of employees. The occupations represented by those taking the tests included; account executive; attorney;career consultant; company president; consultant; corporate consultant; director of training; executive coach; finance and insurance manager; finance lease manager; finance manager; independent sales agent; investment broker; marketing manager; multi-media developer for television; network administrator; office administrator; personnel consultant; private investor; promotional consultant; sales agent; self-employed; sales consultant; seller of insurance; student; teacher; vice-president of professional services; and vice-president of sales and marketing.

The reliability of MAPP remains high even though major changes happen in a person's life. Evidence for the stability of MAPP is based on the fact that the time interval between tests contained major life changes for 11 (34.38%) of the participants in the study. Changes that those people experienced included changing types of work; changing locations of work; changing from searching for jobs to working at them; and broken relationships. However, these life changes were not related to the strength of the correlations between the first and second times that the tests were taken. Rather, changes in life experiences were found as often for those with higher correlations between test administrations as for those with the lower correlations.

Continuing Research about the Reliability of MAPP™ Test

Research on MAPP is continuous to maintain high levels of reliability. Other areas of outcomes to be studied are vocational traits, personal traits, and learning styles. Within vocational traits are 19 areas with 111 outcomes. Personal traits are categorized into nine areas with 109 outcomes. Learning styles have six areas with 51 outcomes.

Studies will be done to look at reliability within specific occupations to make sure that all kinds of workers respond to items consistently over time, with consistent outcomes in worker traits, job ratings, vocational traits, personal traits, and learning styles. Not only the types of workers, but also the types of work experience and the length of time on the job will be considered in terms of reliability for test items and interpretations. Research on reliability will also consider the stability of responses within different cultures and geographic locations.

View Data Study Table