Validation of GAPtest

In 2008, an independent researcher was contracted to evaluate the Global Awareness Profile using ANOVA and regression analysis. The primary goal of the analysis was to evaluate the reliability and validity of the test. A secondary goal of the project was to provide comparative analysis of blind profiles with scores, exporting profile comparison to graphs for reporting international, national and local mean scores in basic areas. Based on the data from 1,136 participants from 13 institutions, the analysis provided conclusive findings and areas for further development and research.

Specifically, analysis determined that The Global Awareness profile is 96.9% reliable, and a valid instrument for assessing global awareness.  The analysis also found that there are a number of predictors for higher scores of global awareness including owning a passport, international travel and speaking more than one language. Higher levels of education also impacted scores. Participants think they know more than they do by predicting higher scores than they receive.  A particularly interesting analysis, if not controversial, is that urban African-Americans score consistently lower on the GAPtest than Caucasian counterparts in early collegiate education. This difference changes with graduate studies. Further study is needed but may point to the global experience available and quality of education provided in urban contexts.   A fairly lengthy summary of the findings follows below.  The full report is available in PDF upon request.  Please include in any request the intended use of the report. Any and all information regarding individuals and schools is confidential and will not be released in any public report.

A Brief History and Development

In 1998 Dr. Corbitt published the Global Awareness Profile through Intercultural Press as a self-scoring printed test after several years of development in Dr. Corbitt’s classes at Eastern University and his training in the civic environment.  In 2005, the copyright and publication were returned to Dr. Corbitt.   Graduate students from St. Joseph’s University assisted in evaluating the test and recommending changes primarily to questions.  An online version was created in 2007 and enhanced by a database programmer in 2008.  The test was updated and placed online in 2007/2008.   Since then over 2500 students have taken the test from 15 schools and universities.  In 2008, an independent evaluator was contracted to evaluate the reliability and validity of the GAPtest.  This had not been done since 1998.  Along with the analysis Dr. Corbitt wanted to know what improvements could be made with the test, as well any unique findings about critical success factors of test-takers.  A second qualitative phase of the test is in progress through interviews with those who have scored above 100 (out of 126 and the upper quartile) on the GAPtest.  Following is a summary of the GAP quantitative analysis.

Summary of the Report

This report describes the results of the analyses performed on the data from the Global Awareness Profile Test (GAP). Several analyses were performed, including Correlations, Regression, One-Way Analysis of Variances (ANOVAs), and Factorial ANOVAs. In addition to these analyses, the Reliability and Validity of the GAP was assessed as well as the usability of the online test. Limitations of the research were discussed and specific recommendations for improvement were proposed. Potential uses for the actual GAP test were also delineated. Finally, future possible research as well as written reports using the GAP test was discussed.

Descriptive Statistics

  • 1,136 participants were sampled from schools in the United States. Some participants were students while other participants were faculty.
  • Test-takers came from 13 Institutions. The Institutions were public high schools, private high schools, as well as post secondary institutions.
  • Participants were born in 51 different countries. The vast majority of the participants were predictably from the United States. In fact, 81.6% of the participants were born in the United States.
  • The age range for the participants was from 13 to 66. However, the largest majority of participants were 18 years old. They made up 18.1% of the participant pool.
  • 76% of the participants spoke only one language fluently.
  • For purposes of simplicity, the numerous categories of ethnic groups were put together into large groups of similar ethnic backgrounds. The nine main categories were: Caucasian, 58%, Not Specified, and 17%, African-American, 8%, Asian, 5%, North American, 3%, Mixed Ethnic Background, 3%, European – Internationals, 2%, Hispanic. 2%, Other, 2%

Summary of Findings for Reliability and Validity

Cronbach’s Alpha was calculated using the test data from the Global Awareness Profile. It was determined to be .969 or 96.9% reliable. Therefore, the GAP test produces reliable results 96.9% of the time.

The GAPtest has content validity, which is concerned with the degree to which a measure of a variable adequately assesses all aspects of that variable. The test must demonstrate that the questions are relevant and representative of the variable being measured, which the GAP test possesses in abundance.

The validity analysis of the GAP test shows that the GAP test is valid. It includes all relevant and representative aspects of the construct of global awareness, but it does not include aspects that it should not be included. Therefore, the GAP test is a valid measure of global awareness.

Mean Score and Participant Predictions

The mean score for the GAP test is 70.30. This mean can be used as a standard to compare various groups such as different classes, schools, or institutions. This is based on data from 2007/2008 participants.

All participants attempted to predict the score they would receive on the GAP test. In Table 2 are the descriptive statistics for the Predicted Score on the GAP variable.

On average, participants guessed that they would achieve a score of 75 out of 126.  In other words, students felt that they would score better than they actually did.

Regression Equation

The purpose of a regression equation was to determine factors that might predict, with certain accuracy, scores on a dependent variable. In this instance researchers were looking for the factors that could predict high scores on the GAPtest.

The regression was run twice in order to attain an equation in which all predictors significantly predicted high scores on the GAP.  The results clearly show that there are eight significant predictors of success on the GAP test. These predictors are:

1.    Highest Level of Education,
2.    Ownership of a Passport,
3.    Gender,
4.    Age,
5.    Country of Birth,
6.    Number of Languages Spoken Fluently,
7.    Institution, and
8.    Ethnicity.

The results from the ANOVA analysis show that although the predictors may only be able to account for 19.6% of the variance in GAP scores, they are still better predictors of GAP scores than chance alone.

Predictors of Higher Scores

Gender

A significant difference in scores on the GAP test between the genders was found. Male participants performed significantly better on the GAP test than their female counterparts.  Even though this finding seems robust since the group sizes were equal, a test for the homogeneity of variance was conducted to ensure the results were reliable and valid. Upon checking the test for the homogeneity of variance a significant result was found. Therefore, the variances were heterogeneous and it may not be accurate to conclude that the differences between the scores in each group were due to gender alone. This is a finding that should be further explored. A significant difference in scores on the GAP test between the genders was found. Male participants performed significantly better on the GAP test than their female counterparts.

Ownership of a Passport

It was believed that by simply owning a passport, one would be more likely to travel and be aware of international affairs. Therefore, a one-way ANOVA was performed to determine if there would be a significant difference in scores on the GAP test between those who owned a passport and those who did not own a passport. From the descriptive statistics it appears that participants who owned a passport performed better than participants who did not own a passport. From this analysis it is clear that there is a significant difference between those who own a passport and those who do not own a passport. There was concern that the unequal sample sizes may result in heterogeneity of variance between the groups. However, a test for the homogeneity of variance was conducted and produced a non-significant result. Therefore, the variances between the groups were homogeneous. Thus, researchers are able to conclude that participants who owned a passport performed significantly better on the GAP test than those who do not own a passport.

National or International Participant

The main purpose of comparing National and International participants was to dispel the myth that Americans know less about global affairs than their international counterparts. If there was a significant difference between the groups, then the group with the higher mean score knew more about global issues than the other group. If there was not a significant difference between the groups, then it could be concluded that the assertion that Americans know less about global issues than the rest of the world is false.  The results suggest that there is not a significant difference between the groups. Therefore, it could be concluded that Americans know as much about global issues as the rest of the world. However, there were issues with unequal group sizes. Since there were considerably more participants in the National group there could be issues with the homogeneity of variance.  A test for the homogeneity of variance was conducted on this analysis, which turned out to be non-significant. Therefore, the variances are homogeneous. Due to this finding, researchers are able to claim that Americans truly do know as much about global issues as the rest of the world.

Number of Citizenships Held

Researchers wanted to determine whether or not there was a significant difference of scores between groups of varying number of citizenships held. It was believed that those who held more citizenships would perform significantly better on the GAP test than participants who did not possess as many citizenships. It should be noted here that the group with three citizenships included participants who had more than three citizenships. The possible reasons for this finding should be discussed at length in a formal research paper. Nevertheless, one possible reason could be that those who have more than one citizenship may either be more interested in international issues because they lived in another country or they may still have family in their second country and are concerned about what is going on in those regions.

As with many other ANOVA analyses performed from this data, the analysis on the Number of Citizenships Held contained unequal groups. A test for the homogeneity of variance was conducted and found to be non-significant. Therefore, the results of the data may be generalizable and researchers are able to conclude that those who possess more than one citizenship can achieve higher scores on the GAP test.

Number of Countries Visited

It would seem that when one visits more countries they would achieve a higher score on the GAP simply because of their exposure or possibly inherent interest in the global community. Therefore, an ANOVA was performed in order to determine if this hypothesis was correct. Before going into further detail, the groups were created by combining participants who had visited: (1) Zero Countries, (2) One Country, (3) Two Countries, and (4) Three or More Countries. It is obvious that there are differences in the means for each groups and that the group with the most number of countries visited had a much higher mean. The results of the ANOVA analysis show that there is a significant difference between the groups. The groups with the higher mean were the ones where participants visited two or more countries. Therefore, it appears as though the more countries one visits, the higher a score one would achieve on the GAP test.

Number of Languages Spoken Fluently

Similar to the analysis performed in the previous section, the analysis on the Number of Languages Spoken Fluently was expected to find that those participants who spoke a higher number of languages would also receive significantly higher scores on the GAP test. Participants were grouped into three groups, which were those who spoke: (1) One Language Fluently, (2) Two Languages Fluently, and (3) Three or More Languages Fluently.

The results show that there was a significant difference between the groups for the number of languages spoken fluently. The group with the highest mean was the one in which participants fluently spoke three or more languages. Thus, those who spoke more languages fluently achieved significantly higher scores on the GAP test. Like with many of the ANOVA analyses performed from this data, there was an issue with the sample sizes. An analysis of the homogeneity of variance produced non-significant results. Therefore, researchers are able to conclude that there was a significant difference between the groups for the Number of Languages Spoken Fluently.

Home Country

Although there was data from participants who were born in many different countries around the world, only data from three countries was used in the current analysis because there was insufficient data from the other countries. An initial analysis included data from seven countries, which were: (1) Canada, (2) China, (3) Germany, (4) Korea, (5) Taiwan, (6) United Kingdom, and (7) United States. However, the results of the analysis that included all seven of these birth countries indicated there was significant heterogeneity of variance. Therefore, the analysis was performed with four of these countries removed. The resulting analysis included only Canada, Korea, and the United States. The data for Korea included both the Republic of Korea as well as the Democratic Republic of Korea. The data suggests that there are mean differences between participants born in different countries. However, at this point it in unclear whether or not these differences are significant. The results of the ANOVA show that there was not a significant difference in scores between the groups based on birth country. It does not appear that the country one is born in can affect the score one receives on the GAP test. There are, of course, issues with unequal sample sizes. However, a test for the homogeneity of variance found that there were homogeneous variances and therefore the results appear to be valid. Nevertheless, future research should strive to include more data from countries around the world.

The Differences of Scores on the GAP Test Between Different Ethnic Groups and Education Levels

[NOTE: A particularly interesting analysis, if not controversial, is that urban African-Americans score lower on the GAPtest than Caucasian counterparts regardless of education. There are confounding aspects to this analysis that are not possible in the data from the profile.  The primary confounding variables include, but are not limited to: cultural/environmental factors, type of education, social class, access to the global experience and motivation. So while the analysis demonstrates a difference, it cannot answer the question of why. Further study is needed and anticipated.  JNC]

The purpose of the analyses in this section was to determine if differences found between two ethnic groups at a lower education level could be eliminated at a higher education level. Basically, could education close the gap in scores between the ethnic groups? In addition to this question, a factorial ANOVA was performed in order to assess whether there was an interaction between ethnicity and education level.

The groups for these analyses were:

1.    African-American and Education Level from High School to First Year College,
2.    Anglo-American and Education Level from High School to First Year College,
3.    African-American and Education Level from Second Year College to Doctorate, and
4.    Anglo-American and Education Level from Second Year College to Doctorate.

These groups were created using self-reported data from the GAP data set. Participants who identified themselves as African-American, Black, or Afro were placed into the African-American group. Conversely, participants who identified themselves as Anglo-American, White, Anglo, as well as national participants who used their European ethnicities were included in the Anglo-American category. The inclusions for the educational groups are clearly described in the group titles.

The initial analysis performed using these groups was a one-way ANOVA between the different ethnic groups at the lower education level. There appears to be a significant difference in scores between the groups. Table 12 summarizes the results from the ANOVA performed for this analysis.

The results of the ANOVA show that there were significant differences between the groups. Specifically, Anglo-Americans at lower education levels performed significantly better on the GAP than African-Americans at lower education levels. A test for the homogeneity of variance produced non-significant results. Thus, this analysis does not violate a key assumption of ANOVAs. This result was not surprising, but what is of real importance is whether or not this difference between the ethnic groups can be eliminated at higher education levels.
The results of the ANOVA show that there are significant differences between the groups. Therefore, at a higher education level, the differences between the ethnic groups are not eliminated. However, an analysis of the homogeneity of variance revealed a significant result, which means that there was heterogeneity. Furthermore, the groups were highly unequal. Therefore, it is likely that these results are not a true reflection of what is going on.

Nevertheless, a factorial ANOVA was performed in order to determine if ethnicity and education level would have a significant main effect and also if there would be a significant interaction between these two variables. There are differences between the ethnic groups as well as the different education levels, but the questions remains as to whether or not these differences are significant.

The results show that there was a significant main effect for ethnicity as well as education level, but there was not a significant interaction between the two variables. This is in stark contrast to the similar analysis performed  (not shown in this abridged report) where the factorial ANOVA found only a significant main effect for ethnicity and a significant interaction between ethnicity and education level.

Usability Assessment of Online Test

Benefits of the Online Test

By having the GAP test online, researchers have resolved many logistical issues that would arise if the test was in paper format. Most importantly, the test can be taken anywhere in the world at any time with minimal cost. Furthermore, by having an online test, researchers are able to ensure that the test is safe and not able to be copied by outside sources. For example, if the test was mailed to institutions, the test could be photocopied by someone and used for their own purposes.

In addition, researchers are able to gather the data in an efficient manner and data entry is eliminated since the online test data can be easily gathered and transferred to programs for statistical analysis. Essentially, by having the test online researchers are better able to get the test to places around the globe and statistically analyze the data more efficiently.

Drawbacks of the Online Test

[NOTE: these issues have been corrected in a recent update of the GAP online]

While the online test is impressive, there are a few areas of concern that should be examined closely. Firstly, when one is submitting an answer to a question they must click “Submit” before clicking “Next” in order for their answer to be recorded. This is an issue of concern because it could result in answers not being recorded if a test-taker fails to click on “Submit” before going to the next question. The implication of the issue is that it would cause an answer to not be recorded and the actual score of the participant would be incorrect. This would affect the reliability of the test since it is not reliably recording the answers of each participant. Furthermore, it would affect the validity of the test since it is a cause for scores to be lower, which would result in unreliable and invalid results.

Additionally, in the demographic portion of the test participants have the ability to select the number of countries they visited by holding the “Control” key down and using the arrows to select multiple countries. This is an issue for a number of reasons. Primarily, it is a time consuming task that many participants may be reluctant to partake in simply because of impatience. Moreover, if participants are taking the test in a classroom setting, there are more than likely time constraints, which may cause participants to not accurately disclose the number of countries visited.

Furthermore, when the author of this report attempted to select the countries she had visited the program failed to register her selections when she clicked “Next” to move on to the next screen. It took a lot of time to select all of the countries and caused the author to not re-select the countries simply because it took so long in the first place. If there are other participants who encountered this issue, they may have also declined to re-enter their selections, which would cause the data collected to be an inaccurate reflection of the true number of countries some participants have visited. This would also affect the results of some analyses as well as the interpretation of the results. It would be prudent to remedy this issue to ensure accurate data is collected.

Validity Assessment of the GAP Test

Validity is the degree to which a measure accurately assesses the construct or trait it claims to measure. A valid measure assesses all components of the construct and only measures that construct. The purpose of the GAP test is to determine the true score of an individual’s knowledge of global issues. If the GAP test is truly valid, it will only measure knowledge of global issues and nothing else.

Before assessing the measurement validity, the dependent variable, or what the researchers want to measure, must first be defined. For the GAP, researchers want to measure participants’ awareness or knowledge of global issues, issues that range from politics to economics as well as geographic issues. Global awareness is the degree to which an individual has knowledge of every facet of societal function in all areas of the world. In order to assess global awareness, researchers divided the construct into two main categories, Geography and Context, which both contain seven subcategories. The subcategories for geography are:

(1) Asia,
(2) Africa,
(3) North America,
(4) South America,
(5) Middle East,
(6) Europe, and
(7) Global.

Additionally, the subcategories for context are:

(1) Environment,
(2) Culture,
(3) Politics,
(4) Geography,
(5) Religion,
(6) Socio-Economic, and
(7) Global.

Each question within the test uses two subcategories from each main category. For example a question may ask participants about the Culture in Africa, which clearly uses one subcategory from each of the main categories.

The categories and subcategories listed about are clearly related to global awareness. Each geographic location as well as each subcategory of context pertains directly to a facet of societal function. The only question would be if there are additional categories or subcategories that could be added to the current ones to ensure that the test accurately and adequately assesses global awareness. An analysis of this concern reveals that the context category covers all relevant and representative aspects of modern civilization. An addition to the subcategories of context would be redundant and may also cause there to be multicollinearity, a condition in which two or more variables are highly related to each other. When examining the subcategories of geography, it is also clear that all geographical regions are included in the test. Thus, both main categories include all relevant as well as representative aspects of global awareness.

Furthermore, an assessment of the specific questions on the GAP test found that the questions were relevant and representative of global issues. There are 126 questions on the test each of which directly relate to one of the fourteen categories available. Specifically, each of the geographical subcategories includes one of the context subcategories within it. Thus, there are eighteen questions that combine one subcategory from Context and one subcategory from Geography. An example would be a question that uses the geographic subcategory of Asia with the context subcategory of Environment:

“The seasonal wind that typically brings heavy rainfall to South Asia is known as,”

This question clearly uses the subcategory of Asia as well as Environment. Therefore, the test has content validity, which is concerned with the degree to which a measure of a variable adequately assesses all aspects of that variable. The test must demonstrate that the questions are relevant and representative of the variable being measured, which the GAP test possesses in abundance.

The aforementioned details also illustrate how the GAP test has convergent validity, which is the extent to which evidence comes together to show that the test is actually testing what it is designed to assess. In addition to having convergent validity, an analysis of whether or not the test does not measure something it is not supposed to assess was conducted. This type of validity is known as discriminant validity. Researchers would want to be sure that the GAP test does not test for constructs or knowledge other than global awareness. Global awareness itself is a multifaceted construct that includes many categories and subcategories listed above. Each category adds to the validity of the test because it is vital to the content validity, which ensures that all relevant and representative aspects of the construct are included in the test. However, if any of these categories were eliminated, it would not add to the validity of the test. Rather, it would lower the validity by not including all facets of global awareness.

The validity analysis of the GAP test shows that the GAP test is valid. It includes all relevant and representative aspects of the construct of global awareness, but it does not include aspects that it should not be included. Therefore, the GAP test is a valid measure of global awareness.

Limitations and Recommendations for the GAP Test

There are several limitations of the GAP test as well as the research itself.
First, the participants used in the study may not be representative of the general population. Another limitation of the research was the fact that there were zero scores that had to be eliminated. Furthermore, a secondary issue with the data was the fact that some variables had to be grouped in order to conduct statistical analyses that were meaningful. Another issue is that there are a multitude of instances where participants did not disclose all demographic information. Additionally, there were problems that arose from possible negative participants.

Recommendations

There are several key recommendations to improve this study. These are to:

1.    Include additional participants from higher levels of education, countries around the world, and diverse ethnic groups.
2.    Include the socio-economic level of the participants as well as the educational level and profession of the parents for the participants in the demographic section of the test.
3.    Remove the demographic variable of Number of Citizenships in Possession.
4.    Remove the need to click “Submit” before clicking “Next” for the online test.
5.    Develop a list of ethnic groups on the demographic section of the test from which participants must make a selection in order to continue the test.
6.    Require all demographic data to be revealed before the test can be continued.

Potential Uses for the Gap Test

An important question is what uses the GAP test has both in research and in the practical setting. The potential uses for the test include using it as a(n):

1.    Diagnostic tool for knowledge about Global Awareness in the general population.
2.    Indicator of the amount of knowledge gained in a class on International Issues.
3.    Measure that additional tests on Global Awareness can be compared to since its reliability is so high.
4.    Test employers can use to assess their employees who deal with International accounts.
5.    Argument for increasing funding towards social and educational programs aiming to improve global awareness of Americans.
6.    By using the GAP test as a diagnostic tool for the general population, researchers will be able to compare American participants to participants from other countries. This comparison may finally enable the researchers to dispel the stereotype that Americans know less about global issues.
7.    The GAP test can be used as a teaching aid for instructors who wish to determine the amount of knowledge students’ gained over the course of a semester. Also, future researchers who want to develop a measure of global awareness may use the GAP test to assess the reliability and validity of the measure they are developing. Or they may choose to simply use the GAP test as their measure of global awareness. Thus, the test can be sold to researchers studying this area.
8.    The results of the research could possibly be used to generate interest in the area of global awareness.
9.    The results could point to large discrepancies between certain groups in society.
10.    Moreover, the results could be used as an argument for education reforms that aim to improve global awareness in American citizens.

Conclusion

The GAP test has proven to be a useful measure for analyzing relationships between various segments of the population. With the inclusion of additional participants, the results from research using the GAP test have potentially powerful implications. These implications can be used in the future to create many positive changes.

Leave a Reply

Your email address will not be published. Required fields are marked *

*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>