Research Committee

Dear WESTOP Members,

My name is Jesse Moya and I am the research chair of WESTOP’s service council. At the request of the WESTOP Board and with the help of Dr. Connie and Dr. Harley Baker, I have created this introductory outline of information about research and evaluation as it pertains to WESTOP and TRIO programs (specifically Upward Bound programs). This knowledge of research can be used to: 1) better understand and/or challenge proposed research methods; and 2) inform, improve and advocate for our programs.

Specifically, the document will provide an introduction of the following topics:

• General issues in research and evaluation;
• A revisiting of the “findings” and critiques of the Mathematica Study (MPR);
• A brief summary of other research findings on TRIO and Upward Bound programs
• Some general findings on college outreach programs.

1) General issues in research and evaluation

• Evaluations can be useful tools in providing better services to our students. Summative evaluations (usually quantitative) can tell us whether our programs achieve their desired results. Formative evaluations (often qualitative) can tell us HOW a program works and identify challenges and areas of success. Both types of evaluations can lead to program improvements, justification for future funding and convincing data to use for student recruitment. Go to: http://www.socialresearchmethods.net/kb/intreval.htm for more information about different types of evaluation.

• However, research and evaluations can be flawed:

• Most studies can only tell a piece of the story: Quantitative and experimental research studies in education look at outcomes and tell us if something “works” or not. For example, does participation in a certain intervention correlate with changed outcomes along certain measurable variables, like college going rates? However, these types of assessments cannot control for all intervening variables nor do they tell us how an intervention works. This is where more qualitative studies, involving in depth interviews, case studies and observations can be helpful in identifying the effects of an intervention. However, these studies are usually small, and many argue that one cannot generalize from these findings. Moreover, some qualitative methods may not be enough to convince skeptics. For example, individual testimonials can be an important part of an evaluation, but by themselves, they do not show that a program is achieving its goals for all its students.

• Education research should not be looked at as a lab science. Neither type of study (quantitative or qualitative) can systematically prove that a certain intervention was the CAUSE for a change because it is impossible to rule out all other possible variables. An educational “treatment” is not an easily replicable intervention like a pill or a surgical procedure; rather it is a series of negotiated interactions between individuals in a dynamic context. Even in medical studies, the results of clinical trials often fail to hold up when subjected to the social variables outside of closely monitored studies (see Erickson & Gutierrez, 2002. It is a helpful 4-page comment on issues in educational research.   http://courses.ed.asu.edu/berliner/readings/ericksonresponse.rtf).

• Studies may not always be asking the right questions. If a quantitative study only collects data on graduation and college going rates, it can miss a number of other ways that an intervention can affect a student, such as college choice, retention or general quality of experiences. Often times, the narrowly defined variables for success do not always capture the ways that programs help students become more successful.

• Research and evaluations are not free from bias: When analyzing data there is always interpretation, judgment and faith involved. The same data can result in different interpretations from individuals with varied political and theoretical perspectives. Many have argued that qualitative research can be biased but uncritically assume that quantitative or scientific research is unbiased. See chapter one of Merriam, Sharan B. (1998) Qualitative research and case study applications in education for a good overview of the different philosophical foundations of research and researchers.

• Erickson and Gutierrez (2002) counter this public perception about scientific research and share that “many scholars have shown that real scientists in their daily work are anything but disinterested and canonically rational. In their daily practice they are passionate and argumentative, profoundly selective in their attention to evidence, and aesthetic in drawing conclusions from it” (p. 22).

• Researchers selectively highlight some findings over others. The introduction of the Mathematica study to the Education department and media reports emphasized the lack of overall impact on high school and college credits earned. This framing led to UB being labeled ineffective, recent threats to funding and the current evaluation. However, in a edited book on summer outreach programs, Myers, the lead researcher in the study, wrote the following in the first paragraph of his chapter: “An ongoing evaluation of the Upward Bound program…has recently shown that the program has large impacts on students who may be more at risk of academic failure and few if any impacts for students who are less at risk” (p. 183, Moore & Myers, 2004). These large impacts on some of the students were not highlighted in the report to the department of education. This focus and framing can lead to very different interpretations of reports.

• Research and its uses are political: TRIO programs represent the second largest federal expenditure for education. Administrations seeking to cut expenses or direct them to new projects, which they can take credit for, may look to decrease funding for TRIO.

• Effective evaluations need the active and reasonable participation of all stakeholders. If not, then the findings can be compromised. In the case of the MPR, study the research team likely did not.

• There are several factors that can pose threats to the validity of findings: This page provides an overview of threats to internal validity. http://www.socialresearchmethods.net/kb/concthre.php.  For a brief list of threats to internal validity see: http://www.evaluationwiki.org/wiki/index.php/Threats_to_Internal_Validity

2) An overview and critique of the Mathematica Study (MPR);

The “findings” of the Mathematica study were:

Upward Bound had no effect on enrollment at postsecondary institutions or postsecondary credits earned by students overall. However, it may have increased enrollment in four-year colleges by about six (6) percentage points but the evidence is not statistically conclusive. Increased enrollment at four-year colleges appears to be offset by reduced enrollment in two-year colleges.

Upward Bound has limited overall impact on students' academic preparation for college. Although Upward Bound slightly increases the number of math credits earned (0.2 credits), the program has no impact on credits earned in other academic areas, total high school credits, Advanced Placement (AP) course-taking, high school graduation or grade point average.

Upward Bound has a substantial impact on high school and postsecondary outcomes for certain groups of students (i.e. lower performing students). In particular, Upward Bound consistently showed a positive impact on students who, when applying for the program, did not expect to earn a B.A. degree. Although these students with "lower educational expectations" comprise only about 20 percent of Upward Bound participants, participation in the program increases the total number of academic credits they earn in high school (two credits) as well as AP credits (0.7). Most significantly, Upward Bound more than doubles, from 18 to 38 percent, the likelihood that these students will enroll in a four-year college, and improves their early college persistence as measured by total credits earned (11 additional credits).

Staying in Upward Bound for longer periods is associated with better student outcomes. The median length of program participation is 19 months. Each additional year that a student remains in Upward Bound is associated with a 9-percentage point increase in attendance at a postsecondary institution.

Although the findings were mixed, the media and government focused primarily on the negative outcomes. For instance, the Chronicle of Higher Education reported on May 3, 2002: 
“The Bush administration's take on UpwardBound is based on a 1999 evaluation of the program conducted by Mathematica Policy Research, Inc., an independent group that analyzes public policy. That study found that UpwardBound "had no impact on students' in-school behavior, participation in extracurricular activities, grade-point average, or credits earned in subjects such as English or science. Furthermore, UpwardBound had no impact on high-school graduation" and ‘no impact on the chances students attended college.’”

The Full report can be found at the following links:
URL: http://www.ed.gov/rschstat/eval/highered/upward/upward-3rd-report.html
Link to PDF: http://www.ed.gov/rschstat/eval/highered/upward/upward-3rd-report.pdf

Issues with the Mathematica Study

As cited above, the reporting of the study highlighted negative findings in the executive summary and several positive outcomes were buried deep in the report. This created a situation where the “sound bite” was that Upward Bound does not work, while that is not the case. The study also failed to document more nuanced outcomes in participants’ lives affected by Upward Bound programs. In addition to these two factors, it also:

• The MPR study had corrupted data – The “control” group for the study consisted of students who had applied to Upward Bound but randomly not chosen to participate. This pool of students is typically motivated towards postsecondary education and likely found similar services elsewhere (Pell Institute, 2002). In fact, 11% of the control group students participated in Talent Search and close to 3% were even in other UB programs (See Baker link below). Moreover, other control group students likely received similar services from other programs, possibly through referrals from Upward Bound staff. This means that about one in seven of the “control” group students received TRIO services, thus possibly positively skewing the statistics for this group. In addition, a much higher percentage of the control group students participated in non-TRIO college support services. The MPR evaluators assert that after they adjusted for the overlap in TRIO participation, there were no changes in the results. However, as Dr. Harley Baker has pointed, this is almost statistically impossible. MPR has refused to make their data available even though it should be public record. In sum, the study did not show that UB services were ineffective, it showed that there was not considerable differences between the two groups, many of which likely received similar support services. For additional information on these points, see the attachment documents from the Council for Opportunity in Education (COE) website and the WESTOP website below and the link to Dr. Baker’s analysis: http://www.coenet.us/files/available_research-mathematica_policy_april_1999.pdf http://westop.csuchico.edu/issues/talk-points.html  http://westop.csuchico.edu/resources/mathamatica/HBaker/HBaker.pdf

• MPR misappropriated the medical model of research. There was no double blind procedure in the study. Basically, the control group knew they were the control. There is research showing that knowing you are in the control group can change behaviors. In this case, it could cause students to be more motivated and work harder.

• The study did not follow standard and professional guidelines for reporting. The study did not have a bibliography until three years after the first available draft. In addition, it had little to no literature review of prior research on Upward Bound programs (much of which finds positive conclusions about the program). Generally, these two points reflect poor research and reporting practice. By omitting these pieces, they fail to show the type of in-depth understanding of the field that would make them quality researchers on this topic.

• Despite all of these flaws, the study is presented to government researchers as a kind of gold standard. HOWEVER, the Mathematica group was not asked to do the follow-up study, which suggests that their work may not have been such high quality.

3) A brief summary of other research findings on TRIO and Upward Bound programs

While the MPR study highlighted its negative findings, it also found that Upward Bound had positive effects on many students. There is a substantial amount of literature that supports the effectiveness of Upward Bound and other TRIO programs. However, many of the studies are not the experimental research model that the government prefers. In fact, the literature on college access programs generally lacks these types of studies (Gandara, 2001; Gullat & Jan, 2003) Nonetheless, they do represent legitimate research that should be referenced. Many of these articles can be found by searching through the following website: / In addition, there are scores of dissertations and theses that document the effectiveness of Upward Bound programs as well as the anecdotes and observations of those of us who work in these programs. Below, I have provided a summary and links to a few published resources.

The Journal Of Negro Education published by the Howard University School of Education devoted an entire issue of the journal in 1998 to research about TRIO programs. In that journal, McElroy & Armesto’s article entitled “TRIO and Upward Bound: History, Programs, and Issues-Past, Present, and Future,” “traces the history of the TRIO programs and provides evidence of their effectiveness in closing educational opportunity gaps in U.S. society…In looking at TRIO's Upward Bound program specifically, the article presents a synopsis of over 30 years of research and program evaluations of this initiative, along with recommendations for improving Upward Bound's effectiveness” (p. 373). It offers a summary of the Mathematica study and concludes that programs like Upward Bound should be coupled with other educational reforms (e.g. quality teachers and high standards for all students) in order to achieve significant progress in improving educational equity. The article can be found at:
http://findarticles.com/p/articles/mi_qa3626/is_199810/ai_n8811686

In the same journal, the Coles article entitled “TRIO Achievers: The Promise of the Future”, offers a positive assessment of TRIO programs by qualitatively highlighting the accomplishments of ten "TRIO Achievers." They all share multiple positive effects that their TRIO participation had on their educational pathways. This type of study is helpful in showing that TRIO programs can significantly influence the educational success of low-income first generation college bound youth. However, the article does not highlight how the programs had these positive effects on them. For instance, a more in depth qualitative case study of an Upward Bound program and its programming and its effects would be useful to highlight best practices. The article can be found at: http://findarticles.com/p/articles/mi_qa3626/is_199810/ai_n8823445
            Lastly, McLure, & Child’s article, “Upward Bound Students Compared to Other College-Bound Students: Profiles of Nonacademic Characteristics and Academic Achievement” reveals several positive effects of participation in Upward Bound. The authors found that “although issues related to causality were beyond the study's scope, several patterns of similarities and differences emerged from these comparisons. On average, educational aspirations of UB students were as high as those of non-UB students; however, UB participants may have gained better understandings of available college support options, as evidenced by their greater willingness to seek formal assistance in key skill areas” (p. 346). In short, they are sharing that UB provides important “college knowledge” as well as “help-seeking” skills that help students prosper in college. This article can be found at: http://findarticles.com/p/articles/mi_qa3626/is_199810/ai_n8825463

4) Some general findings on college outreach programs.

As stated above, there are a limited amount of systematic research studies of college outreach programs. Below are some resources that have synthesized the research in the field and reported on best practices. However, most of these reports call for further study to understand both the effects of college outreach programs and the practices that lead to these positive results. While the government almost exclusively favors experimental design studies, we have seen that they can have many flaws, such as in the Mathematica study.  Programs and researchers should also consider other strategies for evaluation.

The following information has been extracted from the following website: http://www.collegeaccess.org/NCAN/

• Gandara, Patricia. (September 2001) "Paving the Way to Postsecondary Education: K-12 Intervention Programs for Underrepresented Youth" National Center for Education Statistics. (Sept. 2001)
• Finding: Students who attend summer programs on college campuses are more likely to take social science and math courses to prepare for college.
• Finding: Students who receive tutoring, are mentored, or hear motivational speakers are 11% more likely to attend a 4-year college.
• Finding: Hispanic students who participated in a 2-year college-prep English class were 9% more likely to go on to college.
• Finding: Programs that provide resources to allow schools to make necessary changes in curriculum significantly increase the college-going rate of students at those schools. Furthermore, students are 17% more likely to stay in college after their first year, and 9% less likely to drop out.
• Gullatt, Yvette and Wendy Jan. "How Do Pre-Collegiate Academic Outreach Programs Impact College-Going Among Underrepresented Students?" Boston: Pathways to College Network Clearinghouse. http://www.prel.org/products/pr_/compendium05/GullatJan.pdf
• Finding: Providing a formal, long-term mentoring program that stresses academic goals and has contact at least once a week causes participants to have higher GPAs and higher rates of college attendance.
• Finding: Providing a program of educational enrichment, mentoring, career exploration, and action plan development and employment causes participants to be more likely to start school on schedule and work less during college.
• Cabrera, Alberto F. et al. "Pathways to a Four-Year Degree: Determinants of Degree Completion among Socioeconomically Disadvantaged Students." (Nov. 2003) – This quantitative work provides a series of informative slides and charts that document the results of analysis of longitudinal NELS data. The slides do a very good job at summarizing the state of inequality in college going rates as wells as indicating some factors that are associated with college going. However, it does not seriously address inequality of educational opportunities as a factor.
• Finding: Peer mentoring to pursue a college degree increased participant’s chance of college degree completion by 5%.
• Finding: A student with a Moderate or High level of academic preparation is 19% and 32% (respectively) more likely to obtain a college degree.
• "Some Things Do Make a Difference for Youth: Career Beginnings" American Youth Policy Forum.
• Finding: Providing career exploration and action plan development, educational enrichment, mentoring and employment led to a 9.7% increase in participants’ college attendance rate immediately after high school, and similar retention success after enrollment.
• Cabrera, Alberto F. "On the path to college: What matters?"
• Finding: Early awareness and college planning programs increased the probability of students becoming college qualified by 18.1%.
• Finding: Students who are qualified for college are 14% more likely to apply to college.
• Lipka, Jerry. "Closing the Gap: Education and Change in New Stuyahok. Case Study." Northwest Regional Education Lab. (October 1999)
• Finding: Students who had a positive community voice and learning experience were 40% more likely to attend college.
• "Raising Minority Academic Achievement: Gateway to Higher Education." American Youth Policy Forum.
• Finding: Collaboration with schools to offer summer enrichment programs, Saturday Academies, tutoring, counseling, internships, college visists, cultural awareness activities, advanced laboratory work, SAT prep classes and required AP courses resulted in a 74% graduation rate from four-year colleges or universities within five years, and 59% of these students remained in a science-related major or profession.
• Finding: Providing educational activities such as tutoring, development activities such as life and family skills or future planning, mentoring services, community service opportunities and financial incentives for participation causes participants to be 26% more likely to enroll in post-secondary education, more likely to take part in community service, and less likely to become teen parents.

• Lerner, J. and B. Brand (2007). The College ladder: Linkng Secondary and Postsecondary Education for Success for All Students, American Youth Policy Forum: 1-176. http://www.aypf.org/projects/LuminaProjectonSPLOs.htm

• The American Youth Policy Forum (AYPF) conducted an extensive literature review to identify and analyze 22 programs that have a third party evaluation or "have participated in a semirigourous data collection effort. However, few of the evaluations had longtidinal or control group data. Page 113 has an interesting table that provides an overview of the outcomes of the 22 programs looked at in this review. The table shows that many of the program evaluations were unable to show statistically significant positive results but most did report qualitatively positive results. This underscores the challenge of statistically quantifying the effects of college access and opportunity programs.

The following websites/articles/books provide further information on research methods and evaluation.

• Trochim, William M. The Research Methods Knowledge Base, 2nd Edition. Internet WWW page, at URL: <http://www.socialresearchmethods.net/kb/> (version current as of October 20, 2006).
• This page gives an intro to the different types of evaluation http://www.socialresearchmethods.net/kb/intreval.htm
• This page provides an overview of threats to internal validity. http://www.socialresearchmethods.net/kb/concthre.php
• This page looks at “statistical power.” It provides good information on how to interpret and understand research statistics. http://www.socialresearchmethods.net/kb/power.php
• The Joint Committee is a coalition of major professional associations concerned with the quality of evaluation. For a list of their standards in program evaluation see: http://www.wmich.edu/evalctr/jc/ OR
• See the American Evaluation Association’s guiding principles for evaluators at: http://www.eval.org/Publications/GuidingPrinciples.asp
• To see a brief list of threats to internal validity see: http://www.evaluationwiki.org/wiki/index.php/Threats_to_Internal_Validity
• Chapter one of the following book provides a good overview of philosophical foundations of research. Merriam, Sharan B. (1998) Qualitative research and case study applications in education. San Francisco: Jossey-Bass.

Below is a list of references cited and other publications concerning college outreach programs.

Erickson, F. & Gutierrez, K. (2002). Comment: Culture, Rigor, and Science in Educational Research: Educational Researcher. 31(8). 21-24.
http://courses.ed.asu.edu/berliner/readings/ericksonresponse.rtf

Gándara, P., (with Bial, D.). (2001). Paving the way to postsecondary
education: K-12 intervention programs for underrepresented youth.
Washington, DC: U.S. Department of Education, National Center for
Educational Statistics.
http://nces.ed.gov/pubs2001/2001205.pdf

Gullatt, Y. & Jan, W. (2003). How do Pre-College Academic Outreach Programs impact college-going among underrepresented students? Pathways to College Network. http://www.pathwaystocollege.net
http://www.prel.org/products/pr_/compendium05/GullatJan.pdf

McLure, G., & Child, R. (1998) Upward Bound Students Compared to Other College-Bound Students: Profiles of Nonacademic Characteristics and Academic Achievement. The Journal of Negro Education, 67(4), 346-363.
http://findarticles.com/p/articles/mi_qa3626/is_199810/ai_n8825463

Myers, D., & Schirm, A. (1999). The impacts of Upward Bound: Final report for phase I of the national evaluation. Washington, DC: U.S. Department of Education, Planning and Evaluation Services

Tierney, W. G., Z. B. Corwin, et al. (2005). Preparing for college : nine elements of effective outreach. Albany, N.Y., State University of New York Press.
           
Wimberly, G. (2002) ACT Policy Report: School Relationships Foster Success for African American Students
http://64.233.169.104/search?q=cache:IIPtv1VQ3rEJ:www.act.org/path/policy/pdf/school_relation.pdf+ACT+African+American+Students+Upward+Bound&hl=en&ct=clnk&cd=1&gl=us&client=safari

 

[Back to Board Reports | Top]