What Makes You Think You’re So Great?

 

Creating An Organizational Model for CTP Success Using a Mixed Methodologies Research Approach

 

Bruce Yelton, Ed. D.

Paula Plonski, MA

Grant Morgan, MS

 

©Praxis Research, Inc., 2006

 

 

 

 

 

This work is dedicated to Kenneth Smith for his years of devotion to the cause of education for all students in North Carolina and for his help in making this research possible.

Introduction

 

The emphasis in College Tech Prep (CTP) evaluation during the past ten years has been predominantly focused on quantitative measures of student participation and impacts on student achievement (Franks & Brown, 2001; Miami Valley, 2002).  This focus has proven quite valuable in those locations where this effort has been carried out in a systematic and conscientious manner and has provided valuable insight for interpreting the impact of CTP, and career-technical education in general, at the state and national levels (Stone, 2003; Visher, Bhandari, & Medrich, 2004; Morgan & Yelton, 2006).  In addition to quantitative analysis of student participation and performance there have been frequent descriptions of program activities and processes that include anecdotal narratives from teachers, parents, business allies, and students on program effects.  However, there has been little or no systematic qualitative analysis of CTP or effort to integrate qualitative data into a comprehensive analysis of CTP that generates a model for why such programs succeed or fail in terms of student outcomes (graduation, academic achievement, required remediation, post-secondary enrollment, employment, etc.).  The study of “exemplary” CTP programs carried out in 2000 by CORD (Walde) was an effort in this direction. 

 

The CORD report does a comprehensive job of describing the selected CTP programs and was designed to provide assistance to local programs that were “working to build rigorous and effective education strategies.”     Moreover, the author describes the overriding goal of the report is “to provide readers with innovative examples and step-by-step procedures by which educators can improve their programs”.  The criteria used to identify the eleven programs selected for the study were centered on nine core areas (articulation, accountability, curriculum integration, career guidance, staff development, transition, partnership with business, work-based learning, equal opportunity, and evaluation).

While the report succeeds in providing a detailed description of programs that can be used as models for improvement it can be argued that the criteria used for the identification of the programs were essentially subjective, based on self-description and not on systematic, independent data collection of either a qualitative or quantitative nature.  The methodology reported also did not establish the validity or reliability for the selection process as described in the published report (Walde, 2000). 

 

Also mentioned in the CORD report is a brief description of a model for the components of successful CTP programs stating that “common elements leading to success are strong leadership, active participation among key partners, and effective outreach to students, parents, educators, and employers”.  While this statement seems to reflect sound common sense; the report does not provide supporting data to underpin the claim or analysis to show the logical links to the real world examples cited in the text.

 

It was at least partially to address the limitations of the previous work as described above that the authors undertook to create the research detailed in this report.  The authors were also were challenged to work with the existing longitudinal quantitative data and newly generated qualitative data in creating a blend of information that would be useful to practitioners in improving their services to students and demonstrating the effectiveness of their programs to decision makers.

 

The North Carolina CTP Case Study Project

 

North Carolina has had an active College Tech Prep program since the early 1990’s.  Currently (2005-06) 34 CTP projects are supported with Perkins Grants by the NC Department of Public Instruction. Each year each state funded project must comply with predetermined evaluation standards and present both written and oral evaluation results to a panel of Public Instruction and NC Community College representatives.  These data, like those described above, have been primarily quantitative and descriptive of program implementation.  These data have been very helpful in tracking changes over time in student participation in CTP, the performance of CTP students versus those enrolled in other courses of study, and the articulation of secondary and post-secondary curricula.  One finding that has been documented from this data is the continued climb in students completing CTP and CTP/College-University Prep (CUP) combination programs over the past four years mirroring a proportionate decline in those students choosing a CUP only pathway.

 

There have been questions however that the quantitative data collected in North Carolina and in other states cannot answer about CTP.  Why are some programs successful in marketing their CTP programs and others not?  What creates the most efficient secondary to post-secondary articulation system?  Information of a more in-depth, descriptive/narrative nature is required to systematically explore these and other issues.  It was for this reason that the researchers chose to qualitatively examine the effectiveness of CTP programs in North Carolina.  Qualitative analysis involves selecting respondents according to a methodology that is usually not random as in quantitative studies, and carefully analyzing observation, interview or similar data that is idiosyncratic.  The validity of the conclusions produced are substantiated by their “grounding” in the data, their corroboration by multiple sources, and by their inductive logic.  Given the limitations described above for studies linking student outcomes to program practice, it was our intent to establish both through existing student longitudinal data and interviews with key stakeholders the critical components for successful program performance. In order to accomplish this, the researchers selected programs identified as exemplary by qualified individuals familiar with their performance over time.

 

 

Methodology

 

This research first involved the selection of CTP sites in North Carolina that were regionally and demographically diverse and that had demonstrated via their past performance reviews that they had effective  programs.  Programs were considered “effective” that had demonstrated success in one or more of six areas:

 

•       articulation

•       marketing

•       sustainability

•       business collaboration

•       student achievement, and

•       evaluation.

 

The primary reason for selecting programs that were effective and regionally diverse was to attempt to identify variables common to these programs that were related to their success and independent of their setting.

 

The reviewers for selecting the programs included the lead researchers, the state CTP director (Dr. Kenneth Smith), and a senior state CTE technical assistance director (Mr. Greg Gift).  Although the reviewers were very familiar with the programs and did make use of annual review data (participation in CTE, GPA’s, remediation rates, post-secondary enrollment, etc.) in making participant selections, there was no formal scoring or rating of programs.  In this way this study very much resembles the “subjectivity” of other studies of “exemplary” programs.  However, analysis of the existing student achievement data presented below will demonstrate some interesting results in terms of the differences of the selected group from other CTP programs.

 

Ten programs were selected, contacted via postal mail, and agreed to provide information for the study (See table below).  Two of these were considered urban programs (Guilford and Asheville, Buncombe, Madison).  The remainder of the programs were rural or rural/suburban in nature.  Of the ten programs selected six were multi-LEA partnerships of school systems, three of these crossing county lines. One of the programs included in the study, Guilford, had also been included in the 2000 CORD study of exemplary programs.

 

Table 1. Programs Selected for Case Study

 

The data for this study consists of a combination of qualitative and quantitative information.  The quantitative data is derived from the annual report submitted in 2004 and 2005 to the NC Department of Public Instruction as part of the required annual report on CTP programs.  These data include both secondary and post-secondary student outcomes and are aggregated and reported to the state annually (Morgan & Yelton, 2006). 

 

The qualitative data for the study was collected via telephone interviews with CTP/CTE directors at the secondary and community college levels in all ten consortia.  A predetermined interview protocol was used to conduct these interviews (see Appendix A).  The same interviewer was used for all calls except one of the twenty-two interviews conducted.  This interviewer was trained and experienced but had not met any of the interviewees previously.  The interviews lasted from between 25 to 60 minutes.  All interviews took place over a three week period in the late summer of 2005.  Analysis of the interview data was done through transcription, construct creation, and reification by each of the authors independently and in a group process.

 

The questions asked of the respondents reflect five central areas of inquiry:

 

•       How do participants identify strength/success within their CTP program?

•       What elements of their programs do they considered their strongest.

•       How have they coped with changes in leadership?

•       What do they consider their greatest achievements?

•       What is their vision for the future of their program?

 

 

Results

 

Quantitative Findings

 

Although the examination of programs was intended to be primarily one of qualitative analysis, one of the objectives of the study was to identify how the selected programs might statistically be different from one another based on student performance outcomes of the previous two years of evaluation reporting.  Many of the comparisons made between selected and non-selected programs showed no differences, but three did show a distinct advantage to students from the selected programs:

 

•       The selected consortia demonstrated significantly lower required remediation rates for those students entering community college in mathematics and for both 2004 and 2005.

 

•       Remediation was also lower for students from selected consortia in English both years.

 

•       A higher proportion of selected program CTP (or CP/CTP) graduates enter community college upon graduating from high school compared to those programs that were not selected.

 

The researchers were also very curious about the fact such a high proportion of the programs selected as exemplary were programs that included multi-LEA programs (six of ten) when multi-LEA programs are not represented in this proportion among all CTP projects in the state.  Analysis of the student outcome data using t-test to compare means revealed that students from multi-LEA consortia, both within our study and in general, had higher average first year community college GPA’s than single LEA consortia (p=.000, t=-5.286).  Multi-LEA programs also had a higher proportion of students graduating from high school in a CTP course of study and a higher proportion of those same students entering community college after graduation. 

 

Closer examination of multi-LEA consortia using z-test analysis also revealed that consortia with three partners had the lowest rate overall of mathematics, English, and reading remediation for their CTP/CUP students who entered community college upon high school graduation.  Again these findings were statistically significant (p<.05).

 

 

 

 

Qualitative Findings

 

In conducting the interviews it was clear that the respondents were proud and protective of their organizations.  Their participation was encouraged in some cases due to their familiarity with the researchers who are widely known in the NC-CTP community and by the positive nature of being identified as “exemplary.”  Analysis of the transcripts of the conversations was carried out by three researchers familiar with CTP and with qualitative analysis.  Common themes and ideas across interviews were noted and discussed among the research team and four constructs related to the participants’ expressions about program success were identified.  These were labeled as:

 

• Internal and External Collaboration

 

• Leadership

 

• Sense of Mission; and

 

• Institutionalization.

 

These constructs repeated themselves in various ways across interviews and with repeated analysis arranged themselves into an organizational model that could be identified as related to positive program and student outcomes.  Indeed it was suggested that what was emerging from the data closely represented the Burke-Litwin model (1992) of organizational development presented below.

 

 

The Burke-Litwin Model was originally presented as a causal model for understanding and affecting organizational performance and change.  The model includes twelve boxes that are separated into two groups, which the authors call transformational and transactional dynamics.  The transactional dynamics have been removed because their impact is beyond the scope of this discussion.  As can be seen above, the external environment is considered to be an all-encompassing factor that has a large impact on performance and change.  Internally, however, leadership, mission/strategy, and culture have the greatest impact.

 

In the Burke-Litwin Model, leadership plays, arguably, the most pivotal role in performance and change because it represents the meeting place of strategy and culture.  Galbraith (1983) asserts that the strategic driving force manifests itself in the leader’s beliefs about how to succeed, which suggests that leadership can be thought of as being in the minds of leaders.  The researchers extended this view of the role of leadership and used it as a logical starting point for the CTP model.  As Burke and Litwin (1992) suggest, it is believed that the leaders provide overall direction and serve as role models for all stakeholders while, at the same time, scanning the external environment, choosing the forces to deal with, and taking appropriate action.

 

Using the Burke-Litwin model as a suggestive template, the researchers took the findings of the case study interviews and created the theoretical model of College Tech Prep program functionality that is postulated in Figure 2.  This model places leadership at the initial position in the process of organizational development.  This priority of leadership flows from the logical necessity of having to have an individual or individuals within a educational structure willing to take responsibility for the program, it’s components and processes to make CTP a reality.  Collaboration is given the second position in the model and it is also a fundamental element in CTP due to the requisite internal and cross-organizational cooperation required for success student outcomes.  Sense of Mission and Institutionalization are characteristic of maturing organizations which are becoming self-sustaining and intrinsic to their communities and were therefore placed at the top of the theoretical model.

 

 

To illustrate how CTP organizational model was developed each of the constructs and some of the related interview data from which they were derived will be discussed in turn below.

 

 

 

 

Leadership

 

Respondents in our study did identify a variety of leaders in positions ranging from school superintendent to Chamber President as critical for the success of their programs.  They seemed eager to give credit to their partners for making their programs work and it is interesting to note that some of the respondents had difficulty in naming a particular leader.  Instead, they preferred to talk about shared leadership and a team effort. The authors heard statements like “we have a team”, “everyone pulls an equal weight according to their personal strengths”, and “everyone is equal” on several interviews. 

 

Respondents also talked about the high level of personal commitment of their leaders to the CTP initiative.  Statements like the ones below are fairly typical.

 

They (leaders) understand what’s at stake.  They have taken initiative.  They know the working level requires faculty and high school teachers meet face to face, they know what their programs are like and are proactive about recommending ways that the programs are going to cooperate.

 

We want what’s best for our students… we have experience and the background and we are like a thorn, we just keep fighting…..You’ve got too have somebody who keeps pushing…we are the ones that have just kept pushing.

 

 

Internal and External Collaboration

 

The construct of collaboration as envisioned here goes deeper than formal agreements, informal arrangements, and individual interactions.  It includes the impact of individual and collaborative action on the organizational environment in which the actors work and its linkages with the larger community.  The researchers were able to identify through the interviews the importance of two related but distinct types of collaboration in creating successful CTP programs.  The first of these is the internal collaboration within the educational system of a community and the second is external collaboration beyond the bounds of the educational arena and touching business, human service, government, and other organizations. Both of these types of collaboration are discussed separately below.

 

 

 

 

Internal Collaboration

 

Internal collaboration can be conceived of as the basic “grease” that makes things happen in the bureaucracy of the educational world.  It allows the individual actors pursuing CTP implementation within their organizations to effect change.  The data that was collected via the interviews provided evidence that this internal collaboration, which the researchers determined includes collaboration across the bounds of secondary and post-secondary education institutions, can enhance the status of the individual CTP consortium member.  These individuals become resources for both information and assets that may not be available to other members of their organization by serving as organizational liaisons.  In the experience of the researchers it is a rare phenomenon for most educators at the secondary and community college level to have on-going contact with others in their field not within their own institutions.  This enhanced status of the individual not only serves to further implementation but reciprocally strengthens the collaboration resulting in the frequent use of words like “family”, “friendship” and “bond” being used to describe internal collaboration. 

 

When respondents were asked “What single event can you remember from the past year that makes you believe that your CTP partnership is working well?” the most frequent answer mentioned was the quality of the collaboration between the schools and their community college partner.  Statements like the following backed up this sentiment.

 

There were bonds formed between the faculty at a retreat we had. It broke down the barriers of “us and them”. They got to see that we're all teachers that face some of the same issues in the classroom. The college faculty who once said ‘what are they teaching them in high school, they don't know anything when they come here’ and the high school teachers that are telling the kids ‘your college instructor is not going to care about you’ --and none of that is true. They really got to know each other and see that.

 

I think seeing those groups and those three entities working seamlessly, like they've been knowing each other for years and working together for years, I think that was a highlight of our year.

 

External Collaboration

 

External collaboration incorporates relationships with business, public, and governmental organizations outside the educational community.  These relationships again make CTP programs and their unique.  They directly tie their organizations to economic leaders and development of their communities.  This proximity to economic decision makers is not usual for most educators, especially not those below the level of school superintendent or community college president.

 

Business partnerships were mentioned as important by many of the consortia.  Strong collaborative relationships with local businesses were a recurring theme in most of the interviews.  Many mentioned having monthly meetings with business representatives

 

The graph in Figure 3 shows the results of asking the respondents “In which of the following three CTP activities do you think your partnership is doing the best job?”  Clearly, the majority of the respondents felt that of this group of three (Business Partnerships, Special Populations, Innovative Teaching) Business Partnerships was where they were being the most effective.  One respondent summed it up by saying-

 

We have an outstanding business partnership and we call it a business alignment, we meet the third Thursday every month, and we have representatives from several businesses. Our last meeting, our room was so full, we had to bring in additional chairs.

 

Sense of Mission

 

The third component that was derived for the model of CTP programs was sense of mission.  As previously seen some leaders feel strongly about their roles and many have a sense of the critical issues facing CTE in 21st century public education.  For some consortia this sense mission provides a great deal energy, human resources, innovative thinking, and moral capital.  Their example of selfless devotion to their students, strong belief in the goals of CTP, and willingness to go beyond the bounds of what would be considered the normal requirements for their job descriptions to get results certainly may serve to increase “buy-in” on the part of both internal and external partners and superiors.

 

Many consortia members spoke of wanting the best for their students.  Some spoke of many years of dedication to CTP students in the face of isolation within the academic community of their own institutions.

 

The priority is AP, IB, college bound, and I say there is a spark sometimes that someone acknowledges that there are other kids out there. We’re the ones that believe it in all our hearts and soul, and we keep on pushing.

 

This statement matches similar ones heard from other respondents and is a theme commonly repeated in many CTP evaluations in which the researchers have participated in recent years.

 

 

Institutionalization

 

This construct is closely related to the sense that CTP is an ongoing, essential, and sustainable component of the educational community.  In educational cultures where institutionalization of CTP has taken place the administration would no more consider opening school without a CTP program in place than they would without English classes or media centers.  The effect of institutionalization is to make CTP a part of the permanent “culture” of the organization.

 

In discussing the creation of sustainable CTP program components, collaboration between high schools and community colleges was prominent, as was innovation and funding.  Reaching this level implies that all stakeholders content area teachers, parents, administrators, the business community, and students are cognizant of CTP and value its contribution to the educational/economic community.

 

We have worked so diligently in getting everyone on board from the top administration, the teachers, counselors and all of those people are members of our various task forces… I feel that if the funding stopped tomorrow, which we certainly do not want to do without, I feel that the scholarship implemented in our consortium and support from industry would sustain itself.

Conclusion

 

This research was originally intended to include more varied types of information from participating programs identified as exemplary.  If time had permitted and additional funding had been available, on-site observations and additional interviews with other stakeholders (students, instructors, parents, etc.) would have provided more data that would have expanded the findings presented here.  However, the authors do not believe that the basic findings would have been altered or the theoretical model presented changed in any substantial way.  Rather the authors believe that additional research, both quantitative and qualitative, will serve to support the findings presented here.  Indeed from the perspective of previous CTP research on effectiveness, as limited as it may be, as well as an intuitive and experience based perspective; it is difficult to argue against the positive impact of strong leadership, effective internal and external collaboration, a sense of mission, and a program that has become institutionalized as an essential part of its organizational community. 

 

What will be interesting in conducting further research to test these components will be to develop measures that capture the level of implementation of each of these constructs and to validate their effects via observations of student outcomes.  Studies of this type could be extremely effective in guiding ineffective programs through troubled bureaucratic and implementation challenges and help to demonstrate improvements in implementation over time.  Moreover, comparisons between “poorly implemented models” and “exemplary models” could be used to demonstrate the positive impact of planned technical pathways linked to post-secondary expectations for students.  Self-assessment criteria developed from these measures could in the long term be developed for local consortia to use as tools in program improvement or validation of “exemplary” programs.

  

Appendix A

 

College Tech Prep

Case Study

Telephone Interviews

 

Partnership ______________________          Date: _________________________

 

Interviewee: _____________________           Position: ______________________

 

 

 

1. What single event can you remember from the past year that makes you believe that your CTP partnership is working well?

 

2. In which of the following three CTP activities do you think your partnership is doing the best job?

 

1. Awarding articulated college credit
2. Work-based learning opportunities
3. Integrated curriculum

 

2a. Why do you believe this is your best area of these three?

 

3. In which of the following three CTP activities do you think your partnership is doing the best job?

 

1. Business partnerships
2. Serving special populations
3. Innovative teaching methods

 

3a. Why do you believe this is your best area of these three?

 

4. In which of the following three CTP activities do you think your partnership is doing the best job?

 

1. Evaluation
2. Teaching technology and critical thinking
3. Creating sustainable CTP program components

 

4a. Why do you believe this is your best area of these three?

 

5. Fill in the blank.  If __________________ resigned tomorrow our CTP program would lose a really valuable leader.

 

5a. Why is this person so valuable?

 

5b. When a leader like this has left in the past how has your partnership adjusted?

6. What makes your CTP partnership unique, unlike any other in the state?

 

 

7. What is the greatest success your CTP partnership has experienced in its history so far?

 

8. In the best of all worlds – what will your CTP partnership look like three years from now?

 

8a.  What changes will have occurred?

8b.  How will student outcomes be different?

8c.  What resources will you need to reach this vision?

References

 

Burke, W. & Litwin, G. (1992). A causal model of organizational

performance and change. Journal of Management, 18(3), 523-545.

 

Franks, R. & Brown, C. (2001).  Tech Prep evaluation in Texas:  Using

the numbers.  Connections, (11)7, 1-2.

 

Galbraith, J.R. (1983). Strategy and organization planning. Human

Resources Management, 22, 63-77.

 

Miami Valley (2002).  Celebrating a decade of success:  Miami Valley,

Ohio Tech Prep consortium.  Connections, (12) 3, 1, 5.

 

Morgan, G. & Yelton, B. T. (2006).  North Carolina College Tech Prep:

2000-2004 Data summary.  Charlotte, NC: Praxis Research.

 

Stone, J. (2003). ACTE national policy seminar.  Paper presented at

Professional Development Institute, March 16-18, 2003, Washington, DC.

 

Visher, M. G., Bhandari, R. & Medrich, E. (2004).  High school career

exploration programs:  Do they work?  Phi Delta Kappan, (86) 2, 135-139.

 

Walde, C. (2000). Exemplary Tech Prep programs: Combining college connection and work-based learning opportunities. Waco, TX, CORD.