Different policies have been developed and implemented to train field teachers at the primary and secondary school levels in Turkey to meet the educational demands of the growing population. For instance, there have been attempts to eliminate the need for teachers as quickly as possible which led to unique administrations such as “correspondence course”, “substitute teacher”, “accelerated education” and “Pedagogical Formation” in different periods. Out of these administrations, the Pedagogical Formation Program -founded in the 1970s- continued in different periods to meet the teacher needs until it was abolished in the 1998-1999 academic year by the Council of Higher Education (Akyüz, 2004). The program has been resumed for some programs in2001, but fully resumed in the 2010-2011 academic year where the graduates of the Faculty of Arts and Sciences were given priority. It is expected that prospective teachers acquire fundamental field and culture knowledge in the departments they graduated from. On the other hand, basic teaching skills are practiced and gained through specifically tailored teaching profession courses (Küçükahmet, 2007). The 2010-2011 program, which also included the 3^rd and 4^th grade students from relevant departments, has been continued onlyfor graduates in the following years. The contents and standards for the program were determined by the Council of Higher Education, and universities were asked to form and provide the content of thecoursesinaccordance withthesestandards. The program lasts for one academic year. Students in the program can take their teaching certificates upon successful completion of 26 credits covered in 10 courses. The courses and their credits are illustrated in Table 1. 

 

 

To sustain a quality education system, it is important to train teachers well both prior to the service as well as in the service(?ahin, 2007). Through the Pedagogical Formation Program, it is aimed to develop prospective teachers' teaching knowledge and skills prior to the teaching profession. Thus, the nature and the quality of the program carry utmost importance.

Evaluation of educationprograms is in a sense decision-making process of the effectiveness of the program (Demirel, 2005). Stufflebeam (1971) defines evaluation as obtaining useful and beneficial information for judging alternative decisions and limiting it. Evaluation of education programs is of great importance for understanding the function of education programs on the target audience and thus increasing the effectiveness of education programs.

Evaluation critiques previous documents, plans, and actions (Ornstein and Hunkins, 2009). Since evaluation results aim to reconsider "documents, plans and actions", complete their deficiencies and eliminate their short-comings, evaluation of education programs basically means the evaluation of teaching (Demirel, 2005).

A variety of evaluation models are used in the evaluation of education programs. Among these there are objectives-oriented evaluation, management-oriented evaluation, consumer-oriented evaluation, expertise-oriented evaluation and participant-oriented evaluation models (Fitzpatrick et al., 2004). It is very important that evaluation within the scope of each model should be made via valid and reliable assessment-evaluation tools. Therefore in the current study, the nature and quality of the program are addressed through the opinions of prospective teachers enrolled in the program. First, an objective-oriented program evaluation scale was developed andadministeredtotheparticipants ofthe program. Then, the emerging scale was confirmed with a subsequent administration.

 

Study I: Exploratory Factor Analysis (EFA)

Participants

The participants of the study were students who registered inthe pedagogical formation program of a Turkish state university in spring2011.320 students, who enrolled in the program, were told about the research and asked to participate in the research on a voluntary basis. They were told that they would be asked to fill in a program evaluation instrument and no personal information would be revealed. 250 students volunteered to participate in the research and the data collection tool was administered to 250 voluntary students out of 320 registered students.

Since 29 of the responseswere considered invalid (e.g., empty, monotonous responding), responses of 221 students were analyzed. Of all valid responses, 175 (79.2%) were females and 46 (20.8%) were males. Different colleges were represented in the sample: Humanities (108; 48.9%), arts and sciences (30; 13.6%), conservatory (22; 10%), physical education (22; 10%), medicine (10; 4.5%), fine arts (9; 4.1%), administration (6; 2.7%), communication (3; 1.4 %), theology (2; 0.9%) and engineering (1; 0.5%). Eight students (3.6%)did not state their departments. 

 

Data collection tool

In order to develop the scale to evaluate the current pedagogical formation program, an item pool was generated which addressed the general aims of the program, course objectives and contents. That is, all formal documents involving the pedagogical formation programwerecollectedandanalyzed.However, theanalysis revealed that formal documents did not include sufficient details regarding the basic educational steps of the program such as aims, content, instructional processes, and measurement and evaluation. Thus, the item pool was extended through the review of relevant literature on teaching competencies, program evaluation and scale development. These endeavors led to a 60-item data collection tool. Items were generated as Likert type ranging from strongly disagree (1) through strongly agree (4).

The item pool was examined by three field experts who have PhD in Curriculum and Instruction. They eliminated the items addressing similar constructs, modified complex items and corrected ambiguous ones. They further added new items to address areas neglected by the researcher. After a consensus has been reached by the three experts, the final form included 40 items for piloting.

 

Procedure

To include as much diversity as possible, no limitations have been set while determining the sample size.The data collection was realized towards the end of the academic years. On the other hand, since students would be stressful during the final exams, data were collected a few weeks before the final exams. A total of 250 students responded voluntarily and handed in the questionnaires where 29 responseswere omitted because of invalid responding (e.g., empty pages, monotonous responding). Therefore, responses of 221 students were analyzed.

The Cronbach Alpha of the 40-item data collection tool for the current 221 participants were 0.961.The Kaiser-Meyer-Olkin (KMO) measure of sampling adequacy was checked to see the adequacy of the current sample for factor analysis, which was 0.935. The KMO value should be higher than 0.60to factorize the test items (Büyüköztürk, 2010). Thus, the current KMO value showed that the sample was good for factor analysis. Different sample size values are accepted in the literature. Comrey and Lee (1992) have stated that 200 people are sufficient in middle level.

Whether the data come from multivariate normal distri-bution was tested through the Bartlett Test of Spherity (BTS). This test examines whether there is a relationship among the investigated variables on the basis of partial correlation (Büyüköztürk, 2010). BTS results showed that the p value vas below 0.001which revealed that the factors can be extracted effectively.

The scree plot and the explained variance revealed that the items wereloaded on fourfactors. Thefactor structure of the scales is as good as the degree of variance explained (Gorsuch, 1972; Lee andComrey, 1979; cited in Tav?anc?l, 2006). Although total explained variance values above 30% can be acceptable for single-factor structures,the value must be higher in multi-factor scales (Büyüköztürk, 2010). For instance, explained variance values between 40% through 60% are considered acceptable in social sciences where each factor eigenvalue should be greater than 1(Scherer et al.,1988; cited in Tav?anc?l, 2006). The current EFA revealed that the variance explained by four factors was 58% which was acceptable.

After extracting the four-factor structure, the factor loadings of each item in the scale were examined and complexitems were checked.Having a factor loading of 0.45 or above is considered good, but in practicethis value could be as low as 0.30 for some items (Büyüköztürk, 2010). Thus, the acceptable factor loading cut-off was considered as 0.45. Similarly, the complexity of the factor loadings was investigated. That is, items with high and very close loadings under two different factorswere considered complex (Akbulut, 2010) and these were removed from the scale structure. These analyses led the researcher to exclude 14 items from the scale. The final form involved 26 Likert-items under four factors which explained 58% of total variance.

The factors can be exposed to a factor rotation in order to provide independence, interpretation and comprehensibility (Brown, 2006; cited in Çokluk et al., 2010). Thus, for a better definition and evaluation, current factors were exposed to a rotation process. Considering that current factors were not correlated with each other, a Varimax rotation was realized.

Fourfactorswerelabeledaftertherotationas illustratedinTable2.Theinternalconsistency coefficients for each factor weregood. The alpha values were 0.898 for the first factor (Evaluation of the overall objectives of the program), 0.884 for the second factor (Evaluation of the teaching-learning processes),0.822 for the third factor (Evaluation of the measurement and evaluation process) and 0.741 for the last factor (Evaluation of the pedagogical formation course contents).

 

 

 

Study II: Confirmatory factor analysis (CFA)

A second study was conducted to confirm the four-factor structure proposed through the EFA. That is, the 26-item scale with four factors was administered to a similar population and exposed to a CFA through LISREL.

 

Participants

The participants of the second study were students who registered in the pedagogical formation program of the same Turkish state university in spring 2012. The data were collected from 215 voluntary students. Of all respondents, 156 were females (72.6%) and 59 were males (27.4%). The distribution of the participants’ genres across EFA is seen in Table 3.

 

 

As it is seen in the table,gender distributions are almost equal both in exploratory and confirmatory factor analysis. This situation reveals the fact that teacher formation education program comprises women mostly.

The distribution of the participants across colleges were as follows: Humanities (55; 25.6%), arts and sciences (50; 23.3%), fine arts (35; 16.3%), conservatory (14; 6.5%), physical education (13; 6%), administration (12; 5.6%), tourism (10; 4.7%), medicine (7; 3.3%), communication (4; 1.9%), theology (2; 0.9%), engineering (2; 0.9%). Eleven students (5.1%)did not state their departments. 

In the literature, there are different values for sample size. Comrey and Lee (1992) state that 200 participants will be sufficient in middle level. Furthermore, the Kaiser-Meyer-Olkin (KMO) measure of sampling adequacy was checked to see the adequacy of the current sample for factor analysis, which was 0.914. The KMO value should be higher than 0.60 to factorize the test items (Büyüköztürk, 2010). Thus, the current KMO value showed that the sample was good for factor analysis. Whether the data come from multivariate normal distribution was tested through the Bartlett Test of Spherity (BTS). BTS results showed that the p value was below 0.000 which revealed that the factors can be extracted effectively.

 

Procedure

Similar to the first study the data collection was realized towards the endoftheacademicyear.Thedata collection was realized exactly one year after the first study with 240 new students who responded voluntarily and handed in the questionnaires. Of all scales, 25cases were omitted because of invalid responding (e.g., empty pages, monotonous responding). Therefore, responses of 215 students were analyzed.

 

Results

The internal consistency coefficientof the 26-item scale which was administered to 215 students was 0.93.The factor structure obtained from the EFA was tested with a CFA. According to the suggested modifications, error covariance of item 7 (This program helped me gain the knowledgeandapplicationskillsregardingthe educational programs of the special field I will teach in) was related to items 2, 4, 10 and 22.Thus item 7 was excluded from the scale.Evaluation of the fit indices after the CFA is providedinTable 4. Somefitvaluesaregoodeven before the item 7 was removed from the scale. Thus, removal of the item was not mandatory according to more liberal resources. 

 

 

Chi Square Goodness of Fit (c²) is not an index to be interpreted on its own. It is interpreted through calculating its ratio to the degreeof freedom (Çokluk et al., 2010). According to Tabachnick and Fidel (2001), if the ratio is 2.00 or below,it refers to goodfit (Akbulutet al., 2010). The ratio was1.832 after the removal of item 7 which referred to a good fit.

According to Steiger (2007), if the Root Mean Square Error of Aproximation (RMSEA) is 0.07 or below, it refers to a good fit. The RMSEA was computed as 0.062 in the final measurement model which revealed that there was notanysignificantdifferencesbetweenthepopulation and the current sample.

According to Brown (2006) and Hu and Bentler (1999), a Standardized Root Mean Square Residual (SRMSR) below 0.08 refers to good fit, which was acceptable both before and after the exclusion of item 7 in the current CFA.

According to Sümer  (2000) and Tabachnick and Fidell (2001), if the Normed Fit Index (NFI) and Non-normed Fit Index (NNFI) valuesare 0.90 or above, the model can be considered acceptable (Akbulut et al., 2010). The NFI was somewhat weak (i.e., 0.81). However, the NNFIwas 0.90 and suggested a good fit.

According to Schumacker and Lomax (1996), Hooper et al. (2008) and Sümer (2000),if the goodness of fit index (GFI) and the adjusted goodness of fit index (AGFI)are 0.90 or above, the scale is said to have a good fit. The GFI and AGFI were computed as 0.84 and 0.81 respectively which revealed a weakness in the goodness of fit. In structural equation models, all fit indices are interpreted in conjunction with each other rather than relying on a single index (Cole, 1987; Jöreskog and Sörbom, 1993; Marsh and Hocevar, 1988). Thus, the model was considered acceptable since most fit indices were above the suggested cut-off limits. The internal consistency coefficients for all factors were computed which were good: 0.846 (factor 1), 0.833 (factor 2), 0.786 (factor 3) and 0.823 (factor 4) is provided in Table 5. 

 

 

As in the exploratory factor analysis, in the confirmatory factor analysis it was analyzed whether there was a significant difference between men and women’s views. When the kurtosis and skewness values and normality test results are analyzed according to gender for each factor, it is found out that the four factors do not meet normal distribution conditions. For these reasons, to learn whether there is a significant difference between participants’ views for all factors Mann-Whitney U test, a non-parametric test, was utilized. As a result of the analysis, a significant difference could not be found between men and women for all factors (f1: p=.06>.05; f2: p=.17>.05; f3: p=.67>.05; f4: p=.83>.05).

The validity and reliability studies of PEs confirmed that it is a highly valid and reliable measurement instrument. It is thought that the scale can be effectively used in programevaluationstudiesorientedtoparticipants’ views.

 

An evaluation of the program by stakeholders is one of the conditions to understand a program's functionality. This scale provides the opportunity to make evaluation according to participants’ opinions regarding objectives, content, teaching-learning process and the assessment and evaluation process that make up the program. Findings related to validity and reliability of the scale showed that it is an appropriate tool in order for teacher candidates to evaluate teacher formation programs.

 

The items of this tool that is developed may be used not only for the evaluation of pedagogical formation programs but also for the evaluation of different programsbased on participants’ opinions. This instrument may beused by teachers and administrators for the evaluation of education programs in their own institutions as well as by researchers in the process of program development within the scope of academic studies. Since the focus group of the scale is teacher candidates, the validity and reliability studies should be conducted again in case the scale is applied with different groups.

This scale can be conducted to a sample group taken from all the pedagogical formation students in Turkey in order to be used efficiently and to raise its dissemination. The obtained data can also beanalyzed in terms of dependent variables such as gender; graduated programme also can be used for a deep evaluation of the curriculum effectiveness.

As aresult, this research aiming to evaluate the effectiveness of programs through participant-oriented approach reached a valid and reliable four-factor measurement instrument. 

The author has not declared any conflict of interests.