ABSTRACT
This study focused on the development and validation of psycho productive skill multiple choice test (PSMCT) for assessment of technical college students’ achievement in mechanical engineering craft. The study adopted the instrumentation research design and was conducted in Nasarawa State, Nigeria. The population of the study was 248 National Technical Certificate (NTC) III students comprising three capability groups (42 high, 140 average and 66 of low capability). A 305 item draft copy of psycho productive skills multiple choice test items was generated and utilized as the instrument for the study. The test was used to assess students in the four Government Technical Colleges in Assakio, Mada Station, Agwada, and Doma in the study area. The data collected were analyzed using split-half technique and Kadder-Richardson (K-R20). The reliability coefficient of the test was 0.84. It was found out that there were significant differences in the mean scores of the three ability groups. It was recommended among others, that examination bodies should integrate the psycho productive skill multiple choice test in their examination process for certification of NTC students.
Key words: Mechanical engineering craft, assessment, psycho productive skills multiple choice test, technical college, students, Nasarawa State.
Mechanical engineering craft is one of the programs in technical colleges, in which students are examined by the National Business and Technical Examination Board (NABTEB) for the award of National Technical Certificate (NTC). The National Board for Technical Education (NBTE, 2003) defined mechanical engineering craft as a body of knowledge and skills capable of being utilized on its own as a foundation knowledge for more advance work in the same or other fields of study. Mechanical engineering craft in NBTE consists of the following modules: General metal work I; General metalwork II; general fitting; turning; milling; shaping; drilling and grinding. The board emphasized that the program when successfully completed by the students can be used for employment purpose. According to Okoro (2012), mechanical engineering craft means preparing metal parts by changing the shape, size and surface finish of metals. Mechanical engineering craft therefore is a program among others that involves skills to produce, assemble, and fit engineering components together.
To Ugbalu (2013), mechanical engineering craft involves making of individual parts from plate or bar material by cutting of metal, marking-out, drilling, turning, milling, tapping, grinding and assembly operations. The mechanical craftsman may be primarily engaged in the adjustment and assembly of unit parts manufactured in the machine shop. NBTE (2003) specified the following as objectives of mechanical engineering craft: (i) to stimulate and sustain students’ interest in mechanical engineering craft; (ii) to enable students acquire useful knowledge and practical skills in mechanical engineering craft; (iii) to prepare students for further learning in mechanical engineering craft; and (iv) to prepare students for occupations in mechanical engineering craft.
In vocational and technical institutions, assessment of students’ learning in relation to achievement of the four stated objectives of mechanical engineering craft is carried out following teachers’ delivery and NABTEB at the final examination. In Mechanical Engineering Craft, the researcher observed that the assessment instrument used by NABTEB only helps to determine students’ achievement of two out of the four objectives of mechanical engineering craft, which are cognitive and affective domains. There are negligible observable results for the achievement of objective (ii) and (iv) which are in the area of psychomotor domains. Kaide (2013) observed that mechanical engineering craft practical examinations conducted by NABTEB and teachers is mere rating of products and not procedures of production. In the same vein, Ombugus (2013) stated that in product rating; marks are awarded based on mere observation and rating the end results of students’ activities. Earlier Eggah (2010) considered the approach as subjective and prone to abuse by the raters. To achieve observable skill acquisition by students of mechanical engineering craft at graduation, there is need to assess their performance through well developed and validated psycho productive skills multiple choice test.
Psycho productive skills multiple choice test in Williams (2012) is an instrument for determining the extent to which students can demonstrate their practical competencies in mechanical engineering craft using production process skills multiple choice items. Psycho productive skills multiple choice test is a device with process skills multiple choice items to be responded to by learners. It connote the presentation of series of process skills multiple choice items to be answered by students not necessary in the workshop. This study is therefore designed to develop and validate psycho productive skills multiple choice test to complement the present product rating method used by teachers and NABTEB to enable students demonstrate the acquisition of production skills in mechanical engineering craft and probably practice them at graduation. The relative advantage of the psycho productive skills multiple choice test over rating scale in Okeme (2011) is that the test can be used to recruit competent technical personnel online.
In technical colleges in Nasarawa State, like other states in Nigeria, NABTEB has been accorded the responsibility of assessing the performance of students in mechanical engineering craft. The examining body has been using product evaluation technique in form of rating scale. This is done at the expense of judging the production process of students through psycho productive skills test. The structure of NABTEB objective questions seems to favor the cognitive domain and does not sufficiently address the psychomotor aspect (Hersbatch, 2011). The practice of the examining body according to Bukar (2012) has given room to product evaluation without students’ process skill development, that is, the objective of mechanical engineering craft is not fully achieved. The teachers assess students during workshop instructions while NABTEB assess students at the final examination. In a vocational/technical education program like mechanical engineering craft, students’ effort is geared toward learning the theoretical principle in order to satisfy the requirement of NABTEB for the award of National Technical Certificate. There are limitations associated with product rating in measuring skills in vocational/technical education programs as pointed out by Ogbu (2011). The author concluded that product rating does not measure process skill development and students can get assistance from outside to produce products presented for final assessment. The teachers and the external examiners merely observe and rate the finished mechanical engineering craft projects produced by students instead of judging the production process skill acquired by the students. The assessment practiced by the teachers during class instructions and final examination by NABTEB have produced graduates of mechanical engineering craft that are unemployable in the field. This explains why many of the graduates are into commercial motorcyclist, popularly known as Okada riders. The fact that a student submits finished product does not constitute a positive proof that he can actually and sequentially cut the parts with acceptable degree of skills himself. The product rating method used by the teachers and examination bodies in measuring performance of the students is defective.
In addition to the limitations above, rating scale involves the bias of the rater, for example, if three raters rate the same product it is quite unlikely that their scores will be the same. The difference in score is due to the bias, resulting from difference in perception of individuals. This, in effect makes it impossible for the achievement of all the objective of mechanical engineering craft as stated in NBTE (2003). The present assessment practice does not ensure that the students of mechanical engineering craft are taught the proper way of carrying out tasks in mechanical engineering craft. If the assessment by the teachers and examining bodies had included process skill development and students are successful as claimed through their results, they should be able to demonstrate some acquired manipulative skills in some relevant mechanical engineering craft occupations. The incompetency of the graduates could be attributed to the wrong scores and conclusion about students’ performance obtained from invalid and unreliable measuring instrument. The method of assessing students which properly affected the instruction by their teachers has also prevented the acquisition of skills by the students. The mechanical engineering craft graduates lack a range of skills and in-depth competence for securing and sustaining employment to make a living from mechanical engineering craft trades. However, the present assessment practice seems invalid and unreliable to provide solution to the observed product evaluation problems. In the absence of valid and reliable instrument for assessment of manipulative skills, the obvious consequence is that students and teachers may make decision in mechanical engineering craft based on data obtained from inappropriate assessment instrument. Hence the need to develop and validate a reliable instrument for assessing acquired skills in mechanical engineering craft at the NTC level.
The major purpose of the study therefore, is to develop and validate psycho productive test items in mechanical engineering craft for assessing skills of students in technical colleges in Nassarawa State. Specifically, the study developed and established the reliability of the psycho productive test items in mechanical engineering craft (grinding, drilling and fitting operations). Two research questions were answered by the study and one hypothesis tested.
The study employed the instrumentation research design, which deals with the process of developing instruments for assessing students’ performances. The study was carried out in Nassarawa state and covered the four technical colleges in the state. The population of the study was 248 NTC III students in government technical colleges in Mada Station, Assakio, Agwada and Federal Science and Technical College Doma comprised three ability groups (high, average and low). The population consisted of 210 males and 38 females within the age of 14 and 18 years. There was no sampling because of the small size of population. The instrument for data collection was 160 psycho productive skills multiple choice test items developed from the mechanical engineering craft curriculum of technical college. Three areas of the curriculum where psychomotor skills are predominant were selected. These were grinding, drilling and fitting operations. The development was based on the table of specifications and Simpson’s taxonomy of psychomotor Domain (Simpson, 1972) with the following levels (Table 1).
The items developed covered the area of grinding operation- 42 items, drilling operation- 50 items and fitting operation-68 items.
Three types of validation, face, content and criterion- referenced were involved. Copies of the psycho productive skill multiple choice test, the table of specification, identified areas of the NTC mechanical engineering craft curriculum and a four-point rating scale questionnaire were submitted to five experts; three were subject matter expert and two experts in measurement and evaluation unit, all from Faculty of Education, University of Nigeria Nsukka. The respondents were asked to rate the items using a four-point response option of Highly Essential (4); Average Essential (3); Slightly Essential (2); and Not Essential (1). A mean of 2.5 was used for each item on the questionnaire to possess face and content validity. The criterion-referenced validity of the instrument was determined by comparing the scores of the students to the predetermined Simpson’s recommended scores of at least 1/3 of all the items in perception, set and adaptation levels and 2/3 of guided response, mechanism and complex overt response levels. The split-half technique and Kudder-Richardson (K-R20) were utilized to determine the internal consistency of the scores. The correlation yielded co-efficiency of 0.87; 0.86 and 0.88 for grinding, drilling and fitting operations respectively with overall coefficient of 0.87 depict that the instrument is reliable. The researcher made use of four research assistants to collect data through the test. Research questions 1 was answered by experts’ comments on face validation results. Kudder-Richardson K-R20 was used to answer research question 2, while the analysis of variance (ANOVA) was utilized to test the hypothesis at 0.05 level of significant.
The results of the study are presented according to headings that correspond to the research questions and the hypothesis tested as indicated below.
Research Question 1
What are the process skills items essential for developing psycho productive skills test for assessing students’ skills in mechanical engineering craft (grinding, drilling and fitting operations) at NTC level?
To answer research question one, all the test items were considered based on face, content and criterion-referenced validation results. After the validation exercise, the researcher administered the validated items on a sample of candidates. Table 2 revealed the distribution of the final test items utilized in developing psycho productive skills multiple choice test for assessing NTC students in mechanical engineering craft.
Table 2 reveals that out of the 316 skill items, 11 items were discarded while 305 items were retained and used for the psycho productive skills test development.
Research Question 2
What is the reliability of the developed psycho productive skill multiple choice test items for assessing students’ skills in mechanical engineering craft (grinding, drilling and fitting operations) at the NTC level?
The data for answering research question two are presented in Table 3.
The analysis of data in Table 3 revealed the reliability coefficients of the instrument in grinding, drilling and fitting operations on six levels of Simpson’s taxonomy. Grinding operation with 46 items had the following reliability coefficients: (1) 0.82, (2) 0.87, (3) 0.91, (4) 0.88, (5) 0.92, (6) 0.83; Drilling operation with 39 items had the following reliability coefficients: (1) 0.93, (2) 0.82, (3) 0.88, (4) 0.89, (5) 0.97, (6) 0.82; Fitting operation with 68 items had the following reliability coefficients: (1) 0.93, (2) 0.82, (3) 0.88, (4) 0.89, (5) 0.97, (6) 0.82. The overall reliability coefficient of the test was 0.87.
These coefficients were greater than 0.81 and judged as high correlation (Federer, 1992). They are also greater than 0.06 but less than unity (Zimmerman et al., 1992). From the above coefficients, the psycho productive skills multiple choice test can be said to have high reliability.
Hypothesis tested
Ho: There is no significant difference in the mean performance of the three groups of students (High, Average and Low ability) on the psycho productive skills multiple choice test in grinding operation. The data for testing the hypothesis are presented in Table 3.
Data in Table 3 revealed that the P-values of the student’s performance in grinding operation based on the six levels of Simpson’s taxonomy were (1) 0.023, (2) 0.016, (3) 0.029, (4) 0.030, (5) 0.019 and (6) 0.015. These values were less than the P-value of 0.05 indicating that there is significant difference in the mean performance of the three groups of students (High, Average and low ability) on the psycho productive skills multiple choice test in grinding operation in the six levels of Simpson’s taxonomy.
The table also showed that the calculated F-ratios for the six levels of Simpson’s taxonomy were (1) 3.792, (2) 4.161, (3) 3.572, (4) 3.541, (5) 4.008 and (6) 4.225. These F-ratios were greater than the F-critical value of 3.0317 at 2 and 672 degrees of freedom. This indicated that there is significant difference in the mean performance of the three groups of students (High, Average and Low ability) on the psycho productive skills multiple choice test in grinding operation in the six levels of Simpson’s taxonomy. Therefore, the null hypothesis of no significant difference in the mean performance of the three groups in grinding operation was rejected for the six levels of Simpson’s taxonomy.
Post-hoc analysis using Tukey-Kramer multiple comparison test was carried out to determine the direction of difference of the mean performance of the three groups (High, Average and Low ability). The test revealed that the difference was significant between the high and low ability but was not significant between the high and average abilities when compared. The implication of the result of Tukey-Kramer post-test is that the psycho productive skills test in mechanical engineering was able to distinguish between high and low ability groups in terms of their performance on the test which is a measure of the validity of the test.
The finding that 305 out of 316 skill items with content validity were considered suitable for inclusion in the psycho productive skills multiple choice test was supported by Audu (2010) in a study on Development and Validation of Decision Making Skills Inventory for secondary school principals in Plateau State; it was found out that 24 items of DMSI were found to be properly loaded and therefore dependable with minimum factor loading index of 0.35 which revealed the content validity of 24 items of the DMSI. This finding agreed with the findings of Williams (2012) in a study on Factorial Validation of Psycho-productive skill test for assessing senior secondary school students in woodwork in Ondo State, where it was found out that the content validity ratio of the items confirmed the content validity.
The findings of this study indicated that the psycho productive skills multiple choice test met the criteria of content and internal validity since the test involved performance in psychomotor objectives as recommended by Okoro (2012). The psycho productive skills multiple choice test was interested in measuring how well the test measures the manipulative learning outcomes in mechanical engineering craft covered during the 3 years NTC program. The psycho productive skills multiple choice test identified the subject matter, topics and behavioral outcomes to be measured. The test was constructed based on a table of specification which specified the sample of items to be used and was constructed to be closely fitted within the table of specification. This agreed with Ugbalu (2013) who stated that for a test to meet the content validity criteria, it must identify the subject matter topics, behavior outcome to be measured and be constructed following a table of specification which has been built to specify the sample of items to be measured.
The study also found that the psycho productive skills multiple choice test items had reliability co-efficiencies of between 0.81 and 0.97 in the six levels of Simpson’s taxonomy. This implied that the items were reliable in the six levels of Simpson’s taxonomy tested. These findings are in agreement with the findings of Okeme (2011) in a study on development and validation of psycho-productive skills multiple choice test items for students in Agricultural science in secondary schools in Kogi State. It was found out that the instrument demonstrated good reliability with Cronbach alpha coefficient of 0.83. The findings are also in conformity with the findings of Ogbu (2011) in a study on development and factorial validation of Basic Electricity Interest Inventory where it was found out that the inventory had a high reliability with Cronbach alpha coefficient of 0.74. The finding of this study on reliability is in consonance with the findings of Bukar (2012) in a study on development of an instrument for evaluating practical project in electronics in Kaduna polytechnic; it was found out that the 50 items in IEPPE had a Kudder-Richardson K-R20 reliability estimate of 0.85, which is a measure of inter-item homogeneity.
The study found out that there was significant difference in the mean performance of the three groups of students (High, Average and Low ability) on psycho productive skills multiple choices test in mechanical engineering craft. Tukey Kramer multiple comparison tests revealed that the difference was significant between the high and low ability but was not significant between the high and average ability when compared. The findings above are in conformity with the findings of Eggah (2010) in a study on Development and Factorial Validation of a mathematics test Anxiety scale for senior secondary school in Nasarawa State, where it was found out that SS 3 students had higher mean scores on MTAS when compared with SS 2 with scores of 63.5 and 61.9 for SS 3 and SS 2 respectively.
The achievement of the objective of mechanical engineering craft curriculum at the NTC level cannot be realized if all the domains (cognitive, psychomotor, and affective) are not assessed by examination bodies. The present mode of assessment of knowledge and cognitive ability achievement of students in mechanical engineering craft made the realization of skill development in students of mechanical engineering craft unachievable in technical colleges. Hence students graduate from these colleges with very little occupational entry based skills for work. This situation c
alls for the development of psycho productive skill multiple choice test to fill the gap created by the teaching and learning of mechanical engineering craft towards achieving the objectives. The inclusion of the developed psycho productive skill multiple choice test in mechanical engineering craft could therefore provide a sound basis for accurate judgment through the internet and as to whether all mentioned objectives have been achieved or not.
The study recommended the following for implementation:
1. The external examination bodies (NABTEB, WAEC and NECO) should integrate psycho productive skills multiple choice test in their examination for certification of NTC students.
2. Teachers should be encouraged by government to make use of psycho productive skills multiple choice test items during teaching and assessing students in mechanical engineering craft especially the curriculum content areas that relates to production such as grinding, drilling and fitting operations.
The authors have not declared any conflict of interests.
REFERENCES
|
Audu E (2010). Development and Validation of Decision making Inventory for Secondary School Principals in Plateau State. Journal of Home Economics Research 18(1):43-52.
|
|
|
|
Bukar J (2012). Development of an Instrument for Evaluating Practical Project in Electronics in Kaduna polytechnic, Kaduna. Nigerian Vocational Association Journal (NVAJ) 17(1):153-162
|
|
|
|
|
Eggah G (2010). Development and Factorial Validation of Mathematic Test Anxiety Scale. (Unpublished M. Ed Thesis). Department of Vocational Teacher Education, University of Nigeria Nsukka.
|
|
|
|
|
Federer JD (1992). Criterion-Reference Test: A User Resource".
|
|
|
|
|
Hersbatch DR (2011). Deriving Instructional Content through Task Analysis, USA. Journal of Industrial Teacher Education 13(3):63-71.
|
|
|
|
|
Kaide AL (2013). An Assessment of Teaching Effectiveness of Pre-Vocational Subject Teachers in Ogun State, Nigeria. Retrieved on 5th March, 2014 from View
|
|
|
|
|
National Board for Technical Education (NBTE) (2003). National Technical Certificate and Advance National Technical Certificate Programs Curriculum and Module Specification in Mechanical Engineering Craft.
|
|
|
|
|
Ogbu EJ (2011). Development and Factorial Validation of Basic Electricity Interest Inventory. Nigerian Vocational Association Journal 14(1):1-12.
|
|
|
|
|
Okeme I (2011). Development and Validity of Psycho Productive skills Multiple Choice Items for Students in Agricultural Science in Secondary Schools. (Unpublished PhD Thesis) submitted to the Department of Vocational Teacher Education, University of Nigeria, Nsukka.
|
|
|
|
|
Okoro EN (2012). Improving Students' Skill Acquisition through Industrial Technical Education in Tertiary Institutions in Anambra State - Nigerian. Vocational Association Journal 17(1):17-22.
|
|
|
|
|
Ombugus DA (2013). Development and Validation of workshop-based process skill test in metal fitting for assessing students' skills in technical colleges in Nasarawa State, Nigeria. Journal of Education and Practice 5(23). Retrieved from
View on 20th Sept. 2014.
|
|
|
|
|
Simpson E (1972). The classification of Educational Objectives in the Psychomotor Domain; The psychomotor domain, Washington, DC: Gryphon House P 3.
|
|
|
|
|
Ugbalu NM (2013). Development and Validation of Metal work Process Evauation Scheme for NCE (Tech) Students. International Institute for Science, Technology and Education. Journal of Education and Practice 5(1):63-72.
|
|
|
|
|
Williams PS (2012). Factorial Validation Psycho-productive Skills Test for assessing Senior Secondary School Students in Woodwork in Ondo State. Nigerian Association of Teachers of Technology (NATT) 22nd Annual National Conference Proceedings pp. 184-188.
|
|
|
|
|
Zimmerman BB, Sudweeks Shelley MF, Wood B (1992). How to prepare Better Tests; Guidelines for University Faculty. Provo. VT: Brigham Young University Testing Services.
|
|
