Since the inception of the Institute in 1961, programme evaluation was done manually by administering questionnaires also called “happy sheets” suggesting that participants might have given favourable comments. Further, there was a likelihood of the facilitator tracing a participant who apparently gave negative feedback. The questionnaire was also too lengthy and most participants did not respond to all questions. There also was much paperwork and tedious analyses especially in large classes before the programme coordinator wrote a report. This led to delayed actions in response to participants’ feedback across departments.
In an attempt to have quality participant feedback provided at the end of the training, there was need for departments to have timely, accurate, reliable, sufficient and readily accessible information provided through an integrated online evaluation system. This would improve efficiency and effectiveness in respective operational areas.
The end-of-event evaluation (KAPES) which is an online process of capturing participant feedback at the end of all KIA training was therefore developed in 2009. This was done on the premise that participant feedback should be acted upon promptly for service improvement.
An electronic data capturing portal was developed by the end of 2009 to capture participant feedback for analysis and reporting. Participants log on to the portal and give their feedback by rating a set of six multiple-choice questions on a scale ranging between Poor (1) and Excellent (5). The portal also gives an open-ended question to allow participants freely make any other comments not covered in the rated questions. The mode of feedback on the portal is set anonymously and automatically, which means that one cannot identify the source of the comments nor the portal administrator. This gives participants freedom to give their very honest feedback.
In order to ensure successful implementation of this system, some changes were made as follows:
i. The evaluation form was re-casted to capture essential areas of operation by reducing the number of questions in the evaluation form from 16 to 6 questions;
ii. The evaluation system was re-engineered so that all the service departments could directly access participant feedback online;
iii. All programme coordinators and assistants were trained to administer the evaluation process effectively, and
iv. Evaluations are systematically scheduled on the last day of the programme
By the end of 2010, a comprehensive review was undertaken to establish the effectiveness of the online evaluation system based on the feedback from the users. Recommendations thereof were used to reengineer the KAPES to meet users’ needs. This led to a more user friendly system that has been in place throughout 2011.
Flow Chart Diagram (Before and After)
To date, 3,800 participants have successfully evaluated KIA services as evident in the portal. The data collected is used by the KIA management to make decisions accordingly. This ensures that the same complain is not raised again which would indicate lack of management response.
Current reports on the portal show that participant satisfaction is above 85%. Aspects such as programme coordination, facilitation, quality of food, meeting objectives and customer expectations on average receive excellence ratings. The cleanliness of both lecture and residential rooms has not been as highly rated.
The ICT section realized the need to train some users of KAPES such as programme assistants who are involved in participant satisfaction audit. This has enabled them perform the exercise diligently and interact more with participants as they undertake the exercise. This also helps them capture the mood and perception of participants about the institute in general.
Based on user feedback, a further review is scheduled to be undertaken in 2012