| 4. In which ways is the initiative creative and innovative?
|
|
- In 2007, due to increasing concerns about the efficiency and quality of the marking procedures at the final examinations, Oman began a series of investigations and pilot tests for improving the efficiency of the examination procedures, and enhancing the overall quality and reliability of the examination outcomes.
- In 2008/2009 a number of meetings were held with different providers with expertise in the field, and there were visits from the ministry staff from countries implementing such projects, including UK, Canada, and Germany. The ministry agreed to a pilot project with two distinguished organizations, namely the House of Arab Educationists in cooperation with Pearson Education Group from Australia, and Media for Computer Services and Training in cooperation with Scantron from USA.
- In June 2010, the Ministry of Education undertook the first pilot of the project. The implementation took place at the Muscat Marking Center for two subjects only. Training was given to key people involved in the project, for example, teachers, students, supervisors, and ministry staff. About one month prior to the initial start, students and teachers in the piloting schools were visited by the ministry staff to show them how to deal with test booklets since it was the first time students were expected to respond in a booklet format. Prior to this they used to respond in a separate sheet of paper. Also in the same period, an assessment design phase took place in the ministry before implementation, which included some of the ministry staff. Booklets were subsequently assembled and printed in preparation of the pilot.
- A second trial was conducted in August 2010 for the re-sit examinations at Al Mussan'a Marking Center only. The trailing took place on some students' answer papers in two subjects, ‘Pure Mathematics’ and ‘Biology in the General Education Diploma’.
- In August 2011 during the following scoring phase, all the efforts were made to ensure that the project succeeded and fulfilled its aims and objectives. The ministry staff along with the experts worked together as a team noting down every bit of information and remarks gained from the markers. Suggestions and recommendations which had been given by scorers and supervisors contributed significantly in delivering a superlative marking experience for all involved stakeholders.
- In January 2011, the Ministry of Education issued a tender for the e-marking project for examinations of grade 12 (Diploma Certificate). The implementation was for all subjects and the e-marking was done in four marking centers.
|
|
| 5. Who implemented the initiative and what is the size of the population affected by this initiative?
|
|
The project purposely targeted a broad range of stakeholders to ensure input from all parties having expertise in setting up similar schemes. By the same token, the aim was also to ensure that all affected parties had a strong voice in the set-up of the solution, including both teachers and students, to ensure accountability. The large group of stakeholders involved in the implementation of the project can be listed as follows:
- Staff of the Ministry of Education, especially the Director General of the educational evaluation directorate, and also examination officers from the test and exams administration department.
- Teachers who worked as markers and used the e-marking program.
- Students who answered the test booklets according to the rules and regulations.
- Subject supervisors who worked as group leaders marking groups of students in the marking centers.
- A group of the designers, programmers, and administrative staff from the House of Arab Educationists who worked day and night to make the project meet the goals and standards of the Ministry of Education.
- The companies from the private sector who undertook the job of providing digital printing for the test booklets.
- Public at large and communities as the beneficiaries of a more robust, efficient and effective examination process.
|
| 6. How was the strategy implemented and what resources were mobilized?
|
|
The initiative required many resources financially, technically and, not least in terms of human resources. Ministry staff analyzed the costs of the new scheme and compared to the expenses of the traditional way. There was a tremendous difference between them. The e-marking initiative is expected to save around two million Omani Rails (5 million US Dollars) each year. Thus it constituted a reasonable investment for the ministry to fund the project. As such, all the funding was provided by the Ministry of Education.
In the piloting period, most of the costs were covered by the expert providers (companies) which participated in the early experiments of the initiative. Later the government funded all the costs of the project. The costs are divided into three parts. The first part is about one million OMR and three hundred thousand OMR ($3,367,875) as a payment or service to the external contractor. The other parts of the balances of the costs are divided between payments for the scorers and the administrators. The payment for scorers’ amount to about five hundred thousand OMR ($1,295,336), and the administrator’s payments amount to around one hundred thousand OMR ($259,067).
|
|
| 7. Who were the stakeholders involved in the design of the initiative and in its implementation?
|
|
There were a number of outputs that contributed to the success of this initiative. The most important ones are as follows:
1. Dramatic Reduction of Human Errors – elimination of almost all human errors in the process of calculation of final raw scores.
2. Greater Speed and Accuracy – the whole correction process has been reduced from 18 to 14 steps. Manual marking used to consume around 60 days. Now, by using e-marking, the whole task is completed within only 30 days. The accuracy and the integrity of the data is also greatly improved in the process.
3. Significant Reduction of Costs Associated with Logistics – a noteworthy sum of money has been saved by removing logistics of ''moving papers'', and also in the number of scorers/markers, as well as the marking centers, in so doing making it easier to deliver instructions and follow-up with the markers whence needed.
4. Enhanced accountability and Transparency – it is now possible to trace each step of the marking process, and examine who made decisions at different points.
5. Availability of Detailed Data – students' results/scores of each item in the tests are available to be examined in detail, which can be utilized for analysis and studies.
6. Digital Archives – all test booklets from students are digitally archived, thus ensuring that records are protected and used in case a dispute would arise at any point.
|
|
| 8. What were the most successful outputs and why was the initiative effective?
|
|
From the beginning of the project committees were formulated for the purpose of monitoring the progress of initiative. In this set-up, the main committee was chaired by the director general of the educational evaluation directorate. Other committees were also formed to follow up issues such as the financial procedures with the government, the technical side of the project, and for providing the infrastructure needed. Another committee was for the management of human resources involved in the project. These committees met on a regular basis, approximately twice a month or when they needed to report their findings and evaluations to the main committee. Different people were involved in these committees such as ministry staff, examination officers, and some subject supervisors.
Just before the printing of booklets started, there was also the designing committee, which was responsible for designing the exam booklets. This committee included a number of specialists, including also national staff. During the marking phase there were a number of committees working on site to receive the booklets from the schools to scan and store booklets. A technical committee was also formed to follow up the progress of marking, which included a specialist from each subject.
|
|
| 9. What were the main obstacles encountered and how were they overcome?
|
|
A key challenge was the knowledge transfer associated with training national staff to be able to take ownership over all the operational parts of the project. The idea was to include some national staff in each phase of the project, and thereby gradually allowing them to acquire the necessary skills and knowledge to run the project. Staff worked in the. This is a continuous process, which takes trial and error of being exposed to all the different processes like design, operations and the technical aspects of the project dealing with the machines used in the project.
Another issue encountered was the lack of a suitable infrastructure for deploying such a comprehensive scheme. The lack of Internet speed to accommodate the requirements of the project, especially outside the capital city of Muscat, presented a major challenge. Efforts were made to overcome this problem from both the Ministry of Education and the main internet service provider. Close collaboration and coordination between these parties, eventually resulted in securing an internet lease line for the four marking centers involved.
|