Ministry of Education

A. Problem Analysis

 1. What was the problem before the implementation of the initiative?
In Oman, the state endeavors to make education accessible to all, and aims at using education “to raise and develop general cultural standards, promote scientific thought, kindle the spirit of research, build a generation that takes pride in its nation’s heritage, and preserves its achievements”. Since the 1970s, Oman has made remarkable achievements in providing students with access to schools. From a total of three schools with approximately 900 male students in 1970, to now, 44 years later, 1,149 public schools with 524,325 male and female students. Due to the complexity of the marking operations associated with the public examinations, alternative strategies for managing the process had been suggested by examination directorate staff on several previous occasions. While a number of strategies were attempted, none provided the desired outcomes. During this time, one of the major problems encountered was the marking bias of the teacher responsible for the exams. Originally, three teachers marked each question. The first marker provided the first score for the question, whereas the second marker (checker) checked the scoring of the first, and the third marker audited the work. Problematically, however, the second marker usually agreed with the scores given by the first marker (uncritically) in most cases. Moreover, there was no adequate way to follow-up the quality of marking and the accuracy of the scores. Despite the number of reviews, the process of manually entering the marks into the computers was difficult to sustain without errors occurring. Transpositional errors, for instance, where someone entered 56 instead of 65, occurred regularly. After conducting a serious of study tours to different countries and meetings with experts, the ministry staff finally came across several promising strategies designed to address the issue. The Director General of the Educational Evaluation Directorate, invited key providers of possible e-marking solutions to Oman, including, for instance, RM, Arab Education, etc., to present their proposed solutions. Based on the reports and the recommendations proposed by these stakeholders, and after gaining the necessary approval from the Minister of Education, the Director General made an executive decision to proceed with implementing the project. The introduction of the e-marking project sorted out issues affecting an important segment of the Omani population, namely teachers and students of grade twelve. As this represents the key transition time of when the student decides on going to the university, further training or joining the workforce, the diploma examinations have a significant impact on a students’ life.

B. Strategic Approach

 2. What was the solution?
Today, the e-marking project use modern technology in the operations of correcting and scoring the student’s answers. This is achieved by using a specific technique that allows the exam papers to be printed, scanned and marked electronically, leading to the formation of a detailed database of students' performance, which can be utilized in the development of the teaching-learning process. The aim of the e-marking is to raise the efficiency of the system of evaluating the performance of students, while at the same time reduce the source of potential errors that could be introduced during the process, in so doing ensuring that the students can have a more trustworthy and fair mark. The aim of the e-marking scheme is to: 1-Create a higher level of accuracy in the marking process 2-Save time and effort in the process of marking 3-Reducing the number of teachers working in marking process 4-Provide a regulatory system which contributes to controlling and monitoring the management of the marking process both from a quantitative and qualitative point of view. The process involves the following steps: 1. Design of the question booklet so that each page of it contains a barcode and QR code including all the students’ information. 2. Students are asked to answer in the question booklet, then every booklet is to be cut from the edges and goes through scanners to take images of students' answers. 3. Usernames and passwords are issued for markers and supervisors so that they can access the e-marking program and start the scoring. At least one supervisor is allocated to each group of markers. 4. Booklets are electronically scanned 5. The students' answers to open ended questions are corrected twice by different markers, which the e-marking program compares to their estimated answers. If they give the same score for an answer the program will approve it. On the other hand, any discrepancies between the two scores will be passed to the supervisor for further validation. The supervisor can communicate with the markers directly and provide appropriate feedback. 6. The objective responses are scored automatically through machines and program. Introduction of e-marking system of such high stake exams allows the ministry to: • Ensure that scores reported are generated on the most reliable and valid basis. • Improve the ministry quality assurance mechanisms of scoring. • Reduce costs and risks related to logistics of moving exam papers from site to another. • Decrease the overall processing time for scoring. • Provide digital images and archive for examinations. • Provide detailed database of students performance results for studying and searching. • Acquire statistical reports on the completion and quality of the scoring process. Electronic marking reduced the logistical operation involving 18 centers, 25,000 markers and 60 days of marking times, to 4 centres, 5000 staff, and 30 marking days.

 3. How did the initiative solve the problem and improve people’s lives?
The implementation of this initiative can be considered creative and innovative in the following ways: - Allows the Ministry of Education in Oman the opportunity to showcase that the people in the Middle East can greatly benefit from such technological development as the e-marking represents, and in the process become a leader in electronic assessments. - Introduces incredible changes to the marking procedures in terms of time spent for scoring responses and the quality and cost of work. - Drives towards a better practices for scoring high-stake examinations.

C. Execution and Implementation

 4. In which ways is the initiative creative and innovative?
- In 2007, due to increasing concerns about the efficiency and quality of the marking procedures at the final examinations, Oman began a series of investigations and pilot tests for improving the efficiency of the examination procedures, and enhancing the overall quality and reliability of the examination outcomes. - In 2008/2009 a number of meetings were held with different providers with expertise in the field, and there were visits from the ministry staff from countries implementing such projects, including UK, Canada, and Germany. The ministry agreed to a pilot project with two distinguished organizations, namely the House of Arab Educationists in cooperation with Pearson Education Group from Australia, and Media for Computer Services and Training in cooperation with Scantron from USA. - In June 2010, the Ministry of Education undertook the first pilot of the project. The implementation took place at the Muscat Marking Center for two subjects only. Training was given to key people involved in the project, for example, teachers, students, supervisors, and ministry staff. About one month prior to the initial start, students and teachers in the piloting schools were visited by the ministry staff to show them how to deal with test booklets since it was the first time students were expected to respond in a booklet format. Prior to this they used to respond in a separate sheet of paper. Also in the same period, an assessment design phase took place in the ministry before implementation, which included some of the ministry staff. Booklets were subsequently assembled and printed in preparation of the pilot. - A second trial was conducted in August 2010 for the re-sit examinations at Al Mussan'a Marking Center only. The trailing took place on some students' answer papers in two subjects, ‘Pure Mathematics’ and ‘Biology in the General Education Diploma’. - In August 2011 during the following scoring phase, all the efforts were made to ensure that the project succeeded and fulfilled its aims and objectives. The ministry staff along with the experts worked together as a team noting down every bit of information and remarks gained from the markers. Suggestions and recommendations which had been given by scorers and supervisors contributed significantly in delivering a superlative marking experience for all involved stakeholders. - In January 2011, the Ministry of Education issued a tender for the e-marking project for examinations of grade 12 (Diploma Certificate). The implementation was for all subjects and the e-marking was done in four marking centers.

 5. Who implemented the initiative and what is the size of the population affected by this initiative?
The project purposely targeted a broad range of stakeholders to ensure input from all parties having expertise in setting up similar schemes. By the same token, the aim was also to ensure that all affected parties had a strong voice in the set-up of the solution, including both teachers and students, to ensure accountability. The large group of stakeholders involved in the implementation of the project can be listed as follows: - Staff of the Ministry of Education, especially the Director General of the educational evaluation directorate, and also examination officers from the test and exams administration department. - Teachers who worked as markers and used the e-marking program. - Students who answered the test booklets according to the rules and regulations. - Subject supervisors who worked as group leaders marking groups of students in the marking centers. - A group of the designers, programmers, and administrative staff from the House of Arab Educationists who worked day and night to make the project meet the goals and standards of the Ministry of Education. - The companies from the private sector who undertook the job of providing digital printing for the test booklets. - Public at large and communities as the beneficiaries of a more robust, efficient and effective examination process.
 6. How was the strategy implemented and what resources were mobilized?
The initiative required many resources financially, technically and, not least in terms of human resources. Ministry staff analyzed the costs of the new scheme and compared to the expenses of the traditional way. There was a tremendous difference between them. The e-marking initiative is expected to save around two million Omani Rails (5 million US Dollars) each year. Thus it constituted a reasonable investment for the ministry to fund the project. As such, all the funding was provided by the Ministry of Education. In the piloting period, most of the costs were covered by the expert providers (companies) which participated in the early experiments of the initiative. Later the government funded all the costs of the project. The costs are divided into three parts. The first part is about one million OMR and three hundred thousand OMR ($3,367,875) as a payment or service to the external contractor. The other parts of the balances of the costs are divided between payments for the scorers and the administrators. The payment for scorers’ amount to about five hundred thousand OMR ($1,295,336), and the administrator’s payments amount to around one hundred thousand OMR ($259,067).

 7. Who were the stakeholders involved in the design of the initiative and in its implementation?
There were a number of outputs that contributed to the success of this initiative. The most important ones are as follows: 1. Dramatic Reduction of Human Errors – elimination of almost all human errors in the process of calculation of final raw scores. 2. Greater Speed and Accuracy – the whole correction process has been reduced from 18 to 14 steps. Manual marking used to consume around 60 days. Now, by using e-marking, the whole task is completed within only 30 days. The accuracy and the integrity of the data is also greatly improved in the process. 3. Significant Reduction of Costs Associated with Logistics – a noteworthy sum of money has been saved by removing logistics of ''moving papers'', and also in the number of scorers/markers, as well as the marking centers, in so doing making it easier to deliver instructions and follow-up with the markers whence needed. 4. Enhanced accountability and Transparency – it is now possible to trace each step of the marking process, and examine who made decisions at different points. 5. Availability of Detailed Data – students' results/scores of each item in the tests are available to be examined in detail, which can be utilized for analysis and studies. 6. Digital Archives – all test booklets from students are digitally archived, thus ensuring that records are protected and used in case a dispute would arise at any point.

 8. What were the most successful outputs and why was the initiative effective?
From the beginning of the project committees were formulated for the purpose of monitoring the progress of initiative. In this set-up, the main committee was chaired by the director general of the educational evaluation directorate. Other committees were also formed to follow up issues such as the financial procedures with the government, the technical side of the project, and for providing the infrastructure needed. Another committee was for the management of human resources involved in the project. These committees met on a regular basis, approximately twice a month or when they needed to report their findings and evaluations to the main committee. Different people were involved in these committees such as ministry staff, examination officers, and some subject supervisors. Just before the printing of booklets started, there was also the designing committee, which was responsible for designing the exam booklets. This committee included a number of specialists, including also national staff. During the marking phase there were a number of committees working on site to receive the booklets from the schools to scan and store booklets. A technical committee was also formed to follow up the progress of marking, which included a specialist from each subject.

 9. What were the main obstacles encountered and how were they overcome?
A key challenge was the knowledge transfer associated with training national staff to be able to take ownership over all the operational parts of the project. The idea was to include some national staff in each phase of the project, and thereby gradually allowing them to acquire the necessary skills and knowledge to run the project. Staff worked in the. This is a continuous process, which takes trial and error of being exposed to all the different processes like design, operations and the technical aspects of the project dealing with the machines used in the project. Another issue encountered was the lack of a suitable infrastructure for deploying such a comprehensive scheme. The lack of Internet speed to accommodate the requirements of the project, especially outside the capital city of Muscat, presented a major challenge. Efforts were made to overcome this problem from both the Ministry of Education and the main internet service provider. Close collaboration and coordination between these parties, eventually resulted in securing an internet lease line for the four marking centers involved.

D. Impact and Sustainability

 10. What were the key benefits resulting from this initiative?
The introduction of this project has made a great contribution towards improving the service that the Ministry of Education deliver to stakeholders and users such as schools, teachers, researchers, and students. By being able to reduce the steps required to mark, significant time is saved. More specifically, introducing this public service has effectively sliced the marking time almost in half from around 60 days to 30 days today. As a result, students can obtain their scores/marks earlier, making it possible for them to know how they are doing, and consequently adjust their performance accordingly. Moreover, e-marking also helped improve the administrative work that related to students results. For instance, it is now possible for schools and researchers to obtain detailed reports regarding the results and performance of students on item level. With regard to administration issue, it is now easer to extract a student's booklet using the digital archive for purposes of appealing and remarking. Ministry staff used to spend time storing and organizing the test booklets after the marking was over, but this is now stored in one hard disk, making the administrative burden of keeping hard copies of the students responses redundant. Feedback from teachers involved in marking has also been positive, as the user-friendly e-marking program has helped making their job easier. Subject supervisors and senior teachers working as group leaders also find it easier to check the performance of the markers, and monitor the quality of the marking process. An important element for the Ministry of Education throughout this process has been to measure the level of satisfaction of the stakeholders using the e-marking program. Accordingly, a questionnaire was designed during the piloting phase of the project. Gathering of information has continued throughout the implementation of the project and is still used to continuously improve the service for the users. A number of suggestions and recommendations stated by different stakeholders have been implemented at different stages. In addition to the very significant efficiency gains that have already been described there have been equally important improvements in terms of data quality and accessibility.

 11. Did the initiative improve integrity and/or accountability in public service? (If applicable)
The e-marking project is considered to be an important requirement for the development of the educational evaluation process in the educational system in Oman. There is total support and desire from the ministry for continuing the project. On the one hand it has been a great opportunity for creating the necessary knowledge transfer from international and local experts to the ministry staff. This combined with the drive from the local communities and teachers in using the latest technology in the field of marking operations can help ensure the sustainability of this project for many years to come. The promising results of the project have encouraged the ministry to implement e-marking in other grades. Moreover, the community of teachers shows positive attitudes towards the project and made effective cooperation during implementation which will likely be important element for the continuation of the project. The existence of positive attitudes from local communities, governmental institutions and the state parliament (Shura Council), all provide a strong endorsement for the project. With the support described above and the successful experience it has been possible to extend the e-marking system to marking of national examinations, and plans are for it to be used for the international assessments in which Oman is a regular participant. During the second phase of the implementation, the ministry received delegates from the countries in the region. A group of examination officers from the Ministry of Education from Kuwait came to observe the implementation in January 2012. In June a team from the assessment authority of Bahrain visited the country to view the experience. In both cases, the delegates were most impressed by the functioning of the solution, and consequently consider a similar set-up in their countries.

 12. Were special measures put in place to ensure that the initiative benefits women and girls and improves the situation of the poorest and most vulnerable? (If applicable)
Shifting from the old manual process to the computerized process represents a challenge that had to be overcome at each stage of the pilot and the following implementation. It is a logical step for the continual development of the educational system, and key part of the countries vision to build an inclusive digital landscape. Throughout this experience it has been vital to take in account the impact that such an innovation would have on the local communities and society at large. In fact, it is all about creating the best benefit of the technology in a way that it suits the requirements of the user, namely the students, teachers, and other stakeholders involved in the process. In short, the continued efforts would enable for the employment of technology in the field of assessment.

Contact Information

Institution Name:   Ministry of Education
Institution Type:   Government Agency  
Contact Person:   Zuwaina Al Meskary
Title:   Dr.  
Telephone/ Fax:  
Institution's / Project's Website:  
E-mail:   zuwaina@moe.om  
Postal Code:  
City:   Muscat

          Go Back

Print friendly Page