Questions/Answers

This initiative deploys artificial intelligence (AI) to analyze retinal images of patients with diabetes to detect those with vision-threatening complications known as diabetic retinopathy (DR) and refer them for timely treatment to prevent blindness. DR is a complication associated with diabetes and occurs as a result of long-term damage to the small blood vessels in the retina. In 2010, it caused 1.9% of moderate or severe visual impairment globally and 2.6% of blindness. This was an alarming increase of 64% and 27% respectively, from 1990 to 2010. In Thailand, a middle-income country, DR is the third most common cause of blindness and is the only cause of diabetic blindness. There are approximately 4.5 million people with diabetes compared with 1,500 ophthalmologists. In addition to this shortage of trained doctors, access to expert care is also a challenge. While half the ophthalmologists reside and practice in the country’s capital, about 85% of the patients live outside this area in more suburban and rural settings. Although screening programs for DR have been conducted for many years, only half of the patients access the program. A national survey in 2013 pointed out that almost 10% of avoidable visual impairment in Thailand was due to DR, however, this number is likely an underestimation. Since the number of patients with diabetes in Thailand is expected to rise dramatically in this new decade, DR, if not properly managed, will continue to be a major public health concern. It is in this context that AI has the potential to further enhance screening systems and was introduced within Thailand’s nationwide DR screening program. The deployment of AI with the objective to expand patient access to expert care and prevent blindness from diabetes may be a model from which many countries can learn from.

This initiative promotes state-of-the-art, AI-based technology to increase access to standard of care to patients with diabetes, especially those in rural areas. We have transformed eye screenings in primary care, community hospitals by introducing digitized workflows with AI. Typically, non-communicable disease personnel manually conduct eye screenings and wait days to weeks before an ophthalmologist reviews the retinal images for patient referral. AI integration into current screening workflows using existing digital retinal cameras has enhanced effectiveness and efficiency of routine service delivery. Patients receive accurate, real-time results, and early diagnosis by AI along with digital tracking for referral and follow-up care.

This initiative aligns with SDG 3: Ensure healthy lives and promote well-being for all at all ages, Target 3.8: To achieve universal health coverage, including access to quality essential healthcare services and access to safe, effective, quality and affordable essential medicines. This initiative provides high-quality eye screening to prevent blindness for patients with diabetes. Diabetes can involve patients of all ages and all genders and is one of the major non-communicable diseases that cause health burdens across all socioeconomic statuses. This initiative extends the existing universal coverage of patients with diabetes who have already been covered for treatment in Thailand to have more accessibility for screening. Since this project is a collaboration between the government of Thailand and Google Health, it aligns with SDG 17: Strengthen the means of implementation and revitalize the global partnership for sustainable development. It corresponds to Target 17.16: Enhance the global partnership for sustainable development, complemented by multi-stakeholder partnerships that mobilize and share knowledge, expertise, and technology, to support the achievement of the sustainable development goals in the prevention of blindness in Thailand, and also Target 17.17: Encourage and promote effective public, public-private partnerships, building on the experience and resourcing strategies of partnerships.

This initiative creates equity in prevention of blindness for all patients with diabetes. Blindness causes a severe burden on the quality of life of the patients; therefore, this initiative should play important roles in social sustainability. From the perspective of care providers, AI decreases workload of healthcare personnel and causes sustainability of human task forces. In economic terms, AI in this initiative is proved to be cost-effective in a standard health-economic analysis (see Evaluation Report). It was implemented into existing screening programs using existing resources of the program. This should cause economic sustainability. There is no issue with environmental problems.

This initiative addresses the significant shortfalls of public services for screening of retinopathy in patients with diabetes in Thailand. These shortfalls include 1) the delay in receiving eye screening results (from days to weeks), 2) the limited performance of the trained personnel who read retinal images to detect referrals to ophthalmologists (accuracy not higher than 85%), and 3) the limited ability to track patients with vision-threatening retinopathy for treatment. Patients receive results immediately in this initiative. The robust performance of AI in terms of diagnostic accuracy, approximately 95%, should detect more patients requiring treatment. Supporting the trained personnel with AI provides them with more time to focus on other important tasks, such as health education or monitoring patient adherence. In addition, AI should work seamlessly with end-to-end digitized workflow of the screening processes providing opportunities to track patients who were referred using digital technology. Digital tracking of referred patients could be monitored at both primary care and tertiary care. On the other hand, the patients who had complete treatment could be referred back to the primary care for monitoring. Monitoring of patient sustainability to the screening program in the following years could also be conducted using current digital tracking technology.

The prevalence of diabetes mellitus (DM) is 5.7%, with a slight difference between males (5.5%) and females (5.9%). Some studies have found type 2 DM to be more common in males while females tend to have more complications. Meanwhile, type 1 DM is usually found in the younger age group (<30 years) of both genders. This initiative includes all patients with diabetes in the screening program. In the prospective phase of this initiative, 5007 out of 7940 patients were females (63%) and 457 patients (5.7%) were type 1; therefore, no females or children were left behind.

The target group is a population of patients with diabetes in Thailand. According to the results of our retrospective data analysis, AI is 25% more sensitive in detecting patients with retinopathy severe enough to be referred to ophthalmologists, compared to trained human graders (97% vs. 74%). The Thai human graders were found to be slightly more specific for detecting these high-risk patients (96% vs. 98%). In the prospective, real-world, phase, AI could provide the screening results to all 7940 screened patients immediately without failure. AI was able to refer 2,461 patients to ophthalmologists (31%), 515 patients as vision-threatening retinopathy (VTDR) (6.5%). There were only 28 misdiagnoses of VTDR (0.8%) detected by the overreader. These prospective detection of VTDR and the misdiagnoses were not measurable in the current system without AI in Thailand. Since the sensitivity of the trained graders in detecting VTDR was not optimal, many patients without VTDR were conservatively referred to ophthalmologists in the current system. AI in this initiative was tuned to sensitively detect VTDR; therefore, much lower proportion of patients were missed. From the survey, qualitatively, AI was generally well accepted by both patients and personnel in this initiative.

There are 3 steps in the current workflow of DR screening: patient consent, retinal analysis, and tracking referrals and adherence. This initiative applied cloud-based AI to create efficiencies and accuracy in the retinal analysis step. The end-to-end digital workflow allows easier monitoring of patients and of the performance of AI compared with manual monitoring and evaluation by humans. The chronology of how DR screening was established in Thailand is as follows: 2006, we conducted and published a study evaluating diagnostic performances, such as sensitivity and specificity, of trained local healthcare personnel at primary care units including nurses and technicians in grading retinal images to identify patients with DR to be referred to ophthalmologists. 2007, we started training courses for local healthcare personnel in local areas to screen DR using retinal photographs in their areas. The courses were then expanded to cover most parts of Thailand with an aim to keep the accuracy of the trained personnel at 85%. 2012, this project of training local personnel to prevent diabetic blindness won the second place of UNPSA in the category of Knowledge Management in the Government. 2014, the Ministry of Public Health of Thailand endorsed screening programs for DR in all health regions using the indicator of 60% of patients with diabetes screened for DR. The Ministry also listed bevacizumab, a cheap off-labelled medication for treatment of DR in the National List of Essential Medication for patients who were indicated without cost. 2017, a plan of using AI for screening DR was started with a retrospective validation on the AI’s performance in Thai population. 2019, a prospective study on real-world deployment of AI in existing screening programs in Thailand was started.

The retinal image analysis by trained personnel faced obstacles from workload since they were not full-time employed for this task and required continuing education to maintain standard performance. Many personnel was not confident in grading retinal images; many images were still sent to ophthalmologists to grade. This caused a delay in reporting screening results. These obstacles were overcome using AI which generated results immediately, with better accuracy and consistency. In the real-world phase, ungradable retinal images, inconsistent internet connection, and poor referral adherence can be addressed by improved capture techniques, secured private internet connection, and improved digital tracking.

This initiative is innovative in deploying AI for automated detecting vision-threatening DR in the screening program in Thailand. This AI algorithm was developed and validated in datasets of patients with diabetes in other populations with robust performance. This initiative validated that this performance was confirmed in the Thai population, which has never been done before. We also conducted a head-to-head comparison between AI and the trained graders and also evaluated the overall performance of this AI in real-world settings. Thailand is one of the first few countries in the world where AI in health care was evaluated retrospectively and prospectively.

This initiative is a continuation of the initiative “Preventing Diabetic Blindness by Local Trained Personnel” which won the second place of UNPSA in 2012. It also drew inspiration from a landmark paper in JAMA 2016, which showed that AI developed from >100,000 retinal images of patients with diabetes, on validation with other datasets of >10,000 retinal images, could detect referrals DR with sensitivity as high as 97% and specificity as high as 93%. Prior to this breakthrough, older versions of AI could detect referrals for DR with sensitivity of 90% but specificity as low as 45%.

The initiative deploys Cloud-based AI technology without the need for additional hardware devices (we availed of the existing 700+ digital retinal cameras used in the national DR screening program in Thailand). The Cloud-based technology enabled integration of AI with many cameras at various sites at the same time without requirement of local software installation. In addition, our work takes advantage of a government-backed initiative on digital telehealth with an internet backbone. This initiative was recently developed between the National Broadcasting and Communication Commission of Thailand and the Ministry of Public Health to elevate digital competency in the healthcare system.

We visited many countries in ASEAN, for example, Laos, Cambodia, and Vietnam to discuss the potential of supporting them to conduct national eye screening programs for DR. We hosted workshops on AI for DR screening with participation by invited ophthalmologists from Indonesia and the Philippines who had responsibility for DR screening in their countries. The screening programs in these countries are not yet well established and still in stride. They are developing the programs and found the potential of using AI in the programs. This initiative was invited to be presented in many international congresses in ophthalmology in the world. This included the World Ophthalmology Congress in 2018 (Barcelona, Spain), European Retina Society Meetings in 2018 (Vienna, Austria) and 2019 (Paris, France), Asia-Pacific Academy of Ophthalmology Congresses in 2019 (Bangkok, Thailand), Asia-Pacific Vitreo-Retina Society Congresses in 2018 (Seoul, South Korea) and 2019 (Shanghai, China), Computer-Human-Interaction Conference in 2020 (Hawaii, USA). Recently, there have been many studies on AI for screening of DR published in the medical literature. However, there are few studies on deployment of AI in the real world. This initiative, especially in the prospective phase, could be a good lesson for other countries to learn from.

This initiative may be transferable to many countries in many ways. First, they may adopt the AI algorithm used in this initiative to be implemented in their own countries. The advantage is the AI algorithm is already developed and provides a reference for real-world implementation using lessons learned from our deployment experience. While the countries may not need to develop AI from scratch, they may need to validate its performance in their patient populations, similar to this initiative. In our case, since we validated this AI in the Thai population, it is likely that other populations in Asia should achieve similar results. Second, they may use other commercially available AI. However, the countries also need to validate if the algorithm performs well in their countries. Third, the countries may develop their own AI algorithm from scratch. They can collect their own data: retinal images of patients, develop AI algorithms and validate. The advantage of this way is that the countries own the intellectual properties of AI in full. The disadvantage is they need to collect an abundance of data for development and validation. These processes should be conducted carefully with acceptable science, technology, research design, and real-world evaluation.

Since the program for DR screening has been conducted as a national program in Thailand since 2013, resources for the screening have already been deployed by the Ministry of Public Health for many years. Patients with diabetes in Thailand are screened for retinopathy without cost. However, the cost for applying AI in the national program is adding only the cost of the software in exchange for the cost of the trained personnel. Except from this, there is no additional cost for deploying AI in screening programs for DR and there is no cost of using AI in this initiative.

The ease and practicality of AI make this initiative sustainable. There is much lesser manpower required for the screening. In addition, the positive sentiments towards AI by personnel who used it and patients who were screened by AI based on interviews conducted by experts on computer-machine interactions is another evidence suggesting sustainability of AI for screening of DR in Thailand. A few studies by other countries have reported AI to be cost-effective in their screening programs. A separate study on cost-utility analysis of AI based on our retrospective data was also conducted and the cost-effectiveness was confirmed in Thailand’s perspectives.

是

AI in this initiative was evaluated externally by College of Medicine, Rangsit University, a private medical school in Thailand. The evaluation was conducted based on both the retrospective and the prospective phases of this initiative. For the retrospective phase, the evaluation was based on a paper published in the journal Nature Digital Medicine; for the prospective phase, the evaluation was based partly on a study presented at the Computer-Human Interaction (CHI) Conference 2020 in Hawaii, USA. The evaluation was also based on the analysis on health economics conducted by the Department of Pharmacy, Mahidol University, Thailand.

For the retrospective phase, diagnostic parameters, such as sensitivity, specificity, positive predictive value, negative predictive value, accuracy, and area under the curve, were used for evaluation of the AI. For the prospective phase, outcome measures in the real world, such as the number of referrals, the number of vision-threatening DR, image gradability, time for AI analysis, the number of analysis failures, the number of false negatives detected by over-reading were also used. Qualitative analysis on using AI was presented in the CHI paper. For health economic evaluation, Incremental Cost-Effectiveness Ratio (ICER) per Quality-Adjusted Life Years (QALY) was used.

In the retrospective phase, sensitivity and specificity of AI were 97% and 96% respectively whereas those from human graders were 72% and 98% respectively. In the prospective phase, there were 1314 patients with image ungradability (17%). The AI system was able to identify 2461 referrals from 7940 patients screened (31%), 515 vision-threatening DR (6.5%), only 28 false negative was identified (0.4%) by overreading. The average duration for uploading retinal images was from 10-50 seconds depending on internet connection at the point-of-care. Once the images were completely uploaded, the results were provided immediately. There were no cases of analysis failure. For the health-economic analysis, ICERs per QALYs gained was 38,628.44 baht which is cheaper than 160,000 baht set as a cut-off point as health economical in Thailand. In summary, it is feasible to deploy AI into the DR screening workflow in Thailand. Socio-environmental and technical challenges are real barriers for scaling AI to be deployed in all health regions. These challenges can be overcome through process and system improvements. By (re)designing workflow around AI, patient outcomes can be improved and the value of AI in this initiative can be demonstrated even further.

This initiative is organized by Rajavithi Hospital, a tertiary-care hospital in the Ministry of Public Health. In the retrospective phase, we conducted this initiative with 13 hospitals, representing each of 13 health regions of Thailand. The total number of retinal images from the 13 health regions included in this phase was based on the prevalence of patients with diabetes in each region, which was >7,000 patients with >12,000 images. In the prospective, real-world, phase, AI was deployed to screen DR in 3 major provinces of Thailand across both rural and urban care settings: the capital (Bangkok), the central part (Patumthani), and the northern part (Chiangmai). AI was deployed to a total of 9 hospitals, including a primary-care clinic in a tertiary-care hospital (Rajavithi Hospital in Bangkok) and 4 community, primary-care, hospitals each in Patumthani and Chiangmai provinces.These hospitals were in collaboration with the Non-Communicable Disease Unit in the Provincial Public Health Office in each province. Hospitals in Pathumthani referred cases to Pathumthani Provincial Hospital whereas hospitals in Chiangmai referred cases to Chiangmai Provincial Hospital. The screening in Rajavithi Hospital referred cases to the retina clinic in the hospital. There were 7940 patients in this phase.

The major stakeholders are patients with diabetes and personnel who conduct the screening. In the retrospective phase, 1) Excellence Center for Retinal Disease at Rajavithi Hospital, Bangkok, which was the leader of this initiative 2) Google Health team who provided AI technology and technical support 3) Representatives of ophthalmologists and trained personnel from health regions who were to conduct this initiative, and 4) Retinal specialists and personnel from Sankara Nethralaya, one of the largest private eye hospitals in India, who would be in the panel of international specialists for adjudicated reading of retinal images as the gold standard for comparison between accuracy of AI and the trained graders, engaged in the initial design, implementation, and evaluation. In the prospective phase, the protocol drafted by Rajavithi Hospital and Google Health was presented for approval by the National Committee of Eye Service Plan, a committee of ophthalmologists from 13 health regions to administer eye care services in Thailand. Institutions in Question 10 are also stakeholders. The Institutional Review Board in each participating hospital approved research protocols in both phases. Administrators who run the screening, policy makers in the Ministry of Public Health, Thai Food Drug Administration for AI approval, are also stakeholders.

The key lesson: AI is a component of a larger picture for prevention of blindness from diabetes. More integrated digital technologies are required to create an end-to-end digital workflow for DR screening to increase patient uptake, referral, and adherence in the following years. Plans to improve this initiative: 1. Improve digital tools to engage and track patients. 2. Improve AI performance. The accuracy, though very high, can still be improved with more training to lower false positives and false negatives. The overreading retinal images within a week of screening in this initiative was a mechanism to lower these cases. 3. Expand this initiative to cover screening of other eye diseases. Since patients with diabetes may have other common eye diseases. 4. Expand this initiative to cover screening other complications of diabetes. 5. Scale up to cover more regions. 6. Registration of the AI with Thai FDA 7. Maintaining data privacy is very important. All of patients’ data and images are de-identified and not allowed to be shared or used in other projects. Patients’ informed consent is acquired for each. The possibility of false positive and negative is explained and written in the consent. All these need to be maintained.