For a technical description of this project, see our companion post here.
Many healthcare systems are still paper-based, and digitally transforming those systems is challenging
Paper forms are still widely used in healthcare systems, especially in low-income countries. One example is how Riders for Health medical sample couriers in Liberia have to capture details in handwritten logbooks. Much more effort is required to retrieve, update, analyse, report, and secure data using paper-based systems compared to digital systems. This translates into less efficient and less effective care for patients. For this reason, there are efforts to digitally transform health systems in many countries across the world. However, this is not an easy task.
In a fully digital health system, data is entered directly into a digital form using a device such as a smartphone. While this approach can be effective, its implementation is challenging, as demonstrated for example by Kenya’s experience of setting up its electronic community health information system (eCHIS). One challenge is that every health worker will need access to a device on which to input data. This requires the purchase and maintenance of thousands of phones or tablets, as well as training thousands of staff on how to use the devices. Electricity and internet also need to be readily available to charge and sync devices, but this basic infrastructure is currently lacking in many locations, especially rural locations. Finally, entering data on a digital device can disrupt face-to-face interaction between healthcare workers and their patients, degrading the care experience.
An alternative to the fully digital approach is for health workers to continue collecting data on paper, and then for that data to be digitised afterwards. A single person or team can be tasked with digitising the data collected within each health facility or district, reducing the amount of equipment, infrastructure, and training that is needed compared to the fully digital approach. To digitise the data, a person reads the handwritten data from each field in each form, and manually copies that data into a digital system. It’s a very time-consuming and tedious task. It’s also a task that previously could not be automated because automation required overcoming the notoriously difficult technical challenge of training an artificial intelligence (AI) model to understand all possible varieties of handwriting and form layouts.
We are revisiting the problem of how to digitise data from handwritten forms
We first explored the problem of automatically digitising handwritten data in 2020, when we supported DataKind to build a prototype tool called ROCR (Riders for Health Optical Character Recognition). The tool extracts and processes handwritten information from paper forms that are routinely used in health facilities throughout Nigeria. We quickly discovered that it was very difficult to build a prototype that could perform the task to the required standard. Even when using the most sophisticated tools that were available at the time, extensive processing and calibration were needed. You can read about our experience here.
Six years on, extracting and processing handwritten information from a paper form is no longer a major technical challenge. We found it surprisingly easy to build a high-performing prototype, using the most advanced Vision Language Models (VLMs) that are available today. VLMs are a type of AI model that combine the ability of large language models (LLMs) to ‘read’ text, with the abilities of computer vision models to ‘see’ images. Using a very basic prompt, we found that one highly advanced VLM (Anthropic’s Claude Sonnet 4.6) extracted essential information about country, age, sex, and underlying conditions from a sample of 80 World Health Organization (WHO) COVID-19 reporting forms, with high accuracy.
Data from handwritten forms can now easily be processed using state-of-the-art AI models, but only if the data is sent to a remote server owned by an external provider
It is incredible that today’s state-of-the-art VLMs can so easily process handwritten information, as this is a complex task that was nearly impossible to automate just a few years ago. The drawback is that because these VLMs are privately owned, and because they need so much power, the models can only be run on huge remote servers. It is the same for LLMs. If you want to use the models, you therefore have to send your data to the servers where the model is running. It’s a setup that forces the user to relinquish some control of their data, and to be dependent on infrastructure that is owned and managed by an external provider. Governments in particular need to carefully consider the risks associated with this, including not being able to enforce local data protection laws once data has left their country, and not having any control over the prices that service providers set. The risks are especially relevant when sensitive personal data, such as health data, of their citizens is involved.
Are there any viable alternatives though? Or is it the case that a government either has to send its citizens’ data to a remote server in another country, or has to miss out on the benefits of the most powerful AI models? To answer these questions, we are turning our attention to ‘small’ and ‘open’ models.
‘Small’ and ‘open’ models can be run on your own equipment, so your data stays fully within your control
The size of a VLM or LLM is measured by the number of parameters that control how the model turns an input, e.g. your prompt and an image, into an output. The more parameters that a model has, the more versatile the model is and the more sophisticated are its outputs.
The largest VLMs and LLMs have over a trillion parameters, which is a ridiculously large number. In contrast, the smallest publicly available models have ‘only’ a few hundred million parameters. Although this is still a lot, these ‘small’ models are small enough to run on a laptop. In fact, a modest laptop can run a model with up to around a billion parameters, while a higher-spec laptop can run a model with around seven billion parameters. If you can run a model on your laptop instead of on a remote server, then all your data and outputs physically remain on your laptop. You are fully in control of both your data and the equipment that the model is running on, and you won’t even need an internet connection.
Besides size, another important distinction is whether a model is proprietary or open. Proprietary models are owned by private companies who charge a fee to use their models. Some companies may offer a level of free usage, but in exchange you must allow your data to be used for further training of the model. You cannot download proprietary models, and therefore cannot run them on your own equipment. On the other hand, open-weight models can be freely downloaded, and their weights (parameters) are publicly published. These are models that you can use freely, running them on whatever equipment you like. You can also tweak the parameters to customise the model to your needs.
With digital and AI sovereignty (meaning the ability for countries to independently control their own digital infrastructure, AI models, software, data, and regulations) being a global priority that was highlighted at the UN’s Open Source Week, it is vital to invest in smaller, open models that enable AI sovereignty, rather than just defaulting to using large proprietary models. The UN’s recent Global Dialogue on AI Governance also emphasises the need for openly available AI models and tools, to ensure that the benefits of AI are shared by all.
We’re developing a tool that digitises handwritten forms using small, open VLMs, so that healthcare staff can work more efficiently while staying in control of their patients’ data and their running costs
We know that, by themselves, small VLMs will not perform as well as larger models when tasked with processing handwriting. A large VLM will ‘just work’ when provided with a basic prompt, whereas small models require more effort to get them to successfully complete the task. However, with focused effort to optimise and enhance their performance, small open models may become a viable alternative to their larger and proprietary counterparts, especially in low-resource settings.
If a health facility is using paper forms and staff want to digitise the data from those forms, the only options are currently either for someone to do it manually, or to pay to use a large, proprietary VLM that can only be run on a remote server which means that the data must be sent to that server. We are developing a third option: a tool that can perform to a similar standard as a large VLM but that uses small, open-weight VLMs that can be run on a laptop without an internet connection. This option will enable healthcare staff to spend more time caring for patients instead of on data entry, to keep their patients’ data in the health facility where the data can be managed according to their own rules, and to avoid unexpected costs. It’s a key step to enabling healthcare systems to benefit from AI while maintaining full control of their infrastructure and data, and upholding their own values and laws.
We are developing a third option: a tool that can perform to a similar standard as a large VLM but that uses small, open-weight VLMs that can be run on a laptop without an internet connection. This option will enable healthcare staff to spend more time caring for patients instead of on data entry, to keep their patients’ data in the health facility where the data can be managed according to their own rules, and to avoid unexpected costs.
Our initial results are encouraging
Our progress so far is encouraging. We have used Anthropic’s large, proprietary Claude Sonnet 4.6 VLM as our benchmark and looked at how the performances of three smaller, open-weight VLMs compared to the benchmark, after we had optimised the smaller models. Focusing on five key fields (country, sex, age, underlying conditions, case identifier) from the WHO reporting form that we are testing (see picture below), we achieved the following results:
- A very small, open model with 3 billion parameters (which can run on a modest laptop) performs comparably to the large proprietary model on 3 out of 5 of the fields.
- A small, open model with 7 billion parameters (which can run on a high-spec laptop) performs comparably to the large proprietary model on 4 out of 5 of the fields, and even performs better than the large model on 2 of those fields.
- An open model with 72 billion parameters (which is too big to run on a laptop, but could be run on an in-house server) performs comparably to the large proprietary model on all 5 of the fields.
A small, open model with 7 billion parameters ... even performs better than the large model on 2 of those fields.
Our main results are displayed in the chart below.
Our next step is to test the tool with real users in real field conditions, and to develop the tool further to ensure it performs well in languages other than English. If you’re interested to learn more, you can find all the technical details in our research blog, or you can get in touch with us!
Acknowledgements
Thanks to DataKind for suggesting the rebuild, and to the original ROCR team (2020) whose work JAMR revisits. Read more:
- “Get Back on the Road: How Riders for Health Intends to Use Computer Vision to Digitize Health Forms”, DataKind
- “Working with DataKind to save time, and lives”, Riders for Health
- “ROCR: Turning State-of-the-art OCR into Automated Form Processing”, Datafoss
About Datafoss. Datafoss is an applied AI and ML consultancy focused on building and evaluating AI tools that work reliably in high-stakes, low-resource settings across global health and social impact.
About DataKind. DataKind is a global nonprofit organization that harnesses data science and AI to address critical humanitarian and social challenges. DataKind partners with mission-driven organizations across education, health, economic opportunity, and humanitarian response to deliver durable solutions that work in the real world.