Building local-first health AI using small open-weight VLMs

← All posts

For a discussion about the motivation and context of this project, see our companion post here.

In 2020, Datafoss supported DataKind to build a prototype tool called ROCR (Riders for Health Optical Character Recognition), a tool for extracting handwritten fields from health forms. It took a four-stage pipeline: computer vision, cloud OCR through Azure and Google, field extraction, and post-processing. We rebuilt it six years later as JAMR, using a vision-language model (VLM). Here are four key takeaways:

• The pipeline collapsed into a single model call. The new tool works by sending an image, a prompt, and a response schema to a vision-language model. We no longer need computer vision pre-processing (so far) and the post-processing validation logic we used to hand-code is absorbed by the VLM inference layer.
• The interesting engineering moved. The interesting work is now a config-driven system that automatically generates the prompt, defines the output schema, and produces an eval harness for free with every new form onboarded.
• Large models perform well, even on hard fields. Across 80 WHO COVID-19 case report forms, Sonnet 4.6 (closed-weights) and Qwen2.5-VL 72B (open-weights) saw near-perfect performance on constrained fields with no additional engineering. Qwen2.5-VL 72B achieved 0.85 accuracy on one of the hardest tasks in our eval: random, alphabetic free-form handwriting.
• Small open models can compete. Qwen2.5-VL 7B (7 billion parameters) on a 2022 MacBook Air, with sampling and aggregation at inference, matches or beats Qwen2.5-VL 72B and Sonnet 4.6 on the hardest field in our eval without fine-tuning. That means it's possible to make this tool work without shipping sensitive health data to third-parties, maintaining data sovereignty.

The pipeline collapsed into a single model call

In the ROCR tool, we extracted key handwritten information in four stages of carefully tuned algorithms and code: computer vision algorithms performing image alignment, OCR and handwriting prediction using Azure and Google cloud services, key field extraction, and bespoke post-processing. In 2020, each one took real engineering to get right. We ran JAMR on the same WHO COVID-19 case report form six years later, and the four-stage pipeline had collapsed into one: a single script sending an image, a prompt describing the fields to extract, and an expected response schema to a vision-language model.

The first time we tried this, we handed Sonnet 4.6 a photo of a filled-in form and asked, in plain English, for a few handwritten fields back. The zero-shot prompting with Sonnet 4.6 just worked perfectly. That’s hardly robust evidence, but it did encourage further development and investigation.

JAMR’s approach simplified so many pieces of ROCR. One key change is that JAMR currently does not require complicated computer vision algorithms for image preprocessing. ROCR required careful form alignment to a template and pixel-coordinate configuration in order to locate the form’s fields of interest. The computer vision algorithms required a lot of iteration to get right and it’s unclear how robust the process would have been to real-world conditions. We need to stress test typical quality of image issues found in the field (e.g. rotation, skew, bad lighting) for JAMR, but our early testing has thus far not required any image preprocessing. Another key change from ROCR to JAMR is shifting the complexity of post-processing into the VLM inference layer itself. In ROCR, turning raw OCR output into valid field values required bespoke logic. For instance, for fields with a list of expected outcomes, we computed the Levenshtein distance between OCR output text and a dictionary of candidates to snap noisy reads onto the nearest valid entry. Today, JAMR employs Anthropic’s structured outputs and Ollama’s constrained decoding to constrain the output to a predefined schema and, where applicable, valid data outputs. (More details on how this works in the next section) So,the validation and normalization we used to hand-code is now absorbed by the inference infrastructure.

JAMR: the system, not the model

The interesting engineering moved. JAMR makes a per-form config file the source of truth for everything. It drives the prompt the model sees, the schema the serving layer enforces, and the evaluation harness. Onboarding a new form means writing one config and labelling a small dataset.

Prompt. The config names each field and gives the model structured hints: roughly where the field sits on the page, any extra instructions derived through prompt engineering¹, and what kind of value it is. In the example config snippet shown, the config describes the following fields (with accompanying data constraints): Unique Case Identifier (free-form handwritten string with a regex constraint), Sex (single-choice checkbox), and Reporting Country (free-form handwritten string drawn from a known list of candidates). The system compiles these into the prompt template that gets sent to the VLM alongside the image.

Response Schema. Each field's declared type becomes part of the response schema the serving layer enforces. In this example, Reporting Country is bound to its enumerated set of options and the Unique Case Identifier is bound to its regular expression. Ollama enforces this through grammar-constrained decoding, masking any token that would violate the constraint at each generation step; the Anthropic API does it through Structured Outputs. Within those bounds the model still chooses the value, but it cannot return something outside them.

Eval harness. The config also declares what the eval measures. Each field's type determines its metric (e.g. exact match for closed-set fields and strings, Jaccard similarity for multi-label checkboxes). Onboarding a new form produces a working evaluation as a significant side effect.

The same config, any backend model

The same config runs against the Anthropic API, OpenAI, or a local model served with Ollama. The VLM is, in the tool's internal shorthand, "just another model" (hence JAMR, Just Another Model for Recognition).

That backend-agnosticism makes the data localization argument possible. ROCR leveraged Azure and Google to establish a proof of concept, but that may have been a quiet dealbreaker. For healthcare settings, putting it into production meant sending images of patient forms to cloud providers potentially out-of-country. For many health systems, that is a non-starter on data-sovereignty and privacy grounds. Cost for utilizing cloud services is also a glaring issue. JAMR lets the same config target closed-weights models (Sonnet 4.6) for proof of concept, but adds a key investigation into the performance and importance of running open-weights large models (i.e. Qwen2.5-VL 72B) and open-weights small local models (i.e. Qwen2.5-VL 7B and Qwen2.5-VL 3B).

Good form design is doing more than you might think

We ran the JAMR system on 80 WHO COVID-19 case report forms and compared the results for four backend VLM models: Claude Sonnet 4.6 (closed-weights large model), Qwen2.5-VL 72B (open-weights large model), Qwen2.5-VL 7B and Qwen2.5-VL 3B (open-weights small models). The forms were filled in by hand by three subjects with synthetic patient data to avoid health data privacy concerns; the handwriting itself is real. We compared performance on six separate fields, measuring Jaccard Similarity scores for the Underlying Conditions field and exact match accuracy for all others. A sample form is shown below. We found that all models performed perfectly on Reporting Country and Sex and the two large models performed near-perfect on Underlying Conditions on this eval set. We expect performance to degrade with model number of parameters (Sonnet 4.6, undisclosed but presumed large > Qwen2.5-VL 72B > Qwen2.5-VL 7B > Qwen2.5-VL 3B), and indeed that trend seems to generally hold for Age where Qwen2.5-VL 3B achieves 0.604 accuracy where the remaining models achieve 0.925, 0.938 and 0.925 in increasing number or model parameters. The Unique Case Identifier fields perform the worst, with Sonnet 4.6 performing the most reliably at 0.725 and Qwen2.5-VL 72B at 0.80, but were purposely designed to be excessively challenging to test the handwriting recognition limits.

One takeaway from this work is that handwriting recognition is still genuinely hard and careful form design can have an impact on JAMR performance. The table below outlines the field’s data taxonomy in order of increasing technical difficulty and decreasing overall baseline JAMR performance. Unsurprisingly, increasing constraints on the output results in an easier problem to solve. Reporting Country and Sex are closed-set classification problems with a small, known solution space, making them the easiest. Age is slightly harder because it could have been written in three different locations on the form (years/months/days), but the characters are all digits and digit recognition is one of the oldest, most-solved recognition tasks (MNIST). Even though Underlying conditions has 11 checkboxes, it allows for zero-or-more selection and thus has a large solution space. The error analysis showed frequent false positives which encourage further investigation in future work. Finally, Unique Case IDs were random free-form handwritten codes with no semantic anchors and little constraints. We purposely designed this field to be challenging to test JAMR’s limits. Despite that, baseline accuracy reached 0.85 (72B) for random alphabetic codes and 0.725 (Sonnet 4.6) for random alphanumeric codes. In production, the form could be designed to leverage our knowledge of the model’s abilities and limitations. For example, a Unique Case Identifier could carry more semantic meaning or more constrained patterns, which would likely lead to further improvement in performance as we saw when we saw improved accuracy by restricting the Unique Case ID code to only letters.

Making small open-weights local models competitive

With digital and AI sovereignty (meaning the ability for countries to independently control their own digital infrastructure, AI models, software, data, and regulations) becoming a growing global priority that Yann LeCun highlighted at the UN's Open Source Week, it is vital to invest in smaller, open models that enable AI sovereignty and not just default to large proprietary models.

Quite predictably, zero-shot single-pass runs of smaller models (Qwen2.5-VL 3B, 7B) don't work as well as larger models. The 7 billion parameter model was the largest open-weights model that could comfortably run on our MacBook Air (2022 model, Apple M2 chip, 24G memory), so we focus on this model as a proof of concept for what we can achieve with modest hardware. The performance gap between the local 7B and remote 72B model was stark at 0.29 improvement for Underlying Conditions and 0.10 improvement for Unique Case Identifier. An obvious approach is fine-tuning the 7B model, but that requires labeled data, GPUs and our users would need to engage machine learning experts to do it. We think it would be interesting to explore in future work, but for now, we explored some alternative methods to close the gap without fine-tuning.

We found we could improve small model performance by sampling multiple inference runs and aggregating the results. We applied two standard inference-time techniques: self-consistency (Wang et al. 2022), which we call majority voting, and a Best-of-N variant with the model as its own judge (Cobbe et al. 2021), which we call re-ranking. The majority voting approach takes the output from the 7B model run with k=5 samples (temperature=0.7) and selects the most common output. This approach improved 7B's performance on Underlying Conditions from 0.72 to 0.87, narrowing the gap to 72B's 0.99 accuracy. Another technique, re-ranking, also ran inference k=5 times to generate multiple candidate solutions and then made one more call asking the model to pick which of the candidates was most likely correct. The intuition behind this approach is that picking the correct answer from a discrete set of options is an easier task for language models than generating the correct answer from scratch. We found majority voting worked better for Underlying Conditions and Age fields and re-ranking for the Unique Case ID field. Indeed, the re-ranking 7B model exceeds the 72B model on the hardest field in our eval: (0.90 vs. 0.85 mean exact match accuracy) for the Unique Case ID with a random alphabetic code. A small model running on a laptop reaches, and on the hardest field exceeds, models with >10x parameters with only prompting and inference-time techniques on our eval data. It's possible this approach won't work everywhere, but it's a strong case that inference-time techniques alone can take smaller models a long way.

A drawback worth noting is that our running of the 7B on our 2022 MacBook Air laptop for one form was slow (~69.9 seconds per inference). We profiled a sample run of the extraction call using Ollama's built-in timing metrics and found that pre-fill (the one-time forward pass over the prompt plus image input context that populates the KV cache before token-by-token generation begins) accounts for 59.6 of the 69.9 seconds single image inference runtime. That’s a significant portion and would be faster on more modern hardware. It is noteworthy that pre-fill only needs to happen once, even when running model inference k times. Thus, the difference on our hardware between running the 7B model once and running it for a k=5 majority vote was 1.47x (102.45 seconds vs. 69.9 seconds). While small model sampling strategies are adding wall-clock time, a 1.47x cost for k=5 majority voting seems like a worthwhile trade-off. Re-ranking would likely require two pre-fill operations (one for the sampling passes and one for the judging pass) and result in a greater increase in wall-clock time, but we believe the trade-off may be acceptable given the benefits.

Future Work

So far we've evaluated JAMR on 80 WHO COVID-19 forms in a controlled setting. The next step is putting it in front of a real program collecting forms at scale, where the inputs are messier than our test set: rotation, skew, uneven lighting, smudges, folds, hurried handwriting and whatever else field conditions produce. We'd want to measure how performance holds up, statistical significance of our results, what breaks first, and which parts of the system need the most work. If you're running a program that could benefit from a tool like this, we'd welcome the conversation (anna@datafoss.ai).

If wall-clock time needs more aggressive prioritization, an alternative we're considering is self-distillation: fine-tuning a 7B model on the outputs of our sampling-plus-aggregation pipeline. Since the k=5 majority vote outperforms single-shot inference, those aggregated labels serve as a higher-quality training signal than the base model produces alone. If it works, single-shot inference on the fine-tuned model could approach the quality of the ensemble, collapsing k calls into one. The trade-off is maintaining a per-form-type model instance and losing the generalizability of the sampling-plus-aggregation approach, which works on new form types without retraining.

Many health systems that would benefit from JAMR are not English-first, which is where local deployment matters most. The WHO forms we tested were English, so we haven't yet measured JAMR's performance on lower-resource languages or non-Latin scripts. The natural next step is to swap in open-weights VLMs built for these languages, and see how far this approach carries for these languages.

Acknowledgements

Thanks to DataKind for suggesting the rebuild, and to the original ROCR team (2020) whose work JAMR revisits. Read more:

• “Get Back on the Road: How Riders for Health Intends to Use Computer Vision to Digitize Health Forms”, DataKind
• “Working with DataKind to save time, and lives”, Riders for Health
• “ROCR: Turning State-of-the-art OCR into Automated Form Processing”, Datafoss

---

About Datafoss. Datafoss is an applied AI and ML consultancy focused on building and evaluating AI tools that work reliably in high-stakes, low-resource settings across global health and social impact.

About DataKind. DataKind is a global nonprofit organization that harnesses data science and AI to address critical humanitarian and social challenges. DataKind partners with mission-driven organizations across education, health, economic opportunity, and humanitarian response to deliver durable solutions that work in the real world.