Cost-Optimized Intelligent Document Extraction
A regulated income-verification platform runs an intelligent document-processing (IDP) service that turns paystubs, tax forms, and eligibility documents into validated, structured fields. It worked - and it was expensive, because every document paid full managed-frontier prices for capability most documents never needed. I led the re-architecture that moved the common path onto self-hosted open-weight models and open-source OCR, keeping the frontier model as a confidence-gated escalation for the hard tail. This page walks through the pipeline, the cost call, and how quality stayed non-negotiable.
Hire Me
What the Service Does
Benefits-eligibility and income-verification workflows generate large volumes of document images - paystubs, W-2s, 1040s, enrollment certifications, and work-requirement exemptions. The service turns those pixels into clean, typed fields: it rejects images too poor to read before spending money on AI, recovers text, classifies the document type, and extracts the fields that matter for that type - each with a confidence score and verification flags so downstream consumers know which values to trust. Documents are the fallback path when a benefit recipient cannot be verified directly through a payroll or financial-data provider.
Inputs
Document images - paystubs, tax forms, enrollment and exemption certifications - submitted synchronously or as asynchronous, idempotent jobs.
Outputs
Validated, schema-typed fields with per-field confidence scores and verification flags - low-confidence values auto-routed to human review.
Architecture
A multi-tenant Python FastAPI service on AWS, fronted by API Gateway with machine-to-machine Cognito auth, DynamoDB and S3 for job state and document storage, and Step Functions for asynchronous work. Every document runs a four-stage pipeline: a quality gate rejects blurry, rotated, or multi-page scans before any paid AI runs; OCR recovers text; a model classifies the document type; and a schema-driven pass extracts category-specific fields with guaranteed-valid structured output, followed by internal verification against the OCR text.
The document schemas live in a registry - adding a new document type is a matter of adding a field-schema module, and the classification and extraction prompts are generated from it. That registry is the constant; what changed underneath it is which models do the work.
The Cost Call
The original pipeline used a managed OCR service and a managed frontier model for classification and field extraction. It was fast to build and it worked. It was also billed per page and per token, sized for peak capability - and an audit made the problem plain: the platform was paying full frontier-model prices for a level of capability that most documents never needed. A clean paystub or a standard W-2 does not require a frontier model to read; the hard cases do.
So I made the call to split the workload. The common path - the bulk of well-formed, in-distribution documents - moves to self-hosted open-weight models and open-source OCR running inside our own compliance boundary. The managed frontier model stays, but only as a confidence-gated escalation: when the cheap path is not sure enough, the document escalates to the expensive one. You pay frontier prices only for the documents that actually earn them.
Two things made this defensible rather than reckless. First, self-hosting keeps sensitive applicant data inside the VPC rather than shipping it to an external endpoint - in a regulated income-verification workload, that is a compliance win as much as a cost win. Second, the cut was evidence-based: nothing shipped until a labeled evaluation set proved the cheaper path held extraction accuracy at the frontier baseline. Cost optimization that quietly degrades quality is not a win - it is a liability moved off the invoice and onto the applicant.
Production Patterns
- Confidence-Gated Model Cascade
Every document hits the cheap self-hosted path first. Each extracted field carries a confidence score; when the aggregate confidence for a document clears the threshold, the result stands. When it does not, the document escalates automatically to the managed frontier model.
This is the pattern that makes the cost cut safe: the platform spends frontier dollars on exactly the documents that are ambiguous, damaged, or out of distribution - and nothing else.
- Self-Hosted Open-Weight Extraction on vLLM
The common path runs a Qwen open-weight vision-language model served on vLLM, with grammar-constrained (guided) JSON decoding so the extractor always returns schema-valid output - the same guarantee the managed frontier model gave through forced tool-calling, now without the per-token bill.
Serving is right-sized: batched inference and scale-to-zero on idle keep GPU cost proportional to load rather than provisioned for a peak that rarely arrives.
- Open-Source OCR and a Cheap Quality Gate
The managed OCR service is replaced on the common path by open-source OCR (PaddleOCR / Tesseract class) behind an OpenCV quality gate - Laplacian-variance blur detection, deskew, rotation and page-count checks - that rejects unreadable images before any model runs. The cheapest inference is the one you never make.
- Managed Frontier as a Governed Fallback
The frontier model is not deleted - it is demoted to the escalation tier and wrapped in the same resilience the rest of the pipeline uses: per-operation timeouts, exponential backoff with jitter, and a clean split between retryable and terminal errors mapped to stable API error codes. The cascade degrades gracefully instead of failing hard.
Cost and Quality Discipline
Targets gated by a labeled evaluation harness in CI - no cost change ships if it regresses quality:
- Route the large majority of in-distribution documents through the self-hosted path; escalate only sub-threshold-confidence documents to the frontier model.
- Order-of-magnitude lower marginal cost per document on the common path versus the all-frontier baseline.
- Hold extraction accuracy on the benchmark set at or above the frontier baseline - zero-tolerance for silent quality regression.
- Keep applicant data inside the compliance boundary on the self-hosted path; preserve per-field confidence scoring and human-review routing.
What Leadership on This Build Looked Like
The highest-leverage thing I did here was the judgment call: where a cheaper open-weight model is genuinely good enough, and where a frontier model still earns its keep. That call is only credible because I have been writing software for two decades - I know what these models are good at, where they quietly introduce risk, and how to prove the difference instead of guessing at it.
Player-coach in practice meant partnering with my US-based data science team on model selection and the evaluation harness, owning the cascade and fallback architecture, and holding the line that no cost win ships without passing the quality gate. The result is a pipeline that spends frontier money only where it changes the answer - and a team that can defend every dollar of it.