§Case study
← All projectsDocuFlow AI – Multi-Tenant AI Document Processing Platform
An asynchronous document processing SaaS. Upload PDFs and images, extract structured data with AI, review results, and integrate via a developer-friendly API. BullMQ workers, dead-letter queues, and HMAC-signed webhooks.
- Shipped
- Reading
- 3 min read
- Stack
- TypeScript
- Next.js
- NestJS
- BullMQ
- PostgreSQL
- Gemini
- AWS ECS
- Terraform
“Upload a document. Get structured data back. That’s the promise — the engineering is in making it reliable.”
In plain English
DocuFlow AI is a document processing platform: you upload a PDF or image, AI extracts the data you need (invoice line items, totals, dates, vendor info), and you get structured JSON back. The upload returns immediately; workers handle the heavy lifting in the background. When they’re done, a signed webhook tells your system.
The hard part isn’t the AI extraction — it’s everything else: making it multi-tenant, reliable, secure, and observable.
📦 GitHub Repository: https://github.com/amirfaisalz/docuflow
The problem
Document processing at scale has a few failure modes that demos never show:
- Jobs fail. LLM calls time out, files are corrupt, extraction returns garbage. You need retries, dead-letter queues, and visibility into what went wrong.
- Tenants leak. Without careful scoping, one organization’s documents can surface in another’s dashboard. This isn’t just a bug — it’s a compliance incident.
- Webhooks break. The receiving server is down, returns 500, or takes too long. You need retry logic, delivery history, and tamper-proof signatures.
- Usage must be metered. Every document processed, every LLM token consumed — tracked transactionally, not eventually.
What I built
Asynchronous processing pipeline
The API accepts uploads and dispatches BullMQ jobs immediately. Workers handle the pipeline independently: OCR, LLM extraction, Zod validation, persistence, and webhook delivery. Each step is observable and retryable.
- 3 attempts with exponential backoff per job
- Dead-letter queue (
document-processing-failed) for jobs that exhaust retries - Idempotency via unique
UsageRecord.jobId— replay a job, don’t double-count - State-aware updates — a job in
COMPLETEDstate can’t regress toPROCESSING
Multi-tenancy
Every resource is scoped to an organization. RBAC enforces four roles: Owner, Admin, Member, Developer. Tenant isolation is enforced at the service layer and the database layer. Accessing another tenant’s resource returns 404, not 403 — no information leakage about what exists.
Document security
- Cloudinary stores documents with authenticated (private) access
- Upload params are signed server-side
- Download URLs are short-lived and signed
- 10 MiB cap, PDF/JPEG/PNG only
- Webhook target URLs reject private/internal addresses
Structured AI output
Gemini multimodal extraction produces structured invoice data validated against a strict Zod schema. The LLM output is treated as untrusted input — parsed, validated, and sanitized before persistence. Malformed output is flagged, not silently accepted.
API keys and usage metering
- One-time
df_live_...secrets, SHA-256 hashed at rest with a searchable prefix - Immediate revocation — delete the key, and all in-flight requests using it fail
- Transactional usage metering: documents processed, LLM tokens consumed, processing duration — all recorded atomically
Webhooks
- HMAC-signed payloads (
t=<unix>,v1=<hex>) — recipients can verify authenticity - Signing secrets are encrypted at rest
- 5 delivery attempts with exponential backoff
- Per-attempt history — timestamp, status code, response time, error
Observability
- JSON structured logs with automatic PII redaction
- OpenTelemetry spans across the full pipeline: HTTP → queue → worker → Gemini → DB → webhook
/health/readywith dependency checks (DB, Redis, queue)
Production infrastructure
- AWS ECS Fargate with ALB health probes on
/health/ready - Circuit-breaker rollbacks — unhealthy tasks are replaced automatically
- Autoscaling based on queue depth and CPU
- Prisma migrations as a one-off ECS task before deployment
Architecture
Client → API (NestJS) → BullMQ Queue → Workers
├─ Cloudinary (storage)
├─ Gemini (extraction)
├─ PostgreSQL (persistence)
└─ Webhook delivery
The API never does heavy work. It validates, queues, and returns. Workers are horizontally scalable and stateless.
Status
In active development. Core pipeline, multi-tenancy, API keys, webhooks, and observability are implemented. Dashboard UI and public documentation are in progress.
Next case study ↓
03 / 2026
procure-flow
Procurement Approvals Without a Web Framework
A compact procurement approval service with an explicit state machine, RBAC, append-only audit trail, idempotent writes, and an async event pipeline — all from the Go standard library.
- Go
- PostgreSQL
- Redis
- RabbitMQ
- Prometheus
- Docker