Skip to content
Index
Get in touch

§Case study

← All projects

DocuFlow AI – Multi-Tenant AI Document Processing Platform

An asynchronous document processing SaaS. Upload PDFs and images, extract structured data with AI, review results, and integrate via a developer-friendly API. BullMQ workers, dead-letter queues, and HMAC-signed webhooks.

Shipped
Reading
3 min read
Stack
  • TypeScript
  • Next.js
  • NestJS
  • BullMQ
  • PostgreSQL
  • Gemini
  • AWS ECS
  • Terraform

“Upload a document. Get structured data back. That’s the promise — the engineering is in making it reliable.”

In plain English

DocuFlow AI is a document processing platform: you upload a PDF or image, AI extracts the data you need (invoice line items, totals, dates, vendor info), and you get structured JSON back. The upload returns immediately; workers handle the heavy lifting in the background. When they’re done, a signed webhook tells your system.

The hard part isn’t the AI extraction — it’s everything else: making it multi-tenant, reliable, secure, and observable.

📦 GitHub Repository: https://github.com/amirfaisalz/docuflow

The problem

Document processing at scale has a few failure modes that demos never show:

  • Jobs fail. LLM calls time out, files are corrupt, extraction returns garbage. You need retries, dead-letter queues, and visibility into what went wrong.
  • Tenants leak. Without careful scoping, one organization’s documents can surface in another’s dashboard. This isn’t just a bug — it’s a compliance incident.
  • Webhooks break. The receiving server is down, returns 500, or takes too long. You need retry logic, delivery history, and tamper-proof signatures.
  • Usage must be metered. Every document processed, every LLM token consumed — tracked transactionally, not eventually.

What I built

Asynchronous processing pipeline

The API accepts uploads and dispatches BullMQ jobs immediately. Workers handle the pipeline independently: OCR, LLM extraction, Zod validation, persistence, and webhook delivery. Each step is observable and retryable.

  • 3 attempts with exponential backoff per job
  • Dead-letter queue (document-processing-failed) for jobs that exhaust retries
  • Idempotency via unique UsageRecord.jobId — replay a job, don’t double-count
  • State-aware updates — a job in COMPLETED state can’t regress to PROCESSING

Multi-tenancy

Every resource is scoped to an organization. RBAC enforces four roles: Owner, Admin, Member, Developer. Tenant isolation is enforced at the service layer and the database layer. Accessing another tenant’s resource returns 404, not 403 — no information leakage about what exists.

Document security

  • Cloudinary stores documents with authenticated (private) access
  • Upload params are signed server-side
  • Download URLs are short-lived and signed
  • 10 MiB cap, PDF/JPEG/PNG only
  • Webhook target URLs reject private/internal addresses

Structured AI output

Gemini multimodal extraction produces structured invoice data validated against a strict Zod schema. The LLM output is treated as untrusted input — parsed, validated, and sanitized before persistence. Malformed output is flagged, not silently accepted.

API keys and usage metering

  • One-time df_live_... secrets, SHA-256 hashed at rest with a searchable prefix
  • Immediate revocation — delete the key, and all in-flight requests using it fail
  • Transactional usage metering: documents processed, LLM tokens consumed, processing duration — all recorded atomically

Webhooks

  • HMAC-signed payloads (t=<unix>,v1=<hex>) — recipients can verify authenticity
  • Signing secrets are encrypted at rest
  • 5 delivery attempts with exponential backoff
  • Per-attempt history — timestamp, status code, response time, error

Observability

  • JSON structured logs with automatic PII redaction
  • OpenTelemetry spans across the full pipeline: HTTP → queue → worker → Gemini → DB → webhook
  • /health/ready with dependency checks (DB, Redis, queue)

Production infrastructure

  • AWS ECS Fargate with ALB health probes on /health/ready
  • Circuit-breaker rollbacks — unhealthy tasks are replaced automatically
  • Autoscaling based on queue depth and CPU
  • Prisma migrations as a one-off ECS task before deployment

Architecture

Client → API (NestJS) → BullMQ Queue → Workers
                                          ├─ Cloudinary (storage)
                                          ├─ Gemini (extraction)
                                          ├─ PostgreSQL (persistence)
                                          └─ Webhook delivery

The API never does heavy work. It validates, queues, and returns. Workers are horizontally scalable and stateless.

Status

In active development. Core pipeline, multi-tenancy, API keys, webhooks, and observability are implemented. Dashboard UI and public documentation are in progress.

Next case study ↓

03 / 2026

procure-flow

Procurement Approvals Without a Web Framework

procure-flow product screenshot

A compact procurement approval service with an explicit state machine, RBAC, append-only audit trail, idempotent writes, and an async event pipeline — all from the Go standard library.

  • Go
  • PostgreSQL
  • Redis
  • RabbitMQ
  • Prometheus
  • Docker