Why Your IDP Doesn't Know About Your APIs

Document processing tools and API integration tools are built in separate product categories, sold to separate buyers, and never talk to each other. The gap between them is where most enterprise AI workflows break.

This pairs with the $150,000 invoice on what enterprise IDP actually costs, the agent memory stack, enterprise ontology, the institutional knowledge tax, the twelve layers of LLM cost, and memory residency in the Hardened Agentic Stack — documents, APIs, and memory only compound when the system can carry provenance across all three.

The Q1 invoice problem

A finance team had 400 vendor invoices from Q1. The invoices arrived as PDFs: some generated cleanly from accounting systems, some scanned, some emailed as image-heavy attachments, some with line items that wrapped across pages.

The team bought an IDP tool. It extracted invoice number, vendor name, date, subtotal, tax, and total with roughly 85% accuracy. That sounded good in the demo. In production it meant 60 invoices still needed manual correction, and every invoice still needed manual validation against the company’s vendor contracts and ERP records.

The extraction was not the hard part. The hard part was what happened after extraction:

1. Check whether the vendor existed in the ERP.
2. Match the purchase order.
3. Compare line-item prices against the contract.
4. Flag tax or currency mismatches.
5. Route exceptions to the right approver.
6. Archive the redacted PDF.
7. Leave an audit trail that explained why the system approved or rejected the invoice.

The IDP tool produced JSON. The ERP had an API. The contract repository had search. Slack had a workflow endpoint. None of them shared a common execution layer.

So a human copied fields out of the IDP review screen into the ERP workflow. Four minutes per invoice. Four hundred invoices. About 27 hours every quarter spent moving already-extracted data from one product category into another.

That is “automating the easy part.” The OCR worked. The extraction mostly worked. The workflow still depended on a person because the document system didn’t know how to ask the API system anything, and the API system didn’t know where the document evidence came from.

Lesson: invoice extraction is not invoice automation. If the system cannot validate extracted fields against live APIs and preserve the evidence chain, it has only automated the first step.

Why these systems don’t talk to each other

IDP and API integration come from different software markets.

IDP is sold to finance, legal, insurance operations, and back-office automation teams. The buyer asks about OCR accuracy, field extraction, redaction, document retention, human review queues, and compliance.

iPaaS and API automation are sold to IT, RevOps, platform engineering, and business systems teams. The buyer asks about connectors, workflow triggers, retries, rate limits, credentials, and SaaS integration coverage.

The two product categories rarely share a core object:

• The IDP system thinks the object is a document.
• The iPaaS system thinks the object is an event.
• The ERP thinks the object is a purchase order.
• The contract repository thinks the object is a file.
• The LLM thinks the object is whatever prompt text it was handed.

This is why the “integration” layer usually becomes one of four brittle options.

Zapier or Make. Good for simple event routing. Weak for document provenance, schema drift, retry semantics, large files, and regulated audit trails.

Custom code. Flexible, but now the team owns every mapping between extracted fields and API parameters. The code is rarely a product priority, so it decays quietly until a vendor changes a field name.

RPA. Useful when no API exists. Fragile when browser screens change. Nearly impossible to reason about cryptographically because the automation is copying pixels and fields across UIs.

Manual review. Reliable enough for a quarter-end process. Expensive forever. It also teaches the organization that “AI automation” means “a queue humans clear faster.”

None of those options gives the agent a single way to discover document operations and API operations, execute both, and keep a shared audit trail.

Lesson: IDP and iPaaS don’t fail to integrate because HTTP is hard. They fail to integrate because they were built around different buyers, different objects, and different definitions of completion.

What the gap costs

The visible cost is labor. Four minutes per invoice times 400 invoices is 1,600 minutes, or about 27 hours, every quarter. At loaded finance-ops cost, that is easy to price.

The hidden cost is latency. An invoice that could have been processed in two minutes waits in a review queue for two days because the validation step is manual. Exceptions pile up near close. Vendors ask for status. Finance staff build spreadsheets to track which system is currently the “real” state.

Labor and latency compound:

The team does not experience this as an architecture problem. They experience it as “the IDP tool is only 85% accurate” or “the ERP API is annoying” or “the LLM hallucinates.” Those are symptoms. The structural problem is that document work and API work are split across tools that cannot share context.

Lesson: the expensive part is not one bad extraction. It is the compounding tax of every handoff after extraction.

The three categories of document work

Most enterprise document workflows fall into three categories.

Category 1: extraction only. Pull fields from a document and put them somewhere structured. OCR, classification, key-value extraction, and table extraction live here. A lot of IDP demos stop here because the output is easy to show.

{
  "invoice_number": "INV-2026-0315",
  "vendor": "Northwind Components",
  "total": 18420.5,
  "currency": "USD"
}

Category 2: extraction plus validation. Check the extracted fields against systems of record. Does the vendor exist? Does the PO match? Is the amount within contract terms? Is the approver allowed to authorize this amount?

This category requires APIs. A document tool alone cannot answer these questions because the truth lives outside the document.

Category 3: extraction plus validation plus analysis and action. Decide what to do. Approve, reject, route to human review, ask the vendor for clarification, create an ERP record, archive the document, notify the approver, and write the audit trail.

This category requires document processing, API execution, organizational knowledge, model reasoning, policy enforcement, and memory. It is where most “enterprise AI workflow” promises actually live.

The Q1 invoice process was Category 3. The team bought a Category 1 product and expected it to behave like a Category 3 system.

Lesson: if the workflow requires validation or action, IDP is a component. It is not the workflow.

What document processing actually requires

A serious document pipeline is not one OCR endpoint. It is a chain of boring, necessary tools:

Apache Tika detects file types, extracts metadata, and parses text from PDFs, Office documents, email attachments, and strange enterprise formats that nobody remembers approving.

Gotenberg converts Office documents, HTML, and images into consistent PDFs so downstream processing sees a uniform representation instead of every vendor’s favorite attachment format.

Stirling-PDF handles PDF operations: OCR, page splitting, merging, rotation, redaction, metadata stripping, and per-step hashing.

Paperless-ngx provides document management primitives: ingest, tagging, correspondence, archival, and review workflows.

Enterprise knowledge systems provide the context that documents reference but don’t contain: pricing policies, vendor master data, signed contracts, exception memos, and Slack threads explaining why a vendor has non-standard terms.

None of those systems is exotic. The point is that document work is already a multi-system workflow before the first ERP API call happens.

Raw attachment
  ├─ detect format and metadata
  ├─ normalize to PDF
  ├─ OCR and redact
  ├─ extract fields and tables
  ├─ archive processed artifact
  ├─ index for semantic search
  └─ link to enterprise knowledge

The orchestration layer has to know what happened at each step. If the final decision says “approved because contract pricing matches,” the audit trail needs the source PDF hash, the extracted fields, the contract clause, the API response, and the model verdict.

Lesson: document processing is a pipeline of evidence transformations. Treating it as a single “upload PDF” step destroys provenance before the workflow gets interesting.

The cross-reference problem

The invoice itself does not know whether it is correct. Correctness lives in other systems.

Here is the real shape of the validation step:

from decimal import Decimal


def validate_invoice(invoice: dict, erp, contracts) -> dict:
    vendor = erp.get_vendor(invoice["vendor_id"])
    if vendor is None:
        return {
            "status": "needs_review",
            "reason": "vendor_not_found",
        }

    po = erp.get_purchase_order(invoice["purchase_order"])
    if po is None:
        return {
            "status": "needs_review",
            "reason": "purchase_order_not_found",
        }

    contract = contracts.find_active_contract(
        vendor_id=invoice["vendor_id"],
        as_of=invoice["invoice_date"],
    )

    discrepancies = []

    for line in invoice["line_items"]:
        contract_price = contract.price_for(
            sku=line["sku"],
            quantity=line["quantity"],
        )
        observed = Decimal(str(line["unit_price"]))

        if observed != contract_price:
            discrepancies.append({
                "sku": line["sku"],
                "invoice_price": str(observed),
                "contract_price": str(contract_price),
                "delta": str(observed - contract_price),
            })

    if discrepancies:
        return {
            "status": "needs_review",
            "reason": "price_mismatch",
            "discrepancies": discrepancies,
        }

    return {
        "status": "approved",
        "reason": "validated_against_erp_and_contract",
        "vendor_record": vendor["id"],
        "purchase_order": po["id"],
        "contract": contract.id,
    }

That function is simple only because it assumes the hard parts already exist:

• erp.get_vendor
• erp.get_purchase_order
• contracts.find_active_contract
• contract.price_for
• A consistent invoice schema
• A reliable document hash
• A place to record the verdict

In most enterprises, those calls cross five products and three teams. The validation function becomes a custom integration project, not a reusable capability.

Lesson: cross-reference is the center of the workflow. Any architecture that treats it as “post-processing” will strand humans in the loop.

Where APIs come in

The agent should not need a custom connector for every document workflow. It needs a way to discover operations and execute them against a constrained surface.

That is the OpenAPI pattern:

// Search the available operation catalog.
const operations = await search({
  query: 'find active vendor contract and purchase order',
  scope: ['erp', 'contracts', 'documents'],
});

// Execute the selected operation with structured parameters.
const contract = await execute({
  operationId: 'contracts.findActiveContract',
  parameters: {
    vendor_id: invoice.vendor_id,
    as_of: invoice.invoice_date,
  },
});

const purchaseOrder = await execute({
  operationId: 'erp.getPurchaseOrder',
  parameters: {
    po_number: invoice.purchase_order,
  },
});

The full API specification stays server-side. The agent gets only the operation slice it needs: name, description, parameters, auth scope, and result schema. The same pattern works for document operations:

const extraction = await execute({
  operationId: 'stirling.extractInvoice',
  parameters: {
    document_id: 'doc_q1_000137',
    schema: 'invoice_v2',
  },
});

Now document processing and API validation are peers. The agent searches one operation space and executes against one gateway. The gateway enforces credentials, budget, redaction, logging, and retry policy.

Lesson: APIs enter the document workflow as discoverable operations, not as one-off glue code written after the IDP system is “done.”

The seven-vendor IDP stack

A complete self-hostable document workflow usually spans seven services:

The pipeline looks like this:

Nextcloud / Email / WebDAV
        │
        ▼
Apache Tika
  parse text, detect MIME, extract metadata
        │
        ▼
Gotenberg
  normalize Office, HTML, and images to PDF
        │
        ▼
Stirling-PDF
  OCR, redact, split, merge, Merkle hash per step
        │
        ▼
Paperless-ngx
  archive, tag, review, retain
        │
        ▼
Onyx
  semantic index over contracts, policies, prior decisions
        │
        ▼
ConeShare
  external review packages and VDR links

Seven services is not bloat. It is the minimum shape of enterprise document work when you include ingest, normalization, extraction, archival, search, and distribution.

The integration problem appears when those seven services live outside the API operation space. If the agent can call the ERP but not Stirling-PDF, it cannot preserve document provenance. If it can extract fields but not query Onyx, it cannot explain why a discrepancy is acceptable. If it can search contracts but not archive the processed artifact, the workflow is incomplete.

Lesson: the seven-vendor stack is manageable when every service exposes operations through the same gateway. It becomes unmanageable when every service is a separate island with a separate audit trail.

Orchestration across layers

Simple document workflows are linear. Production workflows are not. OCR confidence can be low. A vendor can have three active contracts. A price mismatch can be allowed by an exception memo. An ERP call can fail and need retry. A model verdict can require escalation.

ClawQL’s gateway orchestrator, inspired by the Q00/ouroboros five-phase loop, treats the workflow as a sequence of interview, seed, execute, evaluate, and evolve steps. The point is not to name a new workflow engine. The point is to make evaluation and retry first-class instead of hiding them in glue code.

type InvoiceVerdict =
  | { status: 'approved'; reason: string }
  | { status: 'needs_review'; reason: string; evidence: unknown[] }
  | { status: 'rejected'; reason: string; evidence: unknown[] };

class IDPOrchestrator {
  constructor(
    private readonly gateway: GatewayClient,
    private readonly worm: WORMLog
  ) {}

  async processInvoice(documentId: string, correlationId: string): Promise<InvoiceVerdict> {
    // Seed: extract the invoice from the normalized document.
    let extraction = await this.gateway.execute({
      operationId: 'stirling.extractInvoice',
      parameters: {
        document_id: documentId,
        schema: 'invoice_v2',
      },
      correlationId,
    });

    await this.worm.write({
      event_kind: 'DOCUMENT_EXTRACTED',
      correlation_id: correlationId,
      payload_hash: hash(extraction),
      payload: {
        document_id: documentId,
        confidence: extraction.confidence,
        merkle_root: extraction.merkle_root,
      },
    });

    // Evaluate: low-confidence extraction gets a richer schema before human review.
    if (extraction.confidence < 0.85) {
      extraction = await this.gateway.execute({
        operationId: 'stirling.extractInvoice',
        parameters: {
          document_id: documentId,
          schema: 'invoice_detailed_v2',
        },
        correlationId,
      });
    }

    const vendor = await this.gateway.execute({
      operationId: 'erp.getVendor',
      parameters: { vendor_id: extraction.vendor_id },
      correlationId,
    });

    const contract = await this.gateway.execute({
      operationId: 'contracts.findActiveContract',
      parameters: {
        vendor_id: extraction.vendor_id,
        as_of: extraction.invoice_date,
      },
      correlationId,
    });

    const policy = await this.gateway.execute({
      operationId: 'onyx.search',
      parameters: {
        query: `${vendor.name} invoice exception pricing policy`,
        limit: 5,
      },
      correlationId,
    });

    // Evolve: model escalation is available when cheap validation cannot decide.
    const verdict = await this.gateway.infer({
      operation: 'invoice_validation',
      tier: 'frugal',
      escalation: {
        enabled: true,
        max_tier: 'standard',
        reason: 'low_confidence_or_policy_conflict',
      },
      input: { extraction, vendor, contract, policy },
      correlationId,
    });

    await this.worm.write({
      event_kind: 'INVOICE_VERDICT',
      correlation_id: correlationId,
      payload_hash: hash(verdict),
      payload: {
        status: verdict.status,
        model: verdict.model,
        tier: verdict.tier,
        evidence_count: verdict.evidence.length,
      },
    });

    return verdict;
  }
}

Every stage uses the same correlationId. The source document hash, extraction payload hash, ERP response hash, policy search references, model tier, verdict, and final action become one reconstructable chain.

Lesson: orchestration is not scheduling. It is the discipline of evaluating each step, retrying when evidence is weak, escalating when a model cannot decide, and preserving the reason every time.

Honest failure modes

A unified pipeline does not make document automation magical.

Eighty-five percent extraction still means failures. Scanned invoices, handwritten notes, rotated pages, and malformed tables will still break extraction. The architecture should route those to richer schemas, alternate OCR, or human review — not pretend they disappeared.

Format diversity is real. Vendors will send PDFs, DOCX files, images, ZIPs, portal exports, and email bodies. Tika and Gotenberg reduce chaos; they don’t eliminate it.

API brittleness remains. ERP schemas change. Salesforce fields get renamed. Rate limits apply. Credentials expire. The gateway can centralize retries and policy, but the underlying systems still need operational ownership.

Regulatory requirements differ. Redaction, retention, right-to-delete, and audit requirements vary by jurisdiction and industry. WORM logging helps prove what happened, but it must be designed with legal constraints before production data enters the system.

Human review does not vanish. It moves to the right cases: low confidence, missing vendor, policy conflict, high-dollar invoice, or exception routing. Eliminating all review is the wrong metric.

Lesson: the goal is not 100% hands-off document processing. The goal is to make the 397 routine cases routine and the 3 hard cases visible, explainable, and auditable.

What a unified pipeline looks like in practice

Return to the 400 Q1 invoices.

In the original workflow, the IDP tool extracted fields with 85% accuracy and humans copied data into downstream systems. Every invoice touched a manual queue because nobody trusted the automation beyond extraction.

In the unified workflow:

1. New PDFs land in Nextcloud.
2. Tika detects file type and extracts metadata.
3. Gotenberg normalizes non-PDF attachments.
4. Stirling-PDF performs OCR, redaction, and per-step hashing.
5. The invoice schema extractor produces structured fields.
6. The gateway queries ERP vendor and PO records.
7. The gateway searches Onyx for active contracts and exception policies.
8. The validation step compares invoice lines to contract terms.
9. Model escalation handles ambiguous policy interpretation.
10. Paperless-ngx archives the processed artifact.
11. ConeShare packages exceptions for external review when needed.
12. Slack receives a structured notification with document and API citations.

The outcome is not “400 invoices approved automatically.” It is more realistic and more useful:

400 invoices received
397 processed without manual copy
  ├─ 338 approved automatically
  ├─ 42 routed to approvers because amount exceeded threshold
  └─ 17 flagged for vendor or contract mismatch

3 routed to human review before validation
  ├─ 2 unreadable scans
  └─ 1 missing purchase order

The 397 invoices no longer require manual field transfer. The 3 failures are explicit. Every approved invoice has a correlation chain that connects source PDF, extracted fields, ERP records, contract evidence, model verdict, archive location, and notification.

That is the difference between an extraction demo and an enterprise workflow.

Lesson: document intelligence becomes operational only when it can call APIs, use enterprise knowledge, and leave a shared audit trail. Until then, the enterprise has automated the easy part and preserved the expensive part for humans.

---

Reference implementation: docs.clawql.com/vision/idp-platform. Source: ClawQL on GitHub. Related: agent memory stack, enterprise ontology, institutional knowledge tax, twelve layers of LLM cost.