AI contract abstraction for healthcare supplier agreements: where it fails, how to review

A supplier rebate agreement is a calculation written in prose. Somewhere in 60 pages of definitions, exhibits and amendments are the values that decide what the supplier owes you: which purchases count, measured how, against which thresholds, paid when. Abstraction is the work of pulling those values into structured fields a system can calculate from, and language models now produce a first pass in minutes.

Most published advice on AI contract review is written for legal teams and concerns renewal dates and indemnity clauses. Rebate terms fail in different places, and the review protocol below ends with a test only rebate terms allow: run the abstracted terms against a period you have already been paid for.

What abstraction means for a rebate agreement

Clause tagging tells you that a minimum commitment clause exists and where it sits. A rebate calculation needs values, in a structure, with the relationships between them intact.

Field group	What the abstraction has to capture
Parties and eligible purchasers	Supplier entity, authorized distributors, and the facilities whose purchases count
Term and periods	Effective and expiration dates, what “contract year” means, quarterly or annual measurement
Eligible products	The product list or categories, and the exclusions
Measure	Dollars or units, and net of what: returns, freight, credits, distributor purchases
Tier ladder	Each threshold and rate, and whether a rate applies to all purchases or only to the increment
Baseline	The base period for a growth rebate, its value, and how it adjusts
Market share	Numerator, denominator, and which competing products sit in the denominator
Payment and penalties	Who pays, in what form, how many days after period end, claim deadlines, caps, clawbacks

Two things set this apart from generic review. The fields depend on each other: a 3% rate means nothing until you know what it multiplies. And the answer is often spread across the base agreement, an exhibit and one or more amendments.

What current models do well

Public benchmarks are a better guide than a vendor’s accuracy figure, which is measured on the vendor’s documents with the vendor’s definition of correct. A common public benchmark is CUAD from The Atticus Project: 510 commercial contracts carrying more than 13,000 labels across 41 clause types, labeled under the supervision of experienced lawyers and accepted at NeurIPS 2021.

ContractEval, a 2025 study, ran 19 language models (4 proprietary, 15 open-source) against CUAD’s test set of 4,128 data points from 102 contracts. The two best models scored an F1 of about 0.64 overall. On plain categories such as Governing Law and Parties, the best model approached or passed 0.9. The authors conclude that the top proprietary models are “approaching the performance level of junior legal assistants”: able to find the relevant clause, still in need of senior oversight.

By analogy, the rebate fields most likely to come back right are the plain ones: party names, effective and expiration dates, payment timing, and a tier table laid out cleanly on one page.

Note the limit of the evidence. CUAD’s categories were chosen for corporate transaction review. The closest to rebate terms are Minimum Commitment, Volume Restriction, Price Restrictions and Revenue/Profit Sharing, and each asks only whether such a clause exists and where. CUAD does not test whether a model can return a tier ladder as rows, identify a growth baseline, or state a share numerator and denominator. A strong score says the model reads contracts competently. It says nothing about yours.

Where models fail predictably

ContractEval reports three findings worth carrying into any evaluation. Even the top-scoring model scored near zero on nuanced or rare categories, Notice Period to Terminate Renewal among them. Some models, mostly open-source ones, answered that no related clause existed when one did, and the worst model in the study had a false no-clause rate of about 0.31. A reasoning mode brought extracted text closer to the labeled passage but reduced correctness for most models.

A ContractSafe article on extraction accuracy lists the general weak points: multi-part renewal logic, inconsistent party names, amendments disconnected from their parent agreement, and negotiated exceptions captured as if the clause were unconditional. Rebate agreements add their own:

• Superseding amendments. Amendment 2 replaces the tier table in Section 4.1. An extraction that reads the base agreement alone, or reads every document without ordering them, returns the old ladder with full confidence.
• Broken tables. A tier table split across a page break, or embedded in a scanned exhibit, loses rows or shifts columns so thresholds pair with the wrong rates.
• Defined terms. The rate is on page 12. The definition of Net Purchases that says what it multiplies is on page 3, with its exclusions.
• Baselines. A growth rebate pays on the increase over a base period. The base may be a stated figure, a formula, or “prior contract year purchases”, a number that is not in the document at all.
• Share language. The ladder and the commitment that defines the share often sit in different sections. An extraction can capture both and still fail to connect them.
• Product exhibits. Eligibility defined by a 400-line exhibit is a data load, not a field.
• Cross-references. “Subject to Section 7.3” is where the cap or the clawback lives.

Wrong is worse than blank

The same ContractSafe article makes the argument that should shape your evaluation: a missing field sends someone to verify it, while a filled field tells them the work is done. In a rebate system the filled field does more than reassure. It sets an accrual, fires or suppresses a tier alert, and produces the expected amount you compare with the supplier’s payment. A threshold that is off by one row looks exactly as authoritative as a correct one.

So judge a tool on how it handles not knowing. It should be able to leave a field empty and say why, mark low confidence separately from a confident value, and put gaps where a reviewer cannot miss them. When you score a tool, weight a wrong value more heavily than a blank.

Review by risk tier

Reviewing every field with equal care is the quickest way to get the review abandoned. ContractSafe proposes tiers by consequence. Adapted to rebate terms:

Tier	Fields	Review	Reviewer
Financial and date	Thresholds, rates, measure and defined terms, baseline, share definition, eligible purchasers, period dates, claim deadlines, caps	Line by line against the document, signed off before the term drives a calculation or alert	The person who manages the agreement, with finance for accrual terms and legal for disputed wording
Structural	Supplier identity, distributors, contract type, product scope by category	Checked at intake	Contracting or data team
Descriptive	Tags, summaries, notes	Spot check in batches	Any trained user

Show the source for every value

Every extracted value should carry the page and passage it came from, so a reviewer confirms a rate by reading the quoted sentence beside it instead of hunting through a PDF. Treat a value with no passage behind it as unverified, whatever confidence the model reports. Be careful with scans: a page number reported by a model reading an image is weaker evidence than a passage matched word for word against the text, and the tool should tell you which one you are looking at.

The other half is an audit trail: who changed which field, from what, to what, and why. A healthcare contract management vendor’s article on responsible AI notes that vendor agreements carry Anti-Kickback implications and calls for AI outputs that are explainable and fully auditable. When internal audit asks why the accrual used 3%, the answer should be a page number and a name.

Back-test before go-live

Rebate terms produce a number, and for a closed period you already know what the supplier paid. Run the abstracted terms over that period’s purchases and compare.

An illustrative example: a wound care agreement abstracted with a ladder of 2% from $500,000, 3% from $1,000,000 and 4% from $2,500,000 in annual purchases, the rate reached applying to all of the year’s purchases.

Step	Figure
Closed-year purchases matching the abstracted product scope	$2,000,000
Tier reached on the abstracted ladder	3%
Rebate the abstracted terms produce	$60,000
Rebate the supplier paid	$45,000
Gap	$15,000, a quarter of the expected figure

A gap that size has two possible homes. Here, the abstraction had missed an exclusion: the agreement’s definition of Net Purchases leaves out products bought through a distributor, $500,000 of the year. Corrected, $1,500,000 still reaches the 3% tier, and 3% of it is $45,000. If the abstraction had held up against the document, the gap would be a payment question, raised with the supplier as a reconciliation of terms you both signed, with the clause and purchase detail attached. Either finding is worth having.

Set a tolerance before you run the test, since timing and credits usually leave small differences. Remember what a pass does not prove: a tier you never reached was never tested.

Governance and document handling

NIST’s voluntary AI Risk Management Framework, released in January 2023, gives a neutral scaffold in four functions. Applied to abstraction:

Function	What it means here
Govern	Name an owner for the process and decide who may approve a financial field
Map	List where the output goes (accruals, alerts, supplier statements) and what a wrong value costs in each
Measure	Run the pilot below, and again when the model or the document mix changes
Manage	Keep a correction path, with errors logged and reviewed

Supplier agreements usually carry confidentiality clauses, so ask any vendor where documents are processed and stored, whether contract text trains a model, how long it is retained, who at the vendor can read it, and what independent security assessment covers the service.

Collect the documents first. The healthcare vendor article cited above warns that AI working over contracts scattered across shared drives and inboxes gives incomplete results. If an amendment lives in someone’s email, the model abstracts the superseded agreement faithfully.

A 30-day pilot

1. Days 1 to 5: pick 25 agreements that include amendments, scans, a share commitment, a growth rebate and a multi-facility deal. Fix the field list.
2. Days 6 to 12: build a gold set by hand. Two people abstract the financial fields independently and resolve differences.
3. Days 13 to 17: run the tool. Reviewers work from its output without seeing the gold set, and you time them per contract.
4. Days 18 to 24: score each field as correct, wrong, missed, or correctly left blank. Precision is the share of filled fields that are right. Recall is the share of gold-set fields the tool filled correctly. For financial fields, precision matters more.
5. Days 25 to 30: back-test five agreements against a closed period, then decide against a threshold you wrote down on day 1, for example 95 of every 100 filled financial fields correct and review time under half of manual abstraction.

If 30 days is too long, run the afternoon version. Take one agreement you know by heart, with at least one amendment that changes the tier table. Count right, wrong and blank in the financial tier, and check that every right answer shows its passage. A tool that returns the superseded ladder with a confident citation to the base agreement has told you how it handles amendments.