Shadow Evaluation Guide
A structured, no-cost process for payer operations teams to establish a behavioral baseline for incoming AI voice calls — without changing anything in your call flow.
Shadow mode means observe-only. Your existing call handling is unchanged. You measure how today's incoming AI voice traffic behaves against the five NHID-Clinical v1.3 controls, then decide what to do with the findings.
Scope
What this is
- No cost, and no vendor changes. Observe-only: it does not sit in the call path. Your privacy, security, and contractual obligations still apply.
- You run it in shadow mode alongside your current operations.
- Goal: produce a written behavioral baseline showing how AI voice agents currently disclose, escalate, and log calls to your administrative lines.
- Observe-only. There is no certification, no SLA, and no commitment beyond sharing anonymized findings with the community if you choose.
Prerequisites
What you need
- Access to call logs or recordings for a sample of incoming administrative calls (eligibility, claim status, prior authorization).
- Staff time to read this guide and agree the scope internally.
- Willingness to share anonymized findings if you choose to.
Model
Where a shadow evaluation sits
One-way on purpose: nothing it produces re-enters the call.
The word that does the work is observe-only. An evaluation reads call traces you already hold; nothing it produces re-enters the call. The diagram is one-way on purpose — there is no arrow back into the live path, because there is no mechanism for one.
Sequence
How an evaluation unfolds
A sequence, not a schedule.
The stages run in this order because each depends on the one before it; how long each takes is yours to decide, and an evaluation can stop after any of them.
- Stage 1Behavioral baselineLog incoming calls. Identify which callers are AI voice agents. Record disclosure timing, escalation behavior, and audit-trail presence.
- Stage 2Gap analysisEvaluate the identified AI calls against the five v1.3 controls (IDG-01, PDX-01, DBC-01, EIT-01, ATR-01) plus the supplemental bot‑to‑bot rule. Quantify what passes, what fails, what is ambiguous.
- Stage 3Written assessmentCompile the findings into a short written assessment. Share the anonymized results if you choose; they inform the next version of the proposal.
Measurement
What an observe-only run actually returns
How often the controls would have acted, before anything is allowed to act.
Shadow evaluation is a proposition about volume: you find out how often the controls would have acted before anything is allowed to act. This is that measurement over a committed corpus — the decisions the engine returned, with nothing routed, blocked or escalated to produce them.
Measurement
Recorded impersonation latency
The interval the corpus records, by label.
Controls
The controls you are observing
| Control | What it requires |
|---|---|
| IDG-01 | Disclose AI identity at call start, before any data exchange. |
| PDX-01 | No PHI or credentials exchanged before identity disclosure is confirmed. |
| DBC-01 | No claim of human status or licensed-professional standing in the agent’s identity assertion. Evaluated on transcript text, not audio. |
| EIT-01 | Immediate, clean escalation path to a human on request. |
| ATR-01 | Basic audit log — enough to reconstruct when disclosure occurred. |
Start
How to start
Email us — or open a GitHub Discussion. Email contact@nhid-clinical.org with the subject line "Shadow Evaluation". Either way, include your role and the workflows you want to observe.
- KIT
Tier 0 Shadow Pilot Kit →
Capture schema, measurement script, report template. Runs on your own call logs; nothing touches production.
- BRIEF
Executive Brief →
One-page overview for leadership and procurement.
- PACK
Evidence Pack →
Guarantees, a worked failure trace, the audit model.
- REG
Regulatory alignment →
Controls mapped to CMS-0057-F, MACPAC, DOJ FCA, state AI laws. Mapped, not certified.
- LINE
Reference demonstration line →
Hear the disclosure controls in a real call.
Observe today's AI voice traffic against IDG-01, PDX-01, DBC-01, EIT-01, and ATR-01 — log what passes, what fails, what is ambiguous — then decide what to require in the next contract cycle.
For payers
For payers
Establish a behavioral baseline for AI voice agent transparency — no vendor changes, observe-only.
It runs alongside live traffic without sitting in the call path; your privacy, security, and contractual obligations still apply. Observe, collect evidence, decide what to require later.
No payer has adopted NHID-Clinical in production yet. It is a voluntary reference model, not an accredited standard, certification, or regulatory requirement.
AWhat you gain
| Metric | Today (typical) | Target with NHID-Clinical |
|---|---|---|
| Verification latency | 3–5 min or call terminated | < 5 seconds |
| Audit effort per vendor | Manual call review (hours) | ~2 minutes (test suite) |
| RFP disclosure language | Custom per vendor | One standard clause |
| Escalation response time | Untracked | ≤ 2 seconds, logged |
BHow a payer-side evaluation works
-
Add RFP language
Insert this clause into your next voice AI vendor RFP or BAA amendment:
"The vendor's AI agent SHALL produce NHID-Clinical v1.3 JSON trace logs for all B2B administrative calls, including disclosure timestamps and opt-out handling. The payer may run the open-source conformance test suite against vendor output at any time."
Existing contract? Send as a formal amendment request.
-
Vendor sandbox testing
Ask your vendor to run the open-source suite — results in under 5 minutes.
git clone https://github.com/NHID-Clinical/NHID-Clinical.git pip install -r requirements.txt python -m pytest tests/ -v
Send full terminal output (+ optional sample traces).
-
Validate logs yourself
Place vendor JSON traces in
traces/, runpython -m pytest tests/ -v, and verify: disclosure before NPI/member ID; no deceptive artifacts; escalation ≤ 2s on request. -
Measure impact
Verification latency (target < 5s) and escalation volume from identity uncertainty (target > 30% reduction).
-
Decide next steps
Tests pass + metrics improve → consider requiring conformance in future contracts. Tests fail or flat metrics → remediation or disqualify from future bids.
v1.3 covers disclosure behavior and audit trails — suitable for shadow evaluations today. v2 adds cryptographic agent identity (NHID-Auth v2) for when you must verify the agent was authorized by the provider it claims to represent. NPIs are public; v1.3 alone does not prove authorization. Read the v2 roadmap →
Scripts
What transparent disclosure sounds like
Side-by-side examples of AI voice agent openings — the kind of interaction this proposal is trying to encourage, and the kind it is trying to reduce.
The core idea is simple: an AI voice agent should say it is not human before asking for or sharing any operational information. The examples below show what that looks like — and what it looks like when it does not happen.
Good examples — disclosure upfront
What we'd like to see more of
These openings work because the AI identifies itself immediately — before asking for anything. The representative knows what they're dealing with from the first sentence.
General payer operations inquiry
Automated nature disclosed immediately. No data exchanged before identification.
Dental payer — claim status
Non-human status disclosed before any data is requested.
Prior authorization follow-up
Clear, early, unambiguous. The representative can make an informed decision about how to proceed.
Eligibility verification
Discloses first, then asks permission. Gives the representative control.
Problematic examples — what this proposal aims to reduce
Patterns that create impersonation latency
These examples illustrate the problem. Each one involves data being exchanged before the representative knows they are talking to an automated system. The scenarios are based on real patterns observed in healthcare payer–provider calls.
The representative shared claim status and engaged in a full workflow exchange before knowing they were talking to automation. NPI and member ID were already exchanged. There is no record of when — or whether — this would have been disclosed without a direct challenge.
Disclosure comes after the purpose and claim details are established. A busy representative may have already started pulling up records.
Breathing sounds, hesitation pauses, and typing are added to create the impression of a human caller. Even if disclosure eventually comes, the framing is designed to mislead.
Disclosure is contingent on being asked. This is the core pattern this proposal aims to change.
These examples cover B2B administrative voice workflows — AI systems calling payer offices on behalf of providers or vendors. They do not apply to patient-facing calls, secure messaging, or internal systems.
The examples above are illustrative, not exhaustive. If you encounter patterns in real calls that are not covered here, we want to hear about them.
Live
The demo line
Two phone demonstrations of the NHID controls. One is a fixed script with no AI in it; the other is Ida, an AI voice agent that calls you the way it would call a payer. Neither involves a real patient: every detail is fictional.
1. Call the scripted demo line
Dial +1 (717) 670-6772 from any phone.
Each scripted turn runs through the NHID-Clinical engine as it plays, and the decisions appear below while the call is in progress.
2. Or let Ida call you
Ida is an ElevenLabs AI voice agent placing a call on behalf of a fictional dental practice to check a claim. You play the payer’s representative. Ida is instructed to ask the NHID gate (POST /v1/gate/turn) mid-call before it says any patient identifier, and not to say it unless the gate allows it. An instruction is not a guarantee, so when the call ends the whole transcript is graded against the controls, and a turn where it did not follow the gate shows up there. Each call costs real calling minutes, so it is CAPTCHA- and rate-limited.
3. See how the call was evaluated
The most recent demonstration call, from either line. Scripted turns appear as they play; an Ida call appears a few seconds after it ends. For an Ida call, the NHID-Clinical engine’s result is shown beside ElevenLabs’ own grading of the same call, and only the engine’s column is NHID evidence.
A demonstration of the NHID controls, not a certification of either line. The calls are scripted or simulated and the patient is fictional; the engine that evaluates them is the same one the API runs. Read the developer docs →
Open for feedback
Questions? Reach out directly.
Whether you have a process question, a concern about the controls, or you want to share what you're seeing in your own call traffic — feedback from payer operations teams is what shapes this work.
Where to go next
Four ways into NHID-Clinical, whatever you came to do.
Going further: Part III of the NHID-Clinical Playbook covers the full evaluation method, governance and data-handling considerations, and what it cannot establish.