How to evaluate an expert shortlist before the first call

A shortlist is a hypothesis. You find out whether it holds up in the first ten minutes of the first call — and by then it's too late to matter. Teams that ship good diligence don't have luckier networks. They pressure-test the shortlist before anyone gets on a call, against criteria the sales rep can't argue with.

This post is the checklist they use. Eight questions, applicable to any vendor, that separate a defensible shortlist from a flat list of names.

Why the shortlist is the actual bottleneck

The common failure mode in expert-driven diligence isn't "the network was slow." It's "the network was fast at finding plausible names." Candidates clear the surface filters (title, company, geography) and the analyst, trusting the funnel, books the calls. Two hours in, one of the three calls answers the question. The others answer adjacent questions. Shortlist quality silently set the ceiling on the whole project.

Catching that before you book is the cheapest place in the project to course-correct. The criteria below are what you're testing for.

The eight-point evaluation checklist

1. Is there a written match rationale on every candidate?

Each candidate should ship with a 3–5 bullet explanation of why this person fits this question. Not a generic profile. Not a job title plus tenure. A written, specific paragraph that names the products, decisions, or experiences that qualify them.

If you can't read the rationale and immediately understand "why this person, for this brief," the network is asking you to do the matching work for them.

2. Is the relevant experience recent enough?

A twelve-year veteran of a category whose last hands-on role ended in 2019 is a different expert than someone running the same playbook today. Date-stamp every claimed competency. "Senior at Voltura, 2014–2017" is honest; "Battery industry expert" is not.

Recency tolerance depends on the question. Regulatory and competitive intel decay fast. Founding-team experience or category history can be evergreen. Decide the threshold up front; reject candidates whose evidence is older than your tolerance.

3. Does the evidence point to a role, not a title?

Titles are noisy proxies. A "VP, Strategy" might own a $500M P&L or run a deck factory. The shortlist should describe what the candidate actually did, with enough specificity that someone outside the company would recognize the work: which product line, which deal, which decision, which scope.

Reject any candidate whose strongest evidence is "former senior at [adjacent company]" with no specifics underneath.

4. Are conflicts flagged inline?

Every shortlist should arrive with a conflict status per candidate. Current employer? Past employer of a competitor? Active board seats? Recent equity events? You shouldn't have to ask — the network should publish what they checked and what they found.

"Cleared" with no detail on what was checked is weaker than "Cleared on employer + competitor exposure, not checked on board affiliations." Specificity is the audit trail.

5. Are the screening notes attached?

If the network voice-screened the candidate, the notes should be on the shortlist. What did the candidate confirm they can speak to? What did they decline? What questions did they answer well versus dodge?

Notes do two things: they let you spend the first ten minutes of the call on new ground instead of re-confirming the screen, and they create an audit trail if the call later disagrees with what was screened.

6. Do geography, language, and segment fit match the brief?

A US-based senior commenting on the German distribution channel is a partial match. A French-speaking operator in Quebec is a partial match for a Paris-headquartered question. Surface those gaps on the shortlist, not on the call.

The good test: read each candidate's claimed fit and ask "would this person give a different answer than someone perfectly in the segment?" If yes, the network should have flagged it.

7. Is there a confidence reason code — not a score?

A confidence number with no reasoning is worse than no number. What you want is why the network has the confidence it has: "high — direct role evidence + recent + cleared" beats "92%." If the score has no reasons underneath, the score is theater.

8. Can you see what got rejected and why?

The strongest signal of search quality is what didn't make the shortlist. Ask: of the 40 candidates considered, why did 35 fall out? "Insufficient role evidence," "competitor exposure," "geography mismatch," "declined screening." The rejection log tells you whether the funnel actually screened or just sorted by tenure.

A network that won't show the rejection log is selling you the names it surfaced, not the answer to your question.

Apply the checklist to any vendor

The checklist is intentionally vendor-neutral. You can run it against a traditional expert network, an AI-driven one, a boutique that promises bespoke search, or an aggregator that routes briefs to multiple suppliers. The criteria don't depend on the matching technology — they describe what an evidence-backed shortlist looks like.

What does depend on technology is how easy the criteria are to enforce. Hand-curated shortlists from a project manager can produce all eight artifacts, but only if the manager has hours per brief. Systems that bake the criteria into the matching workflow produce them in minutes on every brief, because the workflow doesn't ship a shortlist without them.

Put differently: stop praising the PM and start auditing the workflow.

What changes when the shortlist is auditable

Demanding these artifacts pays back in several places across a project:

• Calls start at minute one on substance, not at minute fifteen after the warm-up confirms what should already be in writing.
• The shortlist becomes the project artifact that's defensible to the investment committee, the partner, or the client, instead of a private working doc.
• Rejection reasons become reusable. The next brief refines the search universe instead of starting from zero.
• Compliance review becomes a one-click export, not a reconstruction.

AI isn't the prerequisite. Treating the shortlist as a deliverable is.

If you can only enforce one, enforce reason codes

If you're evaluating networks and only get to enforce one of the eight, enforce point seven: show me the confidence reason codes. It's the one item that reveals whether the network actually thought about your brief, versus whether they ran a fast keyword search and rank-sorted the survivors. Networks that can produce reason codes on demand have a search process worth paying for. The rest are sorting names, not answering your question.