Lead Deduplication
Lead deduplication is the process of identifying and merging duplicate records that represent the same person or company into a single canonical lead. It keeps the database accurate and uncluttered.
Key takeaways
- Matching finds candidate pairs; survivorship rules decide which field values survive.
- Every additional entry point multiplies duplicates unless submission does a lookup first.
- Loose thresholds destroy data irreversibly; tight ones leave a buyer's history split.
- Shared inboxes and generic numbers defeat matching because the data is genuinely ambiguous.
- Merges need an audit trail, because the operation cannot otherwise be undone.
In depth
Deduplication runs in two stages. First, a matching stage compares records on keys that should be unique: normalised email, phone in a canonical format, company domain, and fuzzy comparisons of name and address for near misses. Candidate pairs above a confidence threshold are flagged. Second, a survivorship stage decides what the merged record looks like, field by field: keep the earliest acquisition source, the most recent job title, the highest score, the union of activity history.
Duplicate volume rises with the number of entry points: multiple forms, imported lists, event scans, chat capture and integrations that create records independently. It also rises when a form asks only for an email, since there is nothing else to match on later. The core trade-off is between false merges and missed duplicates. A loose threshold destroys data that cannot be recovered; a tight one leaves fragments that split a buyer's history and trigger two reps calling the same person.
Most teams run prevention and cleanup together. Prevention means a lookup on submission so an existing contact updates rather than inserting, plus normalisation of email case and phone formatting at the point of entry. Cleanup runs on a schedule against the backlog. Quiz funnels create a specific case: the same person often retakes a scorecard weeks later, and the right behaviour is one lead with a score history, keeping the latest result active while the earlier answers stay visible for context.
Deduplication cannot resolve cases where the data genuinely does not distinguish two people. Shared inboxes such as info@ and generic phone numbers produce records that look identical and are not. It also cannot decide business questions on its own: whether two subsidiaries of the same group are one account, or whether a person who changed employer should keep their old history, are policy choices. And every merge is destructive, so a system without an audit trail or an unmerge path will eventually lose something that mattered.
Example in practice
How to measure it
The headline number is the duplicate rate: distinct people divided by total records, or its inverse. Track it by source, because one integration usually produces most of the problem. Alongside it, watch the false merge rate from the review queue, meaning the share of proposed merges a human rejects. A rising rejection rate means the matching threshold has drifted too loose for the current data.
Operational effects are easier to sell than data quality. Count duplicate outreach incidents reported by sales, the number of contacts receiving the same campaign twice, and the difference between reported lead volume and distinct people. That last gap is what corrects cost per lead. If none of these move after a cleanup, the duplicates being merged were not the ones causing problems.
Common mistakes
The costliest error is merging automatically at a confidence level chosen without testing. Run the rule in report-only mode first, sample the pairs it would merge, and count how many are wrong. Teams are usually surprised by the number of unrelated people sharing a support inbox or a switchboard number. Auto-merge only the exact-key matches, and route fuzzy matches to a review queue.
The second is cleaning the database repeatedly without closing the source of duplicates. A monthly merge job that never touches the forms creating the records is permanent work with no end. Fix intake first: normalise email and phone on submission, look up existing contacts before creating new ones, and stop imports that bypass those rules. Cleanup then shrinks to a manageable backlog instead of a recurring project.
Frequently asked questions
Which fields are best for matching duplicates?
Email is the strongest single key, followed by phone and company domain, with normalized names used for fuzzy matching. Combining two or more keys reduces both false merges and missed duplicates.
What is survivorship in deduplication?
Survivorship is the set of rules that decides which field values win when records merge, such as keeping the most recent quiz score or the earliest acquisition source. Clear survivorship rules prevent good data from being overwritten during a merge.
Why does deduplication matter for billing?
Many plans count unique leads against a monthly limit, so duplicates can inflate usage and skew cost-per-lead. Deduplicating ensures each real person is counted once and your reporting stays accurate.
Which field should be the primary match key?
Normalised email is the usual choice because it is close to unique per person and is captured on almost every form. Normalise it first: lower the case, trim whitespace, and decide explicitly whether to strip plus-addressing. Phone works as a secondary key once formatting is canonical. Company domain matches accounts rather than people and should not merge contacts on its own.
What are survivorship rules?
They decide which value wins when two records disagree on a field. Common choices are most recent for job title and score, earliest for acquisition source and creation date, non-empty over empty for anything sparse, and union for lists such as tags and activity. Write them down per field before the first merge, because a rule chosen in the moment is rarely revisited.
How should duplicate quiz submissions be handled?
Keep one lead and store the submissions as a history. Retaking an assessment is a genuine engagement signal, so overwriting the earlier answers loses information about how the situation changed. Make the most recent result the active score for routing, and record the earlier ones as events. Counting each submission as a new lead inflates volume and distorts cost per lead.
Should duplicates be merged or deleted?
Merged, with the losing record retained in an archived state where the system allows it. Deleting removes the activity history and any consent record attached to that contact, which can matter legally as well as commercially. An archived original also makes an unmerge possible when a rule turns out to be wrong, which happens more often than most teams expect.
How do duplicates affect lead scoring?
They split the evidence. Half a person's page views sit on one record and half on another, so neither crosses the scoring threshold and a qualified buyer looks lukewarm twice. Merging usually raises scores rather than lowering them. Recalculate scores after a large cleanup, because the merged records carry a combined activity history the original scores never saw.
How often should deduplication run?
Continuously at the point of entry, and as a batch job on a cadence matched to intake volume: weekly for high-volume inbound, monthly for slower pipelines. Always run it before a large import and before any campaign send, since those are the two moments duplicates cause visible damage. Ad hoc cleanups before a board report are the least useful timing.