Remove Duplicate Lines

Duplicates are more expensive than they look

A duplicated line rarely stays a cosmetic problem for long. A mailing list with the same address three times sends the same person three copies of your newsletter, which is the fastest route to a spam complaint and real damage to your sending reputation. A keyword list with repeats inflates your apparent coverage and skews whatever analysis you run on it. Duplicate product IDs in an import file can create phantom inventory or overwrite the wrong record. Duplicate entries in a survey export quietly distort the results. In each case the cost is not the wasted line, it is the wrong decision made downstream on the strength of data that looked clean. Deduplicating at the point of entry is dramatically cheaper than untangling the consequences later.

Why case and whitespace options matter so much

Exact matching is the default because it is predictable, but it is often too strict to be useful. Real-world lists are assembled by hand, pasted from several sources, and exported from systems with different conventions, so the same underlying item shows up as 'Alice@Example.com', 'alice@example.com' and 'alice@example.com ' with an invisible trailing space. Those are three different strings to a computer and one person to you. Email addresses in particular are case-insensitive in their domain part by specification, so treating them as distinct is simply wrong. Turning on case-insensitive matching and whitespace trimming usually finds substantially more duplicates than the naive comparison, which is precisely the point — the ones that slip past exact matching are the ones you would never have spotted by eye.