You have 4,000 email addresses pasted out of six spreadsheets. They need deduplicating, and afterwards the count is suspiciously high. The problem is almost never the deduplicator. It is that ben@x.com and ben@x.com are different strings, and you removed duplicates before removing the trailing spaces.
Most text-cleaning frustration is this: the operations are trivial, and the order is not.
The order that works
- Normalise whitespace. Trim leading and trailing spaces on every line, collapse runs of spaces, normalise line endings.
- Remove empty lines. After trimming, because a line containing three spaces is not empty until you trim it.
- Normalise case, but only if case does not matter for your data.
- Deduplicate. Now identical things are actually identical strings.
- Sort. Last, because everything above changes what sorts where.
- Count, to check the result is plausible.
Running whitespace cleanup first is what makes steps 2 and 4 behave. Running empty line removal before trimming misses every line made of spaces or tabs.
Step 3 is the one that needs judgement. Email addresses are case-insensitive in the domain and, in practice, in the local part too, so lower-casing before deduplicating is correct and catches Ben@X.com. Usernames, passwords, product codes and file paths are frequently case-sensitive, and flattening them merges records that were not duplicates.
Sorting has one common trap
Sorting lines is straightforward except when the text is tabular.
The other surprise is that alphabetical sorting is not numeric sorting. Sorted as text, 10 comes before 9, because 1 precedes 9. If the lines start with numbers of varying length, you need a numeric sort or zero-padded values.
And sorting is not stable across locales. Accented characters, uppercase letters and digits order differently depending on the collation rules being applied, which is why the same list sorted in two tools can differ.
Case conversion loses information in one direction
Converting between camelCase, snake_case, kebab-case and PascalCase is mechanical in both directions, because each one marks word boundaries explicitly: a capital letter, an underscore, a hyphen.
Converting to a flat lowercase string discards those boundaries permanently. parseHTTPResponse becomes parsehttpresponse, and nothing can reliably split it again.
Acronyms are where converters disagree even in the good direction. parseHTTPResponse could reasonably become parse_http_response or parse_h_t_t_p_response, and different tools choose differently. If you are converting a list of identifiers, check the ones with acronyms in them rather than assuming the whole list is fine.
A case converter handles the mechanical part; the acronyms are the part to eyeball.
Counting is less defined than it looks
Paste the same paragraph into three tools and you may get three word counts. None of them is wrong, because "word" is not a defined term.
The disagreements are over hyphenated compounds (one word or two), numbers and standalone symbols, URLs and email addresses, and contractions. Different tools split on different boundaries.
This matters when a limit is involved. A 500-word essay limit, a 160-character SMS, a 2,200-character Instagram caption or a 60-character page title are all enforced by a specific implementation, and the only count that matters is that one.
Placeholder text is not filler
Lorem ipsum survives for a reason that is easy to dismiss. Latin-looking text has roughly the word-length distribution of English without being readable, so reviewers evaluate the layout instead of the copy. Type real English into a mockup and every review meeting becomes about the wording.
Two things to be careful about. Lorem ipsum has no long words, no numbers and no unusual characters, so a layout that fits it may break on a real German compound noun or a long product name. And it must never reach production, which it does more often than anyone admits, usually in an error state or an empty-list message nobody tested.
Doing it on real data
The reason this matters more than it sounds: the text people clean is mailing lists, exported user records, support transcripts and log extracts. A mailing list pasted into a random online deduplicator has been disclosed to whoever runs it, whatever the page says.
Keep the order: clean the whitespace first, then remove duplicates, then sort, and count at the end to confirm how many lines survived each step.