About the Remove Duplicate Lines
Paste a list and this tool will remove duplicate lines from it, keeping the first occurrence of each entry exactly where it was and telling you how many it dropped. You can choose whether letter case matters, whether leading and trailing spaces should be ignored when comparing, whether blank lines survive and whether the finished list is sorted alphabetically. It handles tens of thousands of lines comfortably because the deduplication happens in your browser with a hash set, which also means an email list or an export full of customer data never leaves your machine.
How to use the Remove Duplicate Lines
- 1
Paste your list into the 'Lines' box, one item per line.
- 2
Leave 'Ignore letter case' ticked to treat 'Apple' and 'apple' as the same entry, or untick it for an exact match.
- 3
Adjust 'Ignore leading/trailing spaces', 'Keep blank lines' and 'Sort the result A→Z' to suit the list.
- 4
Check the Summary for original, unique and removed counts, then copy the result or download it as unique-lines.txt.
What people use it for
Cleaning an email or subscriber list
Exports merged from two sources almost always overlap. Deduplicating with case ignored catches the same address written as Sam@example.com and sam@example.com, which most mail platforms treat as one mailbox anyway.
Tidying keyword research
Keyword tools return the same phrase from several seed queries. Dropping duplicates before you import gives an accurate count of the terms you actually need to write for.
Deduplicating log or error output
A stack trace repeated 400 times hides the three other failures underneath it. Reducing the file to unique lines shows the real set of distinct problems at a glance.
What counts as a duplicate, and why the options matter
Two lines are duplicates only if they are byte-identical after the options you have chosen are applied. Ignoring case lowercases each line before comparing, so 'LONDON' and 'London' collapse into one, and the version kept is whichever appeared first. Trimming removes leading and trailing spaces and tabs before comparison, which matters more than people expect: a list copied out of a spreadsheet column or an indented code block frequently carries trailing spaces that are invisible on screen yet make two identical-looking rows compare as different. When trimming is on, the line is also stored trimmed, so the output is clean as well as deduplicated. There are duplicates this cannot catch, because they are not textually equal at all. Windows line endings, curly versus straight apostrophes, an accented character stored decomposed rather than composed, and a non-breaking space pasted from a web page all look identical yet differ underneath. If a pair of lines refuses to merge, that is nearly always the reason. Passing the list through Remove Extra Spaces first resolves most whitespace cases.
Order preservation and how it differs from sorting
The default behaviour keeps the first occurrence of each line in its original position and discards every later copy. That is deliberate: many lists carry meaning in their order — a chronological log, a ranked set of keywords, a playlist — and sorting would destroy it. Sorting is available as a separate option for when order is irrelevant and alphabetical grouping is easier to scan. Note that the sort applied here is a straightforward locale comparison, so mixed-case entries interleave and numbers sort as text, meaning 'item10' lands before 'item2'. If you need natural numeric ordering, deduplicate here and then run the result through the Sort Lines tool, which uses a numeric-aware collator. The Summary panel is worth reading rather than skipping: the gap between original and unique lines is often the fastest signal that an export ran twice or that a join in a query fanned out unexpectedly.
Tips
- Untick 'Ignore letter case' when the list contains case-sensitive values such as API keys, hashes or file paths on Linux.
- The removed count is a quick data-quality check — a large number usually means a duplicated export rather than genuine repeats.
- Deduplicate before sorting if you want the smallest possible output, since sorting alone never removes anything.