Remove Duplicate Lines from a List

Paste a list with one item per line and get back the unique lines only, in their original order, with a count of what was dropped.

Runs in your browserFree, no sign-upLast reviewed October 8, 2026

Summary

Original lines: 7
Unique lines: 5
Removed: 2

Result

banana apple cherry date elderberry

Quick answers

How do I remove duplicate lines from a text list?: Paste the list into the Lines box above, one entry per line. Every repeated line is deleted and the first copy stays where it was. Copy the Result or download it as a text file. Nothing has to be installed and no spreadsheet is needed.
Does removing duplicates change the order of my list?: No. The first occurrence of each line keeps its position and later copies are dropped, so b, a, b, c, a becomes b, a, c. The order changes only if you tick Sort the result A→Z.
What are unique lines?: The unique lines of a text are its distinct lines with every repeat counted once. A list of 7 lines in which 2 are repeats has 5 unique lines. This is different from lines that appear only once: here a repeated line is kept once, not discarded entirely.
Is Apple the same as apple when removing duplicates?: With Ignore letter case ticked, yes: Apple, apple and APPLE are one line and the first spelling in the list is the one kept. With it unticked the comparison is exact and all three stay.
Why are two lines that look identical both still there?: They differ in a way you cannot see. Common causes are a non-breaking space, a tab in place of a space, two spaces between words, a curly apostrophe against a straight one, or an accented letter stored in two different ways. Only spaces at the ends of a line are ignored.
Which copy is kept, the first or the last?: The first. If a list contains Sam@example.com and later sam@example.com with case ignored, the output shows Sam@example.com. To keep the last copy instead, reverse the line order before pasting and reverse the result afterwards.

Why use Remove Duplicate Lines

This tool keeps the first copy of every line and deletes each later repeat, without sorting unless you ask for it. The sample list of 7 fruit names contains apple and banana twice, so it comes back as 5 unique lines with 2 removed. Four checkboxes decide what counts as a repeat: Ignore letter case treats Apple and apple as the same line, Ignore leading/trailing spaces strips spaces and tabs from both ends before comparing, Keep blank lines controls whether empty lines survive, and Sort the result A→Z orders the output alphabetically. The first two are on by default. The Summary shows Original lines, Unique lines and Removed, and the result can be copied or downloaded as unique-lines.txt.

What each option does to the same list

Input, 9 lines: Apple, apple, [space]apple, Banana, a blank line, banana[space], Cherry, a blank line, APPLE.

Settings	Result	Unique lines	Removed
Defaults (ignore case and end spaces)	Apple, Banana, Cherry	3	6
Ignore letter case off	Apple, apple, Banana, banana, Cherry, APPLE	6	3
Ignore leading/trailing spaces off	Apple, [space]apple, Banana, banana[space], Cherry	5	4
Both off (exact match)	Apple, apple, [space]apple, Banana, banana[space], Cherry, APPLE	7	2
Defaults plus Keep blank lines	Apple, Banana, one blank line, Cherry	4	5
Ignore letter case off, Sort the result A→Z on	apple, Apple, APPLE, banana, Banana, Cherry	6	3

Removed includes blank lines that were dropped, not only repeats. With Keep blank lines on, blank lines are deduplicated like any other line, so only the first one survives.

Worked examples: emails, logs and URLs

Merging two email lists

Sam@example.com, sam@example.com, jo@example.com, [space]jo@example.com[space], a blank line, SAM@EXAMPLE.COM. Default settings.

Sam@example.com and jo@example.com. Original lines 6, Unique lines 2, Removed 4.

With Ignore letter case unticked the same list keeps 4 lines, because the three spellings of Sam's address count as different.

Finding the distinct messages in a log

ERROR timeout, INFO ok, ERROR timeout, WARN slow, ERROR timeout, INFO ok. Default settings.

ERROR timeout, INFO ok, WARN slow. Original lines 6, Unique lines 3, Removed 3.

Lines that start with a timestamp never match each other. Strip the timestamp column first.

A URL list

https://example.com/page, https://example.com/page/, http://example.com/page, https://example.com/page. Default settings.

Three lines kept, 1 removed.

Matching is on the text of the line. A trailing slash or http against https makes a different line, even if the addresses load the same page.

City names in mixed case, sorted

London, paris, LONDON, Berlin, Paris, london. Sort the result A→Z ticked.

Berlin, London, paris. Original lines 6, Unique lines 3, Removed 3.

paris stays in lower case because that was the first spelling in the list.

Lookalike lines that are not merged

Each pair below was tested with the default settings and stayed as two lines.

Difference	Example pair	How to fix it first
Two spaces inside the line	new[2 spaces]york / new york	Run the list through Remove Extra Spaces
Non-breaking space from a web page	new[nbsp]york / new york	Replace the odd space with a normal one using Find and Replace
Tab against a space	apple[tab]red / apple red	Replace tabs with spaces
Curly against straight apostrophe	it’s / it's	Replace curly quotes with straight ones
Accent stored as a separate mark	cafe + combining accent / café	Retype the accented letter in one of the lines
Trailing slash or different scheme in a URL	example.com/page / example.com/page/	Decide on one form and replace the other
Several items on one line	apple, banana, apple	Put each item on its own line first

Windows line endings are not a problem while Ignore leading/trailing spaces is ticked, because the carriage return at the end of each line is stripped before comparing. With that option unticked, a final line with no line break after it will not match an earlier copy from a Windows file.

How to remove duplicate lines in other programs

The same job in tools you may already have open.

Program	How	Keeps original order
Excel	Select the column, then Data > Remove Duplicates	Yes
Google Sheets	Data > Data cleanup > Remove duplicates, or the formula =UNIQUE(A2:A)	Yes
Notepad++	Edit > Line Operations > Remove Duplicate Lines	Yes
VS Code	Select the lines, open the Command Palette and run Delete Duplicate Lines	Yes
Sublime Text	Edit > Permute Lines > Unique	Yes
Linux or macOS terminal	sort -u file.txt	No, output is sorted
Linux or macOS terminal	awk '!seen[$0]++' file.txt	Yes
Windows PowerShell	Get-Content file.txt \| Select-Object -Unique	Yes
Python	list(dict.fromkeys(lines))	Yes

The uniq command on its own only removes repeats that are next to each other, which is why it is normally used after sort.

Sorting, blank lines and large lists

Sorting is a separate step from deduplicating and is off by default, because order often carries meaning: a log is chronological, a keyword list may be ranked, a playlist is sequenced. When Sort the result A→Z is ticked, the unique lines are ordered with a plain alphabetical comparison. Upper and lower case versions of a word sit together, but numbers are compared as text, so 10 and 100 come before 9 and item10 comes before item2. For numeric or natural ordering, deduplicate here and then use the Sort Lines tool.

Blank lines are removed unless Keep blank lines is ticked. Even then only one blank line is kept, since every blank line is a duplicate of the first, so this option is for keeping a single separator and not for preserving paragraph spacing.

The work is done in your browser by checking each line against a set of lines already seen, which takes one pass however long the list is. Lists of tens of thousands of lines are handled comfortably. Because nothing is uploaded, it is suitable for subscriber exports and other lists you would not want to send to a server.

Step by step

1
Paste your list into Lines, one item per line.
2
Leave Ignore letter case ticked to merge Apple and apple, or untick it when case matters, as it does for passwords, hashes and file paths.
3
Leave Ignore leading/trailing spaces ticked unless indentation is part of the data.
4
Tick Keep blank lines to retain an empty line as a separator, and Sort the result A→Z if order does not matter.
5
Check Removed in the Summary, then copy the Result or download unique-lines.txt.

Frequently asked questions

Does it remove every copy of a duplicated line or leave one?

It leaves one. The first copy of each line is kept and the later ones are deleted. It does not have a mode that deletes all copies of anything that repeats, and it does not list the duplicates on their own.

Can it remove duplicate words or comma-separated items on one line?

No. It compares whole lines. A line reading apple, banana, apple is one line and is left alone. Put each item on its own line first, for example by replacing the commas with line breaks, then paste the list here.

How many lines can it handle?

There is no fixed limit. Tens of thousands of lines are processed quickly because each line is checked once. Very large pastes are limited by your browser's memory, not by the tool.

What does the Removed number include?

Removed is Original lines minus Unique lines. That covers repeated lines and, unless Keep blank lines is ticked, the blank lines that were dropped. A list of 6 lines with one blank line and 3 repeats shows Removed 4.

Are the spaces at the ends of lines kept in the output?

Not when Ignore leading/trailing spaces is ticked. Lines are trimmed before comparing and the trimmed version is what appears in the result, so the output is cleaned as well as deduplicated. Untick the option to keep indentation exactly as pasted.

Can I compare two lists and remove the lines they share?

Not directly. Pasting both lists one after the other gives you the combined set of unique lines, which is the usual aim when merging. To see which lines differ between two versions, use the Text Diff Checker on this site.

Is my list uploaded anywhere?

No. The comparison runs in your browser and the list is not sent to a server. The download button creates the unique-lines.txt file on your own device.