Online Tool Store
Duplicate Line Remover
Remove duplicate lines while keeping the original order, collapse adjacent runs the way uniq does, or list only the repeated lines. Optional case and whitespace insensitivity affects the comparison, never your text. Runs in your browser.
🔒 This tool runs entirely in your browser. Your files are never uploaded to a server.
Everything happens in this page and nothing is uploaded, which matters if the list is a set of email addresses, exported customer records or log output. Note that "first of each" preserves the original order, so it is safe on data where order carries meaning — sorting is a separate choice.
How to use it
- Paste your list, or open a text file.
- Pick what to keep — "first of each" is the usual answer.
- Turn on case or whitespace insensitivity if near-identical lines should count as the same.
- Check the summary and the repeated-lines table, then copy the output.
Four questions that sound like one
The sample loaded above has ten lines. Every mode is a defensible reading of "remove the duplicates", and they disagree:
input 10 lines
first of each 6 apple, Banana, cherry, banana, blank, date
collapse runs 9 only the cherry, cherry pair collapses
only repeated 2 apple, cherry
only one-offs 4 Banana, banana, blank, date
first, ignoring case 5 Banana and banana now collapse
Nothing here is a trick: each number is simply the answer to a different question. Which is why the mode is a visible choice rather than a hidden assumption — the wrong one produces a plausible-looking file that is quietly wrong.
The uniq surprise
"Collapse runs" takes the sample from ten lines to nine while leaving six distinct values. That looks broken and is not: it only removes a line that sits immediately below an identical one. The three cherries are adjacent so two go; the three apples are scattered so none do.
This is precisely what uniq does at a command line, and it is why
the idiom is sort | uniq rather than
uniq on its own. Sorting brings the repeats together so there is
something adjacent to collapse. The mode is here because when you are cleaning up wrapped or repeated log output, collapsing
runs while leaving legitimate later repeats alone is exactly right — and because it explains a behaviour that has confused
people at a terminal for decades.
Normalising the comparison, not the data
The case and whitespace options change only the key used to decide whether two lines are the same. The output keeps whatever
you actually typed. So deduplicating a list of addresses with "ignore case" on collapses
Sam@Example.com and
sam@example.com into one entry, and that entry retains its
original capitalisation rather than being lowercased on your behalf.
Whichever of the pair appeared first is the one kept. If which survivor you get matters, sort before deduplicating so the choice is deterministic rather than an artefact of input order.
A caution about deduplicating records
This compares whole lines as text. That makes it right for a list of values and wrong for a set of records where one field identifies the row: two CSV lines for the same customer with a different phone number are not duplicates by this measure, and neither is the same address with a trailing comma. If you are deduplicating structured data, extract the identifying column first — otherwise a clean-looking result can hide the fact that nothing was actually merged.
FAQ
Why are there four modes rather than one button?
Because "remove duplicate lines" describes four different operations, and on the ten-line sample loaded here they return six, nine, two and four lines. Keeping the first of each is what most people want. Collapsing runs is what the unix uniq does. Listing only the repeated lines is how you audit a list, and listing only the one-offs is how you find the entries that appear in one file but not another. A single button would just be one of these with the choice hidden.
Why does "collapse runs" barely change my file?
Because it only removes a repeat that sits directly below its twin. On the sample it takes ten lines to nine, even though only six are distinct. This is exactly the behaviour of uniq, and it is why the standard idiom is sort piped into uniq rather than uniq alone. If you want every repeat gone regardless of position, use "first of each" — or tick sort as well.
Does ignoring case change my text?
No. Case and whitespace options only affect the key used to decide whether two lines count as the same; the lines themselves are emitted exactly as you wrote them. So with "ignore case" on, Banana and banana collapse to one line and the survivor keeps its original capitalisation. Nothing is silently rewritten.
Is the original order kept?
Yes, unless you tick sort. That matters more than it sounds: deduplicating a log, a changelog or a CSV where row order is meaningful should not reorder anything. Sorting is available because it is often what you want for a plain list of values, but it is never the default.
What counts as a blank line?
A line with nothing in it, or only spaces and tabs, when the drop-blank option is on. They are removed before anything else so they cannot skew the counts. Left on, a blank line is treated as an ordinary value and can itself be deduplicated — which is usually what you want when tidying pasted text with double spacing.
Is my text uploaded?
No, it is processed in the page. Worth knowing here because the lists people deduplicate are so often email addresses, exported customer records or log output.
How we compare
| Feature | Online Tool Store | sort | uniq | Spreadsheet remove duplicates |
|---|---|---|---|
| Text never leaves your device | ✓ | ✓ | Depends |
| Keeps the original order | ✓ | ✗ | ✓ |
| Lists only the repeated lines | ✓ | uniq -d | ✗ |
| Shows how many times each line repeated | ✓ | uniq -c | ✗ |
| Nothing to install or open | ✓ | Terminal | ✗ |
| Deduplicates on one column of many | ✗ | With awk | ✓ |
| Handles a file too large for a browser | ✗ | ✓ | ✗ |
Best when you want to see what is being removed before you commit to it, and when order matters. For a multi-gigabyte log, use the command line — it streams rather than loading everything into memory. For deduplicating on a single column of a spreadsheet, use the spreadsheet.