Text

Duplicate Line Remover

Paste logs, lists, or exports and drop repeated lines. Choose trim-aware matching, case-insensitive comparison, keep first or keep last copy, and optional sorted output. Processing stays on your device — nothing is uploaded to DroidXP.

Ad placement — top banner

“Last occurrence” keeps the final copy of each logical line in file order. Use “Sort output” only when you want alphabetical order instead of original order.

Ad placement — mid rectangle

What does this tool do?

It scans your text line by line and removes lines that match an earlier (or later, if you choose “last occurrence”) line according to the rules you set. It is useful for cleaning duplicate URLs, email lists, log excerpts, CSV columns pasted as lines, and other one-item-per-line data.

Keep first vs keep last

First occurrence keeps the topmost copy and drops later repeats — order of unique lines stays as in the original (unless you sort). Last occurrence keeps the bottommost copy of each repeated line and drops earlier duplicates, which matters when later lines were corrected or updated.

Trim and case options

With trim on, leading and trailing spaces are ignored when comparing lines, so apple and  apple  count as the same line (the kept line keeps its original spacing). With case-insensitive on, Apple and apple match; the kept line is unchanged from the source.

Sort output

Optional sort reorders the deduplicated lines using the browser’s localeCompare (numeric-aware). This destroys original order — use it when you want a sorted unique list, not when sequence matters.

How to use this tool

  1. Step 1: Paste text with one logical item per line (try Load sample).
  2. Step 2: Set trim, case, keep first/last, and sort as needed.
  3. Step 3: Copy the result into your editor, spreadsheet, or script.
  4. Step 4: For huge files, prefer a desktop tool or CLI; browsers may slow on very large pastes.

Frequently Asked Questions

Does DroidXP upload my text?

No. Deduplication runs in your browser. Drafts may be saved in localStorage on this device only.

How are “duplicate lines” detected?

Each line is turned into a comparison key: optionally trimmed and optionally lowercased. Lines with the same key are considered duplicates; one copy is kept according to your “first” or “last” setting.

Are empty lines handled?

Yes. Multiple blank lines usually collapse to one kept empty line, because empty keys match. If you need to preserve repeated blank lines, this tool is not the right fit.

Is this the same as Unix uniq?

Similar goal, but uniq traditionally expects sorted input for a full unique pass. Here you can dedupe without sorting, or sort in the tool when you want alphabetical unique lines.

What localStorage keys are used?

Keys are prefixed with droidxp-dlr-: input text, trim, case-insensitive, keep mode, and sort. Clear removes them.

Will this work with Unicode and emoji?

Yes. Lines are compared as JavaScript strings. Case folding uses toLowerCase(), which follows Unicode rules for many scripts; verify edge cases for your language.

Do Windows (CRLF) line endings matter?

Lines are split on both \n and \r\n. Trailing \r is not usually visible but is part of the line unless you trim.

Can I dedupe across paragraphs?

Only whole lines are compared. Multi-line records should be normalized elsewhere (e.g. one record per line) before using this tool.

Is this safe for confidential lists?

Follow your policy. Nothing is sent to DroidXP, but screen capture, extensions, and backups can still expose clipboard or page content.

Can I use this offline?

After the page is cached, it can work offline. No network is required for deduplication.