Viewing and processing text

Command uniq

Remove or count duplicate lines

Linux · Bash examples · commands run on the selected server

When to use uniq

Use uniq to compare adjacent lines. Sort first when equal records can occur in different parts of a file, but preserve original ordering through another method if sequence matters.

Syntax

uniq [PARAMETERS] [INPUT [OUTPUT]]

Square brackets mark optional arguments; an ellipsis means repetition. Replace example file names, user names and server addresses with your own values.

Common options and arguments

-c
Show number of repetitions.
-d
Show duplicate groups only.
-u
Show only non-repeating lines.

Usage examples

Remove adjacent duplicates

uniq ./sorted.txt

Only consecutive repeated lines are collapsed.

Count identical lines

sort ./names.txt | uniq -c

Sorting groups equal names so occurrence counts cover the entire input.

Show duplicate values

sort ./names.txt | uniq -d

Only names that occur repeatedly are printed after sorting.

Notes and limitations

  • Only adjacent rows are compared: for a shared list, first do sort.
  • The order and concept of equality may depend on the locale and the selected options.

Common questions and troubleshooting

Why are duplicate lines still present?

Only adjacent identical lines are compared. Separate occurrences remain unless you group them first or use a tool with a full-input key set.

Will sort followed by uniq preserve input order?

No. Sorting changes order. If original order matters, use an order-preserving deduplication method and consider the memory needed for its key set.

Further help

Options vary by version and distribution. Check the installed documentation with this command: man uniq.