uniq - report or omit repeated lines
Remove duplicate lines:
uniq file.txt
Display the number of repetitions for each line:
uniq -c file.txt
Display only duplicate lines:
uniq -D file.txt
Display only non duplicate lines:
uniq -u file.txt
Remove duplicate lines skipping the first field:
uniq -f 1 file.txt
Remove duplicate lines skipping five initial characters:
uniq -s 5 file.txt
Limit the comparison to five initial characters:
uniq -w 5 file.txt