7 Linux Uniq Command Examples to Remove Duplicate Lines from File

[securitybreach]

Uniq command is helpful to remove or detect duplicate entries in a file. This tutorial explains few most frequently used uniq command line options that you might find helpful.

The following test file is used in some of the example to understand how uniq command works.

$ cat test

aa

aa

bb

bb

bb

xx

1. Basic Usage

 

Syntax:

$ uniq [-options]

For example, when uniq command is run without any option, it removes duplicate lines and displays unique lines as shown below.

$ uniq test

aa

bb

xx

 

2. Count Number of Occurrences using -c option

 

This option is to count occurrence of lines in file.

$ uniq -c test

2 aa

3 bb

1 xx.........

http://www.thegeekst...mmand-examples/

[ichase]

Always cool these command line tool tutorials you find. Not sure for what reason one may find the uniq tool useful but am sure if trying to find something in a long text file, it may be useful.

[trigggl]

I looked at the uniq man page and there was one very important paragraph

 

The command first compares adjacent lines and then removes the second and succeeding duplications of a line. Duplicated lines must be adjacent.

(Before issuing the uniq command, use the sort command to make all duplicate lines adjacent.)

 

The file has to be sorted before the uniq command can be used. Sorting may not be something you would want to do in some files.

 

So, if you have the following:

 

aa

bb

aa

cc

aa

bb

 

|==[ -7 Uinal and -19 K'in until doomsday! ]==|

| $ cat test

aa

bb

aa

cc

aa

bb

 

|==[ -7 Uinal and -19 K'in until doomsday! ]==|

| $ sort test > test_sorted

 

|==[ -7 Uinal and -19 K'in until doomsday! ]==|

| $ uniq test_sorted

aa

bb

cc

 

By the way, it's -7 Uinal and -19 K'n until doomsday. I made a script to count down according to the Mayan long count.

[V.T. Eric Layton]

Mouse-over, highlight, and delete in MS Word is so much easier than all this techie stuff.

[securitybreach]

By the way, it's -7 Uinal and -19 K'n until doomsday. I made a script to count down according to the Mayan long count.

 

Right

[trigggl]

Mouse-over, highlight, and delete in MS Word is so much easier than all this techie stuff.

 

At work I always have to delete a large number of duplicates in a list of harness numbers. I use Excel to split off the prefix and suffix before using the "Remove Duplicates" function.

[V.T. Eric Layton]

Sometimes, we have to utilize the tools that work best for the circumstances.

[steel]

I use this command a lot in everyday use what i do is create backup copy of everything i work on here is what i mean

the file i want to work on called test

 

 

 

$ cat test

 

aa

cc

dd

ef

gh

gh

dd

de

ab

cc

gh

 

i want no duplicates and all in order in create backup of test

 

sort test | uniq > test2

i created a file called test2 which has

 

 

 

$ cat test2

aa

ab

cc

dd

de

ef

gh

 

I know you guys know all this stuff but maybe it helps others ...

[securitybreach]

Nice

[V.T. Eric Layton]

UH-OH! Someone has hijacked Steel's old account here at Scot's. Oh, wait... maybe that really is Steel posting?

[steel]

and i thought i was typing in invisible ink

[tforsman]

let's say i have:

 

a=nowhere@everywhere
b=nowhere@everywhere
c=nowhere@everywhere
d=nowhere@everywhere

 

 

is there a way to just delete =nowhere@everywhere and nothing else? So I end up with only, a,b,c,d in that file....

[securitybreach]

Something like

sed -i 's/=nowhere@everywhere/ /g' *.txt

 

Basically this:

sed -i 's/old-word/new-word/g' *.txt

[V.T. Eric Layton]

He sed, she sed, they sed... but who really knows?

[sunrat]

Something like

sed -i 's/=nowhere@everywhere/ /g' *.txt

Wouldn't that replace the word with a space? Maybe this would work -

sed -i 's/=nowhere@everywhere//g' *.txt

 

Then again, I'm no expert. I would just use Find/Replace in Kate.