securitybreach Posted May 30, 2013 Share Posted May 30, 2013 Uniq command is helpful to remove or detect duplicate entries in a file. This tutorial explains few most frequently used uniq command line options that you might find helpful.The following test file is used in some of the example to understand how uniq command works. $ cat test aa aa bb bb bb xx 1. Basic Usage Syntax: $ uniq [-options] For example, when uniq command is run without any option, it removes duplicate lines and displays unique lines as shown below. $ uniq test aa bb xx 2. Count Number of Occurrences using -c option This option is to count occurrence of lines in file. $ uniq -c test 2 aa 3 bb 1 xx......... http://www.thegeekst...mmand-examples/ 2 Quote Link to comment Share on other sites More sharing options...
ichase Posted May 30, 2013 Share Posted May 30, 2013 Always cool these command line tool tutorials you find. Not sure for what reason one may find the uniq tool useful but am sure if trying to find something in a long text file, it may be useful. 1 Quote Link to comment Share on other sites More sharing options...
trigggl Posted May 30, 2013 Share Posted May 30, 2013 I looked at the uniq man page and there was one very important paragraph The command first compares adjacent lines and then removes the second and succeeding duplications of a line. Duplicated lines must be adjacent. (Before issuing the uniq command, use the sort command to make all duplicate lines adjacent.) The file has to be sorted before the uniq command can be used. Sorting may not be something you would want to do in some files. So, if you have the following: aa bb aa cc aa bb |==[ -7 Uinal and -19 K'in until doomsday! ]==|| $ cat test aa bb aa cc aa bb |==[ -7 Uinal and -19 K'in until doomsday! ]==| | $ sort test > test_sorted |==[ -7 Uinal and -19 K'in until doomsday! ]==| | $ uniq test_sorted aa bb cc By the way, it's -7 Uinal and -19 K'n until doomsday. I made a script to count down according to the Mayan long count. 1 Quote Link to comment Share on other sites More sharing options...
V.T. Eric Layton Posted May 31, 2013 Share Posted May 31, 2013 Mouse-over, highlight, and delete in MS Word is so much easier than all this techie stuff. 1 Quote Link to comment Share on other sites More sharing options...
securitybreach Posted May 31, 2013 Author Share Posted May 31, 2013 By the way, it's -7 Uinal and -19 K'n until doomsday. I made a script to count down according to the Mayan long count. Right Quote Link to comment Share on other sites More sharing options...
trigggl Posted May 31, 2013 Share Posted May 31, 2013 Mouse-over, highlight, and delete in MS Word is so much easier than all this techie stuff. At work I always have to delete a large number of duplicates in a list of harness numbers. I use Excel to split off the prefix and suffix before using the "Remove Duplicates" function. Quote Link to comment Share on other sites More sharing options...
V.T. Eric Layton Posted May 31, 2013 Share Posted May 31, 2013 Sometimes, we have to utilize the tools that work best for the circumstances. 1 Quote Link to comment Share on other sites More sharing options...
steel Posted July 15, 2013 Share Posted July 15, 2013 I use this command a lot in everyday use what i do is create backup copy of everything i work on here is what i mean the file i want to work on called test $ cat test aa cc dd ef gh gh dd de ab cc gh i want no duplicates and all in order in create backup of test sort test | uniq > test2 i created a file called test2 which has $ cat test2 aa ab cc dd de ef gh I know you guys know all this stuff but maybe it helps others ... 1 Quote Link to comment Share on other sites More sharing options...
securitybreach Posted July 15, 2013 Author Share Posted July 15, 2013 Nice Quote Link to comment Share on other sites More sharing options...
V.T. Eric Layton Posted July 15, 2013 Share Posted July 15, 2013 UH-OH! Someone has hijacked Steel's old account here at Scot's. Oh, wait... maybe that really is Steel posting? Quote Link to comment Share on other sites More sharing options...
steel Posted July 15, 2013 Share Posted July 15, 2013 and i thought i was typing in invisible ink 1 Quote Link to comment Share on other sites More sharing options...
tforsman Posted July 15, 2013 Share Posted July 15, 2013 let's say i have: a=nowhere@everywhere b=nowhere@everywhere c=nowhere@everywhere d=nowhere@everywhere is there a way to just delete =nowhere@everywhere and nothing else? So I end up with only, a,b,c,d in that file.... Quote Link to comment Share on other sites More sharing options...
securitybreach Posted July 15, 2013 Author Share Posted July 15, 2013 Something like sed -i 's/=nowhere@everywhere/ /g' *.txt Basically this: sed -i 's/old-word/new-word/g' *.txt 1 Quote Link to comment Share on other sites More sharing options...
V.T. Eric Layton Posted July 15, 2013 Share Posted July 15, 2013 He sed, she sed, they sed... but who really knows? Quote Link to comment Share on other sites More sharing options...
sunrat Posted July 16, 2013 Share Posted July 16, 2013 Something like sed -i 's/=nowhere@everywhere/ /g' *.txt Wouldn't that replace the word with a space? Maybe this would work - sed -i 's/=nowhere@everywhere//g' *.txt Then again, I'm no expert. I would just use Find/Replace in Kate. 1 Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.