Linux下大文件的排序和去重复
Linux下我们用 sort 与 uniq 的命令来实现去重复行。
去重复行
简单的用法如下,如一个文件名:happybirthday.txt
cat happybirthday.txt (显示文件内容)
Happy Birthday to You!
Happy Birthday to You!
Happy Birthday Dear Tux!
Happy Birthday to You!
cat happybirthday.txt|sort (排序)
Happy Birthday Dear Tux!
Happy Birthday to You!
Happy Birthday to You!
Happy Birthday to You!
cat happybirthday.txt|sort|uniq (去重复行)
Happy Birthday Dear Tux!
Happy Birthday to You!