Linux下大文件的排序和去重复
Linux下我们用 sort 与 uniq 的命令来实现去重复行。
去重复行
简单的用法如下，如一个文件名：happybirthday.txt
cat happybirthday.txt (显示文件内容)
Happy Birthday to You!
Happy Birthday to You!
Happy Birthday Dear Tux!
Happy Birthday to You!
cat happybirthday.txt|sort （排序）
Happy Birthday Dear Tux!
Happy Birthday to You!
Happy Birthday to You!
Happy Birthday to You!
cat happybirthday.txt|sort|uniq (去重复行)
Happy Birthday Dear Tux!
Happy Birthday to You!