文章

Linux下大文件的排序和去重复

Linux下我们用 sortuniq 的命令来实现去重复行。

去重复行

简单的用法如下,如一个文件名:happybirthday.txt

cat happybirthday.txt (显示文件内容)

Happy Birthday to You!
Happy Birthday to You!
Happy Birthday Dear Tux!
Happy Birthday to You!

cat happybirthday.txt|sort (排序)

Happy Birthday Dear Tux!
Happy Birthday to You!
Happy Birthday to You!
Happy Birthday to You!

cat happybirthday.txt|sort|uniq (去重复行)

Happy Birthday Dear Tux!
Happy Birthday to You!

阅读更多