=Start=<\/p>\n
\u5728\u8ba2\u9605\u7684\u535a\u5ba2\u5217\u8868\u4e2d\u770b\u5230\u4e00\u7bc7\u6587\u7ae0\u300c\u5bf9\u5927\u6587\u672c\u6587\u4ef6\u8fdb\u884c\u53bb\u91cd<\/a>\u300d\uff0c\u89c9\u5f97\u5185\u5bb9\u4e0d\u9519\uff0c\u4f46\u662f\u539f\u6587\u7ae0\u6ca1\u6709\u7ed9\u51fa\u4ee3\u7801\u7684\u51fa\u5904\u94fe\u63a5\uff0c\u6240\u4ee5\u624b\u52a8\u641c\u7d22\u4e86\u4e00\u756a\uff0c\u6316\u5230\u4e86\u66f4\u591a\u7684\u77e5\u8bc6\uff0c\u5728\u6b64\u8bb0\u5f55\u4e00\u4e0b\u3002<\/p>\n site:stackoverflow.com MAX_LINES_PER_CHUNK http:\/\/vkundeti.blogspot.com\/2008\/03\/tech-algorithmic-details-of-unix-sort.html<\/a><\/p>\n sorting large files faster with a shell script<\/p>\n \uff08\u591a\u770bmanual<\/strong><\/span>\uff09\u4f7f\u7528-T\u9009\u9879\u624b\u52a8\u6307\u5b9a\u4e34\u65f6\u76ee\u5f55\uff1b\u4f7f\u7528-S\u9009\u9879\u6307\u5b9a\u5141\u8bb8sort\u547d\u4ee4\u4f7f\u7528\u7684\u5185\u5b58\u5927\u5c0f\uff1b\u5982\u679c\u670d\u52a1\u5668\u662f\u591a\u6838\u7684\u8bdd\uff0c\u8fd8\u53ef\u4ee5\u4f7f\u7528–parallel\u9009\u9879\u8bbe\u5b9a\u5e76\u53d1\u4efb\u52a1\u91cf\u4ee5\u63d0\u9ad8\u901f\u5ea6\u3002\u5728\u67d0\u4e9b\u7279\u6b8a\u60c5\u51b5\u4e0b\uff0c\u4f60\u751a\u81f3\u53ef\u4ee5\u901a\u8fc7\u624b\u52a8\u8bbe\u5b9a\u73af\u5883\u53d8\u91cf\u300cLC_ALL=C\u300d\uff0c\u6765\u63d0\u9ad8\u5904\u7406\u901f\u5ea6(\u907f\u514d\u89e3\u6790UTF-8\u6587\u672c\uff0c\u4ee5\u53ca\u8fdb\u884c\u590d\u6742\u7684\u6392\u5e8f\u64cd\u4f5c)\u3002<\/p>\n Look carefully at the options of sort to speed performance and understand it’s impact on your machine and problem. Key parameters on Ubuntu are<\/p>\n The questioner asks “Why no high memory usage?” The answer to that comes from history, older unix machines were small and the default memory size is set small. Adjust this as big as possible for your workload to vastly improve sort performance. Set the working directory to a place on your fastest device that has enough space to hold at least 1.25 * the size of the file being sorted.<\/p>\n ==<\/p>\n Buffer it in memory using -S. For example, to use (up to) 50% of your memory as a sorting buffer do:<\/p>\n Note that modern Unix sort can sort in parallel. My experience is that it automatically uses as many cores as possible. You can set it directly using –parallel. To sort using 4 threads:<\/p>\n So all in all, you should put everything into one file and execute something like:<\/p>\n ==<\/p>\n Using the sort command will probably be the fastest option.<\/p>\n But you’ll probably want to fix the locale to C.<\/p>\n sort -u doesn’t report unique lines, but one of each set of lines that sort the same. In the C locale, 2 different lines necessarily don’t sort the same, but that’s not the case in most UTF-8 based locales on GNU systems.<\/p>\n Also, using the C locale avoids the overhead of having to parse UTF-8 and processing complex sort orders so would improve performance dramatically<\/span><\/strong>.<\/p>\n So:<\/p>\n You can also improve performance by using a faster drive (or a different drive from the one where the input and\/or output files are) for the temporary files (using -T or $TMPDIR environment variable), or by fiddling with the -S option supported by some sort implementations).<\/p>\n For some type of input or for slow storage, using the –compress-program option of GNU sort(for instance with lzop) might improve performance in addition to storage usage.<\/p>\n =EOF=<\/p>\n","protected":false},"excerpt":{"rendered":" =Start= \u5728\u8ba2\u9605\u7684\u535a\u5ba2\u5217\u8868\u4e2d\u770b\u5230\u4e00\u7bc7\u6587\u7ae0\u300c\u5bf9\u5927\u6587\u672c\u6587\u4ef6\u8fdb\u884c\u53bb\u91cd\u300d\uff0c\u89c9\u5f97\u5185\u5bb9\u4e0d\u9519\uff0c\u4f46\u662f\u539f\u6587\u7ae0\u6ca1\u6709\u7ed9\u51fa\u4ee3\u7801\u7684 […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[23,11,7,12],"tags":[30,292,19],"class_list":["post-2609","post","type-post","status-publish","format-standard","hentry","category-knowledgebase-2","category-linux","category-programing","category-tools","tag-linux","tag-sort","tag-tips"],"views":6497,"_links":{"self":[{"href":"https:\/\/ixyzero.com\/blog\/wp-json\/wp\/v2\/posts\/2609","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ixyzero.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/ixyzero.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/ixyzero.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/ixyzero.com\/blog\/wp-json\/wp\/v2\/comments?post=2609"}],"version-history":[{"count":0,"href":"https:\/\/ixyzero.com\/blog\/wp-json\/wp\/v2\/posts\/2609\/revisions"}],"wp:attachment":[{"href":"https:\/\/ixyzero.com\/blog\/wp-json\/wp\/v2\/media?parent=2609"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/ixyzero.com\/blog\/wp-json\/wp\/v2\/categories?post=2609"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/ixyzero.com\/blog\/wp-json\/wp\/v2\/tags?post=2609"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
\n\u641c\u7d22\u5173\u952e\u5b57\/\u53c2\u8003\u94fe\u63a5\uff1a<\/h6>\n
\nhttp:\/\/stackoverflow.com\/questions\/930044\/how-could-the-unix-sort-command-sort-a-very-large-file<\/a><\/p>\n\u641c\u7d22\u5173\u952e\u5b57\/\u53c2\u8003\u94fe\u63a5\uff1a<\/h6>\n
\n
\u53c2\u8003\u89e3\u7b54\uff1a<\/h6>\n
\n
sort -S 50% file<\/pre>\n
sort --parallel=4 file<\/pre>\n
sort -S 50% --parallel=4 file<\/pre>\n
LC_ALL=C sort -u file<\/pre>\n