Reprinted from Multicore CPU To speed up awk, sed, bzip2, grep, wc etc. , To view the original , Please click this link to enter .

------------------------------- I'm the divider Start ---------------------------------------

Want to let Linux The command uses all CPU kernel , We need to use GNU Parallel command , It makes all of us CPU The kernel does magic in a single machine map-reduce operation , Of course , It's with the help of something that's rarely used –pipes Parameters ( It's also called –spreadstdin). such , Your load will be evenly distributed to each CPU On , Really? .


bzip2 It's better than gzip Better compression tools , But it's slow ! Don't bother , We have a way to solve this problem .

Previous practice :

cat bigfile.bin | bzip2 --best > compressedfile.bz2

Now that's it :

cat bigfile.bin | parallel --pipe --recend '' -k bzip2 --best > compressedfile.bz2

Especially for bzip2,GNU parallel In multicore CPU It's super fast . If you don't pay attention , It's done .


If you have a very large text file , You might have been like this before :

grep pattern bigfile.txt

Now you can do this :

cat bigfile.txt | parallel  --pipe grep 'pattern'

Or so :

cat bigfile.txt | parallel --block 10M --pipe grep 'pattern'

This second usage uses –block 10M Parameters , This means that each kernel handles 1 Ten million lines —— You can use this parameter to adjust each CUP How many rows of data does the kernel process .


Here is a awk Command to calculate a very large data file example .

Routine usage :

cat rands20M.txt | awk '{s+=$1} END {print s}'

Now that's it :

cat rands20M.txt | parallel --pipe awk \'{s+=\$1} END {print s}\' | awk '{s+=$1} END {print s}'

It's a little complicated :parallel In the command –pipe Parameter will cat The output is divided into blocks and assigned to awk call , There are a lot of sub computation operations . These subcalculations go through a second pipe into the same awk command , So we can output the final result . first awk There are three backslashes , This is a GNU parallel call awk The need for .


Want to calculate the number of lines in a file as fast as possible ?

Traditional practice :

wc -l bigfile.txt

Now you should :

cat bigfile.txt | parallel  --pipe wc -l | awk '{s+=$1} END {print s}'

Very clever , First use parallel command ‘mapping’ A lot of wc -l call , Formers calculation , Finally, it's piped to awk To summarize .


I want to use it in a huge file sed Command to do a lot of replacement operations ?

routine :

sed s^old^new^g bigfile.txt

Now you can :

cat bigfile.txt | parallel --pipe sed s^old^new^g
 Then you can use the pipeline to store the output to a specified file .

------------------------------- I'm the divider end ---------------------------------------

Use multicore to accelerate Linux More articles on the command line

  1. wpa_supplicant Transplantation and linux Command line mode configuration of wireless Internet

    The content of this paper is linux Command line mode configuration of wireless Internet And wpa_supplicant The process of porting to the development board , For reference only . 1. Source download wpa_supplicant Source code download address :http://hosta ...

  2. Linux Switch between command line environment and desktop environment

    1. Preface In most cases , We are using Linux I'm used to using the command line environment when I'm working , But sometimes you still use the installation desktop environment , So here's how to install desktop environment for systems without desktop environment . With Centos 6.5 For example, demonstrate a ...

  3. Core system command practice Chapter one Linux Command line introduction

    Chapter one Linux Command line introduction 1.1 Linux Command line Overview 1.1.1 Linux Command line on and off Turn on : Login account password to enter the system sign out :exit/logout  Shortcut key :Ctrl+d 1.1.2 Li ...

  4. 40 It's a very interesting Linux Command line eggs and games

    40 An interesting Linux Command line eggs and games , Let you pretend to be an expert hacker . One click installation script is attached , In raspberry pie and ubuntu The test on the virtual machine is successful , Some of them can be in Windows Of DOS Run on command line . This article is a complete set B Stop video :40 individual ...

  5. Use it like a hacker Linux Command line

    Preface Before I saw an introduction in the blog Garden IntelliJ IDEA Configuration article , It uses gif Dynamic pictures to show , I think it's very good . So in my blog posts today and in the future , I will also try to use gif We're going to show it in motion pictures . Make ...

  6. Detailed explanation Linux Command line common svn command

    1.Linux Under the command line, the file checkout Go to the local directory svn checkout path(path It's the directory on the server ) for example :svn checkout svn:// ...

  7. linux Command line Cursor movement skills

    linux Command line Cursor movement skills It's a great experience to see a real expert operating the command line - The cursor moves back and forth between words , Different scrolls on the command line . There's a strong build-up here GUI The developers of the program try to work at the prompt . But it's not ...

  8. stay linux Execute at the command line php Program

    How to be in linux At the command line , perform php Program . Example Print current time php -r "echo time()" Random output of a number php -r "echo rand(1,20) ...

  9. git agent ,windows Command line proxy ,linux Command line proxy

    Download immovable settings agent :git config --global http.proxy config --global https.proxy https:/ ...

Random recommendation

  1. Java Using agents Proxy Access to the network and its verification

    stay Java In network programming , Sometimes the intranet server needs to access the network resources of the extranet , In this case, you need to use a proxy . Setting agent (Proxy) There are two ways : 1. By setting system properties (System.setPropery(String ...

  2. Android Medium Context Detailed explanation

    Preface : I read this article <Android Kernel analysis > The first 7 Chapter   After the formation of reading notes , I would like to know Android Frame book recommends this book . Hello everyone ,  Today, let's introduce our most familiar and unfamiliar friends in application development ---- ...

  3. UI4_ Registration login interface

    // // ViewController.h // UI4_ Registration login interface // // Created by zhangxueming on 15/7/3. // Copyright (c) 2015 year ...

  4. Windows Command line (DOS command ) course -3( Reprint )

    5、 ... and . Common commands DOS There are about a hundred orders in all ( Including text editing . Kill the virus . The configuration file . Batch processing, etc ), Let's introduce in detail 20 commonly used DOS command . Let's first introduce the concept of wildcards . wildcard * and ? * Represents a string ? It's just one ...

  5. Java Array copy

    System Provides a static method arraycopy(), We can use it to copy between arrays . Its function prototype is : public static void arraycopy(Object src, int sr ...

  6. php Realization ‘ Browse record ’ Group by date

    Issue : The existing test data are as follows , It is required to achieve the effect as shown in the figure below : array(4) { [0] => array(6) { ["visit_id"] => int(127) [&quo ...

  7. 4.Node.js Wechat message management

    One . Let me write it out front When the user sends the message to the official account ( Or when events are pushed by certain user actions ), Will produce a POST request , Developers can use the (Get) Return to specific XML structure , To respond to the message . Message push is also ...

  8. sql basic

    SELECT: select * from table select Name from table select DISTINCT Name from table INSERT: insert into t ...

  9. Virtual machine building and installation Hadoop And start up

    hibernate hadoop The first lesson : Virtual machine building and installation hadoop And start up hibernate hadoop The second lesson :hdfs Cluster centralized management and hadoop File operations hibernate hadoop The third class :java Development hdfs hibernate hadoop The first ...

  10. android scheme Open tmall , TaoBao

    Go straight to the code Intent intent = new Intent(); intent.setAction("android.intent.action.VIEW"); /* Str ...