Improve fwrite and fprintf Functional I/O performance

http://www.matlabsky.com/thread-34861-1-1.html

 

 

Today we are going to talk about the famous fwrite(fprintf) function , They are used for binary ( Text ) File write operation . because fwrite Function is the bottom layer I/O function , And it's used very often , Many users will question , How could it have room for performance improvement , If there is MathWorks It's already updated .

 

Flushing and Buffer

 

Unlike C/C++ Language , stay MATLAB Call in fwirte(fprintf) Function time ,MATLAB Will automatically refresh (flush) Output buffer (buffer), This point MATLAB There are no direct instructions in the help document for , It's just fopen There is an implicit reference to

 

See here , Many users should guess what to do . When writing data , If there is no cache (buffer), So every time I call fwrite function , A file write operation is needed , This way, it will seriously reduce I/O performance !

 

Let's look at a set of comparative data , First of all, we don't use caching

data = randi(250,1e6,1);  % Generate a set of data , For testing I/O performance

% Standard form , No application cache output - slow

fid = fopen('demo.dat', 'wb'); % w It's lowercase ,b For binary

tic, for idx = 1:length(data), fwrite(fid,data(idx)); end, toc

fclose(fid);

Elapsed time is 14.983201 seconds.

Consumed about 15s, Let's take a look at how caching is used

% Cache output mode – fast 3 times

fid = fopen('demo.dat', 'Wb'); % Be careful W It's capital ,b For binary

tic, for idx = 1:length(data), fwrite(fid,data(idx)); end, toc

fclose(fid);

Elapsed time is 5.616357 seconds.

After using the cache, the time is reduced to 5.6s, It seems that the efficiency has been improved a lot !

We can't understand MathWorks Why will fopen The default is non cached mode , But maybe they have a reason ! But when you're writing large data files , Recommend or use caching mode ('w') Well !

 

Chunking I/O

 

Write with cache , In fact, it is to reduce the number of access to data files , So in MATLAB in , If all the data is ready first , And then call it all at once fwrite Function to write it to a file

fid = fopen('demo.dat', 'wb'); % Notice that it's lowercase w

tic, fwrite(fid,data); toc

fclose(fid);

Elapsed time is 0.034816 seconds.

You can see that even in non cached mode , write in 30ms Our efficiency is still very high .

 

But if we read and write web files , It may take a long time due to network reasons , At this time, big data is broken into many small pieces , And then it's a wise choice to deal with it in blocks .

h = waitbar(0, 'Saving data...', 'Name','Saving data...');

cN = 100;  % number of steps/chunks

% Divide the data into chunks (last chunk is smaller than the rest)

dN = length(data);

dataIdx = [1 : round(dN/cN) : dN, dN+1];  % cN+1 chunk location indexes

% Save the data

fid = fopen('test.dat', 'Wb');

for chunkIdx = 0 : cN-1

   % Update the progress bar

   fraction = chunkIdx/cN;

   msg = sprintf('Saving data... (%d%% done)', round(100*fraction));

   waitbar(fraction, h, msg);

   % Save the next data chunk

   chunkData = data(dataIdx(chunkIdx+1) : dataIdx(chunkIdx+2)-1);

   fwrite(fid,chunkData);

end

fclose(fid);

close(h);

in general , The techniques in this paper are also applicable to fprintf and fwrite function , But storing and reading binaries (fwrite/fread) Far faster than text files (fprintf/fscanf/textscan), So if the data is not for human readability , Try to save in binary !

 

Source document <http://blog.sina.com.cn/s/blog_61c0518f0101cckt.html>

MATLAB To improve fwrite and fprintf Functional I/O More articles on performance

  1. MFC in CString Type use fprintf Function written to the file garbled solution

    It was recorded in the last article that fprintf Function to write content to a file , But there's a problem : The generated file string is garbled . terms of settlement : use _ftprintf function in addition , It is said that : unicode If you want to use fwprintf    ...

  2. matlab Solving linear equations in rref function

    Excerpt from :http://www.maybe520.net/blog/987/ matlab How to solve a system of linear equations in ? matlab Cramer's law can be used to solve linear equations in (Cramer's Rule) That is, through det( ...

  3. Matlab The training function and learning function of neural network are very common in computer science

    One . Training functions 1.traingd Name:Gradient descent backpropagation ( Gradient descent back propagation algorithm ) Description:triangd is a networ ...

  4. matlab in m File with the m Learning and understanding of functions

    1. m File with the m Difference of function So-called MATLAB Program , It can be roughly divided into two categories : M Script files (M-Script) and M function (M-function), They are all ordinary ASCII A file made up of codes . M Script ...

  5. MATLAB Import data from :importdata function

    use load Function import mat You're all going to . But today I got a piece of data , The file suffix is '.data'. How to read it ? I just use it matlab Interface Workspace Regional "import data&quo ...

  6. matlab Medium size(),length(),ndims() How to use the function

    1.size() Usage method : size(a) Represents the length of each dimension of a matrix example size([1 2 3;4 5 6]) be equal to [2 3]: It means that he has 2 That's ok 3 Column . size([1 2 3]) be equal to [1 3]: Express ...

  7. matlab Medium linkage and cluster function

    Linkage: Agglomerative hierarchical cluster tree( Agglomerate into a hierarchical clustering tree ) grammar : explain : Z=linkage(x), return Z, It's a X A hierarchical clustering tree of rows in a matrix ( use ...

  8. Matlab Medium cell、size function

    Refer to the website :http://blog.sina.com.cn/s/blog_5efed5800100exmj.html Cell function If p For a number , that h(1)=p, There is no problem . If p It's a vector , that ...

  9. matlab in fix, floor, ceil, round How to use the function

    Reprint : https://www.ilovematlab.cn/thread-91895-1-1.html Matlab Integral functions have : fix, floor, ceil, round. The specific application methods are as follows : ...

Random recommendation

  1. MyBatis Source code analysis (2)—— Plugin principle

    @(MyBatis)[Plugin] MyBatis Source code analysis --Plugin principle Plugin principle Plugin The implementation of Java Dynamic proxy for , Applied the chain of responsibility design pattern InterceptorChain ...

  2. JavaScript And FileSystemObject

    What is? FileSystemObject( abbreviation FSO)  FSO File system objects , It's a list Windows Disk directories and files , Delete directories and files . newly build . Copy . shear . Mobile and other operating technologies . Use FSO Website ...

  3. Use WiX Toolset establish .NET The app Bootstrapper( Install policy management )( One )----- First time to know WiX ( turn )

    Original address :http://blog.csdn.net/rryqsh/article/details/8274832 Visual Studio Seven sins of packing and installing Development .NET People who , Sure to use Visual ...

  4. PowerDesigner How to add common fields in each table and add comments automatically

    PowerDesigner How to add common fields in each table : Sometimes when creating a table, there are some common fields , But every table has to be created , It's troublesome to do so , Especially this repetitive work , If you don't pay attention, there will be problems , In fact, PD There's something like this ...

  5. opengl note

    GL_ARRAY_BUFFER( Represents vertex data ) GL_ELEMENT_ARRAY_BUFFER( Represents index data ) GL_PIXEL_UNPACK_BUFEER( To convey to O p e n G L The pixel ...

  6. Mac maven Environment variable configuration

    I've been learning to use Macbook, Record all the problems here Origin of problem : 1.Macbook Installed Eclipse,Eclipse Loading plug-ins maven & git , But in the git in clone ...

  7. C#Winform Implement automatic update

    Server side : [WebMethod] public string GetNewService(string version) { // Compare by version number if (version == "v1.0 ...

  8. Android Get current network status

    Android Get current network status design sketch There's a network No network Source code Download address (Android Studio engineering ):http://download.csdn.net/detail/q4878802/9052 ...

  9. 20155324《 Cyber warfare 》Exp07 Network fraud prevention

    20155324< Cyber warfare >Exp07 Network fraud prevention Practice content Simple application SET Tools to build fake websites ettercap DNS spoof Combining the two technologies , use DNS spoof Guide specific access to ...

  10. 2072. Kirill the Gardener 3

    http://acm.timus.ru/problem.aspx?space=1&num=2072 recall #include <iostream> #include <st ...