To read an entire file efficiently you need to know the file size FIRST.
So whats the fastest way to programatically find the file size. If you read stackoverflow you’ll find a number of suggestions, but many use fseek to jump to the end of the file, find your location and use that as the file size…
TL;DR – This works, but is 30% slower than using stat or fstat
I just ran a bunch of tests comparing using
fstat also comparing using file streams and file descriptors to see what seems to be the fastest. For the test I create a 100M file.
TL;DR – using file descriptors, fstat and read was the fastest and using file streams and seek was the slowest. Go to the bottom to see the real slowest.
For the test I ran this on a small Linux box I have running a headless ArchLinux server. I ran the test: checking the file size,
malloc a buffer, read the entire file into the buffer, close the file, free the buffer.
I ran the test 3 times with 1000 cycles each time and using
clock_gettime to calculate the elapsed time.
Just simply comparing JUST the time it takes to get the file size using
fstat were at least 30% faster than using
Comparing just the speed of using file streams vs file descriptors, they were pretty nearly the same – descriptors were about 1-3% faster.
In comparing getting the file size, opening the file,
malloc a buffer, read the entire 100M, close the file and free the buffer — using file descriptors and fstat were 6-8% faster than using
lseek. Probably because the bulk of the time is spent in the file read vs the getting the file size, which dilutes the overall performance benefit.
BTW – do not use
fgetc and read the file 1 character at a time. This is crazy inefficient and really really slow! Like 1700% slower!!!!