[PVFS-users] PVFS API Performance with Multiple Processes

pvfstester at fastmail.com.au pvfstester at fastmail.com.au
Tue Dec 7 06:14:25 EST 2004

Hello all,

I have been experimenting with PVFS for a while, and recently I have run
into some problems I can't figure out. I was hoping that someone might
know what I am doing wrong or have some suggestions for improving

I have three machines connected by a gigabit network, each acting as a
PVFS client and an IOD. Two machines have one 1GHz processor each and
1GB of RAM; the third has two 3GHz Xeons with HyperThreading and 2GB of
RAM. Each machine has a single IDE drive which contains the operating
system, software, and the PVFS root directory. All machines are running
RedHat AS 3.

I am testing out an existing piece of software which was written using
standard UNIX I/O and which does all of its reading and writing in large
blocks (multiple MB per read/write). The application is generally I/O
bound, and I am using a fairly large (60GB) data file to perform my
tests. During the course of execution, the application reads the entire
input file, performs some manipulation of the data, writes the file back
out to temporary several files a bit at a time, reads the temporary
files back in, manipulates the data some more, and writes the data out
to the final output file.

The goal of my testing is to determine if there is a gain in speed when
the application is run on all three machines rather than just one. I
have had success running the application (as-is) from a single machine
(the dual-Xeon machine) with all input and output on the PVFS
filesystem. I have also had success running the application (as-is) from
all three machines at once by having each machine lseek() and then
process its 1/3 of the file. However, when I replace the UNIX I/O calls
with PVFS I/O (read() -> pvfs_read(), write() -> pvfs_write(), open() ->
pvfs_open(), close() -> pvfs_close(), stat() -> pvfs_stat(), and lseek()
-> pvfs_lseek()), the application's performance decreases significantly.

If I run the application (with PVFS I/O calls) from one node, it slows
to a crawl unless I tell it to write the temporary files to local disk.
If I run the application (with PVFS I/O calls) from all three nodes, it
slows to a crawl no matter how I distribute the input/temporary/output
files. If I run a simple disk benchmark using similar I/O buffer sizes,
it shows significant improvement when the UNIX I/O calls are replaced by
PVFS I/O calls even if I run the benchmark on all three machines at

Has anyone else run into similar performance problems? Any suggestions
would be greatly appreciated.

Thank you in advance,
  pvfstester at fastmail.com.au

More information about the PVFS-users mailing list