[PVFS2-users] PVFS2 performance

[Robert Latham]

On Tue, Mar 01, 2005 at 02:49:42PM +0000, Angel de Vicente wrote:
> Hi all,
> 
> I just installed PVFS2 in a test 4 nodes cluster (head node metadata
> server, the four nodes IO nodes). Also, following the instructions
> on section 5 of the quick start guide, I installed the kernel
> interface in the four machines, and run the pvfs2-client. It all
> seems OK, I can see /mnt/pvfs2 from all machines, and I can copy,
> move, etc. files around with the regular commands: ls, cp, etc.
> 
> Now, just copying (with cp) a big file to /mnt/pvfs2 or to another
> place through NFS gives much better performance to NFS, but I
> understand that's probably normal, so I try to run the mpi-io-test
> at ftp://ftp.parl.clemson.edu/pub/pvfs2/tests/
> 
> I have mpich-1.2.6, the code compiles with no problems and then I test.

If possible, you should use mpich2
(http://www.mcs.anl.gov/mpi/mpich2/).  The PVFS2 driver in mpich-1.2.6
is somewhat older, won't perform as well, and probably has bugs that
were fixed in mpich2.

> First, to a NFS mounted directory:
> 
> [beo at nodo0 ~]$ mpirun -np 4 mpi-io-test -i 10 -f /net/nodo0/scratch/foo.out -p /etc/pvfs2tab
> # Using mpi-io calls.
> nr_procs = 4, nr_iter = 10, blk_sz = 16777216
> # total_size = 671088640
> # Write: min_t = 5.339317, max_t = 66.259956, mean_t = 50.615525, var_t = 911.269773
> # Read:  min_t = 1.053671, max_t = 48.641617, mean_t = 36.042611, var_t = 544.747562
> Write bandwidth = 10.128118 Mbytes/sec
> Read bandwidth = 13.796594 Mbytes/sec

Your variance is unusually high, which ususally means some sort of
serialization is happening.  

> And then to /mnt/pvfs2
> 
> [beo at nodo0 ~]$ mpirun -np 4 mpi-io-test -i 10 -f /mnt/pvfs2/foo.out -p /etc/pvfs2tab
> # Using mpi-io calls.
> nr_procs = 4, nr_iter = 10, blk_sz = 16777216
> # total_size = 671088640
> # Write: min_t = 43.714029, max_t = 46.666999, mean_t = 45.611818, var_t = 1.691665
> # Read:  min_t = 44.097879, max_t = 47.578564, mean_t = 46.287230, var_t = 2.755476
> Write bandwidth = 14.380368 Mbytes/sec
> Read bandwidth = 14.104853 Mbytes/sec
> [beo at nodo0 ~]$

These variances are more like what we would expect.  

> As you can see, the gain is not great. Is this the way to run it?
> Shouldn't I get a better gain? Do I have to install the kernel
> interface for these tests?

If you can cp and ls /mnt/pvfs2 from your nodes, then you already have
the kernel interface installed.  

Can you see what happens if you prefix the file name with 'pvfs2:',
just to ensure that we are using the PVFS2 functions and not the
unix-like routines.

==rob

-- 
Rob Latham
Mathematics and Computer Science Division    A215 0178 EA2D B059 8CDF
Argonne National Labs, IL USA                B29D F333 664A 4280 315B