[PVFS2-users] PVFS2 performance blues...

[Rob Ross]

Hi Angel,

I think the reason that you're getting the performance improvement is 
that you're now using the ROMIO PVFS2 support, while it's likely you 
were going through the UFS layer before, which has more overhead.

We're aware of the date issue; we know how to fix that one and will do 
so soon.

PVFS2 is in use on the Chiba City machine at ANL and the large P4 
cluster at OSU, and is being tested at Univ. of Utah for production use 
and for use on the Jazz machine and the IBM BG/L machine at ANL. 
Remember that we only released the 1.0 version in November, so we're 
happy that we've had this many people jump on board so soon!

It sounds like there will be packaged versions of PVFS2 with some of the 
cluster distributions soon as well, which I'm sure will encourage adoption.

Regards,

Rob

Angel de Vicente wrote:
> Hi,
> 
> I wasn't sure I could fix the problems as they were, so I decided to go the
> usual fix-it-all method: start all over. I deleted all the PVFS2 partitions, and
> reinstalled, and I got further than before...
> 
>  > On Wed, Mar 30, 2005 at 05:47:01PM +0100, Angel de Vicente wrote:
>  > > I simply did 10 times 
>  > > qsub -l nodes=4,walltime=08:00:00 mpi-io_i1000_delete_mpich2.pbs
>  > > where mpi-io_i1000_delete_mpich2.pbs is:
>  > > 
>  > > #!/bin/bash
>  > > NP=$(wc -l $PBS_NODEFILE | awk '{print $1}')
>  > > rm /mnt/pvfs2/foo1000_delete_mpich2.out
>  > > cd $PBS_O_WORKDIR
>  > > mpirun -np $NP ./mpi-io-test_mpich-2.1.0 -i 1000 -f /mnt/pvfs2/foo1000_delete_mpich2.out -p /etc/pvfs2tab
>  > 
>  > We just recently fixed a problem in ROMIO where PVFS2 files would not
>  > be treated like UFS files if you left off the "pvfs2:" prefix.  
>  > 
> 
> Funnily, in my previous attempts, this didn't give me an error message. When I
> tried to run it now, it gave me a problem which said something about UFS
> (probably what you mention above), so I changed the line:
> 
> mpirun -np $NP ./mpi-io-test_mpich-2.1.0 -i 1000 -f /mnt/pvfs2/foo1000_delete_mpich2.out -p /etc/pvfs2tab
> 
> to
> 
> mpiexec -n $NP ./mpi-io-test_mpich-2.1.0 -i 1000 -f pvfs2:/mnt/pvfs2/foo1000_delete_mpich2.out -p /etc/pvfs2tab
> 
> With this it all seems to work fine now (using mpich2). The files are being
> deleted and recreated apparently without problems, and actually I got much
> better performance than before (due to what, I do not know, MPICH2,
> ROMIO?). Before I got performance in the order of 14MB/s both read and write and
> now I got almost linear improvement (on NFS I got around 10MB/s, and with 4
> PVFS2 IO server I got around 35MB/s), which is excellent!
> 
> -----------
> nodes (4 cpu total):
> nodo3
> nodo2
> nodo1
> nodo0
> 
> # Using mpi-io calls.
> nr_procs = 4, nr_iter = 1000, blk_sz = 16777216
> # total_size = 67108864000
> # Write: min_t = 1778.461002, max_t = 1783.518778, mean_t = 1781.081419, var_t = 4.943779
> # Read:  min_t = 1828.179753, max_t = 1881.055890, mean_t = 1858.338576, var_t = 623.435660
> Write bandwidth = 37.627226 Mbytes/sec
> Read bandwidth = 35.676167 Mbytes/sec
> --------------
> 
> So, it all looks very good, except the creation date of the file as you can see
> below (I wasn't even around then :-)). When I create the file through touch, cp,
> etc. the date is OK, though.
> 
> [beo at nodo0 PVFS2-tests]$ ls -lh /mnt/pvfs2/
> total 63G
> -r-x-w-r-x  1 beo  beo   63G Jan  8  1970 foo1000_delete_mpich2.out
> drwxrwxrwx  1 root root 4.0K Apr  5 09:41 lost+found
> [beo at nodo0 PVFS2-tests]$
> 
> Is this a common issue? Any ideas?
> 
> Thanks,
> Angel de Vicente
> 
> ps. By the way, we are in the process of installing a new cluster, and I'm
> trying to push into the PVFS2 direction, but the convincing job would be easier
> if we knew of other production clusters using it. Do you have such a list, any
> statistics, something to show to the other people involved in the decision
> making?