[PVFS2-users] Low performance of PVFS2 on our small cluster

[Silvestre Zabala]

* Rob Ross <rross at mcs.anl.gov> [2004-10-27 15:17]:

Hi,

> To start with, you are asking a lot from those nodes to run two compute 
> processes and the pvfs2-client *and* the pvfs2-server.  What does 
> performance look like if you only run 5 compute processes total instead of 
> 10?  Perhaps you're just seeing the impact of too many processes on a 
> node?

Ok, now I've run the pvfs2-servers on 5 dedicated IO-nodes and 5
b_eff_io processes on 5 dedicated compute-nodes.

b_eff_io =   35.041 MB/s on 5 processes with 1000 MByte/PE

But since we have SMP nodes I've doubled the b_eff_io process count:

b_eff_io =   38.948 MB/s on 10 processes with 500 MByte/PE

Then I mixed the processes again. So now we have 10 nodes each 
running one pvfs2-server and one b_eff_io process:

b_eff_io =   50.945 MB/s on 10 processes with 1000 MByte/PE

And again, "utilizing" the second CPU, by running two b_eff_io
processes:

b_eff_io =   46.055 MB/s on 20 processes with 500 MByte/PE

So, what do all this number mean? I think running as many pvfs2-servers
as possible is best and running two b_eff_ios per nodes may be too much.
(However, with so few data points, one could conclude many things :) )

In fact doubling the number of pvfs2-servers doubled the b_eff_io score,
which is nice.

Now, next, would be trying Nathan's suggestions. And of course using
ext2 instead of ext3 (To avoid switching to another on-disk format).

One datapoint more: 
I have a super-simple MPI program that reads and writes large 
not overlapping chunks into a file. Running on 10 nodes it gets
a read bandwidth of 450 MB/s, which is very good, as that's exactly the
aggregated bw of all disks, but a write bw of only 30 MB/s. I hadn't
expected such a difference. 

Best regards,
  Silvestre

-- 
http://silvestre.zabala.name/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://www.beowulf-underground.org/pipermail/pvfs2-users/attachments/20041027/74ab8290/attachment.bin