[PVFS2-users] Low performance of PVFS2 on our small cluster
silvestre at zabala.name
Wed Oct 27 22:15:57 EDT 2004
* Rob Ross <rross at mcs.anl.gov> [2004-10-27 15:17]:
> To start with, you are asking a lot from those nodes to run two compute
> processes and the pvfs2-client *and* the pvfs2-server. What does
> performance look like if you only run 5 compute processes total instead of
> 10? Perhaps you're just seeing the impact of too many processes on a
Ok, now I've run the pvfs2-servers on 5 dedicated IO-nodes and 5
b_eff_io processes on 5 dedicated compute-nodes.
b_eff_io = 35.041 MB/s on 5 processes with 1000 MByte/PE
But since we have SMP nodes I've doubled the b_eff_io process count:
b_eff_io = 38.948 MB/s on 10 processes with 500 MByte/PE
Then I mixed the processes again. So now we have 10 nodes each
running one pvfs2-server and one b_eff_io process:
b_eff_io = 50.945 MB/s on 10 processes with 1000 MByte/PE
And again, "utilizing" the second CPU, by running two b_eff_io
b_eff_io = 46.055 MB/s on 20 processes with 500 MByte/PE
So, what do all this number mean? I think running as many pvfs2-servers
as possible is best and running two b_eff_ios per nodes may be too much.
(However, with so few data points, one could conclude many things :) )
In fact doubling the number of pvfs2-servers doubled the b_eff_io score,
which is nice.
Now, next, would be trying Nathan's suggestions. And of course using
ext2 instead of ext3 (To avoid switching to another on-disk format).
One datapoint more:
I have a super-simple MPI program that reads and writes large
not overlapping chunks into a file. Running on 10 nodes it gets
a read bandwidth of 450 MB/s, which is very good, as that's exactly the
aggregated bw of all disks, but a write bw of only 30 MB/s. I hadn't
expected such a difference.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Size: 189 bytes
Desc: Digital signature
Url : http://www.beowulf-underground.org/pipermail/pvfs2-users/attachments/20041027/74ab8290/attachment.bin
More information about the PVFS2-users