[Pvfs2-users] PVFS2 on BlueGene

[Sam Lang]

Hi Matthew,

I think the version of PVFS in the Zepto release is pvfs2-1.5.1.   
Besides some performance improvements in the latest release  
(pvfs-2.6.3), there was a specific bugfix made in PVFS for largish  
mpi-io jobs.  If you could try the latest (at http://www.pvfs.org/),  
it would help us to verify that you're not running into the same  
problem.

Regarding config options for PVFS on BGL, make sure you have jumbo  
frames enabled, i.e.

ifconfig eth0 mtu 8000 up

Also, you should probably set the tcp buffer sizes explicitly in the  
pvfs config file, fs.conf:

<Defaults>
	...
         TCPBufferSend 524288
         TCPBufferReceive 1048576
</Defaults>

You might also see better performance with an alternative trove  
method for doing disk io:

<StorageHints>
	...
	TroveMethod alt-aio
</StorageHints>


Thanks,

-sam

On Apr 20, 2007, at 4:25 PM, Matthew Woitaszek wrote:

>
> Good afternoon,
>
> Michael Oberg and I are attempting to get PVFS2 working on NCAR's 1- 
> rack
> BlueGene/L system using ZeptoOS. We ran into a snag at over 8 BG/L  
> I/O nodes
> (>256 compute nodes).
>
> We've been using the mpi-io-test program shipped with PVFS2 to test  
> the
> system. For cases up to and including 8 I/O nodes (256 coprocessor  
> or 512
> virtual node mode tasks), everything works fine. Larger jobs fail  
> with file
> not found error messages, such as:
>
>    MPI_File_open: File does not exist, error stack:
>    ADIOI_BGL_OPEN(54): File /pvfs2/mattheww/_file_0512_co does not  
> exist
>
> The file is created on the PVFS2 filesystem and has a zero-byte  
> size. We've
> run the tests with 512 tasks on 256 nodes, and it successfully  
> created a
> 8589934592-byte file. Going to 257 nodes fails.
>
> Has anyone seen this behavior before? Are there any PVFS2 server or  
> client
> configuration options that you would recommend for a BG/L  
> installation like
> this?
>
> Thanks for your time,
>
> Matthew
>
>
>
> _______________________________________________
> Pvfs2-users mailing list
> Pvfs2-users at beowulf-underground.org
> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
>