[Pvfs2-developers] Re: patch for client data corruption with 2.4kernels

[David Metheny]

I'm running the same types of tests I did with a RHEL3 and RHEL4 system. But
not seeing the same results on the RHEL5 systems. So... I'm not sure if it
really is a problem or not. 

Watching 'top' or using 'ps' I can see the VIRT and RSS sizes pretty much
match on initial start for pvfs2-client-core. With pvfs2 idle, I'm running
some external program that eats up the available memory, using more than our
physical RAM. 

On RHEL3/RHEL4 with the patch, the RSS sizes shows that memory gets swapped
out a bit, but leaves around 40MB which matches our size for I/O Buffers
(--desc-count 5 --desc-size 8388608).

Without the patch, on RHEL3, the RSS size would get really low, around 1K or
so, and when PVFS2 was used again, it would start putting it back in memory
and the RSS size would grow, however we would see that the data in the I/O
transfers at that point was wrong.

For the RHEL5 systems, I'm able to use the external program to eat up
memory, and see the RSS (shows up as RES in top) size get really low, but
I'm not able to push it back into memory, and can't get the data corruption
issue to trigger. 



-----Original Message-----
From: Rob Ross [mailto:rross at mcs.anl.gov] 
Sent: Friday, November 21, 2008 11:38 AM
To: david.metheny at gmail.com
Cc: pvfs2-developers at beowulf-underground.org
Subject: Re: [Pvfs2-developers] Re: patch for client data corruption with
2.4kernels

is there any way to do a quick verification that the buffer is still  
in the right place, on the kernel side, and then force a remapping?

rob

On Nov 21, 2008, at 11:19 AM, David Metheny wrote:

> I know this fixes RHEL3 (2.4 kernels) and the RHEL4 U6/U7 (2.6.9  
> kernels), but testing this with RHEL5 U2 (2.6.18-92.el5PAE) shows  
> that the memory is still being swapped out which would result in  
> client data corruption.
>
> From: pvfs2-developers-bounces at beowulf-underground.org
[mailto:pvfs2-developers-bounces at beowulf-underground.org 
> ] On Behalf Of Bart Taylor
> Sent: Thursday, November 20, 2008 4:32 PM
> To: pvfs2-developers at beowulf-underground.org
> Subject: [Pvfs2-developers] Re: patch for client data corruption  
> with 2.4kernels
>
> I neglected the header file; attached is an additional patch to  
> catch it. Both of these patches should apply cleanly to head.
>
> Bart.
>
> On Thu, Nov 20, 2008 at 1:54 PM, Bart Taylor <batayl at gmail.com> wrote:
> Hey guys,
>
> This patch fixes a client data corruption issue on Linux 2.4  
> systems. We worked with Phil off-list and determined that the  
> kernel's mapped buffers are not staying pinned in memory as they  
> should be. We were able to verify that under heavy swapping the  
> pvfs2-client-core memory was being moved, invalidating the pointers.  
> This fix uses mlock to hold the two buffers in place, and we were  
> able to verify that it fixes the swapping issue. There are compile  
> time flags to ensure it is only added for Linux 2.4 systems.
>
> Bart.
>
> _______________________________________________
> Pvfs2-developers mailing list
> Pvfs2-developers at beowulf-underground.org
> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers