[Pvfs2-developers] Re: patch for client data corruption with 2.4kernels

[Phil Carns]

Maybe the memory accounting is just handled differently in RHEL5?

-Phil

David Metheny wrote:
> I'm running the same types of tests I did with a RHEL3 and RHEL4 system. But
> not seeing the same results on the RHEL5 systems. So... I'm not sure if it
> really is a problem or not. 
> 
> Watching 'top' or using 'ps' I can see the VIRT and RSS sizes pretty much
> match on initial start for pvfs2-client-core. With pvfs2 idle, I'm running
> some external program that eats up the available memory, using more than our
> physical RAM. 
> 
> On RHEL3/RHEL4 with the patch, the RSS sizes shows that memory gets swapped
> out a bit, but leaves around 40MB which matches our size for I/O Buffers
> (--desc-count 5 --desc-size 8388608).
> 
> Without the patch, on RHEL3, the RSS size would get really low, around 1K or
> so, and when PVFS2 was used again, it would start putting it back in memory
> and the RSS size would grow, however we would see that the data in the I/O
> transfers at that point was wrong.
> 
> For the RHEL5 systems, I'm able to use the external program to eat up
> memory, and see the RSS (shows up as RES in top) size get really low, but
> I'm not able to push it back into memory, and can't get the data corruption
> issue to trigger. 
> 
> 
> 
> -----Original Message-----
> From: Rob Ross [mailto:rross at mcs.anl.gov] 
> Sent: Friday, November 21, 2008 11:38 AM
> To: david.metheny at gmail.com
> Cc: pvfs2-developers at beowulf-underground.org
> Subject: Re: [Pvfs2-developers] Re: patch for client data corruption with
> 2.4kernels
> 
> is there any way to do a quick verification that the buffer is still  
> in the right place, on the kernel side, and then force a remapping?
> 
> rob
> 
> On Nov 21, 2008, at 11:19 AM, David Metheny wrote:
> 
>> I know this fixes RHEL3 (2.4 kernels) and the RHEL4 U6/U7 (2.6.9  
>> kernels), but testing this with RHEL5 U2 (2.6.18-92.el5PAE) shows  
>> that the memory is still being swapped out which would result in  
>> client data corruption.
>>
>> From: pvfs2-developers-bounces at beowulf-underground.org
> [mailto:pvfs2-developers-bounces at beowulf-underground.org 
>> ] On Behalf Of Bart Taylor
>> Sent: Thursday, November 20, 2008 4:32 PM
>> To: pvfs2-developers at beowulf-underground.org
>> Subject: [Pvfs2-developers] Re: patch for client data corruption  
>> with 2.4kernels
>>
>> I neglected the header file; attached is an additional patch to  
>> catch it. Both of these patches should apply cleanly to head.
>>
>> Bart.
>>
>> On Thu, Nov 20, 2008 at 1:54 PM, Bart Taylor <batayl at gmail.com> wrote:
>> Hey guys,
>>
>> This patch fixes a client data corruption issue on Linux 2.4  
>> systems. We worked with Phil off-list and determined that the  
>> kernel's mapped buffers are not staying pinned in memory as they  
>> should be. We were able to verify that under heavy swapping the  
>> pvfs2-client-core memory was being moved, invalidating the pointers.  
>> This fix uses mlock to hold the two buffers in place, and we were  
>> able to verify that it fixes the swapping issue. There are compile  
>> time flags to ensure it is only added for Linux 2.4 systems.
>>
>> Bart.
>>
>> _______________________________________________
>> Pvfs2-developers mailing list
>> Pvfs2-developers at beowulf-underground.org
>> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
> 
> _______________________________________________
> Pvfs2-developers mailing list
> Pvfs2-developers at beowulf-underground.org
> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers