回复:Re: [Pvfs2-users] Samba over PVFS2 performance

[Murali Vilayannur]

Hi Jiang,
> I do some improve of Samba write over PVFS2.
> And 2 IO nodes,1 client node can provide 160MB/s write performance from Samba clients now.
>   
Can you tell us how you got the speedups? Did you have to tune Samba 
clients or was this a setup issue?
> But the Samba read performance is not good,just 80MB/s.  :(
> 1.PVFS2 client seems not use the read cache,Can I use it?
>   
Again, we do not do caching, although it won't be too hard to do so 
using the generic page cache functions *if* you don't mind stale data 
and terrible performance.
As an aside, I am guilty of not finishing up a patch that should speed 
up kernel reads (for a special class of files with the immutable bits 
set) through the page-cache by batching them up. Hopefully, I will get 
some time to do it later this week/month, but no guarantees on when I 
can finish it though.
If you want to work on it, let me know and I can tell you how it would work.

> 2.Can I use the "readahead" in PVFS2?
> 3.What shall I do,modify any PVFS2 source code?
>   
Again, all of this requires support for reading from the page-cache in 
Linux.
Implicit readaheads and fadvise () based hints will then come for free.
The only issue that needs to be resolved is whether or not you can 
tolerate stale caches.
thanks,
Murali
> Please give me some advice,thanks so much.wait for your letter.
> Thank you Murali.
>
>
> Best regards
> --------------------------------------------------
>
>
>   
>> Hi Jiang,
>>
>>     
>>> HI Murali,
>>> I test the write performance just on pvfs2 client with big 
>>>       
>> blocks,the> performance seems good.
>>     
>>> and I test samba with COPY form windows clients to local disk(JBOD)
>>> without pvfs2,the local file system is XFS,and the write 
>>>       
>> performance is
>>     
>>> good too.about 100MB/s.
>>> But when I add the pvfs2 file system,the write performance is 
>>>       
>> bad,just> 20MB/s.
>>
>> Hmm.. This definitely seems to indicate that Samba over PVFS2 uses 
>> verysmall block sizes..
>>
>>     
>>> 1.you said " Due to the absence of any client-side caches, 
>>>       
>> multiple round
>>     
>>> trips ensues and performance drops.",shall I use the client-sid 
>>>       
>> caches?How> can I use it ?
>>
>> Nope. We haven't implemented this yet.
>>
>>     
>>> 2.you said "Can you check if the smbd server uses pvfs2 writev 
>>>       
>> function> instead of write? (perhaps by strace'ing the server?)",
>>     
>>>    How can I check it ?
>>>       
>> If the samba server is a UNIX userspace process, could you start 
>> up the
>> samba server like so
>> strace -f -o /tmp/log.txt -e file /path/to/smbd <arguments>
>> AFter the server is started, remount samba on all clients and redo 
>> yourtests.
>>
>> Check the contents of the file /tmp/log.txt and see if there are 
>> calls to
>> writev or write()?If there are calls to write(), check the sizes 
>> of the
>> system call arguments..
>> Hopefully that will point us to the problems and possibly solutions.
>> THanks,
>> Murali
>>
>>     
>>> Thank for help me very much!
>>>
>>> Best regards
>>> ----------------------------------------------
>>>
>>> ----- Original Message -----
>>> From: "Murali Vilayannur" <vilayann at mcs.anl.gov>
>>> Cc: <pvfs2-users at beowulf-underground.org>
>>> Sent: Monday, September 25, 2006 11:43 PM
>>> Subject: Re: [Pvfs2-users] Samba over PVFS2 performance
>>>
>>>
>>>       
>>>> Hi Philips,
>>>>
>>>>         
>>>>>     I met a Samba over PVFS2 prolem.So would anybody help me 
>>>>>           
>> please?>> >     I use the pvfs2 file system and export the files 
>> with Samba.The
>>     
>>>> hardware are two JBOD with
>>>>         
>>>>>     146G 10k FC disk, 4 pvfs2 IO servers,8 pvfs2 clients and some
>>>>>           
>>>> windows clients.Every IO server
>>>>         
>>>>>     manages 6+1 RAID5.the stripe size is 64k.
>>>>>     My problem is that How can we tuning the read/write 
>>>>>           
>> performance of
>>     
>>>> this system ?
>>>>         
>>>>>     this architecture,I test the performance with 2G 
>>>>>           
>> files.and find
>>     
>>>> the read performance is fast,but the
>>>>         
>>>>>      write performance is bad.Would you tell me what should 
>>>>>           
>> be care
>>     
>>>> about Samba over PVFS2?
>>>>
>>>> Can you verify whether PVFS2 performance (no Samba export, just 
>>>>         
>> native>> PVFS2) is bad on your setup first? What
>>     
>>>> benchmark did you use to measure the performance on your system?
>>>> Just wanted to make sure that it is a pvfs2 performance problem 
>>>>         
>> and not
>>     
>>>> a
>>>> samba tuning issue..
>>>>
>>>> I suspect what might happen with writes is that the VFS/smbd 
>>>>         
>> writes  out
>>     
>>>> single pages and hence PVFS2 sees only 4K/8K bytes at a time.
>>>> Due to the absence of any client-side caches, multiple round trips
>>>> ensues
>>>> and performance drops.
>>>> Can you check if the smbd server uses pvfs2 writev function 
>>>>         
>> instead of
>>     
>>>> write? (perhaps by strace'ing the server?)
>>>> thanks,
>>>> Murali
>>>>
>>>>         
>>>       
>>
>>     
>
> _______________________________________________
> Pvfs2-users mailing list
> Pvfs2-users at beowulf-underground.org
> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
>
>