[Pvfs2-users] pvfs2 performance with btio

[Sam Lang]

On Jul 17, 2007, at 2:26 PM, Murali Vilayannur wrote:

> Hi florin,
> I suspect this might have to do with your platform's AIO libraries
> being broken and us having to work around that.
> Could you try the alt-aio stuff and see if that improves things a bit?
> Sam, there are some config file settings for this, isn't it? I forget
> what we need to modify..

In the <StorageHints> context:

TroveMethod alt-aio

Unless the data he's just written is sitting in the kernel buffers, I  
would expect reads to have the same problem as writes if aio is the  
cause.  What makes you suspect AIO libraries for his platform?

> Would it make any difference if configure works around aio-threaded- 
> callbacks?

Do you mean configure is disabling threaded callbacks for his build,  
or that we should ask it to?  AIO results we've seen without threaded  
callbacks are worse than with them.

>
> I am really tempted to just write our own aio libraries and on-disk
> backends instead of relying on Linux/glibc and ext3..sigh

:-)  I hear you.  Are you running over ext3 Murali?  I've seen  
results that suggest xfs might be better for large IOs and multiple  
threads.

-sam

> Thanks,
> Murali
>
> On 7/17/07, Florin Isaila <florin.isaila at gmail.com> wrote:
>> Hi Sam, we start the pvfs2 servers on different machines than the
>> compute nodes (picking the nodes from the list provided by the batch
>> system). Was that your question?
>>
>> And I should have said, all the measurements are done with collective
>> I/O of ROMIO.
>>
>> On 7/17/07, Sam Lang <slang at mcs.anl.gov> wrote:
>> >
>> > Ah, I read your email wrong.  Hmm...so writes really tank.  Are you
>> > using the storage nodes as servers, or other compute nodes?
>> >
>> > -sam
>> >
>> > On Jul 17, 2007, at 11:15 AM, Sam Lang wrote:
>> >
>> > >
>> > > Hi Florin,
>> > >
>> > > Just one clarification question...are those are bandwidth numbers
>> > > not seconds as the plot label suggests?
>> > >
>> > > -sam
>> > >
>> > > On Jul 17, 2007, at 11:03 AM, Florin Isaila wrote:
>> > >
>> > >> Hi everybody,
>> > >>
>> > >> I have a question about the PVFS2 write performance.
>> > >>
>> > >> We did some measurements with BTIO over PVFS2 on lonestar at  
>> TACC
>> > >> (http://www.tacc.utexas.edu/services/userguides/lonestar/)
>> > >>
>> > >> and we get pretty bad write results with classes B and C:
>> > >>
>> > >> http://www.arcos.inf.uc3m.es/~florin/btio.htm
>> > >>
>> > >> We used 16 I/O servers, the default configuration parameters  
>> and upto
>> > >> 100 processes. We realized that all I/O servers were used  
>> also as
>> > >> metadata servers, but BTIO uses just one file.
>> > >>
>> > >> The times are in seconds, contain only I/O time (no compute  
>> time) and
>> > >> are aggregated per each BTIO run (BTIO performs several writes).
>> > >>
>> > >> TroveSyncMeta was set to yes (by default). Could this cause  
>> the I/
>> > >> O to
>> > >> be serialized? It looks as if there were a serialization.
>> > >>
>> > >> Or could the fact that all nodes were also launched as metadata
>> > >> managers affect the performance?
>> > >>
>> > >> Any clue why this happens?
>> > >>
>> > >> Many thanks
>> > >> Florin
>> > >> _______________________________________________
>> > >> Pvfs2-users mailing list
>> > >> Pvfs2-users at beowulf-underground.org
>> > >> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
>> > >>
>> > >
>> >
>> >
>> _______________________________________________
>> Pvfs2-users mailing list
>> Pvfs2-users at beowulf-underground.org
>> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
>>
>