[Pvfs2-users] File affinity and CPU resource scheduling concerns when using compute node as a PVFS server

[Chris Worley]

On Wed, Oct 21, 2009 at 12:31 PM, Kyle Schochenmaier <kschoche at gmail.com> wrote:
> Chris -
>
> A lot of work has been done with pvfs2 and infiniband.
> Pete Wyckoff has many publications with it -  (I cant locate them
> right now from my old link to his publications)
> There are some documents from my work at AmesLab floating around, and
> a careful search of the mailing list archives here will yield some
> user-contributed data.
>
> If you specify what type of performance metrics you are looking for I
> might be able to help further.  We can take that offline.

I've got drives that do 800MB/s locally, and scale linearly (locally)
up to 6GB/s as you add drives.

Exporting these via SRP from one target, I can get 2.7GB/s on the
initiator using one QDR IB port, and 3.7GB/s using two QDR IB ports
(off two different HCA's on the same target).

How would PVFS compare and scale?

Thanks,

Chris
P.S.  thanks to Rob for the sanity check.
>
> Hope that helps.
>
> Kyle Schochenmaier
>
>
>
> On Wed, Oct 21, 2009 at 1:22 PM, Rob Ross <rross at mcs.anl.gov> wrote:
>> Hi Chris,
>>
>> Yes, you can certainly run PVFS servers on compute nodes. That's what we
>> used to do all the time back in "the day", but it's still possible.
>>
>> The file system does not migrate files. In part this is because it would
>> dramatically complicate metadata management, in part because we are
>> frequently running in environments where there's no locality to exploit.
>>
>> You can choose to distribute over fewer (or just one) node, and there is
>> functionality to discover where a file is stored, so you can map your
>> computation onto the storage.
>>
>> Obviously yes, doing work on some other node will slow down computation on
>> that node, how much depends on many things.
>>
>> Rob
>>
>> On Oct 21, 2009, at 12:40 PM, Chris Worley wrote:
>>
>>> I haven't looked at PVFS for nearly a decade, so my understanding
>>> could be way out of date.  Sorry if this is a stupid question; I'm
>>> hoping PVFS might fill some requirements I'm looking at.
>>>
>>> Is it typical to use compute nodes as PVFS file servers?  Is it even
>>> possible (i.e. it isn't w/ Lustre)?
>>>
>>> If so, while striping across I/O nodes looks to be the default, is
>>> there any provision for file affinity... i.e. files migrating to the
>>> nodes using them?
>>>
>>> If striping is the only choice, and compute nodes serve the I/O too,
>>> then a job running on one set of nodes could be using files striped
>>> across another set of nodes in use by another job, causing both jobs
>>> to loose performance, correct?
>>>
>>> Are there any links to PVFS performance over IB?
>>>
>>> Thanks,
>>>
>>> Chris
>>
>> _______________________________________________
>> Pvfs2-users mailing list
>> Pvfs2-users at beowulf-underground.org
>> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
>>
>