[Pvfs2-users] Expected/ToExpect vs. real performance ??

[Michael Will]

Depending on what you are trying to do, this might or might not be the 
right filesystem for you.

I have only tested pvfs2 in its default configuration with no 
fine-tuning, but so far I see pvfs2 strengths are:
1. bandwidth scaling: gets you more i/o bandwidth with additional i/o nodes
2. parallelism: multiple clients reading at the same time
3. write-speed over read-speed: aggragate write speed scales much better 
than the read-speed

If you have only one client (say video player or video editor) running 
at a time, and not enough
i/o nodes to make up for the overhead of splitting the data across 
servers, then you might be better
off running just an nfs-server on a single beefy node and put all the 
disks in there in a raid10 or raid0.

If you plan to support multiple clients, or if you can add enough i/o 
nodes, then pvfs2 is very capable.

One thing to try could be to decouple the application and the i/o 
generation: Run your application on a
machine that is not also a data-server since then your video/audio 
mixing will not be competing for cycles
with the data-producing servers.

Try to have only two i/o nodes and one client instead of all three being 
i/o nodes if you only have two servers.

I ran some benchmarks on a small cluster with 6 clients and 4 i/o nodes 
each of which only had a single sata disk
and compared it to the nfs-server running on the headnode of the 
cluster. This nfs-server was pretty slow, however
I found that a single client would perform better in read with the 
single NFS server. For four or  six clients,
the NFS-server would be caving in badly though where PVFS2 would then 
give me a nice 280MB/s aggregate
write-bandwidth. Read was still only 45MB/s aggragate.

I hope to run more tests on a much larger cluster with tons of storage 
this week (>100 i/o nodes with 4.6TB each = Relion 2612 2U server with 
12 sata drives)

Michael Will

belcampo wrote:
> belcampo wrote:
>> Hi all,
>>
>> New to pvfs and related stuff, so try to be kind with me ;-)
>> I installed according the pvfs2-quickstart guide.
>> pvfs2-ping -m /mnt/pvfs2
>>
>> (1) Parsing tab file...
>>
>> (2) Initializing system interface...
>>
>> (3) Initializing each file system found in tab file: /etc/fstab...
>>
>>    PVFS2 servers: tcp://server:3334
>>    Storage name: pvfs2-fs
>>    Local mount point: /mnt/pvfs2
>>    /mnt/pvfs2: Ok
>>
>> (4) Searching for /mnt/pvfs2 in pvfstab...
>>
>>    PVFS2 servers: tcp://server:3334
>>    Storage name: pvfs2-fs
>>    Local mount point: /mnt/pvfs2
>>
>>    meta servers:
>>    tcp://mmulti:3334
>>
>>    data servers:
>>    tcp://mmulti:3334
>>    tcp://mm1:3334
>>    tcp://server:3334
>>
>> (5) Verifying that all servers are responding...
>>
>>    meta servers:
>>    tcp://mmulti:3334 Ok
>>
>>    data servers:
>>    tcp://mmulti:3334 Ok
>>    tcp://mm1:3334 Ok
>>    tcp://server:3334 Ok
>>
>> (6) Verifying that fsid 533592664 is acceptable to all servers...
>>
>>    Ok; all servers understand fs_id 533592664
>>
>> (7) Verifying that root handle is owned by one server...
>>
>>    Root handle: 1048576
>>      Ok; root handle is owned by exactly one server.
>>
>> =============================================================
>>
>> The PVFS2 filesystem at /mnt/pvfs2 appears to be correctly configured.
>>
>> Copying files to /mnt/pvfs limited by network, so OK.
>>
>> Did a high IO-demanding muxing of audio/video first locally and then 
>> on /mnt/pvfs2 both from the same machine which is one of the data 
>> servers.
>>
>> Local
>>
>> Saving to timetest.mp4: 0.500 secs Interleaving
>> 7.58user 19.71system 1:52.26elapsed 24%CPU (0avgtext+0avgdata 
>> 0maxresident)k
>> 0inputs+0outputs (77major+6054minor)pagefaults 0swaps
>>
>> on /mnt/pvfs2
>>
>> Saving to timetest.mp4: 0.500 secs Interleaving
>> 37.56user 61.05system 41:54.96elapsed 3%CPU (0avgtext+0avgdata 
>> 0maxresident)k
>> 0inputs+0outputs (68major+6063minor)pagefaults 0swaps
>>
>> System load user about times 5, system > times 20, needed time > 
>> times 20.
>>
>> What could be the reason it is, like it is.
>>
>> Regards
>>
>> Henk Schoneveld
>>
>> _______________________________________________
>> Pvfs2-users mailing list
>> Pvfs2-users at beowulf-underground.org
>> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
> Additional info
> PVF version 2.7.0
> kernel 2.6.22.9-desktop586-1mdv
> x86-32 tcp/ip realtek 8139too on all
> no MPI of MPI-IO
> logs only tell
> Client
> D 15:48:13.061859] [INFO]: Mapping pointer 0xb6769000 for I/O
> Server
> D 02/04 15:47] PVFS2 Server version 2.7.0 starting.
>
> _______________________________________________
> Pvfs2-users mailing list
> Pvfs2-users at beowulf-underground.org
> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users