[Pvfs2-users] Expected/ToExpect vs. real performance ??
Michael Will
mwill at penguincomputing.com
Mon Feb 11 15:02:52 EST 2008
I had a few hours to play on a cluster with 4TB I/O nodes.
Straight dd read / write of a single large file locally on a software
raid0 of the
two 6-drive raid5 volumes in each i/o node would give 345 MB/s read and
205 MB/s
write throughput.
PVFS single client and single server over single gigabit ethernet would
result in 84 MB/s read, 77 MB/s write.
Now I set it up with 8 I/O nodes and 8 clients, the resulting PVFS2
filesystem was 35TB large. However when
doing my benchmarking runs I got these errors:
pvfs2: pvfs2_get_sb -- wait timed out; aborting attempt.
pvfs2_get_sb: mount request failed with -110
/var/log/messages:Feb 9 05:59:27 10.54.1.100 n100 pvfs2-server[8620]: segfault at 0000000000000010 rip 0000003b56e6960d rsp 0000007fbffff160 error 6
/var/log/messages:Feb 9 05:59:38 10.54.1.117 .117 pvfs2_file_read: error in vectored read from handle 1048571, FILE: largefile..117.1
/var/log/messages:Feb 9 05:59:38 10.54.1.117 .117 pvfs2_file_read: error in vectored read from handle 1048571, FILE: largefile..117.1
/var/log/messages:Feb 9 05:59:38 10.54.1.111 .111 pvfs2_file_read: error in vectored read from handle 1048570, FILE: largefile..111.1
/var/log/messages:Feb 9 05:59:38 10.54.1.107 .107 pvfs2_file_read: error in vectored read from handle 1048579, FILE: largefile..107.1
/var/log/messages:Feb 9 05:59:38 10.54.1.111 .111 pvfs2_file_read: error in vectored read from handle 1048570, FILE: largefile..111.1
/var/log/messages:Feb 9 05:59:38 10.54.1.107 .107 pvfs2_file_read: error in vectored read from handle 1048579, FILE: largefile..107.1
/var/log/messages:Feb 9 05:59:38 10.54.1.119 .119 pvfs2_file_read: error in vectored read from handle 1048574, FILE: largefile..119.1
/var/log/messages:Feb 9 05:59:38 10.54.1.106 .106 pvfs2_file_write: error in vectored write to handle 1048581, FILE: largefile..106.1
/var/log/messages:Feb 9 05:59:38 10.54.1.104 .104 pvfs2_file_read: error in vectored read from handle 1048580, FILE: largefile..104.1
/var/log/messages:Feb 9 05:59:38 10.54.1.104 .104 pvfs2_file_read: error in vectored read from handle 1048580, FILE: largefile..104.1
/var/log/messages:Feb 9 05:59:38 10.54.1.103 .103 pvfs2_file_read: error in vectored read from handle 1048573, FILE: largefile..103.1
/var/log/messages:Feb 9 05:59:38 10.54.1.103 .103 pvfs2_file_read: error in vectored read from handle 1048573, FILE: largefile..103.1
/var/log/messages:Feb 9 05:59:38 10.54.1.109 .109 pvfs2_file_read: error in vectored read from handle 1048572, FILE: largefile..109.1
/var/log/messages:Feb 9 05:59:38 10.54.1.109 .109 pvfs2_file_read: error in vectored read from handle 1048572, FILE: largefile..109.1
/var/log/messages:Feb 9 05:59:38 10.54.1.106 .106 pvfs2_file_write: error in vectored write to handle 1048581, FILE: largefile..106.1
/var/log/messages:Feb 9 06:08:18 10.54.1.118 .118 pvfs2: pvfs2_fs_umount -- wait timed out; aborting attempt.
Unfortunately I did not get to play with this anymore since these where customers systems that needed to be cleaned up and shipped,
so I cannot do any additional troubleshooting or find out why pvfs2_server died with a segfault on node n100, but it could have to do
with the naming scheme on the cluster (node 100 has hostname .100 and an alias n100, n101 is .101 etc.).
The size of the filesystem should not have been an issue, right?
Michael
Michael Will wrote:
> Depending on what you are trying to do, this might or might not be the
> right filesystem for you.
>
> I have only tested pvfs2 in its default configuration with no
> fine-tuning, but so far I see pvfs2 strengths are:
> 1. bandwidth scaling: gets you more i/o bandwidth with additional i/o
> nodes
> 2. parallelism: multiple clients reading at the same time
> 3. write-speed over read-speed: aggragate write speed scales much
> better than the read-speed
>
> If you have only one client (say video player or video editor) running
> at a time, and not enough
> i/o nodes to make up for the overhead of splitting the data across
> servers, then you might be better
> off running just an nfs-server on a single beefy node and put all the
> disks in there in a raid10 or raid0.
>
> If you plan to support multiple clients, or if you can add enough i/o
> nodes, then pvfs2 is very capable.
>
> One thing to try could be to decouple the application and the i/o
> generation: Run your application on a
> machine that is not also a data-server since then your video/audio
> mixing will not be competing for cycles
> with the data-producing servers.
>
> Try to have only two i/o nodes and one client instead of all three
> being i/o nodes if you only have two servers.
>
> I ran some benchmarks on a small cluster with 6 clients and 4 i/o
> nodes each of which only had a single sata disk
> and compared it to the nfs-server running on the headnode of the
> cluster. This nfs-server was pretty slow, however
> I found that a single client would perform better in read with the
> single NFS server. For four or six clients,
> the NFS-server would be caving in badly though where PVFS2 would then
> give me a nice 280MB/s aggregate
> write-bandwidth. Read was still only 45MB/s aggragate.
>
> I hope to run more tests on a much larger cluster with tons of storage
> this week (>100 i/o nodes with 4.6TB each = Relion 2612 2U server with
> 12 sata drives)
>
> Michael Will
>
> belcampo wrote:
>> belcampo wrote:
>>> Hi all,
>>>
>>> New to pvfs and related stuff, so try to be kind with me ;-)
>>> I installed according the pvfs2-quickstart guide.
>>> pvfs2-ping -m /mnt/pvfs2
>>>
>>> (1) Parsing tab file...
>>>
>>> (2) Initializing system interface...
>>>
>>> (3) Initializing each file system found in tab file: /etc/fstab...
>>>
>>> PVFS2 servers: tcp://server:3334
>>> Storage name: pvfs2-fs
>>> Local mount point: /mnt/pvfs2
>>> /mnt/pvfs2: Ok
>>>
>>> (4) Searching for /mnt/pvfs2 in pvfstab...
>>>
>>> PVFS2 servers: tcp://server:3334
>>> Storage name: pvfs2-fs
>>> Local mount point: /mnt/pvfs2
>>>
>>> meta servers:
>>> tcp://mmulti:3334
>>>
>>> data servers:
>>> tcp://mmulti:3334
>>> tcp://mm1:3334
>>> tcp://server:3334
>>>
>>> (5) Verifying that all servers are responding...
>>>
>>> meta servers:
>>> tcp://mmulti:3334 Ok
>>>
>>> data servers:
>>> tcp://mmulti:3334 Ok
>>> tcp://mm1:3334 Ok
>>> tcp://server:3334 Ok
>>>
>>> (6) Verifying that fsid 533592664 is acceptable to all servers...
>>>
>>> Ok; all servers understand fs_id 533592664
>>>
>>> (7) Verifying that root handle is owned by one server...
>>>
>>> Root handle: 1048576
>>> Ok; root handle is owned by exactly one server.
>>>
>>> =============================================================
>>>
>>> The PVFS2 filesystem at /mnt/pvfs2 appears to be correctly configured.
>>>
>>> Copying files to /mnt/pvfs limited by network, so OK.
>>>
>>> Did a high IO-demanding muxing of audio/video first locally and then
>>> on /mnt/pvfs2 both from the same machine which is one of the data
>>> servers.
>>>
>>> Local
>>>
>>> Saving to timetest.mp4: 0.500 secs Interleaving
>>> 7.58user 19.71system 1:52.26elapsed 24%CPU (0avgtext+0avgdata
>>> 0maxresident)k
>>> 0inputs+0outputs (77major+6054minor)pagefaults 0swaps
>>>
>>> on /mnt/pvfs2
>>>
>>> Saving to timetest.mp4: 0.500 secs Interleaving
>>> 37.56user 61.05system 41:54.96elapsed 3%CPU (0avgtext+0avgdata
>>> 0maxresident)k
>>> 0inputs+0outputs (68major+6063minor)pagefaults 0swaps
>>>
>>> System load user about times 5, system > times 20, needed time >
>>> times 20.
>>>
>>> What could be the reason it is, like it is.
>>>
>>> Regards
>>>
>>> Henk Schoneveld
>>>
>>> _______________________________________________
>>> Pvfs2-users mailing list
>>> Pvfs2-users at beowulf-underground.org
>>> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
>> Additional info
>> PVF version 2.7.0
>> kernel 2.6.22.9-desktop586-1mdv
>> x86-32 tcp/ip realtek 8139too on all
>> no MPI of MPI-IO
>> logs only tell
>> Client
>> D 15:48:13.061859] [INFO]: Mapping pointer 0xb6769000 for I/O
>> Server
>> D 02/04 15:47] PVFS2 Server version 2.7.0 starting.
>>
>> _______________________________________________
>> Pvfs2-users mailing list
>> Pvfs2-users at beowulf-underground.org
>> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
> _______________________________________________
> Pvfs2-users mailing list
> Pvfs2-users at beowulf-underground.org
> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
More information about the Pvfs2-users
mailing list