[Pvfs2-users] PVFS2 over Infiniband

Mon Sep 11 02:38:03 EDT 2006

Thanks for your replay.

I measured performance using bonnie and pvfs2-cp.

When I run bonnie at mount directory by pvfs2(/mnt/pvfs2), the result was

3~8MB/seconds(write/block). When pvfs-cp(from local to /mnt/pvfs2) was run,
the result was about 17~20MB/seconds (write).

What is the problem? I can not find the reason...

This is my configuration information.

I used OFDE-1.0 and Linux 2.6.9-5.ELsmp.

I used 4 servers(intel Xeon 2.8GHz) and Flextronics 24 4X port switch and
Topspin HCA Card(PCI-X).

My configuration is like this.

============================================================================

[root at rose3 pvfs2]# configure --with-kernel=/usr/src/kernels/2.6.9-5.EL-smp-

i686 --with-openib=/usr/local/ofed --with-openib-
includes=/usr/local/ofed/include --with-openib-libs=/usr/local/ofed/lib

**********************************************************************

    Welcome to the PVFS2 Configuration Generator:

This interactive script will generate configuration files suitable for use
with a new PVFS2 file system.  Please see the PVFS2 quickstart guide for
details.

**********************************************************************

You must first select the network protocol that your file system will use.

The only currently supported options are "tcp", "gm", and "ib".

(For multi-homed configurations, use e.g. "ib,tcp".)

* Enter protocol type [Default is tcp]: ib

Choose a TCP/IP port for the servers to listen on for IB communications.

Note that this

script assumes that all servers will use the same port number.

* Enter port number [Default is 3335]: 

Choose a directory for each server to store data in.

* Enter directory name: [Default is /pvfs2-storage-space]: /home2/pvfs2-
storage-space

Choose a file for each server to write log messages to.

* Enter log file location [Default is /tmp/pvfs2-server.log]: /tmp/pvfs2-
server.log

Next you must list the hostnames of the machines that will act as I/O
servers.  Acceptable syntax is "node1, node2, ..." or "node{#-#,#,#}".

* Enter hostnames [Default is localhost]: 11.0.0.1, 11.0.0.2, 11.0.0.3,

11.0.0.4

Now list the hostnames of the machines that will act as Metadata servers.
This list may or may not overlap with the I/O server list.

* Enter hostnames [Default is localhost]: 11.0.0.4

Configured a total of 4 servers:

4 of them are I/O servers.

1 of them are Metadata servers.

* Would you like to verify server list (y/n) [Default is n]? y

****** I/O servers:

****** Metadata servers:

11.0.0.4

* Does this look ok (y/n) [Default is y]? y

Writing fs config file... Done.

Writing 4 server config file(s)... Done.

11.0.0.3

11.0.0.2

11.0.0.1

11.0.0.4

=================================================================

[root at rose1 ~]# cat /etc/pvfs2-fs.conf

<Defaults>

        UnexpectedRequests 50

        LogFile /tmp/pvfs2-server.log

        EventLogging none

        LogStamp datetime

        BMIModules bmi_ib

        FlowModules flowproto_multiqueue

        PerfUpdateInterval 1000

        ServerJobBMITimeoutSecs 30

        ServerJobFlowTimeoutSecs 30

        ClientJobBMITimeoutSecs 300

        ClientJobFlowTimeoutSecs 300

        ClientRetryLimit 5

        ClientRetryDelayMilliSecs 2000

</Defaults>

<Aliases>

        Alias 11.0.0.1 ib://11.0.0.1:3335

        Alias 11.0.0.2 ib://11.0.0.2:3335

        Alias 11.0.0.3 ib://11.0.0.3:3335

        Alias 11.0.0.4 ib://11.0.0.4:3335 </Aliases>

<Filesystem>

        Name pvfs2-fs

        ID 1069359617

        RootHandle 1048576

        <MetaHandleRanges>

                Range 11.0.0.1 4-1073741826

        </MetaHandleRanges>

        <DataHandleRanges>

                Range 11.0.0.1 1073741827-2147483649

                Range 11.0.0.2 2147483650-3221225472

                Range 11.0.0.3 3221225473-4294967295

                Range 11.0.0.4 4294967296-5368709118

        </DataHandleRanges>

        <StorageHints>

                TroveSyncMeta yes

                TroveSyncData no

        </StorageHints>

</Filesystem>

=================================================================
[root at rose1 ~]# cat /etc/pvfs2-server.conf-11.0.0.1 StorageSpace /pvfs2-
storage-space HostID "ib://11.0.0.1:3335"

LogFile /tmp/pvfs2-server.log

[root at rose1 ~]# cat /etc/pvfs2-server.conf-11.0.0.2 StorageSpace /pvfs2-
storage-space HostID "ib://11.0.0.2:3335"

LogFile /tmp/pvfs2-server.log

[root at rose1 ~]# cat /etc/pvfs2-server.conf-11.0.0.3 StorageSpace /pvfs2-
storage-space HostID "ib://11.0.0.3:3335"

LogFile /tmp/pvfs2-server.log

[root at rose1 ~]# cat /etc/pvfs2-server.conf-11.0.0.4 StorageSpace /pvfs2-
storage-space HostID "ib://11.0.0.4:3335"

LogFile /tmp/pvfs2-server.log

=================================================================

[root at rose4 pvfs2-ib-config]# cat pvfs2tab 

ib://11.0.0.4:3335/pvfs2-fs /mnt/pvfs2 pvfs2 default,noauto 0 0
=================================================================

[root at rose4 linux]# cat mount_pvfs2.sh

#!/bin/bash

if [ "`echo $PATH | grep -c /usr/src/modtools/sbin`" = "0" ]; then

   echo "Loading up bashrc to get latest mod-tools"

   . ~/.bashrc

fi

if [ "`lsmod | grep -c pvfs2`" = "0" ]; then

   if test "x`uname -r | cut -b 1-3`" = "x2.4"; then

        echo "Loading the 2.4.x pvfs2 module"

        insmod ../../../../src/kernel/linux-2.4/pvfs2.o $1

   else

        echo "Loading the 2.6.x pvfs2 module"

        insmod ../../../../src/kernel/linux-2.6/pvfs2.ko $1

   fi

fi

CUR_DEV="pvfs2-flow"

for f in `cat /proc/devices | grep pvfs2 | awk '{print $1}'`; do

   if [ ! -c /dev/$CUR_DEV ]; then

      echo "Creating device file /dev/$CUR_DEV with major number $f"

      mknod /dev/$CUR_DEV c $f 0 -m 666

   fi

   CUR_DEV="pvfs2-req"

done

if [ ! -d /mnt/pvfs2 ]; then

   echo "Creating /mnt/pvfs2"

   mkdir /mnt/pvfs2

fi

echo "Starting pvfs2-client"

./pvfs2-client -p ./pvfs2-client-core

if [ "`mount | grep -c pvfs2`" = "0" ]; then

   if test "x`uname -r | cut -b 1-3`" = "x2.4"; then

        echo "Mounting 2.4.x pvfs2 on /mnt/pvfs2"

        mount -t pvfs2 pvfs2 /mnt/pvfs2 -o ib://11.0.0.4:3335/pvfs2-fs

   else

        echo "Mounting 2.6.x pvfs2 on /mnt/pvfs2"

        mount -t pvfs2 ib://11.0.0.4:3335/pvfs2-fs /mnt/pvfs2

   fi

else

   echo "Uh...I think pvfs2 is already mounted."

fi

============================================================================

[root at rose3 bonnie]# mount | grep pvfs

ib://11.0.0.4:3335/pvfs2-fs on /mnt/pvfs2 type pvfs2 (rw)

============================================================================

This is the performance result.

============================================================================

[root at rose1 etc]# pvfs2-cp -t /root/Down/IBGD-1.8.2.tgz /mnt/pvfs2/test-a

Wrote 57914150 bytes in 2.447529 seconds. 22.566122 MB/seconds

============================================================================

[root at rose1 bonnie]# pwd

/mnt/pvfs2/bonnie

[root at rose1 bonnie]# Bonnie

File './Bonnie.14537', size: 104857600

Writing with putc()...done

Rewriting...done

Writing intelligently...done

Reading with getc()...done

Reading intelligently...done

Seeker 1...Seeker 2...Seeker 3...start 'em...done...done...done...

              -------Sequential Output-------- ---Sequential Input-- --
Random--

              -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --
Seeks---

Machine    MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU  /sec
%CPU

          100 20574 86.1  6615  1.2  1759  0.6  8396 37.4  3240  1.4 265.7
1.7

============================================================================

What is the problem?...

Regards,

Hyeyoung cho

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.beowulf-underground.org/pipermail/pvfs2-users/attachments/20060911/bba22d58/attachment-0001.htm