[Pvfs2-users] Problems with intense parallel i/o via kernel VFS
bircoph at gmail.com
Thu May 17 09:31:51 EDT 2012
On Wed, 16 May 2012 14:49:31 -0400 Becky Ligon wrote:
> I was able to run "make -j10" without any problems. The primary difference
> is the Linux kernel version. I am running 2.6.32-220.4.1.el6.x86_64. I
> also had selinux disabled.
I do not use SELinux, my kernel is vanilla 3.2.14 with a custom
configuration, SELinux is removed from the the kernel completely. I
can send you the kernel config if this is required.
> My environment used orangefs-2.8.5, 3 servers and 1 client, all on the same
> machine. My OFS configure command looked like:
> ./configure --prefix=/home/bligon/orangefs-2.8.5/install
> --with-kernel=/usr/src/kernels/2.6.32-220.4.1.el6.x86_64 --with-db=/opt/db4
> I ran the following commands:
> cd /mnt/pvfs2
> tar -xzf torque-3.0.5.tar.gz
> cd torque-3.0.5
> make -j10
> To narrow down the problem, could you please configure your OFS
> installation as I did with mine?
I can do this for testing purposes, but for a limited time only; next
week I need to give it to users at least for some simple tasks.
I want to clarify: do I need only to configure OFS package as yours
or I need to modify its setup to 3 servers and 1 client on the single
host as well?
There is another issue: running the same configure options will not
be enough to get the same configuration, because OFS configure
performs package checks and will enable some features if
corresponding packages were found, e.g. gtk or openssl. I wouldn't be
able to build gtk code anyway, because this cluster is an X-less
setup. So it would be good if I can sync with your config.log.
> The with-db option is only needed if your
> installation of Berkeley DB is not in the standard location. BTW, which
> version of BDB are you using? I am running 4.8.30. --prefix isn't
> required either. If you're not running 4.8.30 or later, you need to be.
I'm using db-4.8.30.
Another thing to consider is CFLAGS and LDFLAGS being used as well
as compiler version. As you can see I use rather sophisticated flags,
but they were well tested on the entire system, because all system
was build using this flags. However, sometimes these optimization
levels trigger code bugs not seen otherwise, especially memory
alignment and region's overlap issues. I will try to compile OFS
without any *FLAGS at all, this will effectively disable all
I'll run some test with your configure options and mail results later.
For now this will be done in 3 variations:
1) current server layout will be kept: 15 servers, both data and
meta, standalone client, all on different hosts;
2) 3 servers (both data and meta), 1 client, all on different hosts;
3) as above, but on all on a single host. This will involve a shared
usage of a single local drive, performance may be really bad, so I'm
not sure this test will be successful.
Also I'll try to compile with and without optimization for each test,
so 6 test will take some considerable time.
If you need, I can compile OFS with debug information, but I will not
be able to debug kernel module, because kernel debugging facilities
were mostly disabled in the kernel during configuration.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Size: 198 bytes
Desc: not available
Url : http://www.beowulf-underground.org/pipermail/pvfs2-users/attachments/20120517/b1218fda/attachment.bin
More information about the Pvfs2-users