[PVFS-users] Filesystem limitations on pvfs volumes

[Max Okumoto]

Hi,
I am using pvfs-1.6.0 on a rocks 3.1 (linux cluster), and I am running
into some strange size limitations.  I can't seem to create files over
35Gbytes in size.  Is there a maximum filesystem size for pvfs?

I created a pvfs volume with 15 data back end servers.  Each IO node
has 322 Gbytes of space for a total about 5 Tbytes of space.  But when
I do a df on the pvfs volume, I get 1.1 Tbytes of space.

# .iodtab
# Created by mkiodtab - Wed Jun  9 23:46:26 2004
#
# node:port #
cp0-5:7000
cp0-6:7000
cp0-7:7000
cp0-8:7000
cp0-9:7000
cp0-10:7000
cp0-11:7000
cp0-12:7000
cp0-13:7000
cp0-14:7000
cp0-15:7000
cp0-16:7000
cp0-17:7000
cp0-18:7000
cp0-19:7000


32771
0
0
0040777
3000
comp-pvfs-0-0
/state/partition1/pvfs-meta

/dev/sda3             332G  428K  315G   1% /state/partition1

head:/state/partition1/pvfs-meta
Filesystem           1k-blocks      Used Available Use% Mounted on
                      1092528512 1055464960  37063552  97% /mnt/pvfs


After I create a large file using dd.  Keep getting a no space on device 
for any new
files on the volume.

[storage pvfs]# dd if=/dev/zero of=large bs=1024k
[storage pvfs]# pvstat large
large: base = 0, pcount = 15, ssize = 65536
[storage pvfs]# dd if=dev/zero of=large.1 bs=1024k
dd: writing `large.1': No space left on device
1+0 records in
0+0 records out

[storage pvfs]# ls -la
total 37026820
drwxrwxrwx    1 root     root            0 Jun 10 22:31 .
drwxr-xr-x    5 root     root         4096 Mar 25 23:10 ..
-rw-r--r--  32767 root     root      1048576 Jun 10 22:31 large.1
-rw-r--r--  32767 root     root     37914411008 Jun 10 22:30 large


Has anyone else run into this problem?

				Max Okumoto