[Pvfs2-users] goofy permission denied error
Robert Olson
olson at mcs.anl.gov
Mon Sep 25 14:06:37 EDT 2006
This is 1.5.1 on PPC (PPC64 kernel), running Debian 3.1. G5 Xserve
cluster. 4 I/O nodes.
> On closer look, it looks like the permission errors that you are
> seeing may be from the files that you are trying to copy from and not
> pvfs2 related...
> Can you check the permissions of the files under simtab.0917 (local
> file
> system)?
Nope - they're all readable. If I rerun the copy for the things that
didn't copy the first time, the copy-every-other-file behavior
repeats itself again.
Client log is attached, server logs follow.
Thanks,
--bob
[D 09/15 21:10] PVFS2 Server version 1.5.1 starting.
[E 09/15 21:10] TROVE:DBPF:Berkeley DB: illegal record number of 0
[E 09/15 21:10] failed to set cursor position at 512
[E 09/15 21:10] dbpf_dspace_iterate_handles_op_svc: Invalid argument
[E 09/25 12:26] tried to remove non-existant dataspace
[E 09/25 12:26] tried to remove non-existant dataspace
[E 09/25 12:26] Flow proto error cleanup started on 0x100ec948,
error_code: -1073741973
[E 09/25 12:26] Flow proto 0x100ec948 canceling a total of 0 BMI or
Trove operations
[E 09/25 12:26] Flow proto error cleanup finished 0x100ec948,
error_code: -1073741973
[E 09/25 12:29] tried to remove non-existant dataspace
[E 09/25 12:29] tried to remove non-existant dataspace
[E 09/25 12:29] tried to remove non-existant dataspace
[E 09/25 12:29] tried to remove non-existant dataspace
[E 09/25 12:29] tried to remove non-existant dataspace
[E 09/25 12:29] tried to remove non-existant dataspace
[D 08/10 10:04] PVFS2 Server version 1.5.1 starting.
[E 09/25 12:26] Flow proto error cleanup started on 0xf6033810,
error_code: -1073741973
[E 09/25 12:26] Flow proto 0xf6033810 canceling a total of 0 BMI or
Trove operations
[E 09/25 12:26] Flow proto error cleanup finished 0xf6033810,
error_code: -1073741973
[D 08/10 10:05] PVFS2 Server version 1.5.1 starting.
[E 09/25 12:27] Flow proto error cleanup started on 0xf66b3680,
error_code: -1073741973
[E 09/25 12:27] Flow proto 0xf66b3680 canceling a total of 0 BMI or
Trove operations
[E 09/25 12:27] Flow proto error cleanup finished 0xf66b3680,
error_code: -1073741973
[D 08/10 10:05] PVFS2 Server version 1.5.1 starting.
--bob
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pvfs2-client.log
Type: application/octet-stream
Size: 69190 bytes
Desc: not available
Url : http://www.beowulf-underground.org/pipermail/pvfs2-users/attachments/20060925/46d5f98f/pvfs2-client-0001.obj
-------------- next part --------------
On Sep 25, 2006, at 12:53 PM, Murali Vilayannur wrote:
> Hi,
> Is this with CVS HEAD or with pvfs2-1.5.1? what platform and distro?
> Can you enable kmod debugging, rerun cp and send the relevant parts
> of the
> logs?
> sudo sh -c "echo 1 > /proc/sys/pvfs2/debug" (for pvfs2-1.5.1)
> sudo sh -c "echo 32767 > /proc/sys/pvfs2/debug" (for cvs head)
> thanks!
> Murali
>
> On Mon, 25 Sep 2006, Robert Olson wrote:
>
>> This is weird. I appear to getting permission denied on every other
>> file to be copied from a local filesystem to the PVFS2 filesystem.
>> This ring a bell with anyone?
>>
>> Thanks,
>> --bob
>>
>> olson at bio-ppc-1:/scratch$ cp -r simtab.0917 /disks/pvfs2/olson
>> cp: cannot create regular file `/disks/pvfs2/olson/simtab.0917/sims.
>> 10': Permission denied
>> cp: cannot create regular file `/disks/pvfs2/olson/simtab.0917/sims.
>> 101': Permission denied
>> cp: cannot create regular file `/disks/pvfs2/olson/simtab.0917/sims.
>> 103': Permission denied
>> cp: cannot create regular file `/disks/pvfs2/olson/simtab.0917/sims.
>> 105': Permission denied
>>
>>
>> olson at bio-ppc-1:/scratch$ ls -l /disks/pvfs2/olson/simtab.0917/
>> total 88044
>> -rw-r--r-- 1 olson seed-users 37994316 Sep 25 17:26 sims.1
>> -rw-r--r-- 1 olson seed-users 30049 Sep 25 17:26 sims.100
>> -rw-r--r-- 1 olson seed-users 7226 Sep 25 17:26 sims.102
>> -rw-r--r-- 1 olson seed-users 1915643 Sep 25 17:26 sims.104
>> -rw-r--r-- 1 olson seed-users 50200576 Sep 25 17:26 sims.106
>>
>> olson at bio-ppc-1:/scratch$ for i in simtab.0917/*; do echo $i; /vol/
>> pvfs2/bin/pvfs2-cp -t $i /disks/pvfs2/olson/$i; done
>> simtab.0917/sims.1
>> Wrote 37994316 bytes in 0.437222 seconds. 82.873696 MB/seconds
>> simtab.0917/sims.10
>> open: Permission denied
>> could not open /disks/pvfs2/olson/simtab.0917/sims.10
>> Could not open /disks/pvfs2/olson/simtab.0917/sims.10
>> simtab.0917/sims.100
>> Wrote 30049 bytes in 0.001754 seconds. 16.339859 MB/seconds
>> simtab.0917/sims.101
>> Wrote 18867 bytes in 0.035888 seconds. 0.501365 MB/seconds
>> simtab.0917/sims.102
>> Wrote 7226 bytes in 0.001195 seconds. 5.766959 MB/seconds
>> simtab.0917/sims.103
>> open: Permission denied
>> could not open /disks/pvfs2/olson/simtab.0917/sims.103
>> Could not open /disks/pvfs2/olson/simtab.0917/sims.103
>> simtab.0917/sims.104
>> Wrote 1915643 bytes in 0.022985 seconds. 79.482314 MB/seconds
>> simtab.0917/sims.105
>> Wrote 113037406 bytes in 3.774026 seconds. 28.563891 MB/seconds
>> simtab.0917/sims.106
>> Wrote 177660236 bytes in 5.191931 seconds. 32.633331 MB/seconds
>> simtab.0917/sims.107
>> open: Permission denied
>> could not open /disks/pvfs2/olson/simtab.0917/sims.107
>> Could not open /disks/pvfs2/olson/simtab.0917/sims.107
>> simtab.0917/sims.108
>> Wrote 152382253 bytes in 5.079284 seconds. 28.610931 MB/seconds
>> simtab.0917/sims.109
>> open: Permission denied
>> could not open /disks/pvfs2/olson/simtab.0917/sims.109
>> Could not open /disks/pvfs2/olson/simtab.0917/sims.109
>> simtab.0917/sims.11
>> Wrote 55995114 bytes in 1.893243 seconds. 28.206152 MB/seconds
>> simtab.0917/sims.110
>> open: Permission denied
>> could not open /disks/pvfs2/olson/simtab.0917/sims.110
>> Could not open /disks/pvfs2/olson/simtab.0917/sims.110
>> simtab.0917/sims.111
>> Wrote 85578997 bytes in 2.864490 seconds. 28.491805 MB/seconds
>> simtab.0917/sims.112
>> open: Permission denied
>> could not open /disks/pvfs2/olson/simtab.0917/sims.112
>> Could not open /disks/pvfs2/olson/simtab.0917/sims.112
>> simtab.0917/sims.113
>> Wrote 159349826 bytes in 5.469951 seconds. 27.782303 MB/seconds
>> simtab.0917/sims.114
>> open: Permission denied
>> could not open /disks/pvfs2/olson/simtab.0917/sims.114
>> Could not open /disks/pvfs2/olson/simtab.0917/sims.114
>> simtab.0917/sims.115
>> Wrote 385509 bytes in 0.043475 seconds. 8.456602 MB/seconds
>> simtab.0917/sims.116
>> open: Permission denied
>> could not open /disks/pvfs2/olson/simtab.0917/sims.116
>> Could not open /disks/pvfs2/olson/simtab.0917/sims.116
>> simtab.0917/sims.117
>> Wrote 125580543 bytes in 4.246360 seconds. 28.203670 MB/seconds
>> simtab.0917/sims.118
>> open: Permission denied
>> could not open /disks/pvfs2/olson/simtab.0917/sims.118
>> Could not open /disks/pvfs2/olson/simtab.0917/sims.118
>> simtab.0917/sims.119
>> Wrote 74459114 bytes in 2.527315 seconds. 28.096912 MB/seconds
>> simtab.0917/sims.12
>> open: Permission denied
>> could not open /disks/pvfs2/olson/simtab.0917/sims.12
>> Could not open /disks/pvfs2/olson/simtab.0917/sims.12
>> [etc]
>>
>>
>> pvfs-statfs is happy:
>>
>> olson at bio-ppc-1:~$ /vol/pvfs2/bin/pvfs2-statfs -m /mnt/pvfs2
>>
>> aggregate statistics:
>> ---------------------------------------
>>
>> fs_id: 1239905522
>> total number of servers (meta and I/O): 4
>> handles available (meta and I/O): 5365714538
>> handles total (meta and I/O): 5368709115
>> bytes available: 1218774892544
>> bytes total: 1495224942592
>>
>> NOTE: The aggregate total and available statistics are calculated
>> based
>> on an algorithm that assumes data will be distributed evenly; thus
>> the free space is equal to the smallest I/O server capacity
>> multiplied by the number of I/O servers. If this number seems
>> unusually small, then check the individual server statistics below
>> to look for problematic servers.
>>
>> meta server statistics:
>> ---------------------------------------
>>
>> server: tcp://bio-ppc-head-4:3334
>> RAM bytes total : 4107878400
>> RAM bytes free : 13307904
>> uptime (seconds) : 4045972
>> load averages : 4704 9088 5664
>> handles available: 2145227856
>> handles total : 2147483646
>> bytes available : 310944141312
>> bytes total : 373806235648
>> mode: serving both metadata and I/O data
>>
>>
>> I/O server statistics:
>> ---------------------------------------
>>
>> server: tcp://bio-ppc-head-4:3334
>> RAM bytes total : 4107878400
>> RAM bytes free : 13307904
>> uptime (seconds) : 4045972
>> load averages : 4704 9088 5664
>> handles available: 2145227856
>> handles total : 2147483646
>> bytes available : 310944141312
>> bytes total : 373806235648
>> mode: serving both metadata and I/O data
>>
>> server: tcp://bio-ppc-head-1:3334
>> RAM bytes total : 4107878400
>> RAM bytes free : 2062213120
>> uptime (seconds) : 832980
>> load averages : 30624 12352 3872
>> handles available: 1073741272
>> handles total : 1073741823
>> bytes available : 307329638400
>> bytes total : 373806235648
>> mode: serving only I/O data
>>
>> server: tcp://bio-ppc-head-2:3334
>> RAM bytes total : 4107878400
>> RAM bytes free : 12697600
>> uptime (seconds) : 4045977
>> load averages : 10592 6720 1792
>> handles available: 1073372712
>> handles total : 1073741823
>> bytes available : 304693723136
>> bytes total : 373806235648
>> mode: serving only I/O data
>>
>> server: tcp://bio-ppc-head-3:3334
>> RAM bytes total : 3578347520
>> RAM bytes free : 12324864
>> uptime (seconds) : 4046045
>> load averages : 13056 9376 3136
>> handles available: 1073372698
>> handles total : 1073741823
>> bytes available : 309065900032
>> bytes total : 373806235648
>> mode: serving only I/O data
>>
>> _______________________________________________
>> Pvfs2-users mailing list
>> Pvfs2-users at beowulf-underground.org
>> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
>>
>>
>
More information about the Pvfs2-users
mailing list