[Pvfs2-users] goofy permission denied error
Sam Lang
slang at mcs.anl.gov
Mon Sep 25 19:50:17 EDT 2006
Hi Bob,
I think I found the bug and have a fix for this, but I just want to
make sure. Do you remember the last time you restarted the pvfs2
servers? I think this error is caused at server startup on ppc64
after at least 512 files are created. If your servers were re-
started recently (with a populated file system), then I know what's
wrong. Otherwise, if they were started once when the file system was
created and have been running ever since, I've just found a bug
elsewhere in our code, and I'll need to keep looking for yours. Let
me know.
Thanks,
-sam
On Sep 25, 2006, at 1:51 PM, Robert Olson wrote:
> This was built new for 1.5.1, and I've been using it on and off for
> a few weeks.
>
> --bob
>
> On Sep 25, 2006, at 1:37 PM, Sam Lang wrote:
>
>>
>> Hi Bob,
>>
>> Is this from a new storage space, or had things been working and
>> then you started getting the permission denied errors? If its a
>> storage space that's been around for a while, did you migrate from
>> 1.3.x or 1.4.0?
>>
>> Based on the logfiles you sent, it doesn't look like a kernel
>> module problem. We haven't seen those specific problems with
>> 1.5.1, but we didn't do much testing of the 1.5.1 release on 64
>> bit platforms (esp. ppc). Would it be possible to get access to
>> that system so that I can do some debugging?
>>
>> Thanks,
>>
>> -sam
>>
>> On Sep 25, 2006, at 12:34 PM, Robert Olson wrote:
>>
>>> This is weird. I appear to getting permission denied on every
>>> other file to be copied from a local filesystem to the PVFS2
>>> filesystem. This ring a bell with anyone?
>>>
>>> Thanks,
>>> --bob
>>>
>>> olson at bio-ppc-1:/scratch$ cp -r simtab.0917 /disks/pvfs2/olson
>>> cp: cannot create regular file `/disks/pvfs2/olson/simtab.0917/
>>> sims.10': Permission denied
>>> cp: cannot create regular file `/disks/pvfs2/olson/simtab.0917/
>>> sims.101': Permission denied
>>> cp: cannot create regular file `/disks/pvfs2/olson/simtab.0917/
>>> sims.103': Permission denied
>>> cp: cannot create regular file `/disks/pvfs2/olson/simtab.0917/
>>> sims.105': Permission denied
>>>
>>>
>>> olson at bio-ppc-1:/scratch$ ls -l /disks/pvfs2/olson/simtab.0917/
>>> total 88044
>>> -rw-r--r-- 1 olson seed-users 37994316 Sep 25 17:26 sims.1
>>> -rw-r--r-- 1 olson seed-users 30049 Sep 25 17:26 sims.100
>>> -rw-r--r-- 1 olson seed-users 7226 Sep 25 17:26 sims.102
>>> -rw-r--r-- 1 olson seed-users 1915643 Sep 25 17:26 sims.104
>>> -rw-r--r-- 1 olson seed-users 50200576 Sep 25 17:26 sims.106
>>>
>>> olson at bio-ppc-1:/scratch$ for i in simtab.0917/*; do echo $i; /
>>> vol/pvfs2/bin/pvfs2-cp -t $i /disks/pvfs2/olson/$i; done
>>> simtab.0917/sims.1
>>> Wrote 37994316 bytes in 0.437222 seconds. 82.873696 MB/seconds
>>> simtab.0917/sims.10
>>> open: Permission denied
>>> could not open /disks/pvfs2/olson/simtab.0917/sims.10
>>> Could not open /disks/pvfs2/olson/simtab.0917/sims.10
>>> simtab.0917/sims.100
>>> Wrote 30049 bytes in 0.001754 seconds. 16.339859 MB/seconds
>>> simtab.0917/sims.101
>>> Wrote 18867 bytes in 0.035888 seconds. 0.501365 MB/seconds
>>> simtab.0917/sims.102
>>> Wrote 7226 bytes in 0.001195 seconds. 5.766959 MB/seconds
>>> simtab.0917/sims.103
>>> open: Permission denied
>>> could not open /disks/pvfs2/olson/simtab.0917/sims.103
>>> Could not open /disks/pvfs2/olson/simtab.0917/sims.103
>>> simtab.0917/sims.104
>>> Wrote 1915643 bytes in 0.022985 seconds. 79.482314 MB/seconds
>>> simtab.0917/sims.105
>>> Wrote 113037406 bytes in 3.774026 seconds. 28.563891 MB/seconds
>>> simtab.0917/sims.106
>>> Wrote 177660236 bytes in 5.191931 seconds. 32.633331 MB/seconds
>>> simtab.0917/sims.107
>>> open: Permission denied
>>> could not open /disks/pvfs2/olson/simtab.0917/sims.107
>>> Could not open /disks/pvfs2/olson/simtab.0917/sims.107
>>> simtab.0917/sims.108
>>> Wrote 152382253 bytes in 5.079284 seconds. 28.610931 MB/seconds
>>> simtab.0917/sims.109
>>> open: Permission denied
>>> could not open /disks/pvfs2/olson/simtab.0917/sims.109
>>> Could not open /disks/pvfs2/olson/simtab.0917/sims.109
>>> simtab.0917/sims.11
>>> Wrote 55995114 bytes in 1.893243 seconds. 28.206152 MB/seconds
>>> simtab.0917/sims.110
>>> open: Permission denied
>>> could not open /disks/pvfs2/olson/simtab.0917/sims.110
>>> Could not open /disks/pvfs2/olson/simtab.0917/sims.110
>>> simtab.0917/sims.111
>>> Wrote 85578997 bytes in 2.864490 seconds. 28.491805 MB/seconds
>>> simtab.0917/sims.112
>>> open: Permission denied
>>> could not open /disks/pvfs2/olson/simtab.0917/sims.112
>>> Could not open /disks/pvfs2/olson/simtab.0917/sims.112
>>> simtab.0917/sims.113
>>> Wrote 159349826 bytes in 5.469951 seconds. 27.782303 MB/seconds
>>> simtab.0917/sims.114
>>> open: Permission denied
>>> could not open /disks/pvfs2/olson/simtab.0917/sims.114
>>> Could not open /disks/pvfs2/olson/simtab.0917/sims.114
>>> simtab.0917/sims.115
>>> Wrote 385509 bytes in 0.043475 seconds. 8.456602 MB/seconds
>>> simtab.0917/sims.116
>>> open: Permission denied
>>> could not open /disks/pvfs2/olson/simtab.0917/sims.116
>>> Could not open /disks/pvfs2/olson/simtab.0917/sims.116
>>> simtab.0917/sims.117
>>> Wrote 125580543 bytes in 4.246360 seconds. 28.203670 MB/seconds
>>> simtab.0917/sims.118
>>> open: Permission denied
>>> could not open /disks/pvfs2/olson/simtab.0917/sims.118
>>> Could not open /disks/pvfs2/olson/simtab.0917/sims.118
>>> simtab.0917/sims.119
>>> Wrote 74459114 bytes in 2.527315 seconds. 28.096912 MB/seconds
>>> simtab.0917/sims.12
>>> open: Permission denied
>>> could not open /disks/pvfs2/olson/simtab.0917/sims.12
>>> Could not open /disks/pvfs2/olson/simtab.0917/sims.12
>>> [etc]
>>>
>>>
>>> pvfs-statfs is happy:
>>>
>>> olson at bio-ppc-1:~$ /vol/pvfs2/bin/pvfs2-statfs -m /mnt/pvfs2
>>>
>>> aggregate statistics:
>>> ---------------------------------------
>>>
>>> fs_id: 1239905522
>>> total number of servers (meta and I/O): 4
>>> handles available (meta and I/O): 5365714538
>>> handles total (meta and I/O): 5368709115
>>> bytes available: 1218774892544
>>> bytes total: 1495224942592
>>>
>>> NOTE: The aggregate total and available statistics are calculated
>>> based
>>> on an algorithm that assumes data will be distributed evenly; thus
>>> the free space is equal to the smallest I/O server capacity
>>> multiplied by the number of I/O servers. If this number seems
>>> unusually small, then check the individual server statistics below
>>> to look for problematic servers.
>>>
>>> meta server statistics:
>>> ---------------------------------------
>>>
>>> server: tcp://bio-ppc-head-4:3334
>>> RAM bytes total : 4107878400
>>> RAM bytes free : 13307904
>>> uptime (seconds) : 4045972
>>> load averages : 4704 9088 5664
>>> handles available: 2145227856
>>> handles total : 2147483646
>>> bytes available : 310944141312
>>> bytes total : 373806235648
>>> mode: serving both metadata and I/O data
>>>
>>>
>>> I/O server statistics:
>>> ---------------------------------------
>>>
>>> server: tcp://bio-ppc-head-4:3334
>>> RAM bytes total : 4107878400
>>> RAM bytes free : 13307904
>>> uptime (seconds) : 4045972
>>> load averages : 4704 9088 5664
>>> handles available: 2145227856
>>> handles total : 2147483646
>>> bytes available : 310944141312
>>> bytes total : 373806235648
>>> mode: serving both metadata and I/O data
>>>
>>> server: tcp://bio-ppc-head-1:3334
>>> RAM bytes total : 4107878400
>>> RAM bytes free : 2062213120
>>> uptime (seconds) : 832980
>>> load averages : 30624 12352 3872
>>> handles available: 1073741272
>>> handles total : 1073741823
>>> bytes available : 307329638400
>>> bytes total : 373806235648
>>> mode: serving only I/O data
>>>
>>> server: tcp://bio-ppc-head-2:3334
>>> RAM bytes total : 4107878400
>>> RAM bytes free : 12697600
>>> uptime (seconds) : 4045977
>>> load averages : 10592 6720 1792
>>> handles available: 1073372712
>>> handles total : 1073741823
>>> bytes available : 304693723136
>>> bytes total : 373806235648
>>> mode: serving only I/O data
>>>
>>> server: tcp://bio-ppc-head-3:3334
>>> RAM bytes total : 3578347520
>>> RAM bytes free : 12324864
>>> uptime (seconds) : 4046045
>>> load averages : 13056 9376 3136
>>> handles available: 1073372698
>>> handles total : 1073741823
>>> bytes available : 309065900032
>>> bytes total : 373806235648
>>> mode: serving only I/O data
>>>
>>> _______________________________________________
>>> Pvfs2-users mailing list
>>> Pvfs2-users at beowulf-underground.org
>>> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
>>>
>>
>
More information about the Pvfs2-users
mailing list