[Pvfs2-users] goofy permission denied error
Robert Olson
olson at mcs.anl.gov
Tue Sep 26 12:15:38 EDT 2006
Thanks for the quick fix! I presume you mean rebuild the software,
not the volumes? Patch applied and built fine, will let you know how
things go.
--bob
On Sep 26, 2006, at 10:18 AM, Rob Ross wrote:
> Thanks for reporting the bug Bob.
>
> Rob
>
> Sam Lang wrote:
>> Bob,
>> The attached patch should apply cleanly to the 1.5.1 release, and
>> fix the problems you are seeing on ppc 64. You will have to
>> rebuild after patching, and then restart the pvfs servers after
>> rebuilding. I've committed the fixes to cvs and they'll get
>> incorporated in the next release of pvfs. If you get a chance to
>> try the patch, let me know.
>> Thanks,
>> -sam
>> On Sep 25, 2006, at 10:15 PM, Robert Olson wrote:
>>> One of the servers (on bio-ppc-head-1) was rebooted fairly
>>> recently, while
>>> the others were not.
>>>
>>> --bob
>>>
>>> On Mon, 25 Sep 2006, Sam Lang wrote:
>>>
>>>>
>>>> Hi Bob,
>>>>
>>>> I think I found the bug and have a fix for this, but I just want to
>>>> make sure. Do you remember the last time you restarted the pvfs2
>>>> servers? I think this error is caused at server startup on ppc64
>>>> after at least 512 files are created. If your servers were re-
>>>> started recently (with a populated file system), then I know what's
>>>> wrong. Otherwise, if they were started once when the file
>>>> system was
>>>> created and have been running ever since, I've just found a bug
>>>> elsewhere in our code, and I'll need to keep looking for yours.
>>>> Let
>>>> me know.
>>>>
>>>> Thanks,
>>>>
>>>> -sam
>>>>
>>>> On Sep 25, 2006, at 1:51 PM, Robert Olson wrote:
>>>>
>>>>> This was built new for 1.5.1, and I've been using it on and off
>>>>> for
>>>>> a few weeks.
>>>>>
>>>>> --bob
>>>>>
>>>>> On Sep 25, 2006, at 1:37 PM, Sam Lang wrote:
>>>>>
>>>>>>
>>>>>> Hi Bob,
>>>>>>
>>>>>> Is this from a new storage space, or had things been working and
>>>>>> then you started getting the permission denied errors? If its a
>>>>>> storage space that's been around for a while, did you migrate
>>>>>> from
>>>>>> 1.3.x or 1.4.0?
>>>>>>
>>>>>> Based on the logfiles you sent, it doesn't look like a kernel
>>>>>> module problem. We haven't seen those specific problems with
>>>>>> 1.5.1, but we didn't do much testing of the 1.5.1 release on 64
>>>>>> bit platforms (esp. ppc). Would it be possible to get access to
>>>>>> that system so that I can do some debugging?
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> -sam
>>>>>>
>>>>>> On Sep 25, 2006, at 12:34 PM, Robert Olson wrote:
>>>>>>
>>>>>>> This is weird. I appear to getting permission denied on every
>>>>>>> other file to be copied from a local filesystem to the PVFS2
>>>>>>> filesystem. This ring a bell with anyone?
>>>>>>>
>>>>>>> Thanks,
>>>>>>> --bob
>>>>>>>
>>>>>>> olson at bio-ppc-1:/scratch$ cp -r simtab.0917 /disks/pvfs2/olson
>>>>>>> cp: cannot create regular file `/disks/pvfs2/olson/simtab.0917/
>>>>>>> sims.10': Permission denied
>>>>>>> cp: cannot create regular file `/disks/pvfs2/olson/simtab.0917/
>>>>>>> sims.101': Permission denied
>>>>>>> cp: cannot create regular file `/disks/pvfs2/olson/simtab.0917/
>>>>>>> sims.103': Permission denied
>>>>>>> cp: cannot create regular file `/disks/pvfs2/olson/simtab.0917/
>>>>>>> sims.105': Permission denied
>>>>>>>
>>>>>>>
>>>>>>> olson at bio-ppc-1:/scratch$ ls -l /disks/pvfs2/olson/simtab.0917/
>>>>>>> total 88044
>>>>>>> -rw-r--r-- 1 olson seed-users 37994316 Sep 25 17:26 sims.1
>>>>>>> -rw-r--r-- 1 olson seed-users 30049 Sep 25 17:26 sims.100
>>>>>>> -rw-r--r-- 1 olson seed-users 7226 Sep 25 17:26 sims.102
>>>>>>> -rw-r--r-- 1 olson seed-users 1915643 Sep 25 17:26 sims.104
>>>>>>> -rw-r--r-- 1 olson seed-users 50200576 Sep 25 17:26 sims.106
>>>>>>>
>>>>>>> olson at bio-ppc-1:/scratch$ for i in simtab.0917/*; do echo $i; /
>>>>>>> vol/pvfs2/bin/pvfs2-cp -t $i /disks/pvfs2/olson/$i; done
>>>>>>> simtab.0917/sims.1
>>>>>>> Wrote 37994316 bytes in 0.437222 seconds. 82.873696 MB/seconds
>>>>>>> simtab.0917/sims.10
>>>>>>> open: Permission denied
>>>>>>> could not open /disks/pvfs2/olson/simtab.0917/sims.10
>>>>>>> Could not open /disks/pvfs2/olson/simtab.0917/sims.10
>>>>>>> simtab.0917/sims.100
>>>>>>> Wrote 30049 bytes in 0.001754 seconds. 16.339859 MB/seconds
>>>>>>> simtab.0917/sims.101
>>>>>>> Wrote 18867 bytes in 0.035888 seconds. 0.501365 MB/seconds
>>>>>>> simtab.0917/sims.102
>>>>>>> Wrote 7226 bytes in 0.001195 seconds. 5.766959 MB/seconds
>>>>>>> simtab.0917/sims.103
>>>>>>> open: Permission denied
>>>>>>> could not open /disks/pvfs2/olson/simtab.0917/sims.103
>>>>>>> Could not open /disks/pvfs2/olson/simtab.0917/sims.103
>>>>>>> simtab.0917/sims.104
>>>>>>> Wrote 1915643 bytes in 0.022985 seconds. 79.482314 MB/seconds
>>>>>>> simtab.0917/sims.105
>>>>>>> Wrote 113037406 bytes in 3.774026 seconds. 28.563891 MB/seconds
>>>>>>> simtab.0917/sims.106
>>>>>>> Wrote 177660236 bytes in 5.191931 seconds. 32.633331 MB/seconds
>>>>>>> simtab.0917/sims.107
>>>>>>> open: Permission denied
>>>>>>> could not open /disks/pvfs2/olson/simtab.0917/sims.107
>>>>>>> Could not open /disks/pvfs2/olson/simtab.0917/sims.107
>>>>>>> simtab.0917/sims.108
>>>>>>> Wrote 152382253 bytes in 5.079284 seconds. 28.610931 MB/seconds
>>>>>>> simtab.0917/sims.109
>>>>>>> open: Permission denied
>>>>>>> could not open /disks/pvfs2/olson/simtab.0917/sims.109
>>>>>>> Could not open /disks/pvfs2/olson/simtab.0917/sims.109
>>>>>>> simtab.0917/sims.11
>>>>>>> Wrote 55995114 bytes in 1.893243 seconds. 28.206152 MB/seconds
>>>>>>> simtab.0917/sims.110
>>>>>>> open: Permission denied
>>>>>>> could not open /disks/pvfs2/olson/simtab.0917/sims.110
>>>>>>> Could not open /disks/pvfs2/olson/simtab.0917/sims.110
>>>>>>> simtab.0917/sims.111
>>>>>>> Wrote 85578997 bytes in 2.864490 seconds. 28.491805 MB/seconds
>>>>>>> simtab.0917/sims.112
>>>>>>> open: Permission denied
>>>>>>> could not open /disks/pvfs2/olson/simtab.0917/sims.112
>>>>>>> Could not open /disks/pvfs2/olson/simtab.0917/sims.112
>>>>>>> simtab.0917/sims.113
>>>>>>> Wrote 159349826 bytes in 5.469951 seconds. 27.782303 MB/seconds
>>>>>>> simtab.0917/sims.114
>>>>>>> open: Permission denied
>>>>>>> could not open /disks/pvfs2/olson/simtab.0917/sims.114
>>>>>>> Could not open /disks/pvfs2/olson/simtab.0917/sims.114
>>>>>>> simtab.0917/sims.115
>>>>>>> Wrote 385509 bytes in 0.043475 seconds. 8.456602 MB/seconds
>>>>>>> simtab.0917/sims.116
>>>>>>> open: Permission denied
>>>>>>> could not open /disks/pvfs2/olson/simtab.0917/sims.116
>>>>>>> Could not open /disks/pvfs2/olson/simtab.0917/sims.116
>>>>>>> simtab.0917/sims.117
>>>>>>> Wrote 125580543 bytes in 4.246360 seconds. 28.203670 MB/seconds
>>>>>>> simtab.0917/sims.118
>>>>>>> open: Permission denied
>>>>>>> could not open /disks/pvfs2/olson/simtab.0917/sims.118
>>>>>>> Could not open /disks/pvfs2/olson/simtab.0917/sims.118
>>>>>>> simtab.0917/sims.119
>>>>>>> Wrote 74459114 bytes in 2.527315 seconds. 28.096912 MB/seconds
>>>>>>> simtab.0917/sims.12
>>>>>>> open: Permission denied
>>>>>>> could not open /disks/pvfs2/olson/simtab.0917/sims.12
>>>>>>> Could not open /disks/pvfs2/olson/simtab.0917/sims.12
>>>>>>> [etc]
>>>>>>>
>>>>>>>
>>>>>>> pvfs-statfs is happy:
>>>>>>>
>>>>>>> olson at bio-ppc-1:~$ /vol/pvfs2/bin/pvfs2-statfs -m /mnt/pvfs2
>>>>>>>
>>>>>>> aggregate statistics:
>>>>>>> ---------------------------------------
>>>>>>>
>>>>>>> fs_id: 1239905522
>>>>>>> total number of servers (meta and I/O): 4
>>>>>>> handles available (meta and I/O): 5365714538
>>>>>>> handles total (meta and I/O): 5368709115
>>>>>>> bytes available: 1218774892544
>>>>>>> bytes total: 1495224942592
>>>>>>>
>>>>>>> NOTE: The aggregate total and available statistics are
>>>>>>> calculated
>>>>>>> based
>>>>>>> on an algorithm that assumes data will be distributed evenly;
>>>>>>> thus
>>>>>>> the free space is equal to the smallest I/O server capacity
>>>>>>> multiplied by the number of I/O servers. If this number seems
>>>>>>> unusually small, then check the individual server statistics
>>>>>>> below
>>>>>>> to look for problematic servers.
>>>>>>>
>>>>>>> meta server statistics:
>>>>>>> ---------------------------------------
>>>>>>>
>>>>>>> server: tcp://bio-ppc-head-4:3334
>>>>>>> RAM bytes total : 4107878400
>>>>>>> RAM bytes free : 13307904
>>>>>>> uptime (seconds) : 4045972
>>>>>>> load averages : 4704 9088 5664
>>>>>>> handles available: 2145227856
>>>>>>> handles total : 2147483646
>>>>>>> bytes available : 310944141312
>>>>>>> bytes total : 373806235648
>>>>>>> mode: serving both metadata and I/O data
>>>>>>>
>>>>>>>
>>>>>>> I/O server statistics:
>>>>>>> ---------------------------------------
>>>>>>>
>>>>>>> server: tcp://bio-ppc-head-4:3334
>>>>>>> RAM bytes total : 4107878400
>>>>>>> RAM bytes free : 13307904
>>>>>>> uptime (seconds) : 4045972
>>>>>>> load averages : 4704 9088 5664
>>>>>>> handles available: 2145227856
>>>>>>> handles total : 2147483646
>>>>>>> bytes available : 310944141312
>>>>>>> bytes total : 373806235648
>>>>>>> mode: serving both metadata and I/O data
>>>>>>>
>>>>>>> server: tcp://bio-ppc-head-1:3334
>>>>>>> RAM bytes total : 4107878400
>>>>>>> RAM bytes free : 2062213120
>>>>>>> uptime (seconds) : 832980
>>>>>>> load averages : 30624 12352 3872
>>>>>>> handles available: 1073741272
>>>>>>> handles total : 1073741823
>>>>>>> bytes available : 307329638400
>>>>>>> bytes total : 373806235648
>>>>>>> mode: serving only I/O data
>>>>>>>
>>>>>>> server: tcp://bio-ppc-head-2:3334
>>>>>>> RAM bytes total : 4107878400
>>>>>>> RAM bytes free : 12697600
>>>>>>> uptime (seconds) : 4045977
>>>>>>> load averages : 10592 6720 1792
>>>>>>> handles available: 1073372712
>>>>>>> handles total : 1073741823
>>>>>>> bytes available : 304693723136
>>>>>>> bytes total : 373806235648
>>>>>>> mode: serving only I/O data
>>>>>>>
>>>>>>> server: tcp://bio-ppc-head-3:3334
>>>>>>> RAM bytes total : 3578347520
>>>>>>> RAM bytes free : 12324864
>>>>>>> uptime (seconds) : 4046045
>>>>>>> load averages : 13056 9376 3136
>>>>>>> handles available: 1073372698
>>>>>>> handles total : 1073741823
>>>>>>> bytes available : 309065900032
>>>>>>> bytes total : 373806235648
>>>>>>> mode: serving only I/O data
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Pvfs2-users mailing list
>>>>>>> Pvfs2-users at beowulf-underground.org
>>>>>>> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>
>> ---------------------------------------------------------------------
>> ---
>> _______________________________________________
>> Pvfs2-users mailing list
>> Pvfs2-users at beowulf-underground.org
>> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
>
More information about the Pvfs2-users
mailing list