[Pvfs2-users] goofy permission denied error
Rob Ross
rross at mcs.anl.gov
Tue Sep 26 11:18:10 EDT 2006
Thanks for reporting the bug Bob.
Rob
Sam Lang wrote:
>
> Bob,
>
> The attached patch should apply cleanly to the 1.5.1 release, and fix
> the problems you are seeing on ppc 64. You will have to rebuild after
> patching, and then restart the pvfs servers after rebuilding. I've
> committed the fixes to cvs and they'll get incorporated in the next
> release of pvfs. If you get a chance to try the patch, let me know.
>
> Thanks,
>
> -sam
>
>
> On Sep 25, 2006, at 10:15 PM, Robert Olson wrote:
>
>> One of the servers (on bio-ppc-head-1) was rebooted fairly recently,
>> while
>> the others were not.
>>
>> --bob
>>
>> On Mon, 25 Sep 2006, Sam Lang wrote:
>>
>>>
>>> Hi Bob,
>>>
>>> I think I found the bug and have a fix for this, but I just want to
>>> make sure. Do you remember the last time you restarted the pvfs2
>>> servers? I think this error is caused at server startup on ppc64
>>> after at least 512 files are created. If your servers were re-
>>> started recently (with a populated file system), then I know what's
>>> wrong. Otherwise, if they were started once when the file system was
>>> created and have been running ever since, I've just found a bug
>>> elsewhere in our code, and I'll need to keep looking for yours. Let
>>> me know.
>>>
>>> Thanks,
>>>
>>> -sam
>>>
>>> On Sep 25, 2006, at 1:51 PM, Robert Olson wrote:
>>>
>>>> This was built new for 1.5.1, and I've been using it on and off for
>>>> a few weeks.
>>>>
>>>> --bob
>>>>
>>>> On Sep 25, 2006, at 1:37 PM, Sam Lang wrote:
>>>>
>>>>>
>>>>> Hi Bob,
>>>>>
>>>>> Is this from a new storage space, or had things been working and
>>>>> then you started getting the permission denied errors? If its a
>>>>> storage space that's been around for a while, did you migrate from
>>>>> 1.3.x or 1.4.0?
>>>>>
>>>>> Based on the logfiles you sent, it doesn't look like a kernel
>>>>> module problem. We haven't seen those specific problems with
>>>>> 1.5.1, but we didn't do much testing of the 1.5.1 release on 64
>>>>> bit platforms (esp. ppc). Would it be possible to get access to
>>>>> that system so that I can do some debugging?
>>>>>
>>>>> Thanks,
>>>>>
>>>>> -sam
>>>>>
>>>>> On Sep 25, 2006, at 12:34 PM, Robert Olson wrote:
>>>>>
>>>>>> This is weird. I appear to getting permission denied on every
>>>>>> other file to be copied from a local filesystem to the PVFS2
>>>>>> filesystem. This ring a bell with anyone?
>>>>>>
>>>>>> Thanks,
>>>>>> --bob
>>>>>>
>>>>>> olson at bio-ppc-1:/scratch$ cp -r simtab.0917 /disks/pvfs2/olson
>>>>>> cp: cannot create regular file `/disks/pvfs2/olson/simtab.0917/
>>>>>> sims.10': Permission denied
>>>>>> cp: cannot create regular file `/disks/pvfs2/olson/simtab.0917/
>>>>>> sims.101': Permission denied
>>>>>> cp: cannot create regular file `/disks/pvfs2/olson/simtab.0917/
>>>>>> sims.103': Permission denied
>>>>>> cp: cannot create regular file `/disks/pvfs2/olson/simtab.0917/
>>>>>> sims.105': Permission denied
>>>>>>
>>>>>>
>>>>>> olson at bio-ppc-1:/scratch$ ls -l /disks/pvfs2/olson/simtab.0917/
>>>>>> total 88044
>>>>>> -rw-r--r-- 1 olson seed-users 37994316 Sep 25 17:26 sims.1
>>>>>> -rw-r--r-- 1 olson seed-users 30049 Sep 25 17:26 sims.100
>>>>>> -rw-r--r-- 1 olson seed-users 7226 Sep 25 17:26 sims.102
>>>>>> -rw-r--r-- 1 olson seed-users 1915643 Sep 25 17:26 sims.104
>>>>>> -rw-r--r-- 1 olson seed-users 50200576 Sep 25 17:26 sims.106
>>>>>>
>>>>>> olson at bio-ppc-1:/scratch$ for i in simtab.0917/*; do echo $i; /
>>>>>> vol/pvfs2/bin/pvfs2-cp -t $i /disks/pvfs2/olson/$i; done
>>>>>> simtab.0917/sims.1
>>>>>> Wrote 37994316 bytes in 0.437222 seconds. 82.873696 MB/seconds
>>>>>> simtab.0917/sims.10
>>>>>> open: Permission denied
>>>>>> could not open /disks/pvfs2/olson/simtab.0917/sims.10
>>>>>> Could not open /disks/pvfs2/olson/simtab.0917/sims.10
>>>>>> simtab.0917/sims.100
>>>>>> Wrote 30049 bytes in 0.001754 seconds. 16.339859 MB/seconds
>>>>>> simtab.0917/sims.101
>>>>>> Wrote 18867 bytes in 0.035888 seconds. 0.501365 MB/seconds
>>>>>> simtab.0917/sims.102
>>>>>> Wrote 7226 bytes in 0.001195 seconds. 5.766959 MB/seconds
>>>>>> simtab.0917/sims.103
>>>>>> open: Permission denied
>>>>>> could not open /disks/pvfs2/olson/simtab.0917/sims.103
>>>>>> Could not open /disks/pvfs2/olson/simtab.0917/sims.103
>>>>>> simtab.0917/sims.104
>>>>>> Wrote 1915643 bytes in 0.022985 seconds. 79.482314 MB/seconds
>>>>>> simtab.0917/sims.105
>>>>>> Wrote 113037406 bytes in 3.774026 seconds. 28.563891 MB/seconds
>>>>>> simtab.0917/sims.106
>>>>>> Wrote 177660236 bytes in 5.191931 seconds. 32.633331 MB/seconds
>>>>>> simtab.0917/sims.107
>>>>>> open: Permission denied
>>>>>> could not open /disks/pvfs2/olson/simtab.0917/sims.107
>>>>>> Could not open /disks/pvfs2/olson/simtab.0917/sims.107
>>>>>> simtab.0917/sims.108
>>>>>> Wrote 152382253 bytes in 5.079284 seconds. 28.610931 MB/seconds
>>>>>> simtab.0917/sims.109
>>>>>> open: Permission denied
>>>>>> could not open /disks/pvfs2/olson/simtab.0917/sims.109
>>>>>> Could not open /disks/pvfs2/olson/simtab.0917/sims.109
>>>>>> simtab.0917/sims.11
>>>>>> Wrote 55995114 bytes in 1.893243 seconds. 28.206152 MB/seconds
>>>>>> simtab.0917/sims.110
>>>>>> open: Permission denied
>>>>>> could not open /disks/pvfs2/olson/simtab.0917/sims.110
>>>>>> Could not open /disks/pvfs2/olson/simtab.0917/sims.110
>>>>>> simtab.0917/sims.111
>>>>>> Wrote 85578997 bytes in 2.864490 seconds. 28.491805 MB/seconds
>>>>>> simtab.0917/sims.112
>>>>>> open: Permission denied
>>>>>> could not open /disks/pvfs2/olson/simtab.0917/sims.112
>>>>>> Could not open /disks/pvfs2/olson/simtab.0917/sims.112
>>>>>> simtab.0917/sims.113
>>>>>> Wrote 159349826 bytes in 5.469951 seconds. 27.782303 MB/seconds
>>>>>> simtab.0917/sims.114
>>>>>> open: Permission denied
>>>>>> could not open /disks/pvfs2/olson/simtab.0917/sims.114
>>>>>> Could not open /disks/pvfs2/olson/simtab.0917/sims.114
>>>>>> simtab.0917/sims.115
>>>>>> Wrote 385509 bytes in 0.043475 seconds. 8.456602 MB/seconds
>>>>>> simtab.0917/sims.116
>>>>>> open: Permission denied
>>>>>> could not open /disks/pvfs2/olson/simtab.0917/sims.116
>>>>>> Could not open /disks/pvfs2/olson/simtab.0917/sims.116
>>>>>> simtab.0917/sims.117
>>>>>> Wrote 125580543 bytes in 4.246360 seconds. 28.203670 MB/seconds
>>>>>> simtab.0917/sims.118
>>>>>> open: Permission denied
>>>>>> could not open /disks/pvfs2/olson/simtab.0917/sims.118
>>>>>> Could not open /disks/pvfs2/olson/simtab.0917/sims.118
>>>>>> simtab.0917/sims.119
>>>>>> Wrote 74459114 bytes in 2.527315 seconds. 28.096912 MB/seconds
>>>>>> simtab.0917/sims.12
>>>>>> open: Permission denied
>>>>>> could not open /disks/pvfs2/olson/simtab.0917/sims.12
>>>>>> Could not open /disks/pvfs2/olson/simtab.0917/sims.12
>>>>>> [etc]
>>>>>>
>>>>>>
>>>>>> pvfs-statfs is happy:
>>>>>>
>>>>>> olson at bio-ppc-1:~$ /vol/pvfs2/bin/pvfs2-statfs -m /mnt/pvfs2
>>>>>>
>>>>>> aggregate statistics:
>>>>>> ---------------------------------------
>>>>>>
>>>>>> fs_id: 1239905522
>>>>>> total number of servers (meta and I/O): 4
>>>>>> handles available (meta and I/O): 5365714538
>>>>>> handles total (meta and I/O): 5368709115
>>>>>> bytes available: 1218774892544
>>>>>> bytes total: 1495224942592
>>>>>>
>>>>>> NOTE: The aggregate total and available statistics are calculated
>>>>>> based
>>>>>> on an algorithm that assumes data will be distributed evenly; thus
>>>>>> the free space is equal to the smallest I/O server capacity
>>>>>> multiplied by the number of I/O servers. If this number seems
>>>>>> unusually small, then check the individual server statistics below
>>>>>> to look for problematic servers.
>>>>>>
>>>>>> meta server statistics:
>>>>>> ---------------------------------------
>>>>>>
>>>>>> server: tcp://bio-ppc-head-4:3334
>>>>>> RAM bytes total : 4107878400
>>>>>> RAM bytes free : 13307904
>>>>>> uptime (seconds) : 4045972
>>>>>> load averages : 4704 9088 5664
>>>>>> handles available: 2145227856
>>>>>> handles total : 2147483646
>>>>>> bytes available : 310944141312
>>>>>> bytes total : 373806235648
>>>>>> mode: serving both metadata and I/O data
>>>>>>
>>>>>>
>>>>>> I/O server statistics:
>>>>>> ---------------------------------------
>>>>>>
>>>>>> server: tcp://bio-ppc-head-4:3334
>>>>>> RAM bytes total : 4107878400
>>>>>> RAM bytes free : 13307904
>>>>>> uptime (seconds) : 4045972
>>>>>> load averages : 4704 9088 5664
>>>>>> handles available: 2145227856
>>>>>> handles total : 2147483646
>>>>>> bytes available : 310944141312
>>>>>> bytes total : 373806235648
>>>>>> mode: serving both metadata and I/O data
>>>>>>
>>>>>> server: tcp://bio-ppc-head-1:3334
>>>>>> RAM bytes total : 4107878400
>>>>>> RAM bytes free : 2062213120
>>>>>> uptime (seconds) : 832980
>>>>>> load averages : 30624 12352 3872
>>>>>> handles available: 1073741272
>>>>>> handles total : 1073741823
>>>>>> bytes available : 307329638400
>>>>>> bytes total : 373806235648
>>>>>> mode: serving only I/O data
>>>>>>
>>>>>> server: tcp://bio-ppc-head-2:3334
>>>>>> RAM bytes total : 4107878400
>>>>>> RAM bytes free : 12697600
>>>>>> uptime (seconds) : 4045977
>>>>>> load averages : 10592 6720 1792
>>>>>> handles available: 1073372712
>>>>>> handles total : 1073741823
>>>>>> bytes available : 304693723136
>>>>>> bytes total : 373806235648
>>>>>> mode: serving only I/O data
>>>>>>
>>>>>> server: tcp://bio-ppc-head-3:3334
>>>>>> RAM bytes total : 3578347520
>>>>>> RAM bytes free : 12324864
>>>>>> uptime (seconds) : 4046045
>>>>>> load averages : 13056 9376 3136
>>>>>> handles available: 1073372698
>>>>>> handles total : 1073741823
>>>>>> bytes available : 309065900032
>>>>>> bytes total : 373806235648
>>>>>> mode: serving only I/O data
>>>>>>
>>>>>> _______________________________________________
>>>>>> Pvfs2-users mailing list
>>>>>> Pvfs2-users at beowulf-underground.org
>>>>>> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
>>>>>>
>>>>>
>>>>
>>>
>>>
>>
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Pvfs2-users mailing list
> Pvfs2-users at beowulf-underground.org
> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
More information about the Pvfs2-users
mailing list