[Pvfs2-users] goofy permission denied error
Sam Lang
slang at mcs.anl.gov
Tue Sep 26 11:11:52 EDT 2006
Bob,
The attached patch should apply cleanly to the 1.5.1 release, and fix
the problems you are seeing on ppc 64. You will have to rebuild
after patching, and then restart the pvfs servers after rebuilding.
I've committed the fixes to cvs and they'll get incorporated in the
next release of pvfs. If you get a chance to try the patch, let me
know.
Thanks,
-sam
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ledger-fix.patch
Type: application/octet-stream
Size: 2463 bytes
Desc: not available
Url : http://www.beowulf-underground.org/pipermail/pvfs2-users/attachments/20060926/880d808f/ledger-fix.obj
-------------- next part --------------
On Sep 25, 2006, at 10:15 PM, Robert Olson wrote:
> One of the servers (on bio-ppc-head-1) was rebooted fairly
> recently, while
> the others were not.
>
> --bob
>
> On Mon, 25 Sep 2006, Sam Lang wrote:
>
>>
>> Hi Bob,
>>
>> I think I found the bug and have a fix for this, but I just want to
>> make sure. Do you remember the last time you restarted the pvfs2
>> servers? I think this error is caused at server startup on ppc64
>> after at least 512 files are created. If your servers were re-
>> started recently (with a populated file system), then I know what's
>> wrong. Otherwise, if they were started once when the file system was
>> created and have been running ever since, I've just found a bug
>> elsewhere in our code, and I'll need to keep looking for yours. Let
>> me know.
>>
>> Thanks,
>>
>> -sam
>>
>> On Sep 25, 2006, at 1:51 PM, Robert Olson wrote:
>>
>>> This was built new for 1.5.1, and I've been using it on and off for
>>> a few weeks.
>>>
>>> --bob
>>>
>>> On Sep 25, 2006, at 1:37 PM, Sam Lang wrote:
>>>
>>>>
>>>> Hi Bob,
>>>>
>>>> Is this from a new storage space, or had things been working and
>>>> then you started getting the permission denied errors? If its a
>>>> storage space that's been around for a while, did you migrate from
>>>> 1.3.x or 1.4.0?
>>>>
>>>> Based on the logfiles you sent, it doesn't look like a kernel
>>>> module problem. We haven't seen those specific problems with
>>>> 1.5.1, but we didn't do much testing of the 1.5.1 release on 64
>>>> bit platforms (esp. ppc). Would it be possible to get access to
>>>> that system so that I can do some debugging?
>>>>
>>>> Thanks,
>>>>
>>>> -sam
>>>>
>>>> On Sep 25, 2006, at 12:34 PM, Robert Olson wrote:
>>>>
>>>>> This is weird. I appear to getting permission denied on every
>>>>> other file to be copied from a local filesystem to the PVFS2
>>>>> filesystem. This ring a bell with anyone?
>>>>>
>>>>> Thanks,
>>>>> --bob
>>>>>
>>>>> olson at bio-ppc-1:/scratch$ cp -r simtab.0917 /disks/pvfs2/olson
>>>>> cp: cannot create regular file `/disks/pvfs2/olson/simtab.0917/
>>>>> sims.10': Permission denied
>>>>> cp: cannot create regular file `/disks/pvfs2/olson/simtab.0917/
>>>>> sims.101': Permission denied
>>>>> cp: cannot create regular file `/disks/pvfs2/olson/simtab.0917/
>>>>> sims.103': Permission denied
>>>>> cp: cannot create regular file `/disks/pvfs2/olson/simtab.0917/
>>>>> sims.105': Permission denied
>>>>>
>>>>>
>>>>> olson at bio-ppc-1:/scratch$ ls -l /disks/pvfs2/olson/simtab.0917/
>>>>> total 88044
>>>>> -rw-r--r-- 1 olson seed-users 37994316 Sep 25 17:26 sims.1
>>>>> -rw-r--r-- 1 olson seed-users 30049 Sep 25 17:26 sims.100
>>>>> -rw-r--r-- 1 olson seed-users 7226 Sep 25 17:26 sims.102
>>>>> -rw-r--r-- 1 olson seed-users 1915643 Sep 25 17:26 sims.104
>>>>> -rw-r--r-- 1 olson seed-users 50200576 Sep 25 17:26 sims.106
>>>>>
>>>>> olson at bio-ppc-1:/scratch$ for i in simtab.0917/*; do echo $i; /
>>>>> vol/pvfs2/bin/pvfs2-cp -t $i /disks/pvfs2/olson/$i; done
>>>>> simtab.0917/sims.1
>>>>> Wrote 37994316 bytes in 0.437222 seconds. 82.873696 MB/seconds
>>>>> simtab.0917/sims.10
>>>>> open: Permission denied
>>>>> could not open /disks/pvfs2/olson/simtab.0917/sims.10
>>>>> Could not open /disks/pvfs2/olson/simtab.0917/sims.10
>>>>> simtab.0917/sims.100
>>>>> Wrote 30049 bytes in 0.001754 seconds. 16.339859 MB/seconds
>>>>> simtab.0917/sims.101
>>>>> Wrote 18867 bytes in 0.035888 seconds. 0.501365 MB/seconds
>>>>> simtab.0917/sims.102
>>>>> Wrote 7226 bytes in 0.001195 seconds. 5.766959 MB/seconds
>>>>> simtab.0917/sims.103
>>>>> open: Permission denied
>>>>> could not open /disks/pvfs2/olson/simtab.0917/sims.103
>>>>> Could not open /disks/pvfs2/olson/simtab.0917/sims.103
>>>>> simtab.0917/sims.104
>>>>> Wrote 1915643 bytes in 0.022985 seconds. 79.482314 MB/seconds
>>>>> simtab.0917/sims.105
>>>>> Wrote 113037406 bytes in 3.774026 seconds. 28.563891 MB/seconds
>>>>> simtab.0917/sims.106
>>>>> Wrote 177660236 bytes in 5.191931 seconds. 32.633331 MB/seconds
>>>>> simtab.0917/sims.107
>>>>> open: Permission denied
>>>>> could not open /disks/pvfs2/olson/simtab.0917/sims.107
>>>>> Could not open /disks/pvfs2/olson/simtab.0917/sims.107
>>>>> simtab.0917/sims.108
>>>>> Wrote 152382253 bytes in 5.079284 seconds. 28.610931 MB/seconds
>>>>> simtab.0917/sims.109
>>>>> open: Permission denied
>>>>> could not open /disks/pvfs2/olson/simtab.0917/sims.109
>>>>> Could not open /disks/pvfs2/olson/simtab.0917/sims.109
>>>>> simtab.0917/sims.11
>>>>> Wrote 55995114 bytes in 1.893243 seconds. 28.206152 MB/seconds
>>>>> simtab.0917/sims.110
>>>>> open: Permission denied
>>>>> could not open /disks/pvfs2/olson/simtab.0917/sims.110
>>>>> Could not open /disks/pvfs2/olson/simtab.0917/sims.110
>>>>> simtab.0917/sims.111
>>>>> Wrote 85578997 bytes in 2.864490 seconds. 28.491805 MB/seconds
>>>>> simtab.0917/sims.112
>>>>> open: Permission denied
>>>>> could not open /disks/pvfs2/olson/simtab.0917/sims.112
>>>>> Could not open /disks/pvfs2/olson/simtab.0917/sims.112
>>>>> simtab.0917/sims.113
>>>>> Wrote 159349826 bytes in 5.469951 seconds. 27.782303 MB/seconds
>>>>> simtab.0917/sims.114
>>>>> open: Permission denied
>>>>> could not open /disks/pvfs2/olson/simtab.0917/sims.114
>>>>> Could not open /disks/pvfs2/olson/simtab.0917/sims.114
>>>>> simtab.0917/sims.115
>>>>> Wrote 385509 bytes in 0.043475 seconds. 8.456602 MB/seconds
>>>>> simtab.0917/sims.116
>>>>> open: Permission denied
>>>>> could not open /disks/pvfs2/olson/simtab.0917/sims.116
>>>>> Could not open /disks/pvfs2/olson/simtab.0917/sims.116
>>>>> simtab.0917/sims.117
>>>>> Wrote 125580543 bytes in 4.246360 seconds. 28.203670 MB/seconds
>>>>> simtab.0917/sims.118
>>>>> open: Permission denied
>>>>> could not open /disks/pvfs2/olson/simtab.0917/sims.118
>>>>> Could not open /disks/pvfs2/olson/simtab.0917/sims.118
>>>>> simtab.0917/sims.119
>>>>> Wrote 74459114 bytes in 2.527315 seconds. 28.096912 MB/seconds
>>>>> simtab.0917/sims.12
>>>>> open: Permission denied
>>>>> could not open /disks/pvfs2/olson/simtab.0917/sims.12
>>>>> Could not open /disks/pvfs2/olson/simtab.0917/sims.12
>>>>> [etc]
>>>>>
>>>>>
>>>>> pvfs-statfs is happy:
>>>>>
>>>>> olson at bio-ppc-1:~$ /vol/pvfs2/bin/pvfs2-statfs -m /mnt/pvfs2
>>>>>
>>>>> aggregate statistics:
>>>>> ---------------------------------------
>>>>>
>>>>> fs_id: 1239905522
>>>>> total number of servers (meta and I/O): 4
>>>>> handles available (meta and I/O): 5365714538
>>>>> handles total (meta and I/O): 5368709115
>>>>> bytes available: 1218774892544
>>>>> bytes total: 1495224942592
>>>>>
>>>>> NOTE: The aggregate total and available statistics are calculated
>>>>> based
>>>>> on an algorithm that assumes data will be distributed evenly; thus
>>>>> the free space is equal to the smallest I/O server capacity
>>>>> multiplied by the number of I/O servers. If this number seems
>>>>> unusually small, then check the individual server statistics below
>>>>> to look for problematic servers.
>>>>>
>>>>> meta server statistics:
>>>>> ---------------------------------------
>>>>>
>>>>> server: tcp://bio-ppc-head-4:3334
>>>>> RAM bytes total : 4107878400
>>>>> RAM bytes free : 13307904
>>>>> uptime (seconds) : 4045972
>>>>> load averages : 4704 9088 5664
>>>>> handles available: 2145227856
>>>>> handles total : 2147483646
>>>>> bytes available : 310944141312
>>>>> bytes total : 373806235648
>>>>> mode: serving both metadata and I/O data
>>>>>
>>>>>
>>>>> I/O server statistics:
>>>>> ---------------------------------------
>>>>>
>>>>> server: tcp://bio-ppc-head-4:3334
>>>>> RAM bytes total : 4107878400
>>>>> RAM bytes free : 13307904
>>>>> uptime (seconds) : 4045972
>>>>> load averages : 4704 9088 5664
>>>>> handles available: 2145227856
>>>>> handles total : 2147483646
>>>>> bytes available : 310944141312
>>>>> bytes total : 373806235648
>>>>> mode: serving both metadata and I/O data
>>>>>
>>>>> server: tcp://bio-ppc-head-1:3334
>>>>> RAM bytes total : 4107878400
>>>>> RAM bytes free : 2062213120
>>>>> uptime (seconds) : 832980
>>>>> load averages : 30624 12352 3872
>>>>> handles available: 1073741272
>>>>> handles total : 1073741823
>>>>> bytes available : 307329638400
>>>>> bytes total : 373806235648
>>>>> mode: serving only I/O data
>>>>>
>>>>> server: tcp://bio-ppc-head-2:3334
>>>>> RAM bytes total : 4107878400
>>>>> RAM bytes free : 12697600
>>>>> uptime (seconds) : 4045977
>>>>> load averages : 10592 6720 1792
>>>>> handles available: 1073372712
>>>>> handles total : 1073741823
>>>>> bytes available : 304693723136
>>>>> bytes total : 373806235648
>>>>> mode: serving only I/O data
>>>>>
>>>>> server: tcp://bio-ppc-head-3:3334
>>>>> RAM bytes total : 3578347520
>>>>> RAM bytes free : 12324864
>>>>> uptime (seconds) : 4046045
>>>>> load averages : 13056 9376 3136
>>>>> handles available: 1073372698
>>>>> handles total : 1073741823
>>>>> bytes available : 309065900032
>>>>> bytes total : 373806235648
>>>>> mode: serving only I/O data
>>>>>
>>>>> _______________________________________________
>>>>> Pvfs2-users mailing list
>>>>> Pvfs2-users at beowulf-underground.org
>>>>> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
>>>>>
>>>>
>>>
>>
>>
>
More information about the Pvfs2-users
mailing list