[Pvfs2-developers] Re: Pvfs2-developers Digest, Vol 72, Issue 8
Dimokritos Stamatakis
dimstamat at gmail.com
Mon Feb 20 12:27:10 EST 2012
Hello and thanks for your answer!
The problem was the amazon ec2-disassociate command, where the public IP became unreachable. I now invoke just the ec2-associate and everything is fine!!
Thanks to all for trying to help,
Dimos.
On 20 Φεβ 2012, at 5:42 μ.μ., pvfs2-developers-request at beowulf-underground.org wrote:
> Send Pvfs2-developers mailing list submissions to
> pvfs2-developers at beowulf-underground.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
> or, via email, send a message with subject or body 'help' to
> pvfs2-developers-request at beowulf-underground.org
>
> You can reach the person managing the list at
> pvfs2-developers-owner at beowulf-underground.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Pvfs2-developers digest..."
>
>
> Today's Topics:
>
> 1. no space left on device when running on amazon ec2
> (Dimos Stamatakis)
> 2. Re: no space left on device when running on amazon ec2
> (Dimos Stamatakis)
> 3. Re: no space left on device when running on amazon ec2
> (Dimos Stamatakis)
> 4. Re: Re: no space left on device when running on amazon ec2
> (Becky Ligon)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Mon, 20 Feb 2012 02:17:01 +0200
> From: Dimos Stamatakis <dimstamat at gmail.com>
> Subject: [Pvfs2-developers] no space left on device when running on
> amazon ec2
> To: pvfs2-developers at beowulf-underground.org
> Message-ID:
> <CAB8CW0ET2U3OtOXTgZMXBPRiBmvR4=0FpyM8gzDg8s0A4KSf5w at mail.gmail.com>
> Content-Type: text/plain; charset="iso-8859-1"
>
> Hello!
> I have successfully run a pvfs installation on a eucalyptus cloud, but when
> I moved to Amazon EC2, I get a very strange error.
> When I run a metadata server it says:
>
> [S 02/20 00:04] PVFS2 Server ready.
>
> and then it says:
>
> [E 02/20 00:04] batch_create request got: No space left on device
> ....... And this error repeats ......
>
> I checked all of my devices and there is plenty of space, so I don't think
> there is not enough space left...
> Can you explain that?
> What is the batch_create function? And where is it trying to write?
>
> Here is the output of the df -h on the data node:
>
> Filesystem Size Used Avail Use% Mounted on
> /dev/sda1 9.9G 2.7G 6.8G 29% /
> tmpfs 308M 0 308M 0% /lib/init/rw
> udev 10M 108K 9.9M 2% /dev
> tmpfs 308M 4.0K 308M 1% /dev/shm
>
> and on the meta data node:
>
> Filesystem Size Used Avail Use% Mounted on
> /dev/sda1 9.9G 2.0G 7.5G 21% /
> tmpfs 308M 0 308M 0% /lib/init/rw
> udev 10M 108K 9.9M 2% /dev
> tmpfs 308M 4.0K 308M 1% /dev/shm
>
> Many thanks,
> Dimos.
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: http://www.beowulf-underground.org/pipermail/pvfs2-developers/attachments/20120220/e5087dbc/attachment-0001.htm
>
> ------------------------------
>
> Message: 2
> Date: Mon, 20 Feb 2012 03:49:50 +0200
> From: Dimos Stamatakis <dimstamat at gmail.com>
> Subject: [Pvfs2-developers] Re: no space left on device when running
> on amazon ec2
> To: pvfs2-developers at beowulf-underground.org
> Message-ID:
> <CAB8CW0EvXb6012tqNWpAsg_bug4+ZuLniKv8cTY0tX1fnic=gQ at mail.gmail.com>
> Content-Type: text/plain; charset="iso-8859-1"
>
> I forgot to tell you that I use ec2-associate-address commands to tell the
> new master to grab the elastic IP address. If I do not use replication and
> I use the normal IP addresses it works fine!
> Is there a way to have High availability to amazon EC2 without use of
> elastic IPs??
>
> Many thanks,
> Dimos.
>
>
> On Mon, Feb 20, 2012 at 2:17 AM, Dimos Stamatakis <dimstamat at gmail.com>wrote:
>
>> Hello!
>> I have successfully run a pvfs installation on a eucalyptus cloud, but
>> when I moved to Amazon EC2, I get a very strange error.
>> When I run a metadata server it says:
>>
>> [S 02/20 00:04] PVFS2 Server ready.
>>
>> and then it says:
>>
>> [E 02/20 00:04] batch_create request got: No space left on device
>> ....... And this error repeats ......
>>
>> I checked all of my devices and there is plenty of space, so I don't think
>> there is not enough space left...
>> Can you explain that?
>> What is the batch_create function? And where is it trying to write?
>>
>> Here is the output of the df -h on the data node:
>>
>> Filesystem Size Used Avail Use% Mounted on
>> /dev/sda1 9.9G 2.7G 6.8G 29% /
>> tmpfs 308M 0 308M 0% /lib/init/rw
>> udev 10M 108K 9.9M 2% /dev
>> tmpfs 308M 4.0K 308M 1% /dev/shm
>>
>> and on the meta data node:
>>
>> Filesystem Size Used Avail Use% Mounted on
>> /dev/sda1 9.9G 2.0G 7.5G 21% /
>> tmpfs 308M 0 308M 0% /lib/init/rw
>> udev 10M 108K 9.9M 2% /dev
>> tmpfs 308M 4.0K 308M 1% /dev/shm
>>
>> Many thanks,
>> Dimos.
>>
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: http://www.beowulf-underground.org/pipermail/pvfs2-developers/attachments/20120220/ffdfe9c8/attachment-0001.htm
>
> ------------------------------
>
> Message: 3
> Date: Mon, 20 Feb 2012 05:48:30 +0200
> From: Dimos Stamatakis <dimstamat at gmail.com>
> Subject: [Pvfs2-developers] Re: no space left on device when running
> on amazon ec2
> To: pvfs2-developers at beowulf-underground.org
> Message-ID:
> <CAB8CW0Er_JzyH9uO7z0wQekXHk3_yrNn+kd700SrKkSR8qV_Qg at mail.gmail.com>
> Content-Type: text/plain; charset="iso-8859-1"
>
> Hello again!
> I want to help you realize what is going wrong by telling you that the
> client blocks at a pvfs2-ls (it does not say connection refused
> immediatelly). It can also ping the new elastic IP normally!
>
> I redirected the metadata server output to a file and when I checked it i
> didn't find anything wrong... It did all the gets - puts that are happening
> everytime the DB is created. Here is the output tail:
>
> get (handle: 4611686018427387903)()(key_sz:8) -> (511)(4)
> put (handle: 4611686018427387903)()(key_sz:8) -> (512)(4)
> [1329709072:419164][4413/140213703595776] TROVE:DBPF:Berkeley DB: bulk_msg:
> Send buffer after copy due to PERM
> [1329709072:419173][4413/140213703595776] TROVE:DBPF:Berkeley DB:
> send_bulk: Send 160 (0xa0) bulk buffer bytes
> [1329709072:419183][4413/140213703595776] TROVE:DBPF:Berkeley DB:
> //pvfs2-storage-space/27c41225/ rep_send_message: msgv = 7 logv 19 gen = 1
> eid -1, type bulk_log, LSN [1][217660] perm
> [1329709072:419193][4413/140213703595776] TROVE:DBPF:Berkeley DB:
> rep_send_function returned: -30975
>
> How can I find out why this metadata server refuses serving the client
> requests?
>
> Thanks again,
> Dimos.
>
>
>
> On Mon, Feb 20, 2012 at 3:49 AM, Dimos Stamatakis <dimstamat at gmail.com>wrote:
>
>> I forgot to tell you that I use ec2-associate-address commands to tell the
>> new master to grab the elastic IP address. If I do not use replication and
>> I use the normal IP addresses it works fine!
>> Is there a way to have High availability to amazon EC2 without use of
>> elastic IPs??
>>
>> Many thanks,
>> Dimos.
>>
>>
>>
>> On Mon, Feb 20, 2012 at 2:17 AM, Dimos Stamatakis <dimstamat at gmail.com>wrote:
>>
>>> Hello!
>>> I have successfully run a pvfs installation on a eucalyptus cloud, but
>>> when I moved to Amazon EC2, I get a very strange error.
>>> When I run a metadata server it says:
>>>
>>> [S 02/20 00:04] PVFS2 Server ready.
>>>
>>> and then it says:
>>>
>>> [E 02/20 00:04] batch_create request got: No space left on device
>>> ....... And this error repeats ......
>>>
>>> I checked all of my devices and there is plenty of space, so I don't
>>> think there is not enough space left...
>>> Can you explain that?
>>> What is the batch_create function? And where is it trying to write?
>>>
>>> Here is the output of the df -h on the data node:
>>>
>>> Filesystem Size Used Avail Use% Mounted on
>>> /dev/sda1 9.9G 2.7G 6.8G 29% /
>>> tmpfs 308M 0 308M 0% /lib/init/rw
>>> udev 10M 108K 9.9M 2% /dev
>>> tmpfs 308M 4.0K 308M 1% /dev/shm
>>>
>>> and on the meta data node:
>>>
>>> Filesystem Size Used Avail Use% Mounted on
>>> /dev/sda1 9.9G 2.0G 7.5G 21% /
>>> tmpfs 308M 0 308M 0% /lib/init/rw
>>> udev 10M 108K 9.9M 2% /dev
>>> tmpfs 308M 4.0K 308M 1% /dev/shm
>>>
>>> Many thanks,
>>> Dimos.
>>>
>>
>>
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: http://www.beowulf-underground.org/pipermail/pvfs2-developers/attachments/20120220/8f30dcd1/attachment-0001.htm
>
> ------------------------------
>
> Message: 4
> Date: Mon, 20 Feb 2012 10:42:21 -0500
> From: Becky Ligon <ligon at omnibond.com>
> Subject: Re: [Pvfs2-developers] Re: no space left on device when
> running on amazon ec2
> To: Dimos Stamatakis <dimstamat at gmail.com>
> Cc: pvfs2-developers at beowulf-underground.org
> Message-ID:
> <CAJBnF338CSi4q80XPCcRY7kPm+U7bzwkPmvFhS7J1UxUO2CLwQ at mail.gmail.com>
> Content-Type: text/plain; charset="iso-8859-1"
>
> When a server is started, it sends a batch_create request to every other
> server in the filesystem. The batch_create request asks the receiving
> server to send back a list of unused data handles (owned by that particular
> server). For those handles in the list, the receiving server sets an
> attribute in the local database to indicate that the handle is in use. You
> may see a batch_create request after your servers have been running for a
> while, since a server will request another batch of handles if its current
> store gets low. This entire process is a performance enhancement, which
> allows a file's data handles to be assigned by the metadata server without
> contacting the data handle servers, thus reducing the time it takes to
> create a file.
>
> With all of that said, it seems that one of your servers is having trouble
> accessing the database or communicating with another server. I'm not
> exactly sure without further research. Think about the above description
> and see if you can't pinpoint which server is causing the trouble.
>
> Becky
>
> On Sun, Feb 19, 2012 at 10:48 PM, Dimos Stamatakis <dimstamat at gmail.com>wrote:
>
>> Hello again!
>> I want to help you realize what is going wrong by telling you that the
>> client blocks at a pvfs2-ls (it does not say connection refused
>> immediatelly). It can also ping the new elastic IP normally!
>>
>> I redirected the metadata server output to a file and when I checked it i
>> didn't find anything wrong... It did all the gets - puts that are happening
>> everytime the DB is created. Here is the output tail:
>>
>> get (handle: 4611686018427387903)()(key_sz:8) -> (511)(4)
>> put (handle: 4611686018427387903)()(key_sz:8) -> (512)(4)
>> [1329709072:419164][4413/140213703595776] TROVE:DBPF:Berkeley DB:
>> bulk_msg: Send buffer after copy due to PERM
>> [1329709072:419173][4413/140213703595776] TROVE:DBPF:Berkeley DB:
>> send_bulk: Send 160 (0xa0) bulk buffer bytes
>> [1329709072:419183][4413/140213703595776] TROVE:DBPF:Berkeley DB:
>> //pvfs2-storage-space/27c41225/ rep_send_message: msgv = 7 logv 19 gen = 1
>> eid -1, type bulk_log, LSN [1][217660] perm
>> [1329709072:419193][4413/140213703595776] TROVE:DBPF:Berkeley DB:
>> rep_send_function returned: -30975
>>
>> How can I find out why this metadata server refuses serving the client
>> requests?
>>
>> Thanks again,
>> Dimos.
>>
>>
>>
>> On Mon, Feb 20, 2012 at 3:49 AM, Dimos Stamatakis <dimstamat at gmail.com>wrote:
>>
>>> I forgot to tell you that I use ec2-associate-address commands to tell
>>> the new master to grab the elastic IP address. If I do not use replication
>>> and I use the normal IP addresses it works fine!
>>> Is there a way to have High availability to amazon EC2 without use of
>>> elastic IPs??
>>>
>>> Many thanks,
>>> Dimos.
>>>
>>>
>>>
>>> On Mon, Feb 20, 2012 at 2:17 AM, Dimos Stamatakis <dimstamat at gmail.com>wrote:
>>>
>>>> Hello!
>>>> I have successfully run a pvfs installation on a eucalyptus cloud, but
>>>> when I moved to Amazon EC2, I get a very strange error.
>>>> When I run a metadata server it says:
>>>>
>>>> [S 02/20 00:04] PVFS2 Server ready.
>>>>
>>>> and then it says:
>>>>
>>>> [E 02/20 00:04] batch_create request got: No space left on device
>>>> ....... And this error repeats ......
>>>>
>>>> I checked all of my devices and there is plenty of space, so I don't
>>>> think there is not enough space left...
>>>> Can you explain that?
>>>> What is the batch_create function? And where is it trying to write?
>>>>
>>>> Here is the output of the df -h on the data node:
>>>>
>>>> Filesystem Size Used Avail Use% Mounted on
>>>> /dev/sda1 9.9G 2.7G 6.8G 29% /
>>>> tmpfs 308M 0 308M 0% /lib/init/rw
>>>> udev 10M 108K 9.9M 2% /dev
>>>> tmpfs 308M 4.0K 308M 1% /dev/shm
>>>>
>>>> and on the meta data node:
>>>>
>>>> Filesystem Size Used Avail Use% Mounted on
>>>> /dev/sda1 9.9G 2.0G 7.5G 21% /
>>>> tmpfs 308M 0 308M 0% /lib/init/rw
>>>> udev 10M 108K 9.9M 2% /dev
>>>> tmpfs 308M 4.0K 308M 1% /dev/shm
>>>>
>>>> Many thanks,
>>>> Dimos.
>>>>
>>>
>>>
>>
>> _______________________________________________
>> Pvfs2-developers mailing list
>> Pvfs2-developers at beowulf-underground.org
>> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
>>
>>
>
>
> --
> Becky Ligon
> OrangeFS Support and Development
> Omnibond Systems
> Anderson, South Carolina
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: http://www.beowulf-underground.org/pipermail/pvfs2-developers/attachments/20120220/06b36737/attachment.htm
>
> ------------------------------
>
> _______________________________________________
> Pvfs2-developers mailing list
> Pvfs2-developers at beowulf-underground.org
> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
>
>
> End of Pvfs2-developers Digest, Vol 72, Issue 8
> ***********************************************
More information about the Pvfs2-developers
mailing list