[Pvfs2-developers] Re: Pvfs2-developers Digest, Vol 72, Issue 8

Dimokritos Stamatakis dimstamat at gmail.com
Mon Feb 20 12:27:10 EST 2012


Hello and thanks for your answer!
The problem was the amazon ec2-disassociate command, where the public IP became unreachable. I now invoke just the ec2-associate and everything is fine!!

Thanks to all for trying to help,
Dimos.

On 20 Φεβ 2012, at 5:42 μ.μ., pvfs2-developers-request at beowulf-underground.org wrote:

> Send Pvfs2-developers mailing list submissions to
>    pvfs2-developers at beowulf-underground.org
> 
> To subscribe or unsubscribe via the World Wide Web, visit
>    http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
> or, via email, send a message with subject or body 'help' to
>    pvfs2-developers-request at beowulf-underground.org
> 
> You can reach the person managing the list at
>    pvfs2-developers-owner at beowulf-underground.org
> 
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Pvfs2-developers digest..."
> 
> 
> Today's Topics:
> 
>   1. no space left on device when running on amazon    ec2
>      (Dimos Stamatakis)
>   2. Re: no space left on device when running on    amazon ec2
>      (Dimos Stamatakis)
>   3. Re: no space left on device when running on    amazon ec2
>      (Dimos Stamatakis)
>   4. Re: Re: no space left on device when running on    amazon ec2
>      (Becky Ligon)
> 
> 
> ----------------------------------------------------------------------
> 
> Message: 1
> Date: Mon, 20 Feb 2012 02:17:01 +0200
> From: Dimos Stamatakis <dimstamat at gmail.com>
> Subject: [Pvfs2-developers] no space left on device when running on
>    amazon    ec2
> To: pvfs2-developers at beowulf-underground.org
> Message-ID:
>    <CAB8CW0ET2U3OtOXTgZMXBPRiBmvR4=0FpyM8gzDg8s0A4KSf5w at mail.gmail.com>
> Content-Type: text/plain; charset="iso-8859-1"
> 
> Hello!
> I have successfully run a pvfs installation on a eucalyptus cloud, but when
> I moved to Amazon EC2, I get a very strange error.
> When I run a metadata server it says:
> 
> [S 02/20 00:04] PVFS2 Server ready.
> 
> and then it says:
> 
> [E 02/20 00:04] batch_create request got: No space left on device
> ....... And this error repeats ......
> 
> I checked all of my devices and there is plenty of space, so I don't think
> there is not enough space left...
> Can you explain that?
> What is the batch_create function? And where is it trying to write?
> 
> Here is the output of the df -h on the data node:
> 
> Filesystem            Size  Used Avail Use% Mounted on
> /dev/sda1             9.9G  2.7G  6.8G  29% /
> tmpfs                 308M     0  308M   0% /lib/init/rw
> udev                   10M  108K  9.9M   2% /dev
> tmpfs                 308M  4.0K  308M   1% /dev/shm
> 
> and on the meta data node:
> 
> Filesystem            Size  Used Avail Use% Mounted on
> /dev/sda1             9.9G  2.0G  7.5G  21% /
> tmpfs                 308M     0  308M   0% /lib/init/rw
> udev                   10M  108K  9.9M   2% /dev
> tmpfs                 308M  4.0K  308M   1% /dev/shm
> 
> Many thanks,
> Dimos.
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: http://www.beowulf-underground.org/pipermail/pvfs2-developers/attachments/20120220/e5087dbc/attachment-0001.htm
> 
> ------------------------------
> 
> Message: 2
> Date: Mon, 20 Feb 2012 03:49:50 +0200
> From: Dimos Stamatakis <dimstamat at gmail.com>
> Subject: [Pvfs2-developers] Re: no space left on device when running
>    on    amazon ec2
> To: pvfs2-developers at beowulf-underground.org
> Message-ID:
>    <CAB8CW0EvXb6012tqNWpAsg_bug4+ZuLniKv8cTY0tX1fnic=gQ at mail.gmail.com>
> Content-Type: text/plain; charset="iso-8859-1"
> 
> I forgot to tell you that I use ec2-associate-address commands to tell the
> new master to grab the elastic IP address. If I do not use replication and
> I use the normal IP addresses it works fine!
> Is there a way to have High availability to amazon EC2 without use of
> elastic IPs??
> 
> Many thanks,
> Dimos.
> 
> 
> On Mon, Feb 20, 2012 at 2:17 AM, Dimos Stamatakis <dimstamat at gmail.com>wrote:
> 
>> Hello!
>> I have successfully run a pvfs installation on a eucalyptus cloud, but
>> when I moved to Amazon EC2, I get a very strange error.
>> When I run a metadata server it says:
>> 
>> [S 02/20 00:04] PVFS2 Server ready.
>> 
>> and then it says:
>> 
>> [E 02/20 00:04] batch_create request got: No space left on device
>> ....... And this error repeats ......
>> 
>> I checked all of my devices and there is plenty of space, so I don't think
>> there is not enough space left...
>> Can you explain that?
>> What is the batch_create function? And where is it trying to write?
>> 
>> Here is the output of the df -h on the data node:
>> 
>> Filesystem            Size  Used Avail Use% Mounted on
>> /dev/sda1             9.9G  2.7G  6.8G  29% /
>> tmpfs                 308M     0  308M   0% /lib/init/rw
>> udev                   10M  108K  9.9M   2% /dev
>> tmpfs                 308M  4.0K  308M   1% /dev/shm
>> 
>> and on the meta data node:
>> 
>> Filesystem            Size  Used Avail Use% Mounted on
>> /dev/sda1             9.9G  2.0G  7.5G  21% /
>> tmpfs                 308M     0  308M   0% /lib/init/rw
>> udev                   10M  108K  9.9M   2% /dev
>> tmpfs                 308M  4.0K  308M   1% /dev/shm
>> 
>> Many thanks,
>> Dimos.
>> 
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: http://www.beowulf-underground.org/pipermail/pvfs2-developers/attachments/20120220/ffdfe9c8/attachment-0001.htm
> 
> ------------------------------
> 
> Message: 3
> Date: Mon, 20 Feb 2012 05:48:30 +0200
> From: Dimos Stamatakis <dimstamat at gmail.com>
> Subject: [Pvfs2-developers] Re: no space left on device when running
>    on    amazon ec2
> To: pvfs2-developers at beowulf-underground.org
> Message-ID:
>    <CAB8CW0Er_JzyH9uO7z0wQekXHk3_yrNn+kd700SrKkSR8qV_Qg at mail.gmail.com>
> Content-Type: text/plain; charset="iso-8859-1"
> 
> Hello again!
> I want to help you realize what is going wrong by telling you that the
> client blocks at a pvfs2-ls (it does not say connection refused
> immediatelly). It can also ping the new elastic IP normally!
> 
> I redirected the metadata server output to a file and when I checked it i
> didn't find anything wrong... It did all the gets - puts that are happening
> everytime the DB is created. Here is the output tail:
> 
> get (handle: 4611686018427387903)()(key_sz:8) -> (511)(4)
> put (handle: 4611686018427387903)()(key_sz:8) -> (512)(4)
> [1329709072:419164][4413/140213703595776] TROVE:DBPF:Berkeley DB: bulk_msg:
> Send buffer after copy due to PERM
> [1329709072:419173][4413/140213703595776] TROVE:DBPF:Berkeley DB:
> send_bulk: Send 160 (0xa0) bulk buffer bytes
> [1329709072:419183][4413/140213703595776] TROVE:DBPF:Berkeley DB:
> //pvfs2-storage-space/27c41225/ rep_send_message: msgv = 7 logv 19 gen = 1
> eid -1, type bulk_log, LSN [1][217660]  perm
> [1329709072:419193][4413/140213703595776] TROVE:DBPF:Berkeley DB:
> rep_send_function returned: -30975
> 
> How can I find out why this metadata server refuses serving the client
> requests?
> 
> Thanks again,
> Dimos.
> 
> 
> 
> On Mon, Feb 20, 2012 at 3:49 AM, Dimos Stamatakis <dimstamat at gmail.com>wrote:
> 
>> I forgot to tell you that I use ec2-associate-address commands to tell the
>> new master to grab the elastic IP address. If I do not use replication and
>> I use the normal IP addresses it works fine!
>> Is there a way to have High availability to amazon EC2 without use of
>> elastic IPs??
>> 
>> Many thanks,
>> Dimos.
>> 
>> 
>> 
>> On Mon, Feb 20, 2012 at 2:17 AM, Dimos Stamatakis <dimstamat at gmail.com>wrote:
>> 
>>> Hello!
>>> I have successfully run a pvfs installation on a eucalyptus cloud, but
>>> when I moved to Amazon EC2, I get a very strange error.
>>> When I run a metadata server it says:
>>> 
>>> [S 02/20 00:04] PVFS2 Server ready.
>>> 
>>> and then it says:
>>> 
>>> [E 02/20 00:04] batch_create request got: No space left on device
>>> ....... And this error repeats ......
>>> 
>>> I checked all of my devices and there is plenty of space, so I don't
>>> think there is not enough space left...
>>> Can you explain that?
>>> What is the batch_create function? And where is it trying to write?
>>> 
>>> Here is the output of the df -h on the data node:
>>> 
>>> Filesystem            Size  Used Avail Use% Mounted on
>>> /dev/sda1             9.9G  2.7G  6.8G  29% /
>>> tmpfs                 308M     0  308M   0% /lib/init/rw
>>> udev                   10M  108K  9.9M   2% /dev
>>> tmpfs                 308M  4.0K  308M   1% /dev/shm
>>> 
>>> and on the meta data node:
>>> 
>>> Filesystem            Size  Used Avail Use% Mounted on
>>> /dev/sda1             9.9G  2.0G  7.5G  21% /
>>> tmpfs                 308M     0  308M   0% /lib/init/rw
>>> udev                   10M  108K  9.9M   2% /dev
>>> tmpfs                 308M  4.0K  308M   1% /dev/shm
>>> 
>>> Many thanks,
>>> Dimos.
>>> 
>> 
>> 
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: http://www.beowulf-underground.org/pipermail/pvfs2-developers/attachments/20120220/8f30dcd1/attachment-0001.htm
> 
> ------------------------------
> 
> Message: 4
> Date: Mon, 20 Feb 2012 10:42:21 -0500
> From: Becky Ligon <ligon at omnibond.com>
> Subject: Re: [Pvfs2-developers] Re: no space left on device when
>    running on    amazon ec2
> To: Dimos Stamatakis <dimstamat at gmail.com>
> Cc: pvfs2-developers at beowulf-underground.org
> Message-ID:
>    <CAJBnF338CSi4q80XPCcRY7kPm+U7bzwkPmvFhS7J1UxUO2CLwQ at mail.gmail.com>
> Content-Type: text/plain; charset="iso-8859-1"
> 
> When a server is started, it sends a batch_create request to every other
> server in the filesystem.  The batch_create request asks the receiving
> server to send back a list of unused data handles (owned by that particular
> server).  For those handles in the list, the receiving server sets an
> attribute in the local database to indicate that the handle is in use.  You
> may see a batch_create request after your servers have been running for a
> while, since a server will request another batch of handles if its current
> store gets low.  This entire process is a performance enhancement, which
> allows a file's data handles to be assigned by the metadata server without
> contacting the data handle servers, thus reducing the time it takes to
> create a file.
> 
> With all of that said, it seems that one of your servers is having trouble
> accessing the database or communicating with another server.  I'm not
> exactly sure without further research.  Think about the above description
> and see if you can't pinpoint which server is causing the trouble.
> 
> Becky
> 
> On Sun, Feb 19, 2012 at 10:48 PM, Dimos Stamatakis <dimstamat at gmail.com>wrote:
> 
>> Hello again!
>> I want to help you realize what is going wrong by telling you that the
>> client blocks at a pvfs2-ls (it does not say connection refused
>> immediatelly). It can also ping the new elastic IP normally!
>> 
>> I redirected the metadata server output to a file and when I checked it i
>> didn't find anything wrong... It did all the gets - puts that are happening
>> everytime the DB is created. Here is the output tail:
>> 
>> get (handle: 4611686018427387903)()(key_sz:8) -> (511)(4)
>> put (handle: 4611686018427387903)()(key_sz:8) -> (512)(4)
>> [1329709072:419164][4413/140213703595776] TROVE:DBPF:Berkeley DB:
>> bulk_msg: Send buffer after copy due to PERM
>> [1329709072:419173][4413/140213703595776] TROVE:DBPF:Berkeley DB:
>> send_bulk: Send 160 (0xa0) bulk buffer bytes
>> [1329709072:419183][4413/140213703595776] TROVE:DBPF:Berkeley DB:
>> //pvfs2-storage-space/27c41225/ rep_send_message: msgv = 7 logv 19 gen = 1
>> eid -1, type bulk_log, LSN [1][217660]  perm
>> [1329709072:419193][4413/140213703595776] TROVE:DBPF:Berkeley DB:
>> rep_send_function returned: -30975
>> 
>> How can I find out why this metadata server refuses serving the client
>> requests?
>> 
>> Thanks again,
>> Dimos.
>> 
>> 
>> 
>> On Mon, Feb 20, 2012 at 3:49 AM, Dimos Stamatakis <dimstamat at gmail.com>wrote:
>> 
>>> I forgot to tell you that I use ec2-associate-address commands to tell
>>> the new master to grab the elastic IP address. If I do not use replication
>>> and I use the normal IP addresses it works fine!
>>> Is there a way to have High availability to amazon EC2 without use of
>>> elastic IPs??
>>> 
>>> Many thanks,
>>> Dimos.
>>> 
>>> 
>>> 
>>> On Mon, Feb 20, 2012 at 2:17 AM, Dimos Stamatakis <dimstamat at gmail.com>wrote:
>>> 
>>>> Hello!
>>>> I have successfully run a pvfs installation on a eucalyptus cloud, but
>>>> when I moved to Amazon EC2, I get a very strange error.
>>>> When I run a metadata server it says:
>>>> 
>>>> [S 02/20 00:04] PVFS2 Server ready.
>>>> 
>>>> and then it says:
>>>> 
>>>> [E 02/20 00:04] batch_create request got: No space left on device
>>>> ....... And this error repeats ......
>>>> 
>>>> I checked all of my devices and there is plenty of space, so I don't
>>>> think there is not enough space left...
>>>> Can you explain that?
>>>> What is the batch_create function? And where is it trying to write?
>>>> 
>>>> Here is the output of the df -h on the data node:
>>>> 
>>>> Filesystem            Size  Used Avail Use% Mounted on
>>>> /dev/sda1             9.9G  2.7G  6.8G  29% /
>>>> tmpfs                 308M     0  308M   0% /lib/init/rw
>>>> udev                   10M  108K  9.9M   2% /dev
>>>> tmpfs                 308M  4.0K  308M   1% /dev/shm
>>>> 
>>>> and on the meta data node:
>>>> 
>>>> Filesystem            Size  Used Avail Use% Mounted on
>>>> /dev/sda1             9.9G  2.0G  7.5G  21% /
>>>> tmpfs                 308M     0  308M   0% /lib/init/rw
>>>> udev                   10M  108K  9.9M   2% /dev
>>>> tmpfs                 308M  4.0K  308M   1% /dev/shm
>>>> 
>>>> Many thanks,
>>>> Dimos.
>>>> 
>>> 
>>> 
>> 
>> _______________________________________________
>> Pvfs2-developers mailing list
>> Pvfs2-developers at beowulf-underground.org
>> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
>> 
>> 
> 
> 
> -- 
> Becky Ligon
> OrangeFS Support and Development
> Omnibond Systems
> Anderson, South Carolina
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: http://www.beowulf-underground.org/pipermail/pvfs2-developers/attachments/20120220/06b36737/attachment.htm
> 
> ------------------------------
> 
> _______________________________________________
> Pvfs2-developers mailing list
> Pvfs2-developers at beowulf-underground.org
> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
> 
> 
> End of Pvfs2-developers Digest, Vol 72, Issue 8
> ***********************************************



More information about the Pvfs2-developers mailing list