[PVFS-users] Re: Quick Question
Jeffrey B Layton
Thu, 26 Oct 2000 14:42:32 -0400
Robert Ross wrote:
> Good question! This should be in the FAQ.
> First, let's talk about the number of open FDs under linux. It is
> actually a dynamically settable value, but it defaults to 1024 under 2.2.
> If you look at the output of "limit" (under tcsh) or "ulimit -a" (under
> bash), you'll see the 1024 value (or whatever is set on your machine).
Actually, there is a variable called FOPEN_MAX (it's part the
standard C library) that is the minimum number of foiles that the
implementation guarantees can be open simultaneously. That's
kind of what I've been going by (since it's a minimum).
> In the PVFS code we use the value from /usr/include/linux/fs.h, which is
> 1048576. There is some really poor allocation going on WRT this value
> also, I now notice :(. Something that should be fixed.
> So in theory you could have TONS of open files. However, you can't
> really, because we use up FD when we open connections to I/O servers all
> over the place. So really the practical limit of open files for a single
> client under PVFS is N/(I+2), where "I" is the number of I/O nodes in your
> system and "N" is the number of open files for Linux. The constant value
> comes from the connection to the manager (1 more FD) and an FD used as a
Is this for PVFS in general or for each code that is linked
with the pvfs library?
> So, for example in a system with 8 I/O nodes and default resource limits,
> I'd expect you could open as many as 100 PVFS files simultaneously. With
> 48 I/O nodes, you're down to 10 PVFS files.
For large clusters (or at least large number of I/O nodes) this
could be a problem. If you have a fair number of users and a
fair number of tasks, you could run out of open files.
> Obviously there is room for improvement in our resource utilization :).
Is there a way to set a #define or a simple function somewhere
(pvfs.h?) that either defines or computes the maximum number
of open files? This would be a nice little feature for what I'm
doing (but doing at work by the way :) ).
> (I'm CC'ing this to the pvfs-users list for posterity.)
> On Thu, 26 Oct 2000, Jeffrey B Layton wrote:
> > Is there a limit to the number of open files an application can
> > have when using the PVFS library (pvfs_open() )? I know under Linux
> > the usual number is 256.