[PVFS-users] Re: Quick Question

Jeffrey B Layton jeffrey.b.layton@lmco.com
Thu, 26 Oct 2000 14:42:32 -0400

Robert Ross wrote:

> Jeff,
> Good question!  This should be in the FAQ.
> First, let's talk about the number of open FDs under linux.  It is
> actually a dynamically settable value, but it defaults to 1024 under 2.2.
> If you look at the output of "limit" (under tcsh) or "ulimit -a" (under
> bash), you'll see the 1024 value (or whatever is set on your machine).

Actually, there is a variable called FOPEN_MAX (it's part the
standard C library) that is the minimum number of foiles that the
implementation guarantees can be open simultaneously.  That's
kind of what I've been going by (since it's a minimum).

> In the PVFS code we use the value from /usr/include/linux/fs.h, which is
> 1048576.  There is some really poor allocation going on WRT this value
> also, I now notice :(.  Something that should be fixed.
> So in theory you could have TONS of open files.  However, you can't
> really, because we use up FD when we open connections to I/O servers all
> over the place.  So really the practical limit of open files for a single
> client under PVFS is N/(I+2), where "I" is the number of I/O nodes in your
> system and "N" is the number of open files for Linux.  The constant value
> comes from the connection to the manager (1 more FD) and an FD used as a
> placeholder.

Is this for PVFS in general or for each code that is linked
with the pvfs library?

> So, for example in a system with 8 I/O nodes and default resource limits,
> I'd expect you could open as many as 100 PVFS files simultaneously.  With
> 48 I/O nodes, you're down to 10 PVFS files.

For large clusters (or at least large number of I/O nodes) this
could be a problem. If you have a fair number of users and a
fair number of tasks, you could run out of open files.

> Obviously there is room for improvement in our resource utilization :).

Is there a way to set a #define or a simple function somewhere
(pvfs.h?) that either defines or computes the maximum number
of open files? This would be a nice little feature for what I'm
doing (but doing at work by the way :)  ).



> Rob
> (I'm CC'ing this to the pvfs-users list for posterity.)
> On Thu, 26 Oct 2000, Jeffrey B Layton wrote:
> >    Is there a limit to the number of open files an application can
> > have when using the PVFS library (pvfs_open()  )? I know under Linux
> > the usual number is 256.