[PVFS-users] Re: Quick Question

[Robert Ross]

On Thu, 26 Oct 2000, Jeffrey B Layton wrote:

> Actually, there is a variable called FOPEN_MAX (it's part the
> standard C library) that is the minimum number of files that the
> implementation guarantees can be open simultaneously.  That's
> kind of what I've been going by (since it's a minimum).

Gotcha.  That seems reasonable.

> > So in theory you could have TONS of open files.  However, you can't
> > really, because we use up FD when we open connections to I/O servers all
> > over the place.  So really the practical limit of open files for a single
> > client under PVFS is N/(I+2), where "I" is the number of I/O nodes in your
> > system and "N" is the number of open files for Linux.  The constant value
> > comes from the connection to the manager (1 more FD) and an FD used as a
> > placeholder.
> 
> Is this for PVFS in general or for each code that is linked
> with the pvfs library?

It is a per-process limit.  There are other limits you might hit first if
you use lots of processes with not too many open files.

> > So, for example in a system with 8 I/O nodes and default resource limits,
> > I'd expect you could open as many as 100 PVFS files simultaneously.  With
> > 48 I/O nodes, you're down to 10 PVFS files.
> 
> For large clusters (or at least large number of I/O nodes) this
> could be a problem. If you have a fair number of users and a
> fair number of tasks, you could run out of open files.

It could be.  That is why it is important to know how to up the FD limit.
We can do this.  That should be another FAQ entry I suppose.  If someone
needs to know this, I'll post a summary.

> > Obviously there is room for improvement in our resource utilization :).
> 
> Is there a way to set a #define or a simple function somewhere
> (pvfs.h?) that either defines or computes the maximum number
> of open files? This would be a nice little feature for what I'm
> doing (but doing at work by the way :)  ).

It's not that easy unfortunately, because there's no way to know how many
I/O servers someone used for files they might open or create.  We could
come up with an upper bound though probably...

So what are you doing :)?

Rob