[Pvfs2-developers] Re: RFC: Config file overhaul

[Sam Lang]

On Sep 12, 2006, at 5:07 PM, Murali Vilayannur wrote:

> Hey guys,
> Sam and I discussed the option of overhauling the configuration  
> file and
> server setup options today and here is a summary of what we had  
> come up
> with.. (Sam, please correct me if I missed out anything or  
> misunderstood
> something)
>
> Well-known problems with the current setup are
> a) the need for 2 config files (a global one and a per-server  
> config file)
> and the implicit reliance on a savvy admin to keep these consistent  
> and
> synchronized.
> b) Not having a clean way to be able to do a SIGHUP or equivalent  
> to get
> restart functionality on servers.
>
> We were thinking of going about solving this in a 3 step process.
> 1) Eliminate the need for a per-server config file and pass the
> server host id as a command line parameter. The fs.conf file will  
> have a
> new Defaults tag for the storage space in case it is the same on all
> servers (which is usually the case) like the log file pathname.
> Since the fs.conf aliases section maps aliases to host ids, servers  
> can
> use that to obtain the aliases and thence the metahandle, datahandle
> ranges.
> If each server has a different storage space path, we could have a  
> new tag
> <StorageSpace>
> localhost1 /tmp/path1
> localhost2 /tmp/path2
> ...
> </StorageSpace>
> and so on..
> NOTE: If people feel that passing an address to the servers is in some
> sense similar to having the good old config file and does not  
> really solve
> the config problems, we could probably
> just listen on the 0.0.0.0 address and any ephemeral port, and  
> register
> the ephemeral port with the portmapper. More on this issue a little  
> later.
> 2) Now, we eliminate the need for having a synchronized fs.conf  
> file by
> designating a single metadata server (similar to what Julian had  
> written
> earlier today) as the sole authority of the config files for a  
> particular
> file system and having it push the config files to all the servers.
>
> When this metadata server starts up, it parses through all the  
> hostid's
> with the exception of itself and then connects to all the remaining
> servers associated with that fsid and does a putconfig request (a new
> request type).
> Question is, how does this server talk to the remaining servers to  
> push
> the config requests? We could just simply use tcp for this  
> communication
> and have the putconfig act like an implicit SIGHUP which will cause  
> all
> servers to restart and listen on the appropriate interfaces  
> subsequent to
> the putconfig.
> As regards the port numbers to do the putconfig, we could query the
> portmapper or dedicate a range of port numbers on each host for  
> running
> the remaining servers initially.
>
> So what we have is as follows
>
> <Root Server>                 <Other servers>
>
>                             start by providing either the actual  
> bmi url
>                             hostid or any ephemeral port number as  
> cmdline
> 			    parameter.
>
> 			    setup the sm's etc and wait for a putconfig()
> 			    All other requests will be Nack'ed
>
> parse the fs.conf
> files and issue a putconfig()
> to all the remaining servers.
> And until all the remaining
> servers acknowledge, continue
> doing this.
>
> Start waiting for regular
> reqs to show up
>
> It is now enough for
> SIGHUP  to be sent to
> the RootServer. All other
> servers will ignore
> SIGHUP. The root server
> will issue a putconfig()
> to all servers.
>
>                             as soon as we get a putconfig, act like  
> it was
>                             a SIGHUP, parse the config files and  
> restart
> 			    implicitly.
> 			    Now start servicing new requests..
>
> 			    Any subsequent putconfig() requests will again
>                             act like a SIGHUP (wait for current  
> reqs to be
> 			    drained, new reqs NACked) and restart.
>
>
> NOTE: each server maintains a global in-memory session identifier  
> that is
> propagated by the putconfig() from the root-server and this identifier
> is part of every client servreq structure. If it matches with the  
> server
> id, the requests are processed, else the clients are forced to do a
> getconfig().
>
> I still dont think we have addressed all issues here.. For instance  
> server
> (non root) restarts should cause putconfig's implicitly or  
> getconfig's?
> Should the session id be on db on the root servers (and non root  
> servers)
> or is it sufficient to be in memory?
>
> I felt this topic requires some discussion and brain storming before
> prototyping and hence this longish email.
>

To continue the discussion here that I had offline with Murali, what  
about having servers send an initial hello message to the master  
server, so that they don't have to figure out what port to listen  
on?  The master can send that server's HostID, wait for a ready  
response, and then push the config file to that endpoint.

-sam

> Comments and suggestions welcome!
> thanks,
> Murali
>