From batayl at gmail.com Fri May 2 17:00:33 2008 From: batayl at gmail.com (Bart Taylor) Date: Fri May 2 17:01:02 2008 Subject: [Pvfs2-developers] RHEL4 and 2.7.1 mount problems Message-ID: <4ac846300805021400v5b75d258q1272b49819bbe931@mail.gmail.com> Skipped content of type multipart/alternative-------------- next part -------------- A non-text attachment was scrubbed... Name: test-mount-limit.pl Type: application/octet-stream Size: 7712 bytes Desc: not available Url : http://www.beowulf-underground.org/pipermail/pvfs2-developers/attachments/20080502/5d57356a/test-mount-limit.obj From slang at mcs.anl.gov Fri May 2 17:15:04 2008 From: slang at mcs.anl.gov (Sam Lang) Date: Fri May 2 17:15:12 2008 Subject: [Pvfs2-developers] RHEL4 and 2.7.1 mount problems In-Reply-To: <4ac846300805021400v5b75d258q1272b49819bbe931@mail.gmail.com> References: <4ac846300805021400v5b75d258q1272b49819bbe931@mail.gmail.com> Message-ID: <4CD0BEAC-39D6-4BA9-8411-9A88486287D2@mcs.anl.gov> Hi Bart, After loading the pvfs2 kmod, can you do: echo "1" > /proc/sys/pvfs2/debug Then run the same test, and send the dmesg output to me? This should show where the inode allocs/deallocs are going awry. Thanks, -sam On May 2, 2008, at 4:00 PM, Bart Taylor wrote: > Hey guys, > > I have been running some tests against the 271 release, and I am > having some trouble with multiple mounts on one client. My setup > has 2 servers (both meta and io servers on local disk) and one > client all of which are running RHEL4 update 6. All that was done on > the test client is loading the kernel module and starting pvfs2- > client. I can mount the file system once and use it without any > problem, but I have attached a test script - takes file system > information and a number of times to mount it - that keeps failing. > Here are the steps it executes: > > - For the number of mounts requested > - Create a new directory (defaults to /tmp/mount_limit.#) > - Mount the specified file system on the new dir > > - For the number of mounts requested > - Do a recursive ls comparison (keep a copy the first time > through and compare subsequent mounts to the first) > - Unmount the dir > - Delete the dir > > I have been able to consistently reproduce the problem running the > attached script like this: > ./test-mount-limit.pl pvfs2-server1:3334/pvfs2-fs 100 > It stalls every time with either 36 or 37 mounts remaining. The > script has been successfully run on previous versions of pvfs2 up to > several thousand mounts. > > The problem comes at the umount step. Eventually the process just > hangs, strands a bunch of mounts, and umount doesn't work as > expected after that even from the command line. When it stalls, I > start seeing messages like this one in dmesg and syslog: > May 2 15:02:44 client-node kernel: pvfs2_kill_sb: (WARNING) number > of inode allocs (4100) != number of inode deallocs (2665) > > I am running this against an almost empty file system since the > recursive ls would take a while if it were large. Am I doing > something wrong/strange here, or is there a client/kernel problem? > The test seems pretty straight-forward, and I've never had an issue > with the script before. I'm not sure if it was run against the 2.7.0 > release though. > > Bart. > _______________________________________________ > Pvfs2-developers mailing list > Pvfs2-developers@beowulf-underground.org > http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2417 bytes Desc: not available Url : http://www.beowulf-underground.org/pipermail/pvfs2-developers/attachments/20080502/bd6c4781/smime.bin From slang at mcs.anl.gov Fri May 2 17:18:17 2008 From: slang at mcs.anl.gov (Sam Lang) Date: Fri May 2 17:18:24 2008 Subject: [Pvfs2-developers] RHEL4 and 2.7.1 mount problems In-Reply-To: <4ac846300805021400v5b75d258q1272b49819bbe931@mail.gmail.com> References: <4ac846300805021400v5b75d258q1272b49819bbe931@mail.gmail.com> Message-ID: <45947A4C-4A5B-411A-9CA9-5CE8A823BD07@mcs.anl.gov> And can you send your generated pvfs2-config.h file? Thanks, -sam On May 2, 2008, at 4:00 PM, Bart Taylor wrote: > Hey guys, > > I have been running some tests against the 271 release, and I am > having some trouble with multiple mounts on one client. My setup > has 2 servers (both meta and io servers on local disk) and one > client all of which are running RHEL4 update 6. All that was done on > the test client is loading the kernel module and starting pvfs2- > client. I can mount the file system once and use it without any > problem, but I have attached a test script - takes file system > information and a number of times to mount it - that keeps failing. > Here are the steps it executes: > > - For the number of mounts requested > - Create a new directory (defaults to /tmp/mount_limit.#) > - Mount the specified file system on the new dir > > - For the number of mounts requested > - Do a recursive ls comparison (keep a copy the first time > through and compare subsequent mounts to the first) > - Unmount the dir > - Delete the dir > > I have been able to consistently reproduce the problem running the > attached script like this: > ./test-mount-limit.pl pvfs2-server1:3334/pvfs2-fs 100 > It stalls every time with either 36 or 37 mounts remaining. The > script has been successfully run on previous versions of pvfs2 up to > several thousand mounts. > > The problem comes at the umount step. Eventually the process just > hangs, strands a bunch of mounts, and umount doesn't work as > expected after that even from the command line. When it stalls, I > start seeing messages like this one in dmesg and syslog: > May 2 15:02:44 client-node kernel: pvfs2_kill_sb: (WARNING) number > of inode allocs (4100) != number of inode deallocs (2665) > > I am running this against an almost empty file system since the > recursive ls would take a while if it were large. Am I doing > something wrong/strange here, or is there a client/kernel problem? > The test seems pretty straight-forward, and I've never had an issue > with the script before. I'm not sure if it was run against the 2.7.0 > release though. > > Bart. > _______________________________________________ > Pvfs2-developers mailing list > Pvfs2-developers@beowulf-underground.org > http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2417 bytes Desc: not available Url : http://www.beowulf-underground.org/pipermail/pvfs2-developers/attachments/20080502/f97aaa3e/smime-0001.bin From carns at mcs.anl.gov Fri May 2 17:22:24 2008 From: carns at mcs.anl.gov (Phil Carns) Date: Fri May 2 17:22:38 2008 Subject: [Pvfs2-developers] RHEL4 and 2.7.1 mount problems In-Reply-To: <4CD0BEAC-39D6-4BA9-8411-9A88486287D2@mcs.anl.gov> References: <4ac846300805021400v5b75d258q1272b49819bbe931@mail.gmail.com> <4CD0BEAC-39D6-4BA9-8411-9A88486287D2@mcs.anl.gov> Message-ID: <481B8610.9050909@mcs.anl.gov> The dmesg output probably will help, but I think that the inode alloc/dealloc check itself is a little trigger happy: https://trac.mcs.anl.gov/projects/pvfs/ticket/7 I didn't think it would actually break anything other than printing out an unecessary warning, but I never followed up on it. There may actually be something unrelated going wrong. -Phil Sam Lang wrote: > > Hi Bart, > > After loading the pvfs2 kmod, can you do: > > echo "1" > /proc/sys/pvfs2/debug > > Then run the same test, and send the dmesg output to me? This should > show where the inode allocs/deallocs are going awry. > > Thanks, > -sam > > On May 2, 2008, at 4:00 PM, Bart Taylor wrote: > >> Hey guys, >> >> I have been running some tests against the 271 release, and I am >> having some trouble with multiple mounts on one client. My setup has >> 2 servers (both meta and io servers on local disk) and one client all >> of which are running RHEL4 update 6. All that was done on the test >> client is loading the kernel module and starting pvfs2-client. I can >> mount the file system once and use it without any problem, but I have >> attached a test script - takes file system information and a number of >> times to mount it - that keeps failing. Here are the steps it executes: >> >> - For the number of mounts requested >> - Create a new directory (defaults to /tmp/mount_limit.#) >> - Mount the specified file system on the new dir >> >> - For the number of mounts requested >> - Do a recursive ls comparison (keep a copy the first time through >> and compare subsequent mounts to the first) >> - Unmount the dir >> - Delete the dir >> >> I have been able to consistently reproduce the problem running the >> attached script like this: >> ./test-mount-limit.pl pvfs2-server1:3334/pvfs2-fs 100 >> It stalls every time with either 36 or 37 mounts remaining. The >> script has been successfully run on previous versions of pvfs2 up to >> several thousand mounts. >> >> The problem comes at the umount step. Eventually the process just >> hangs, strands a bunch of mounts, and umount doesn't work as expected >> after that even from the command line. When it stalls, I start seeing >> messages like this one in dmesg and syslog: >> May 2 15:02:44 client-node kernel: pvfs2_kill_sb: (WARNING) number of >> inode allocs (4100) != number of inode deallocs (2665) >> >> I am running this against an almost empty file system since the >> recursive ls would take a while if it were large. Am I doing something >> wrong/strange here, or is there a client/kernel problem? The test >> seems pretty straight-forward, and I've never had an issue with the >> script before. I'm not sure if it was run against the 2.7.0 release >> though. >> >> Bart. >> _______________________________________________ >> Pvfs2-developers mailing list >> Pvfs2-developers@beowulf-underground.org >> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers > > > ------------------------------------------------------------------------ > > _______________________________________________ > Pvfs2-developers mailing list > Pvfs2-developers@beowulf-underground.org > http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers From batayl at gmail.com Fri May 2 18:09:30 2008 From: batayl at gmail.com (Bart Taylor) Date: Fri May 2 18:09:43 2008 Subject: [Pvfs2-developers] RHEL4 and 2.7.1 mount problems In-Reply-To: <481B8610.9050909@mcs.anl.gov> References: <4ac846300805021400v5b75d258q1272b49819bbe931@mail.gmail.com> <4CD0BEAC-39D6-4BA9-8411-9A88486287D2@mcs.anl.gov> <481B8610.9050909@mcs.anl.gov> Message-ID: <4ac846300805021509u6bcca292xfc6d3227d0c48706@mail.gmail.com> Skipped content of type multipart/alternative-------------- next part -------------- A non-text attachment was scrubbed... Name: pvfs2-config.h Type: application/octet-stream Size: 13331 bytes Desc: not available Url : http://www.beowulf-underground.org/pipermail/pvfs2-developers/attachments/20080502/4803c714/pvfs2-config-0001.obj From carns at mcs.anl.gov Tue May 13 09:55:53 2008 From: carns at mcs.anl.gov (Phil Carns) Date: Tue May 13 09:56:06 2008 Subject: [Pvfs2-developers] RHEL4 and 2.7.1 mount problems In-Reply-To: <4ac846300805021400v5b75d258q1272b49819bbe931@mail.gmail.com> References: <4ac846300805021400v5b75d258q1272b49819bbe931@mail.gmail.com> Message-ID: <48299DE9.3040502@mcs.anl.gov> Hi Bart, Could you try this patch out and see if it fixes your problem? This is checked into trunk as well. This won't eliminate the inode alloc warning, but I think it does actually fix the umount hang. I also suspect that this same issue may affect a few other cases as well, but it would be good if you could confirm this much for starters. I think the same class of bug is affecting some of the proc file handlers, for example. Cases like this also cause a pvfs2-client-core hang: "for i in `seq 1 100`; do echo $i; cat /proc/sys/pvfs2/perf-counters/acache; done" thanks! -Phil Bart Taylor wrote: > Hey guys, > > I have been running some tests against the 271 release, and I am having > some trouble with multiple mounts on one client. My setup has 2 servers > (both meta and io servers on local disk) and one client all of which are > running RHEL4 update 6. All that was done on the test client is loading > the kernel module and starting pvfs2-client. I can mount the file > system once and use it without any problem, but I have attached a test > script - takes file system information and a number of times to mount it > - that keeps failing. Here are the steps it executes: > > - For the number of mounts requested > - Create a new directory (defaults to /tmp/mount_limit.#) > - Mount the specified file system on the new dir > > - For the number of mounts requested > - Do a recursive ls comparison (keep a copy the first time through > and compare subsequent mounts to the first) > - Unmount the dir > - Delete the dir > > I have been able to consistently reproduce the problem running the > attached script like this: > ./test-mount-limit.pl pvfs2-server1:3334/pvfs2-fs 100 > It stalls every time with either 36 or 37 mounts remaining. The script > has been successfully run on previous versions of pvfs2 up to several > thousand mounts. > > The problem comes at the umount step. Eventually the process just > hangs, strands a bunch of mounts, and umount doesn't work as expected > after that even from the command line. When it stalls, I start seeing > messages like this one in dmesg and syslog: > May 2 15:02:44 client-node kernel: pvfs2_kill_sb: (WARNING) number of > inode allocs (4100) != number of inode deallocs (2665) > > I am running this against an almost empty file system since the > recursive ls would take a while if it were large. Am I doing something > wrong/strange here, or is there a client/kernel problem? The test seems > pretty straight-forward, and I've never had an issue with the script > before. I'm not sure if it was run against the 2.7.0 release though. > > Bart. > > > ------------------------------------------------------------------------ > > _______________________________________________ > Pvfs2-developers mailing list > Pvfs2-developers@beowulf-underground.org > http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers -------------- next part -------------- A non-text attachment was scrubbed... Name: 64-umount-hang.patch Type: text/x-patch Size: 1863 bytes Desc: not available Url : http://www.beowulf-underground.org/pipermail/pvfs2-developers/attachments/20080513/8972c9b6/64-umount-hang.bin From batayl at gmail.com Tue May 13 11:52:15 2008 From: batayl at gmail.com (Bart Taylor) Date: Tue May 13 11:52:28 2008 Subject: [Pvfs2-developers] RHEL4 and 2.7.1 mount problems In-Reply-To: <48299DE9.3040502@mcs.anl.gov> References: <4ac846300805021400v5b75d258q1272b49819bbe931@mail.gmail.com> <48299DE9.3040502@mcs.anl.gov> Message-ID: <4ac846300805130852r49fafb26p95aa798447c5ab47@mail.gmail.com> Thanks Phil, that patch works great! I successfully ran the previous script on a few thousand mounts and did not have any more problems. It looks like you nailed down where the problem lies even if it isn't fixed in every case. I ran the 'for' loop you mentioned that bounces off the acache perf-counter just to see what would happen, and it still causes a hang as I think you expected. Bart. On Tue, May 13, 2008 at 7:55 AM, Phil Carns wrote: > Hi Bart, > > Could you try this patch out and see if it fixes your problem? This is > checked into trunk as well. This won't eliminate the inode alloc warning, > but I think it does actually fix the umount hang. > > I also suspect that this same issue may affect a few other cases as well, > but it would be good if you could confirm this much for starters. > > I think the same class of bug is affecting some of the proc file handlers, > for example. Cases like this also cause a pvfs2-client-core hang: > > "for i in `seq 1 100`; do echo $i; cat > /proc/sys/pvfs2/perf-counters/acache; done" > > thanks! > -Phil > > Bart Taylor wrote: > > > Hey guys, > > > > I have been running some tests against the 271 release, and I am having > > some trouble with multiple mounts on one client. My setup has 2 servers > > (both meta and io servers on local disk) and one client all of which are > > running RHEL4 update 6. All that was done on the test client is loading the > > kernel module and starting pvfs2-client. I can mount the file system once > > and use it without any problem, but I have attached a test script - takes > > file system information and a number of times to mount it - that keeps > > failing. Here are the steps it executes: > > > > - For the number of mounts requested > > - Create a new directory (defaults to /tmp/mount_limit.#) > > - Mount the specified file system on the new dir > > > > - For the number of mounts requested > > - Do a recursive ls comparison (keep a copy the first time through and > > compare subsequent mounts to the first) > > - Unmount the dir > > - Delete the dir > > > > I have been able to consistently reproduce the problem running the > > attached script like this: > > ./test-mount-limit.pl pvfs2-server1:3334/pvfs2-fs 100 > > It stalls every time with either 36 or 37 mounts remaining. The script > > has been successfully run on previous versions of pvfs2 up to several > > thousand mounts. > > > > The problem comes at the umount step. Eventually the process just > > hangs, strands a bunch of mounts, and umount doesn't work as expected after > > that even from the command line. When it stalls, I start seeing messages > > like this one in dmesg and syslog: > > May 2 15:02:44 client-node kernel: pvfs2_kill_sb: (WARNING) number of > > inode allocs (4100) != number of inode deallocs (2665) > > > > I am running this against an almost empty file system since the > > recursive ls would take a while if it were large. Am I doing something > > wrong/strange here, or is there a client/kernel problem? The test seems > > pretty straight-forward, and I've never had an issue with the script before. > > I'm not sure if it was run against the 2.7.0 release though. > > > > Bart. > > > > > > ------------------------------------------------------------------------ > > > > _______________________________________________ > > Pvfs2-developers mailing list > > Pvfs2-developers@beowulf-underground.org > > http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.beowulf-underground.org/pipermail/pvfs2-developers/attachments/20080513/e7c41767/attachment.htm From carns at mcs.anl.gov Tue May 13 14:35:21 2008 From: carns at mcs.anl.gov (Phil Carns) Date: Tue May 13 14:35:37 2008 Subject: [Pvfs2-developers] RHEL4 and 2.7.1 mount problems In-Reply-To: <4ac846300805130852r49fafb26p95aa798447c5ab47@mail.gmail.com> References: <4ac846300805021400v5b75d258q1272b49819bbe931@mail.gmail.com> <48299DE9.3040502@mcs.anl.gov> <4ac846300805130852r49fafb26p95aa798447c5ab47@mail.gmail.com> Message-ID: <4829DF69.80704@mcs.anl.gov> Here is an updated version that catches the other stray cases like perf-counters/acache. It also corrects that warning message about inode allocations. -Phil Bart Taylor wrote: > Thanks Phil, that patch works great! I successfully ran the previous > script on a few thousand mounts and did not have any more problems. > > It looks like you nailed down where the problem lies even if it isn't > fixed in every case. I ran the 'for' loop you mentioned that bounces off > the acache perf-counter just to see what would happen, and it still > causes a hang as I think you expected. > > Bart. > > > > On Tue, May 13, 2008 at 7:55 AM, Phil Carns > wrote: > > Hi Bart, > > Could you try this patch out and see if it fixes your problem? This > is checked into trunk as well. This won't eliminate the inode alloc > warning, but I think it does actually fix the umount hang. > > I also suspect that this same issue may affect a few other cases as > well, but it would be good if you could confirm this much for starters. > > I think the same class of bug is affecting some of the proc file > handlers, for example. Cases like this also cause a > pvfs2-client-core hang: > > "for i in `seq 1 100`; do echo $i; cat > /proc/sys/pvfs2/perf-counters/acache; done" > > thanks! > -Phil > > Bart Taylor wrote: > > Hey guys, > > I have been running some tests against the 271 release, and I am > having some trouble with multiple mounts on one client. My > setup has 2 servers (both meta and io servers on local disk) and > one client all of which are running RHEL4 update 6. All that was > done on the test client is loading the kernel module and > starting pvfs2-client. I can mount the file system once and use > it without any problem, but I have attached a test script - > takes file system information and a number of times to mount it > - that keeps failing. Here are the steps it executes: > > - For the number of mounts requested > - Create a new directory (defaults to /tmp/mount_limit.#) > - Mount the specified file system on the new dir > > - For the number of mounts requested > - Do a recursive ls comparison (keep a copy the first time > through and compare subsequent mounts to the first) > - Unmount the dir > - Delete the dir > > I have been able to consistently reproduce the problem running > the attached script like this: > ./test-mount-limit.pl pvfs2-server1:3334/pvfs2-fs 100 > It stalls every time with either 36 or 37 mounts remaining. The > script has been successfully run on previous versions of pvfs2 > up to several thousand mounts. > > The problem comes at the umount step. Eventually the process > just hangs, strands a bunch of mounts, and umount doesn't work > as expected after that even from the command line. When it > stalls, I start seeing messages like this one in dmesg and syslog: > May 2 15:02:44 client-node kernel: pvfs2_kill_sb: (WARNING) > number of inode allocs (4100) != number of inode deallocs (2665) > > I am running this against an almost empty file system since the > recursive ls would take a while if it were large. Am I doing > something wrong/strange here, or is there a client/kernel > problem? The test seems pretty straight-forward, and I've never > had an issue with the script before. I'm not sure if it was run > against the 2.7.0 release though. > > Bart. > > > ------------------------------------------------------------------------ > > > _______________________________________________ > Pvfs2-developers mailing list > Pvfs2-developers@beowulf-underground.org > > http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers > > > -------------- next part -------------- A non-text attachment was scrubbed... Name: vfs-repost-and-inode-count.patch Type: text/x-patch Size: 19241 bytes Desc: not available Url : http://www.beowulf-underground.org/pipermail/pvfs2-developers/attachments/20080513/3c54c9b0/vfs-repost-and-inode-count.bin From walt at CLEMSON.EDU Tue May 13 15:30:16 2008 From: walt at CLEMSON.EDU (Walter Ligon) Date: Tue May 13 15:31:00 2008 Subject: [Pvfs2-developers] host id Message-ID: <4829EC48.1020004@clemson.edu> Hey guys, we're needing to send a server host identification on the wire. In the config the host_id appears to be a variable length string. Should we just use that, or is there something else of fixed size I'm not aware of? Walt From carns at mcs.anl.gov Tue May 13 16:10:36 2008 From: carns at mcs.anl.gov (Phil Carns) Date: Tue May 13 16:10:51 2008 Subject: [Pvfs2-developers] host id In-Reply-To: <4829EC48.1020004@clemson.edu> References: <4829EC48.1020004@clemson.edu> Message-ID: <4829F5BC.7030001@mcs.anl.gov> I don't think we have any globally valid fixed sized identifier for the servers. The only option I know if would be to use a handle (for example, the first handle in the server's mapping) to represent each server. I think just using the string name would be safer, though. -Phil Walter Ligon wrote: > Hey guys, > > we're needing to send a server host identification on the wire. In the > config the host_id appears to be a variable length string. Should we > just use that, or is there something else of fixed size I'm not aware of? > > Walt > _______________________________________________ > Pvfs2-developers mailing list > Pvfs2-developers@beowulf-underground.org > http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers From slang at mcs.anl.gov Tue May 13 16:17:29 2008 From: slang at mcs.anl.gov (Sam Lang) Date: Tue May 13 16:17:36 2008 Subject: [Pvfs2-developers] host id In-Reply-To: <4829F5BC.7030001@mcs.anl.gov> References: <4829EC48.1020004@clemson.edu> <4829F5BC.7030001@mcs.anl.gov> Message-ID: <8338DD2B-1E5F-4D23-A318-84C25DB5A9BE@mcs.anl.gov> On May 13, 2008, at 3:10 PM, Phil Carns wrote: > I don't think we have any globally valid fixed sized identifier for > the servers. The only option I know if would be to use a handle > (for example, the first handle in the server's mapping) to represent > each server. I think just using the string name would be safer, > though. Index into the list of aliases in the config file works too, as long as you don't store it anywhere. I have some functions that map index to bmi addr if you want them. -sam > > > -Phil > > Walter Ligon wrote: >> Hey guys, >> we're needing to send a server host identification on the wire. In >> the config the host_id appears to be a variable length string. >> Should we just use that, or is there something else of fixed size >> I'm not aware of? >> Walt >> _______________________________________________ >> Pvfs2-developers mailing list >> Pvfs2-developers@beowulf-underground.org >> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers > > _______________________________________________ > Pvfs2-developers mailing list > Pvfs2-developers@beowulf-underground.org > http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2417 bytes Desc: not available Url : http://www.beowulf-underground.org/pipermail/pvfs2-developers/attachments/20080513/99cd2c3e/smime.bin From nlmills at g.clemson.edu Thu May 15 14:10:32 2008 From: nlmills at g.clemson.edu (Nicholas Mills) Date: Thu May 15 14:10:36 2008 Subject: [Pvfs2-developers] host id In-Reply-To: <4829F5BC.7030001@mcs.anl.gov> References: <4829EC48.1020004@clemson.edu> <4829F5BC.7030001@mcs.anl.gov> Message-ID: <9682db3b0805151110t4ea7ef0aq6bf41b1977135f3d@mail.gmail.com> Thanks for your help everyone. David and I have decided to use the first handle in the server's mapping to identify the server. We are at the point now where we need to find a handle range given a host alias. Does anyone know of a quick and easy way to do this? There is also the question of which handle range to use. Do we use the meta handles or the data handles? Also, will having multiple filesystems complicate anything? --Nick On Tue, May 13, 2008 at 4:10 PM, Phil Carns wrote: > I don't think we have any globally valid fixed sized identifier for the > servers. The only option I know if would be to use a handle (for example, > the first handle in the server's mapping) to represent each server. I think > just using the string name would be safer, though. > > -Phil > > > Walter Ligon wrote: > >> Hey guys, >> >> we're needing to send a server host identification on the wire. In the >> config the host_id appears to be a variable length string. Should we just >> use that, or is there something else of fixed size I'm not aware of? >> >> Walt >> _______________________________________________ >> Pvfs2-developers mailing list >> Pvfs2-developers@beowulf-underground.org >> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers >> > > _______________________________________________ > Pvfs2-developers mailing list > Pvfs2-developers@beowulf-underground.org > http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.beowulf-underground.org/pipermail/pvfs2-developers/attachments/20080515/9fbf904b/attachment.htm From rross at mcs.anl.gov Thu May 15 14:23:17 2008 From: rross at mcs.anl.gov (Rob Ross) Date: Thu May 15 14:23:25 2008 Subject: [Pvfs2-developers] host id In-Reply-To: <9682db3b0805151110t4ea7ef0aq6bf41b1977135f3d@mail.gmail.com> References: <4829EC48.1020004@clemson.edu> <4829F5BC.7030001@mcs.anl.gov> <9682db3b0805151110t4ea7ef0aq6bf41b1977135f3d@mail.gmail.com> Message-ID: Yes, having multiple file systems would really mess with you because you could have the same handle for more than one FS. You sure you don't want to just use a string name? Rob On May 15, 2008, at 1:10 PM, Nicholas Mills wrote: > Thanks for your help everyone. > > David and I have decided to use the first handle in the server's > mapping to identify the server. We are at the point now where we > need to find a handle range given a host alias. Does anyone know of > a quick and easy way to do this? > > There is also the question of which handle range to use. Do we use > the meta handles or the data handles? Also, will having multiple > filesystems complicate anything? > > --Nick > > On Tue, May 13, 2008 at 4:10 PM, Phil Carns wrote: > I don't think we have any globally valid fixed sized identifier for > the servers. The only option I know if would be to use a handle > (for example, the first handle in the server's mapping) to represent > each server. I think just using the string name would be safer, > though. > > -Phil > > > Walter Ligon wrote: > Hey guys, > > we're needing to send a server host identification on the wire. In > the config the host_id appears to be a variable length string. > Should we just use that, or is there something else of fixed size > I'm not aware of? > > Walt > _______________________________________________ > Pvfs2-developers mailing list > Pvfs2-developers@beowulf-underground.org > http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers > > _______________________________________________ > Pvfs2-developers mailing list > Pvfs2-developers@beowulf-underground.org > http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers > > _______________________________________________ > Pvfs2-developers mailing list > Pvfs2-developers@beowulf-underground.org > http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers From nlmills at g.clemson.edu Thu May 15 14:41:02 2008 From: nlmills at g.clemson.edu (Nicholas Mills) Date: Thu May 15 14:41:06 2008 Subject: [Pvfs2-developers] host id In-Reply-To: References: <4829EC48.1020004@clemson.edu> <4829F5BC.7030001@mcs.anl.gov> <9682db3b0805151110t4ea7ef0aq6bf41b1977135f3d@mail.gmail.com> Message-ID: <9682db3b0805151141sc2ac9y6353ec10cdeef9ab@mail.gmail.com> I would like to use the string but Dr. Ligon wanted to avoid using a string. I assume it's because they have a variable length. We're going to need to send this across the network. If we were going to use a string would it be better to allocate it dynamically or use a fixed-size buffer (something like 1024 bytes)? --Nick On Thu, May 15, 2008 at 2:23 PM, Rob Ross wrote: > Yes, having multiple file systems would really mess with you because you > could have the same handle for more than one FS. You sure you don't want to > just use a string name? > > Rob > > > On May 15, 2008, at 1:10 PM, Nicholas Mills wrote: > >> Thanks for your help everyone. >> >> David and I have decided to use the first handle in the server's mapping >> to identify the server. We are at the point now where we need to find a >> handle range given a host alias. Does anyone know of a quick and easy way to >> do this? >> >> There is also the question of which handle range to use. Do we use the >> meta handles or the data handles? Also, will having multiple filesystems >> complicate anything? >> >> --Nick >> >> On Tue, May 13, 2008 at 4:10 PM, Phil Carns wrote: >> I don't think we have any globally valid fixed sized identifier for the >> servers. The only option I know if would be to use a handle (for example, >> the first handle in the server's mapping) to represent each server. I think >> just using the string name would be safer, though. >> >> -Phil >> >> >> Walter Ligon wrote: >> Hey guys, >> >> we're needing to send a server host identification on the wire. In the >> config the host_id appears to be a variable length string. Should we just >> use that, or is there something else of fixed size I'm not aware of? >> >> Walt >> _______________________________________________ >> Pvfs2-developers mailing list >> Pvfs2-developers@beowulf-underground.org >> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers >> >> _______________________________________________ >> Pvfs2-developers mailing list >> Pvfs2-developers@beowulf-underground.org >> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers >> >> _______________________________________________ >> Pvfs2-developers mailing list >> Pvfs2-developers@beowulf-underground.org >> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.beowulf-underground.org/pipermail/pvfs2-developers/attachments/20080515/6b6e3379/attachment.htm From slang at mcs.anl.gov Thu May 15 14:48:59 2008 From: slang at mcs.anl.gov (Sam Lang) Date: Thu May 15 14:49:06 2008 Subject: [Pvfs2-developers] host id In-Reply-To: References: <4829EC48.1020004@clemson.edu> <4829F5BC.7030001@mcs.anl.gov> <9682db3b0805151110t4ea7ef0aq6bf41b1977135f3d@mail.gmail.com> Message-ID: <12693F20-0FA5-416D-BCCE-42B81DA4FD53@mcs.anl.gov> Walt, I think for your purposes you should probably just have each server/fs generate a uuid. Keep in mind that everything (including the endpoint string and handle ranges) can change in the config file without recreating the storage spaces. This often happens when admins want to migrate to other servers. It might help to think about the storage space (and each fs within them) as the thing you want to reference with a permanent globally unique-id. Our servers just manage those storage spaces, and the endpoint strings just provide an address (which can change over time). If you were to use a uuid for each server/fs, you probably want random instead of time/host-based, so that you can move servers around without making the id invalid. If you wanted them in the config file, they could be part of the Alias mapping, and genconfig could just generate them with uuidgen or something. We've talked about having something like this for zero-conf, although that was more transient on a per-server start basis, to provide a consistent view if the config changed -- so a different purpose. Alternatively, if that's all too over the top, and you just want an integer instead of a string, the index into the server list is no worse than the endpoint string. -sam On May 15, 2008, at 1:23 PM, Rob Ross wrote: > Yes, having multiple file systems would really mess with you because > you could have the same handle for more than one FS. You sure you > don't want to just use a string name? > > Rob > > On May 15, 2008, at 1:10 PM, Nicholas Mills wrote: >> Thanks for your help everyone. >> >> David and I have decided to use the first handle in the server's >> mapping to identify the server. We are at the point now where we >> need to find a handle range given a host alias. Does anyone know of >> a quick and easy way to do this? >> >> There is also the question of which handle range to use. Do we use >> the meta handles or the data handles? Also, will having multiple >> filesystems complicate anything? >> >> --Nick >> >> On Tue, May 13, 2008 at 4:10 PM, Phil Carns >> wrote: >> I don't think we have any globally valid fixed sized identifier for >> the servers. The only option I know if would be to use a handle >> (for example, the first handle in the server's mapping) to >> represent each server. I think just using the string name would be >> safer, though. >> >> -Phil >> >> >> Walter Ligon wrote: >> Hey guys, >> >> we're needing to send a server host identification on the wire. In >> the config the host_id appears to be a variable length string. >> Should we just use that, or is there something else of fixed size >> I'm not aware of? >> >> Walt >> _______________________________________________ >> Pvfs2-developers mailing list >> Pvfs2-developers@beowulf-underground.org >> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers >> >> _______________________________________________ >> Pvfs2-developers mailing list >> Pvfs2-developers@beowulf-underground.org >> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers >> >> _______________________________________________ >> Pvfs2-developers mailing list >> Pvfs2-developers@beowulf-underground.org >> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers > > _______________________________________________ > Pvfs2-developers mailing list > Pvfs2-developers@beowulf-underground.org > http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2417 bytes Desc: not available Url : http://www.beowulf-underground.org/pipermail/pvfs2-developers/attachments/20080515/ab5faf17/smime.bin From walt at clemson.edu Thu May 15 18:39:41 2008 From: walt at clemson.edu (Walter Ligon) Date: Thu May 15 18:39:58 2008 Subject: [Pvfs2-developers] host id In-Reply-To: <12693F20-0FA5-416D-BCCE-42B81DA4FD53@mcs.anl.gov> References: <4829EC48.1020004@clemson.edu> <4829F5BC.7030001@mcs.anl.gov> <9682db3b0805151110t4ea7ef0aq6bf41b1977135f3d@mail.gmail.com> <12693F20-0FA5-416D-BCCE-42B81DA4FD53@mcs.anl.gov> Message-ID: <482CBBAD.9040700@clemson.edu> Where I think I'm going it to store the Handle of the "owner" of each object as an attribute in the object. For dfiles, this would be the handle of the metafile, for metafiles this would be the handle of the directory (works for now since we don't do hard links). Then We'll send reference the owner handle when we create the capability. This would be translated into a server name, and we tie the key pairs to the servers. Thus, if we change which server handles a data space, the owner object handle will translate to the new server, and it will use that server's keys. The allocating and management of keys will be tied to the servers. this should even allow migration of objects between data spaces. If data is migrated to new servers, probably the new servers will each get new keys. We plan to have a mechanism to flush and rebuild key databases for this eventuality (basically we can request public keys from a server with a PVFS request - they keys are sent in a cert signed by CA). Finally, having this information added might be uf use to the fsck program, as it would allow stray dfiles to be mapped back to the owning metafile, etc. The only problem would come if we actually change the handle of a file, which would require updating all of the dfiles - but we don't do that very often. Thoughts? Walt Sam Lang wrote: > > Walt, > > I think for your purposes you should probably just have each server/fs > generate a uuid. Keep in mind that everything (including the endpoint > string and handle ranges) can change in the config file without > recreating the storage spaces. This often happens when admins want to > migrate to other servers. > > It might help to think about the storage space (and each fs within them) > as the thing you want to reference with a permanent globally unique-id. > Our servers just manage those storage spaces, and the endpoint strings > just provide an address (which can change over time). > > If you were to use a uuid for each server/fs, you probably want random > instead of time/host-based, so that you can move servers around without > making the id invalid. If you wanted them in the config file, they > could be part of the Alias mapping, and genconfig could just generate > them with uuidgen or something. We've talked about having something > like this for zero-conf, although that was more transient on a > per-server start basis, to provide a consistent view if the config > changed -- so a different purpose. > > Alternatively, if that's all too over the top, and you just want an > integer instead of a string, the index into the server list is no worse > than the endpoint string. > > -sam > > > On May 15, 2008, at 1:23 PM, Rob Ross wrote: > >> Yes, having multiple file systems would really mess with you because >> you could have the same handle for more than one FS. You sure you >> don't want to just use a string name? >> >> Rob >> >> On May 15, 2008, at 1:10 PM, Nicholas Mills wrote: >>> Thanks for your help everyone. >>> >>> David and I have decided to use the first handle in the server's >>> mapping to identify the server. We are at the point now where we need >>> to find a handle range given a host alias. Does anyone know of a >>> quick and easy way to do this? >>> >>> There is also the question of which handle range to use. Do we use >>> the meta handles or the data handles? Also, will having multiple >>> filesystems complicate anything? >>> >>> --Nick >>> >>> On Tue, May 13, 2008 at 4:10 PM, Phil Carns wrote: >>> I don't think we have any globally valid fixed sized identifier for >>> the servers. The only option I know if would be to use a handle (for >>> example, the first handle in the server's mapping) to represent each >>> server. I think just using the string name would be safer, though. >>> >>> -Phil >>> >>> >>> Walter Ligon wrote: >>> Hey guys, >>> >>> we're needing to send a server host identification on the wire. In >>> the config the host_id appears to be a variable length string. >>> Should we just use that, or is there something else of fixed size I'm >>> not aware of? >>> >>> Walt >>> _______________________________________________ >>> Pvfs2-developers mailing list >>> Pvfs2-developers@beowulf-underground.org >>> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers >>> >>> _______________________________________________ >>> Pvfs2-developers mailing list >>> Pvfs2-developers@beowulf-underground.org >>> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers >>> >>> _______________________________________________ >>> Pvfs2-developers mailing list >>> Pvfs2-developers@beowulf-underground.org >>> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers >> >> _______________________________________________ >> Pvfs2-developers mailing list >> Pvfs2-developers@beowulf-underground.org >> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers > From walt at clemson.edu Thu May 15 18:40:51 2008 From: walt at clemson.edu (Walter Ligon) Date: Thu May 15 18:41:28 2008 Subject: [Pvfs2-developers] host id In-Reply-To: References: <4829EC48.1020004@clemson.edu> <4829F5BC.7030001@mcs.anl.gov> <9682db3b0805151110t4ea7ef0aq6bf41b1977135f3d@mail.gmail.com> Message-ID: <482CBBF3.1020100@clemson.edu> Shit, forgot about this - we need to include the fsid in the signed structures. that fixes this problem! Walt Rob Ross wrote: > Yes, having multiple file systems would really mess with you because you > could have the same handle for more than one FS. You sure you don't want > to just use a string name? > > Rob > > On May 15, 2008, at 1:10 PM, Nicholas Mills wrote: >> Thanks for your help everyone. >> >> David and I have decided to use the first handle in the server's >> mapping to identify the server. We are at the point now where we need >> to find a handle range given a host alias. Does anyone know of a quick >> and easy way to do this? >> >> There is also the question of which handle range to use. Do we use the >> meta handles or the data handles? Also, will having multiple >> filesystems complicate anything? >> >> --Nick >> >> On Tue, May 13, 2008 at 4:10 PM, Phil Carns wrote: >> I don't think we have any globally valid fixed sized identifier for >> the servers. The only option I know if would be to use a handle (for >> example, the first handle in the server's mapping) to represent each >> server. I think just using the string name would be safer, though. >> >> -Phil >> >> >> Walter Ligon wrote: >> Hey guys, >> >> we're needing to send a server host identification on the wire. In >> the config the host_id appears to be a variable length string. Should >> we just use that, or is there something else of fixed size I'm not >> aware of? >> >> Walt >> _______________________________________________ >> Pvfs2-developers mailing list >> Pvfs2-developers@beowulf-underground.org >> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers >> >> _______________________________________________ >> Pvfs2-developers mailing list >> Pvfs2-developers@beowulf-underground.org >> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers >> >> _______________________________________________ >> Pvfs2-developers mailing list >> Pvfs2-developers@beowulf-underground.org >> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers > > _______________________________________________ > Pvfs2-developers mailing list > Pvfs2-developers@beowulf-underground.org > http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers