[PVFS-users] pvfs 1.6.3 causing possible kernel crash
Shawn Needham
shawn at flash.uchicago.edu
Tue Dec 21 13:46:31 EST 2004
I've started the runs without the modules loaded. I'll let you know how
they go.
Also, do you have a suggestion for a size for the static buffer for the
pvfs module. I'll start perusing the mail list for this information as
well.
Thanks,
Shawn
On Tue, 21 Dec 2004, Rob Ross wrote:
> Thanks Shawn. I think that maxsz is ok. I agree that your users will
> want to have the kernel mounted. I was hoping you could run some
> experiments to see if that in fact eliminated the problem, so that we
> would know for a fact that we weren't barking up the wrong tree.
>
> Can you do that?
>
> Thanks!
>
> Rob
>
> On Tue, 21 Dec 2004, Shawn Needham wrote:
>
> > Hi Rob,
> >
> > On Tue, 21 Dec 2004, Rob Ross wrote:
> >
> > > Hi Shawn,
> > >
> > > Can you send us the configure output from the pvfs-kernel build? I'd like
> > > to know what options it picked.
> >
> > I've attached the log from the pvfs-1.6.3-kernel-2.4 build. If you need
> > more than this let me know.
> > >
> > > Also, when you load the module, are you specifying a specific "buffer="
> > > argument? If you could grab the line out of dmesg that is printed at
> > > module load time, that would be helpful.
> >
> > Rob, I didn't set any special buffer argument. I'm just relying on the
> > default setting for that argument which is to handle it dynamically. I'm
> > guessing that setting this statically could greatly change it's
> > stability/performance. Could let me know a good size to set this
> > buffer to for our configuration.
> >
> > Dec 21 11:38:33 ellipse2 kernel: pvfs: debug = 0x0, maxsz = 16777216
> > bytes, buffer = dynamic, major = 0
> >
> > >
> > > If this is a PVFS problem, it is most likely in the kernel code; otherwise
> > > you shouldn't get hangs.
> > >
> > > I don't understand why you're getting much traffic at all across the
> > > kernel interface with the FLASH code; I think that it is likely that
> > > something is misconfigured there in the I/O stack. Can you try prepending
> > > a "pvfs:" to your file names and see if that helps? Alternatively, you
> > > could create an /etc/pvfstab describing the mount point and not mount the
> > > file system at all, eliminating the pvfs-kernel piece from the equation
> > > entirely.
> >
> > I have created the /etc/pvfstab entry for the mount point. So applications
> > should be able to interface with the filesystem without being mounted.
> > However, without having the kernel loaded and the client running is there
> > a way to access those directories with standard Linux commands (ls, cp,
> > etc). For testing not mounting these directories would be fine, but I'm
> > not sure it would offer an ideal situation down the road where user's
> > would want access to there data while running. I guess the latter point is
> > just something to keep in mind for now.
> >
> > Thanks,
> > Shawn
> >
> > >
> > > Rob
> > >
> > >
> > > On Mon, 20 Dec 2004, Shawn Needham wrote:
> > >
> > > > Greetings,
> > > >
> > > > In the past week I have been attempting to run our FLASH application on
> > > > our cluster using pvfs-1.6.3.
> > > >
> > > > We have been experiencing crashes on one of our compute nodes that
> > > > appears to be caused by software with direct access to the kernel. The
> > > > nodes are freezing, you can no longer ping the interface for the Gb
> > > > switch, the terminal is blacked out and unresponsive to keyboard input,
> > > > and no diagnostic information is written to the log files. The only 2
> > > > modules loaded on our compute nodes are gm (for a Myrinet's MPICH-gm)
> > > > and pvfs, which we are running as a service over the Gb switch.
> > > >
> > > > We have seen the crash occur with the exact same symptoms when trying to
> > > > run our application on a purely Gb based configuration (mpich/hdf5 p4
> > > > configuration and pvfs running over Gb based switch) and with the mixed
> > > > configuration described above. Actually the pure Gb configuration has
> > > > performed much worse and the crashes have always happened much quicker
> > > > in terms the amount of progress in our runs. Some of the Myrinet runs
> > > > have run to completion (though this hasn't been consistent when trying
> > > > the exact same run), but I never had any success with the Gb based runs
> > > > during which neither Myrinet did not have any services running and it's
> > > > module was not loaded.
> > > >
> > > > I was wondering if you have heard of pvfs causing problems similar to
> > > > these. Also if one of the developers would like to have an account on
> > > > the machine to investigate this further that would be tremendous. I have
> > > > been working with developers at Myrinet for the past week and their
> > > > ability to log on has proved beneficial. I'm open to trying pvfs2 if the
> > > > kernel module is more robust than pvfs-1.6.3 but would like to at least
> > > > get your insights into the current problem we are having with this
> > > > installation.
> > > >
> > > > I'm not sure if this relevant, but on two of our earliest runs some
> > > > corrupted file were produced right before our runs crashed. These files
> > > > were written to directories in the pvfs file system.
> > > >
> > > > I just reinstalled a new kernel this afternoon and rebuilt all the
> > > > software, scrapped all old pvfs directories and rebuilt them and my
> > > > Myrinet based run ended up dieing in the same manner as the others.
> > > >
> > > > I've also appending some information about the hardware/software
> > > > configuration of our machine.
> > > >
> > > > Thanks,
> > > > Shawn
> > > >
> > > > Here's the software/hardware information.
> > > >
> > > > Hardware. 17 node (1 master/16 compute) cluster from Aspen Systems with
> > > > the Myrinet M3-E32 switch and a Gb switch (24 port HP Procurve 2724,
> > > > J4897a).
> > > > Nodes: e7505 chipset, dual 2.8Ghz xeon processors 4GB memory. PCIXE-2
> > > > dual channel myrinet cards, cards are in 133Mhz PCI slots, Intel Corp.
> > > > 82545EM Gigabit Ethernet Controller. Nvidia 5950 Geforce FX Ultra.
> > > >
> > > > We are running a stock 2.4.27 kernel from kernel.org. The current
> > > > running kernel is a 2.4.27smp-64GB-noAGPsupport. We initially installed
> > > > RH9 on this system and tried to take as much of the RedHat features out
> > > > as possible, but system parameters are still consistent with RH values.
> > > > Here's /proc/version info.
> > > >
> > > > [root at ellipse0 ~]# cat /proc/version
> > > > Linux version 2.4.27smp-64gb-noagp (root at ellipse0) (gcc version 3.2.2
> > > > 20030222 (Red Hat Linux 3.2.2-5)) #1 SMP Mon Dec 20 12:14:14 CST 2004
> > > >
> > > > Myrinet configuration
> > > >
> > > > gm-2.1.6 (linux-2.4 - ia32)
> > > > pvfs-1.6.3 (running over Gb switch)
> > > > mpich-1.2.6..13b-gm * ifc 8.1 (Version 8.1 (l_fc_p_8.1.018))
> > > > * icc 8.1 (Version 8.1 (l_cc_p_8.1.021))
> > > > hdf5-1.6.2 (built against this mpich)
> > > >
> > > > Gb configuration (all services running over Gb based switch):
> > > >
> > > > pvfs-1.6.3
> > > > mpich-1.2.6-p4
> > > > ifc 8.1 (Version 8.1 Build 20040803Z Package ID:
> > > > l_fc_p_8.1.018)
> > > > gcc 3.2.2 (gcc (GCC) 3.2.2 20030222 (Red Hat Linux 3.2.2-5))
> > > > hdf5-1.6.2-p4 (built against mpich-1.2.6-p4)
> > > >
> > > > I have configured both MPICH libraries with-romio filesupport for the
> > > > following file systems (pvfs,ufs,nfs)
> > > >
> > > >
> > > > _______________________________________________
> > > > PVFS-users mailing list
> > > > PVFS-users at www.beowulf-underground.org
> > > > http://www.beowulf-underground.org/mailman/listinfo/pvfs-users
> > > >
> > > >
> > >
> >
> >
> >
>
More information about the PVFS-users
mailing list