[PVFS-users] pvfs 1.6.3 causing possible kernel crash
Rob Ross
rross at mcs.anl.gov
Tue Dec 21 16:21:47 EST 2004
Did you prefix the filename with "pvfs:"?
Rob
On Tue, 21 Dec 2004, Shawn Needham wrote:
> Hi Rob,
>
> I have it set up so all of our compute node are I/O servers. 16MB or
> more on the clients should not be a problem.
>
> Also, I wrote a bit too soon as I had been doing runs without any I/O when
> writing that last email. When trying to run with I/O through the pvfs
> application interface I'm running into an error when it attempts to open
> the first file for writing.
>
> Attached you will find a run2.log file that captures the output from this
> run.
>
> I built pvfs support into the Myrinet's mpich implementation, but since
> this is not working, I'm not sure if the native pvfs interface is
> configured and available for Myrinet's MPI-IO.
>
> If you would like to set configuration for the Myrinet MPICH installation
> let me know and I can provide those details.
>
> Thank,
> Shawn
> On Tue, 21 Dec 2004, Rob Ross wrote:
>
> > Hi Shawn,
> >
> > The size of the static buffer will define an upper bound on the size of an
> > I/O operation. So it sort of depends on your number of I/O servers and
> > how much free space you have :). If you can spare 16MB of space on your
> > clients, try setting it to that.
> >
> > Rob
> >
> > On Tue, 21 Dec 2004, Shawn Needham wrote:
> >
> > > I've started the runs without the modules loaded. I'll let you know how
> > > they go.
> > >
> > > Also, do you have a suggestion for a size for the static buffer for the
> > > pvfs module. I'll start perusing the mail list for this information as
> > > well.
> > >
> > > Thanks,
> > > Shawn
> > >
> > > On Tue, 21 Dec 2004, Rob Ross wrote:
> > >
> > > > Thanks Shawn. I think that maxsz is ok. I agree that your users will
> > > > want to have the kernel mounted. I was hoping you could run some
> > > > experiments to see if that in fact eliminated the problem, so that we
> > > > would know for a fact that we weren't barking up the wrong tree.
> > > >
> > > > Can you do that?
> > > >
> > > > Thanks!
> > > >
> > > > Rob
> > > >
> > > > On Tue, 21 Dec 2004, Shawn Needham wrote:
> > > >
> > > > > Hi Rob,
> > > > >
> > > > > On Tue, 21 Dec 2004, Rob Ross wrote:
> > > > >
> > > > > > Hi Shawn,
> > > > > >
> > > > > > Can you send us the configure output from the pvfs-kernel build? I'd like
> > > > > > to know what options it picked.
> > > > >
> > > > > I've attached the log from the pvfs-1.6.3-kernel-2.4 build. If you need
> > > > > more than this let me know.
> > > > > >
> > > > > > Also, when you load the module, are you specifying a specific "buffer="
> > > > > > argument? If you could grab the line out of dmesg that is printed at
> > > > > > module load time, that would be helpful.
> > > > >
> > > > > Rob, I didn't set any special buffer argument. I'm just relying on the
> > > > > default setting for that argument which is to handle it dynamically. I'm
> > > > > guessing that setting this statically could greatly change it's
> > > > > stability/performance. Could let me know a good size to set this
> > > > > buffer to for our configuration.
> > > > >
> > > > > Dec 21 11:38:33 ellipse2 kernel: pvfs: debug = 0x0, maxsz = 16777216
> > > > > bytes, buffer = dynamic, major = 0
> > > > >
> > > > > >
> > > > > > If this is a PVFS problem, it is most likely in the kernel code; otherwise
> > > > > > you shouldn't get hangs.
> > > > > >
> > > > > > I don't understand why you're getting much traffic at all across the
> > > > > > kernel interface with the FLASH code; I think that it is likely that
> > > > > > something is misconfigured there in the I/O stack. Can you try prepending
> > > > > > a "pvfs:" to your file names and see if that helps? Alternatively, you
> > > > > > could create an /etc/pvfstab describing the mount point and not mount the
> > > > > > file system at all, eliminating the pvfs-kernel piece from the equation
> > > > > > entirely.
> > > > >
> > > > > I have created the /etc/pvfstab entry for the mount point. So applications
> > > > > should be able to interface with the filesystem without being mounted.
> > > > > However, without having the kernel loaded and the client running is there
> > > > > a way to access those directories with standard Linux commands (ls, cp,
> > > > > etc). For testing not mounting these directories would be fine, but I'm
> > > > > not sure it would offer an ideal situation down the road where user's
> > > > > would want access to there data while running. I guess the latter point is
> > > > > just something to keep in mind for now.
> > > > >
> > > > > Thanks,
> > > > > Shawn
> > > > >
> > > > > >
> > > > > > Rob
> > > > > >
> > > > > >
> > > > > > On Mon, 20 Dec 2004, Shawn Needham wrote:
> > > > > >
> > > > > > > Greetings,
> > > > > > >
> > > > > > > In the past week I have been attempting to run our FLASH application on
> > > > > > > our cluster using pvfs-1.6.3.
> > > > > > >
> > > > > > > We have been experiencing crashes on one of our compute nodes that
> > > > > > > appears to be caused by software with direct access to the kernel. The
> > > > > > > nodes are freezing, you can no longer ping the interface for the Gb
> > > > > > > switch, the terminal is blacked out and unresponsive to keyboard input,
> > > > > > > and no diagnostic information is written to the log files. The only 2
> > > > > > > modules loaded on our compute nodes are gm (for a Myrinet's MPICH-gm)
> > > > > > > and pvfs, which we are running as a service over the Gb switch.
> > > > > > >
> > > > > > > We have seen the crash occur with the exact same symptoms when trying to
> > > > > > > run our application on a purely Gb based configuration (mpich/hdf5 p4
> > > > > > > configuration and pvfs running over Gb based switch) and with the mixed
> > > > > > > configuration described above. Actually the pure Gb configuration has
> > > > > > > performed much worse and the crashes have always happened much quicker
> > > > > > > in terms the amount of progress in our runs. Some of the Myrinet runs
> > > > > > > have run to completion (though this hasn't been consistent when trying
> > > > > > > the exact same run), but I never had any success with the Gb based runs
> > > > > > > during which neither Myrinet did not have any services running and it's
> > > > > > > module was not loaded.
> > > > > > >
> > > > > > > I was wondering if you have heard of pvfs causing problems similar to
> > > > > > > these. Also if one of the developers would like to have an account on
> > > > > > > the machine to investigate this further that would be tremendous. I have
> > > > > > > been working with developers at Myrinet for the past week and their
> > > > > > > ability to log on has proved beneficial. I'm open to trying pvfs2 if the
> > > > > > > kernel module is more robust than pvfs-1.6.3 but would like to at least
> > > > > > > get your insights into the current problem we are having with this
> > > > > > > installation.
> > > > > > >
> > > > > > > I'm not sure if this relevant, but on two of our earliest runs some
> > > > > > > corrupted file were produced right before our runs crashed. These files
> > > > > > > were written to directories in the pvfs file system.
> > > > > > >
> > > > > > > I just reinstalled a new kernel this afternoon and rebuilt all the
> > > > > > > software, scrapped all old pvfs directories and rebuilt them and my
> > > > > > > Myrinet based run ended up dieing in the same manner as the others.
> > > > > > >
> > > > > > > I've also appending some information about the hardware/software
> > > > > > > configuration of our machine.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Shawn
> > > > > > >
> > > > > > > Here's the software/hardware information.
> > > > > > >
> > > > > > > Hardware. 17 node (1 master/16 compute) cluster from Aspen Systems with
> > > > > > > the Myrinet M3-E32 switch and a Gb switch (24 port HP Procurve 2724,
> > > > > > > J4897a).
> > > > > > > Nodes: e7505 chipset, dual 2.8Ghz xeon processors 4GB memory. PCIXE-2
> > > > > > > dual channel myrinet cards, cards are in 133Mhz PCI slots, Intel Corp.
> > > > > > > 82545EM Gigabit Ethernet Controller. Nvidia 5950 Geforce FX Ultra.
> > > > > > >
> > > > > > > We are running a stock 2.4.27 kernel from kernel.org. The current
> > > > > > > running kernel is a 2.4.27smp-64GB-noAGPsupport. We initially installed
> > > > > > > RH9 on this system and tried to take as much of the RedHat features out
> > > > > > > as possible, but system parameters are still consistent with RH values.
> > > > > > > Here's /proc/version info.
> > > > > > >
> > > > > > > [root at ellipse0 ~]# cat /proc/version
> > > > > > > Linux version 2.4.27smp-64gb-noagp (root at ellipse0) (gcc version 3.2.2
> > > > > > > 20030222 (Red Hat Linux 3.2.2-5)) #1 SMP Mon Dec 20 12:14:14 CST 2004
> > > > > > >
> > > > > > > Myrinet configuration
> > > > > > >
> > > > > > > gm-2.1.6 (linux-2.4 - ia32)
> > > > > > > pvfs-1.6.3 (running over Gb switch)
> > > > > > > mpich-1.2.6..13b-gm * ifc 8.1 (Version 8.1 (l_fc_p_8.1.018))
> > > > > > > * icc 8.1 (Version 8.1 (l_cc_p_8.1.021))
> > > > > > > hdf5-1.6.2 (built against this mpich)
> > > > > > >
> > > > > > > Gb configuration (all services running over Gb based switch):
> > > > > > >
> > > > > > > pvfs-1.6.3
> > > > > > > mpich-1.2.6-p4
> > > > > > > ifc 8.1 (Version 8.1 Build 20040803Z Package ID:
> > > > > > > l_fc_p_8.1.018)
> > > > > > > gcc 3.2.2 (gcc (GCC) 3.2.2 20030222 (Red Hat Linux 3.2.2-5))
> > > > > > > hdf5-1.6.2-p4 (built against mpich-1.2.6-p4)
> > > > > > >
> > > > > > > I have configured both MPICH libraries with-romio filesupport for the
> > > > > > > following file systems (pvfs,ufs,nfs)
> > > > > > >
> > > > > > >
> > > > > > > _______________________________________________
> > > > > > > PVFS-users mailing list
> > > > > > > PVFS-users at www.beowulf-underground.org
> > > > > > > http://www.beowulf-underground.org/mailman/listinfo/pvfs-users
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > _______________________________________________
> > PVFS-users mailing list
> > PVFS-users at www.beowulf-underground.org
> > http://www.beowulf-underground.org/mailman/listinfo/pvfs-users
> >
>
More information about the PVFS-users
mailing list