[Pvfs2-users] PVFS 2.6.2 intermittent cmp/diff failure
Sam Lang
slang at mcs.anl.gov
Sat Mar 3 13:13:02 EST 2007
Hi Mark,
These messages in your client logs are the same messages I commented
on back on February 15th. Its possible that the messages in these
logs are just stale, but the timestamps in these logs differ from the
ones you sent on Feb. 15th, so I'm wary that your network issues
haven't been resolved. When I see these logs, they suggest that your
network is not configured properly, and that you're not actually
using PVFS in the way you intend.
To be sure that errors you see in the client logs are recent, you can
pass the
--logstamp=datetime argument to pvfs2-client. That will show full
dates, instead of just times.
I've included some further comments inline.
On Mar 2, 2007, at 8:14 PM, Mark Van De Vyver wrote:
> Hi Rob,
> Thanks for taking time to look at this.
>
> I'm still wondering if the warnings I saw when compiling PVFS 2.6.2
> are related (see my earlier post on upgrading from 1.5.1 to 2.6.2)
I'll try to reply to that email as well. Essentially though you can
ignore those warnings.
>
>> Are you seeing any block device errors in your dmesg/syslog output?
>> Given the fsck that you had to perform previously, it is possible
>> that
>> there's a problem with the local disk or FS.
>
> I'm not sure hat lay behind those disk errors - I've changed a power
> supply and haven't seen any more. I've also completely rebuilt the
> cluster since then - so that PVFS 2.6.2 was upgraded froma on a fresh
> install of 1.5.1.
>
> No errors in dmesg, messages or syslog - I now see some in the pvfs2-
> client.log (attached, see below for a description of what I do on the
> machines during this time)
>
>> What local FS are you using, by the way?
>
> ext3 on a raid 0, all nodes are partitioned identically.
>
> To elaborate/clarify my earlier post:
>
> I take 3 DVD that have previously been copied to the PVFS area: DVD A,
> B, C, thr files on these all passed the diff verification after they
> were copied.
>
> I boot the cluster - the two PVFS2 I/O compute nodes start fine,
> the frontend hangs when loading the PVFS2-client (server loads [ OK
> ]).
This correlates to the client log on the frontend. It sounds like
you're booting the PVFS IO nodes and frontend client node
simultaneously? The PVFS servers have to be done initializing before
you can attempt to mount the volume on the frontend, otherwise the
mount will fail (which is what you're seeing). If I'm reading the
frontend node logs right, it looks like the mount attempt occurs
before your network is even setup, giving you those 'No route to
host' errors.
> After a hard reset, then checking the filesystem the frontend
> boots fine.
Out of curiousity, do you perform this booting/hard reset step for a
particular reason, or is that just what windows has trained you to
do? ;-) PVFS is implemented to allow you to leave these machines
running, and perform the necessary updates and configuration changes
without rebooting each time you make a change.
>
> Load DVD-A on to the frontend and run the script
> copy-taq-dvd-monitor.sh, which calls copy-taq-dvd.sh (both attached)
> Two binary files fail the verification check, they are deleted,
> coppied and verified - both pass.
> Now all the files on DVD-A have been verified once.
>
> Reinsert DVD-A on the frontend and rerun the script - all files pass
> the verification - no files need to be deleted/recopied/verified.
> Now all the files on the PVFS2 area have been _twice_ verified as the
> same as the DVD.
>
> Insert DVD-B and DVD-C on the two nodes and DVD-A on the frontend, run
> the same script on all three machines.
> Now 3 files on DVD-A fail the initial verification. One of these is
> the same that failed the first 'run'.
> DVD-B and C have 3-4 binary files that fail.
> All failed files are deleted/copied/verified and pass the
> verification.
>
> Stop the script running on the two compute nodes (DVD-B, C), restart
> the script on the frontend to re-process DVD-A.
> The diff fails reporting a 'broken pipe', attempts to restart the
> script result in the `cp` operation for this file hanging.
> At this point I stopped.
Based on these steps, it sounds like your setup consists of trying to
use PVFS in a symmetric mode, where you run three servers as part of
your file system, and mount that file system from those same three
nodes with servers running on them. This of course can be done, but
its an uncommon setup for PVFS, and you may benefit from an
alternative setup.
Besides fixing the corruption problems you're seeing, we might be
able to help with your setup as well. What workload will you be
using PVFS for mostly? By that I mean, do you intend to just copy
DVDs to the file system once, and then access them repeatedly at a
(hopefully) high bandwidth? Did you want to be able to distribute
the DVDs over the nodes in your cluster so they can be accessed from
anywhere? How big are the disks on each of the nodes in your cluster?
Could you send us the fs.conf file you're using on each of the nodes?
>
> I have 3 terminals open showing:
> tail -f /var/log/messages
> tail -f /var/log/dmsg
> tail -f /var/log/syslog
> I see no output to these files throughout this exercise.
>
> The pvfs2-client.log files from each machine are attached.
>
> It seems this problem occurs when PVFS2 is under load?
Possibly. When you get to the end of those steps, are you still able
to access files in the file system?
Thanks,
-sam
>
> Hope this helps.
> Mark
>
>> Thanks,
>>
>> Rob
>>
>> Mark Van De Vyver wrote:
>> > Thanks Steve,
>> > I don't see any problem until I run the diff or cmp and even then
>> > these indicate the files are identical if the cmp is run
>> _immediately_
>> > after the file copy.
>> > cmp and diff only indicate a difference when a file is 'checked'
>> after
>> > some other files have been copied-checked.
>> >
>> > The files are from the NYSE trade and quote (TAQ) DVD's, so they
>> are
>> > text stored as binary.
>> >
>> > You might be able to try the following with a dozen or so large
>> binary
>> > files, I have approx 300-400GB stored in the PVFS area.
>> >
>> > Ideally the following should be run on two or more PVFS2 servers at
>> > the same time, apply this to several DVD's that have not been
>> copied
>> > to the PVFS area, then reapply the script to the same DVD's
>> after they
>> > have been copied.
>> > The following is a slightly simplified version of my script -
>> here I
>> > don't delete and re-copy when an existing file fails the cmp
>> > verification:
>> >
>> > # untested script start
>> > for fn in `ls /dvd/*large.bin|sed -e 's/\/dev\//g'`
>> > do
>> > if [ -f /mnt/pvfs2/${fn} ]
>> > then
>> > # This should 'fail' more frequently than the cmp in
>> the else
>> > clause
>> > cmp ${fn} /mnt/pvfs2/${fn}
>> > if [ $? != 0 ]
>> > then
>> > echo "Prexisting copy not exact - more frequent and
>> random?"
>> > fi
>> > else
>> > cp ${fn} /mnt/pvfs2/${fn}
>> > cmp ${fn} /mnt/pvfs2/${fn}
>> > if [ $? != 0 ]
>> > then
>> > echo " Initial copy not exact - less frequent
>> and random"
>> > fi
>> > done
>> > # untested script end
>> >
>> > Regards
>> > Mark
>> >
>> > On 3/2/07, Steve <steve at bov.nu> wrote:
>> >> My setup is a little different in that at the moment I have 2 I/O
>> >> services
>> >> running on one box, a metadata on another and a client/samba
>> server on a
>> >> third. I have moved in the data via samba. We have copied in
>> mp3's and
>> >> avi/mpg's as well as large ISO's plus software exe's. Surely after
>> >> several
>> >> week of use we would notice some problem ?
>> >>
>> >>
>> >>
>> >> I do have another box set up as a client that happens to have a
>> dvd ROM
>> >> drive in it.
>> >>
>> >>
>> >>
>> >> What type of files ? A vob ?
>> >>
>> >> What sequence of commands would I need to do you test your
>> problem ?
>> >>
>> >> If I get a little spare time I could try for U ?
>> >>
>> >>
>> >>
>> >> Steve
>> >>
>> >>
>> >>
>> >> -------Original Message-------
>> >>
>> >>
>> >>
>> >> From: Mark Van De Vyver
>> >>
>> >> Date: 02/03/2007 08:18:11
>> >>
>> >> To: Steve
>> >>
>> >> Subject: Re: [Pvfs2-users] PVFS 2.6.2 intermittent cmp/diff
>> failure
>> >>
>> >>
>> >>
>> >> Hi Steve,
>> >>
>> >>
>> >>
>> >> > Not sure if this helps any but I have copied over 500gb of media
>> >> files to
>> >>
>> >> > pvfs2 running on old dell's 533 to 866 CPU with very little ram
>> >> running on
>> >>
>> >>
>> >> > caos3 beta 3. Although I havent done any checks other than
>> using the
>> >> media
>> >>
>> >>
>> >> > I havent noticed any problems.
>> >>
>> >> >
>> >>
>> >>
>> >>
>> >> The failures might be spurious....?
>> >>
>> >>
>> >>
>> >> > Could you have problems with the dvd device ?
>> >>
>> >>
>> >>
>> >> I doubt it - but it may not be impossible?
>> >>
>> >> This happens with the DVD drives on all three nodes, and when I
>> just
>> >>
>> >> Have one node 'working the diif/cmp failures either don't occur or
>> >>
>> >> Very, very rarely. Start all three nodes 'working' and I see
>> roughly
>> >>
>> >> 1 out of 2 binary files fail the initial diff/cmp check, but
>> very very
>> >>
>> >> Few (one every couple of DVD's fail the cmp/diff check immediately
>> >>
>> >> After the copy is done.....
>> >>
>> >>
>> >>
>> >> Thanks
>> >>
>> >> Mark
>> >>
>> >>
>> >>
>> >> >
>> >>
>> >> > -------Original Message-------
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> > From: Mark Van De Vyver
>> >>
>> >> >
>> >>
>> >> > Date: 02/03/2007 03:26:40
>> >>
>> >> >
>> >>
>> >> > To: pvfs2-users at beowulf-underground.org
>> >>
>> >> >
>> >>
>> >> > Subject: [Pvfs2-users] PVFS 2.6.2 intermittent cmp/diff failure
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> > Hi,
>> >>
>> >> >
>> >>
>> >> > This is a follow up on an earlier email where I reported that
>> PVFS
>> >>
>> >> >
>> >>
>> >> > 1.5.1 failed copy binary files from several DVD's.
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> > I'm running a 3 node Rocks 4.2.1 Cluster, CentOS4.4, x86_64,
>> nodes are
>> >>
>> >> >
>> >>
>> >> > Connected via an unmanaged switch.
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> > I have reinstalled the Rocks Cluster (all nodes), including the
>> >> PVFS2 Roll
>> >>
>> >>
>> >> >
>> >>
>> >> > The cluster is set up with the frontend as the metadaat
>> server and the
>> >>
>> >> >
>> >>
>> >> > Other two nodes are PVFS2 I/O servers and clients. The /
>> mnt.pvfs2
>> >>
>> >> >
>> >>
>> >> > Area is on a 3 disk RAID 0 partition formatted as ext3.
>> >>
>> >> >
>> >>
>> >> > After installing I ran the test steps in the "PVFS2 Quick Start
>> >>
>> >> >
>> >>
>> >> > Guide". The test steps ran without error.
>> >>
>> >> >
>> >>
>> >> > I upgraded to PVFS 2.6.2 on all nodes and re-ran the test
>> steps, again
>> >>
>> >> >
>> >>
>> >> > No errors or problems.
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> > I build PVFS 2.6.2 with the following:
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> > ./configure --with-kernel=</path/to/kernel26/>
>> >>
>> >> >
>> >>
>> >> > --enable-kernel-sendfile --prefix=/usr/local/pvfs2/
>> >>
>> >> >
>> >>
>> >> > Then type
>> >>
>> >> >
>> >>
>> >> > Make all
>> >>
>> >> >
>> >>
>> >> > Make kmod_install
>> >>
>> >> >
>> >>
>> >> > Make install
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> > On each node I have a script that lists the files on the DVD
>> disc
>> >>
>> >> >
>> >>
>> >> > Loaded on that node.
>> >>
>> >> >
>> >>
>> >> > Each file is copied if it does not exist on the HDD (PVFS
>> area) and
>> >>
>> >> >
>> >>
>> >> > The copy is immediately verified:
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> > Cp /dvd/file1 /mnt/pvfs2/file1
>> >>
>> >> >
>> >>
>> >> > Cmp /dvd/file1 /mnt/pvfs2/file1
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> > `cmp` does not report any error.
>> >>
>> >> >
>> >>
>> >> > This has been done for 60-70 DVD.
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> > If I insert a DVD that has previously been copied my script
>> finds that
>> >>
>> >> >
>> >>
>> >> > A file exists in the PVFS area and does a `cmp` with the DVD
>> file, if
>> >>
>> >> >
>> >>
>> >> > The file fails this comparison the file is deleted, copied,
>> verified
>> >>
>> >> >
>> >>
>> >> > (cmp).
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> > I notice that frequently and randomly the previously copied
>> files will
>> >>
>> >> >
>> >>
>> >> > Fail the _initial_ `cmp` check if more than one node is
>> 'active', I.e.
>> >>
>> >> >
>> >>
>> >> > Processing a DVD.
>> >>
>> >> >
>> >>
>> >> > Once deleted and copied the second `cmp` check is passed.
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> > Some details:
>> >>
>> >> >
>> >>
>> >> > The files do not fail the `cmp` check immediately after being
>> copied -
>> >>
>> >> >
>> >>
>> >> > Only when checking a previously copied file.
>> >>
>> >> >
>> >>
>> >> > The `cmp` result indicates a different byte at which the
>> files differ.
>> >>
>> >> >
>> >>
>> >> > Re-inserting the same dvd several times results if different
>> files
>> >>
>> >> >
>> >>
>> >> > Failing the first `cmp` check.
>> >>
>> >> >
>> >>
>> >> > The second check (immediately after the copy is finished) is
>> always
>> >> passed
>> >>
>> >>
>> >> >
>> >>
>> >> > This occurs rarely, if at all (I.e. I haven't noticed it),
>> when only
>> >>
>> >> >
>> >>
>> >> > One node is processing a DVD.
>> >>
>> >> >
>> >>
>> >> > This only occurs with binary files - which are relatively large
>> >> 200MB - 2
>> >> GB
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> > This never occurs with text files - which are also small 100'sKB
>> >>
>> >> >
>> >>
>> >> > The pvfs2-client.log file is empty on each node.
>> >>
>> >> >
>> >>
>> >> > I have tried using diff and experience the same results.
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> > This is similar to an error I was seeing in PVFS 1.5.1 -
>> hence the
>> >>
>> >> >
>> >>
>> >> > Upgrade. I've also changed my previous script which `dd`
>> copied the
>> >>
>> >> >
>> >>
>> >> > DVD to memory (approx 8GB), then wrote this ISO file to the
>> PVFS2 area
>> >>
>> >> >
>> >>
>> >> > - this worked fine for initial copies, but failed for re-
>> copies. At
>> >>
>> >> >
>> >>
>> >> > That time I wasn't verifiying the copy, so it was the copy to
>> the
>> >>
>> >> >
>> >>
>> >> > PVFS2 area that failed.....
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> > Finally, on one occasion when manually running `cmp` on a file I
>> >>
>> >> >
>> >>
>> >> > Noticed the following sequence.
>> >>
>> >> >
>> >>
>> >> > Cmp file1 file2 (pass)
>> >>
>> >> >
>> >>
>> >> > Cmp file1 file2 (pass)
>> >>
>> >> >
>> >>
>> >> > Difffile1 file2 (fail)
>> >>
>> >> >
>> >>
>> >> > Cmp file1 file2 (fail)
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> > Is this known behavior with a known workaround/configuration
>> setting?
>> >>
>> >> >
>> >>
>> >> > The behavior I see made me guess a caching or network issue
>> (there are
>> >>
>> >> >
>> >>
>> >> > No other machines on the cluster network).
>> >>
>> >> >
>> >>
>> >> > Can anyone suggest PVFS configuration settings that will make
>> PVFS more
>> >>
>> >> > robust.
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> > I'm not a programmer or linux guru - I just spent this summer
>> >>
>> >> >
>> >>
>> >> > Converting from winxp...
>> >>
>> >> >
>> >>
>> >> > I'm happy to explore some possible fixes, but don't assume
>> too much :)
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> > Thanks in advance
>> >>
>> >> >
>> >>
>> >> > Mark
>> >>
>> >> >
>> >>
>> >> > _______________________________________________
>> >>
>> >> >
>> >>
>> >> > Pvfs2-users mailing list
>> >>
>> >> >
>> >>
>> >> > Pvfs2-users at beowulf-underground.org
>> >>
>> >> >
>> >>
>> >> > http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >> >
>> >>
>> >>
>> >>
>> > _______________________________________________
>> > Pvfs2-users mailing list
>> > Pvfs2-users at beowulf-underground.org
>> > http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
>> >
>>
>> <copy-taq-dvd-monitor.sh>
>> <copy-taq-dvd.sh>
>> <pvfs2-client.log.frontend>
>> <pvfs2-client.log.compute-0-0>
>> <pvfs2-client.log.compute-0-1>
> _______________________________________________
> Pvfs2-users mailing list
> Pvfs2-users at beowulf-underground.org
> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
More information about the Pvfs2-users
mailing list