[Pvfs2-users] Pvfs performance over 10Ge

[Salmon, Rene]

Hi,

Yeah you are right. I setup a tmpfs file system (/scratch) in RAM on my
test client and here is what I got.  Sorry the local SATA disk on the
client is not fast enough to keep up so I have to use RAM.

Read from pvfs2 write to RAM tmpfs (/scratch)
----------------------------------------------
hpca4000(salmr0)177:pvfs2-cp -t -b 4194304 /pvfs2-fs/salmr0/Hmig-8GB
/scratch/
Wrote 8590895600 bytes in 18.850570 seconds. 434.624338 MB/seconds

Write to pvfs2 read from RAM tmpfs (/scratch)
---------------------------------------------
hpca4000(salmr0)179:pvfs2-cp -t -b 4194304 /scratch/Hmig-8GB
/pvfs2-fs/salmr0/Hmig-8GB-junk
Wrote 8590895600 bytes in 13.541110 seconds. 605.040242 MB/seconds

Read from pvfs2 write to pvfs2
------------------------------
hpca4000(salmr0)182:pvfs2-cp -t -b 4194304 /pvfs2-fs/salmr0/Hmig-8GB
/pvfs2-fs/salmr0/Hmig-8GB-junk2
Wrote 8590895600 bytes in 21.963378 seconds. 373.026249 MB/seconds

I will give the patched pvfs2-cp a try.

Thanks
Rene


> -----Original Message-----
> From: Sam Lang [mailto:slang at mcs.anl.gov] 
> Sent: Wednesday, October 31, 2007 3:20 PM
> To: Salmon, Rene
> Cc: Scott Atchley; pvfs2-users
> Subject: Re: [Pvfs2-users] Pvfs performance over 10Ge
> 
> 
> On Oct 31, 2007, at 2:58 PM, Salmon, Rene wrote:
> 
> > Hi Sam,
> >
> > Thanks for the reply.  You where right.  Taking out all the 
> IB stuff 
> > made a big difference.  Now I am seeing much better 
> performance using 
> > bmi_tcp.  Thanks.
> 
> Hi Rene,
> 
> Good to hear it worked out.  The configure script will warn 
> about enabling multiple BMI methods at once, although, I 
> think it actually assumes tcp is the slow one, whereas in 
> your case its IB.
> 
> >
> > hpca4000(salmr0)140:dd if=/dev/zero of=/pvfs2-fs/salmr0/foo1 bs=4M 
> > count=2000
> > 2000+0 records in
> > 2000+0 records out
> > 8388608000 bytes (8.4 GB) copied, 21.3792 seconds, 392 MB/s
> >
> > hpca4000(salmr0)137:pvfs2-cp -b4194304 -t /pvfs2-fs/salmr0/Hmig-8GB 
> > /pvfs2-fs/salmr0/Hmig-8GB-copy Wrote 8590895600 bytes in 50.296533 
> > seconds. 162.892271 MB/seconds
> >
> > hpca4000(salmr0)142:pvfs2-cp -t -b 4194304 /vol0/salmr0/Hmig-8GB
> > /pvfs2-fs/salmr0/foo-4
> > Wrote 8590895600 bytes in 14.417761 seconds. 568.251657 MB/seconds
> 
> That's interesting, I would have expected the second test (write to
> pvfs) to be about twice as fast as the first test (read from 
> pvfs, write to pvfs).  You're seeing writes that are much 
> faster than reads.  What numbers do you get copying a file 
> from pvfs to local?
> 
> Also, pvfs2-cp is a little inefficient because it does 
> blocking reads and writes using a single buffer.  I've 
> attached a patch to pvfs2-cp that uses multiple buffers.  If 
> you're inclined, you might see even better numbers with it.
> 
> -sam
> 
>