[PVFS-users] PVFS Lockup on ia64 & how to reproduce

[Christoph Biardzki]

Hello,



it seems I finally got a grip on debugging our ia64-Problems with 
Gaussian after loading pvfsd with "pvfsd -d 0xFFFF":



Jun 16 14:02:51 lxsrv163 kernel: Process 21504 entered pvfs_file_write
Jun 16 14:02:51 lxsrv163 kernel: (file.c, 391): pvfs_file_write called 
for 192.168.36.237:3001/pvfsmetadata/pvfs64/root/Gau-21504.rwf 
(885041), offset 189595648, size 731588544
Jun 16 14:02:51 lxsrv163 kernel: (ll_pvfs.c, 640): ll_pvfs_file_write 
called for 192.168.36.237:3001/pvfsmetadata/pvfs64/root/Gau-21504.rwf 
(885041), size 16777216, loc 189595648
Jun 16 14:02:51 lxsrv163 kernel: (pvfsdev.c, 1437): exceeded buffer 
size: sleeping.


It seems that Gaussian is writing >700 MB using one call and PVFS locks 
up in ll_pvfs.c/setup_buffer() - the Gaussian process just hangs and 
PVFS does not leave the loop in setup_buffer()


The problem can be reproduced with iozone


./iozone -t1 -i0 -i1 -r32m -s128m -F /pvfs64/root/testdatei


iozone will just hang.


Notice the -r32m for a 32 MB record - ironically the defalt max. iozone 
record size is equal to PVFS's default maximum transfer size (16 MB). 
So you will have to change

#define MAXBUFFERSIZE (16*1024*1024)

to

#define MAXBUFFERSIZE (64*1024*1024)

in iozone.c otherwise it will not accept the -r32m argument.



Jun 16 15:25:44 lxsrv163 kernel: (file.c, 391): pvfs_file_write called 
for 192.168.36.237:3001/pvfsmetadata/pvfs64/root/testdatei.DUMMY.0 
(885088), offset 0, size 33554432
Jun 16 15:25:44 lxsrv163 kernel: (ll_pvfs.c, 640): ll_pvfs_file_write 
called for 
192.168.36.237:3001/pvfsmetadata/pvfs64/root/testdatei.DUMMY.0 
(885088), size 16777216, loc 0
Jun 16 15:25:44 lxsrv163 kernel: (pvfsdev.c, 1437): exceeded buffer 
size: sleeping.



Can somebody try this to reproduce the problem? Any ideas how to fix it?


Thanks!

  Christoph






-- 
Leibniz Rechenzentrum München (LRZ)
http://www.lrz.de
Abteilung Hochleistungssysteme
Barer Str. 21 - 80333 München - Germany
Tel. ++49-(0)89 / 289-28853, Raum S1527