[Pvfs2-users] 2.6.3, file hole issues?

[Sam Lang]

On Oct 5, 2007, at 12:00 PM, Anthony Tong wrote:

> I'm getting file holes with pvfs 2.6.3 on linux 2.6 (rhel4 kernels  
> i386,
> and a vanilla 2.6 as well) on a test system and can consistently  
> reproduce
> them. Holes are about 16k+ of zeros.
>
> Simple setup: 4 io servers, 1 metadata, over TCP, these nodes also  
> mount
> the filesystem.
>
> I am writing gigabyte files sequentially from a client via the vfs
> interface
>
> I finally had some time to do some debugging this morning and here's
> what I have found so far. "io,client" is the gossip mask on for
> pvfs2-client-core-threaded.

Hi Anthony,

Are you able to reproduce the problem with pvfs2-client-core?  Also,  
are you running pvfs2-client-core-threaded directly or through pvfs2- 
client --threaded?

 From the traces you've included below, it looks like you're mounting/ 
unmounting the filesystem over and over between each IO.  Any reason  
to do that?
-sam


>
> For the first instance of the hole, I see "Posted UNKNOWN" in
> the log. The offset (1301371504) corresponds with where the first
> hole is in my test file.
>
> Snippet from output of cmp -l good.file corrupt.file
> 1301371505 127   0
> 1301371506 376   0
> 1301371509 115   0
> 1301371510 221   0
> 1301371511 132   0
> ... (and so forth till)..
> 1301438544 110   0
> 1571986033   7   0
>
> Searching for other "Posted UNKNOWN" messages and if there's a
> file_req_off nearby, it corresponds for other holes as well.
>
> Gossip snippets
>
> [D 11:09:40.531507] * mem req size is 67040, file_req size is 67040  
> (bytes)
> [D 11:09:40.531534]   bstream_size = 325343856, datafile nr=1,  
> ct=4, file_req_off = 1301371504
> [D 11:09:40.531712]   posted flow for context 0xb4bfd720
> [D 11:09:40.531790]   preposting write ack for context 0xb4bfd720.
> [D 11:09:41.563356] Posted UNKNOWN (waiting for test)
> [D 11:09:41.563558] Posted UNKNOWN (waiting for test)
> [D 11:09:41.640702] get_config state: server_get_config_setup_msgpair
> [D 11:09:41.641900] Posted PVFS_SYS_FS_ADD (waiting for test)
> [D 11:09:41.644099] * Adding new dynamic mount point <DYNAMIC-1> [7,0]
> [D 11:09:41.644148] PINT_server_config_mgr_add_config: adding  
> config 0x84e6680
> [D 11:09:41.644177]     mapped fs_id 1867692515 => config 0x84e6680
> [D 11:09:41.644218] Set min handle recycle time to 360 seconds
> [D 11:09:41.644249] Reloading handle mappings for fs_id 1867692515
> [D 11:09:41.644472] PVFS_isys_io entered [1048186]
> [D 11:09:41.644548] (0x84f0c68) io state: io_init
> [D 11:09:41.644582] (0x84f0c68) getattr_setup_msgpair
> [D 11:09:41.644702] Posted PVFS_SYS_IO (waiting for test)
> [D 11:09:41.645097] trying to add object reference to acache
> [D 11:09:41.645138] (0x84f0c68) getattr state: getattr_cleanup
> [D 11:09:41.645169] (0x84f0c68) io state: io_datafile_setup_msgpairs
> [D 11:09:41.645201] - io_find_target_datafiles called
> [D 11:09:41.645279] io_find_target_datafiles: datafile[1] might  
> have data (out=1)
> [D 11:09:41.645319] io_find_target_datafiles: datafile[2] might  
> have data (out=2)
>
> ...
>
> [D 11:09:55.609389] * mem req size is 100272, file_req size is  
> 100272 (bytes)
> [D 11:09:55.609417]   bstream_size = 393019392, datafile nr=0,  
> ct=4, file_req_off = 1571986032
> [D 11:09:55.609526]   posted flow for context 0xb55fd318
> [D 11:09:55.609554]   preposting write ack for context 0xb55fd318.
> [D 11:09:56.627065] Posted UNKNOWN (waiting for test)
> [D 11:09:56.627238] Posted UNKNOWN (waiting for test)
> [D 11:09:56.693300] get_config state: server_get_config_setup_msgpair
> [D 11:09:56.694529] Posted PVFS_SYS_FS_ADD (waiting for test)
> [D 11:09:56.700558] * Adding new dynamic mount point <DYNAMIC-1> [7,0]
> [D 11:09:56.700620] PINT_server_config_mgr_add_config: adding  
> config 0x83c6680
> [D 11:09:56.700650]     mapped fs_id 1867692515 => config 0x83c6680
> [D 11:09:56.700692] Set min handle recycle time to 360 seconds
> [D 11:09:56.700735] Reloading handle mappings for fs_id 1867692515
> [D 11:09:56.700954] PVFS_isys_io entered [1048186]
> [D 11:09:56.701033] (0x83d0c68) io state: io_init
> [D 11:09:56.701066] (0x83d0c68) getattr_setup_msgpair
> [D 11:09:56.701188] Posted PVFS_SYS_IO (waiting for test)
> [D 11:09:56.701589] trying to add object reference to acache
> [D 11:09:56.701631] (0x83d0c68) getattr state: getattr_cleanup
> [D 11:09:56.701663] (0x83d0c68) io state: io_datafile_setup_msgpairs
>
>
> _______________________________________________
> Pvfs2-users mailing list
> Pvfs2-users at beowulf-underground.org
> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
>