[PVFS-users] opendx, fgets, big files problem

[Marcelo Matus]

Here are some traces:

When opening the file, it reports:

read(3, "file = /scratch2/cdineen/Ml.bin."..., 65536) = 244
close(3)                                = 0
munmap(0x2a95670000, 65536)             = 0
brk(0)                                  = 0x8a57000
brk(0x9a57000)                          = 0x9a57000
open("/scratch2/cdineen/Ml.bin.out", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=25600000, ...}) = 0
brk(0)                                  = 0x9a57000
brk(0xa857000)                          = 0xa857000
fstat(3, {st_mode=S_IFREG|0644, st_size=25600000, ...}) = 0
mmap(NULL, 65536, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 
0) = 0x2a95670000
read(3, 0x9857a70, 15990784)            = -1 ENOMEM (Cannot allocate memory)
close(3)                                = 0
munmap(0x2a95670000, 65536)             = 0
times({tms_utime=2, tms_stime=0, tms_cutime=0, tms_cstime=0}) = 1207585
write(7, "|41|$int|22|end   /main:0/Import"..., 36) = 36
write(7, "|0|$err|114| 0:  ERROR:  41::/ma"..., 128) = 128
select(8, [7], NULL, NULL, {0, 0})      = 0 (Timeout)
....


and near to the error:

read(3, "\231x>\0004n>\331\263\257>@N\245>\211\222\352>`[\307<\225"..., 
4096) =
2418
lseek(3, 40960, SEEK_SET)               = 40960
read(3, "tions\" value 259\ncomponent \"conn"..., 4096) = 4096
lseek(3, 69632, SEEK_SET)               = 69632
read(3, "\231x>\0004n>\331\263\257>@N\245>\211\222\352>`[\307<\225"..., 
4096) =
2418
lseek(3, 40960, SEEK_SET)               = 40960
read(3, "tions\" value 259\ncomponent \"conn"..., 4096) = 4096
lseek(3, 69632, SEEK_SET)               = 69632
read(3, "\231x>\0004n>\331\263\257>@N\245>\211\222\352>`[\307<\225"..., 
4096) =
2418
lseek(3, 40960, SEEK_SET)               = 40960
read(3, "tions\" value 259\ncomponent \"conn"..., 4096) = 4096
lseek(3, 69632, SEEK_SET)               = 69632
read(3, "\231x>\0004n>\331\263\257>@N\245>\211\222\352>`[\307<\225"..., 
4096) =
2418
lseek(3, 40960, SEEK_SET)               = 40960
read(3, "tions\" value 259\ncomponent \"conn"..., 4096) = 4096
lseek(3, 69632, SEEK_SET)               = 69632
read(3, "\231x>\0004n>\331\263\257>@N\245>\211\222\352>`[\307<\225"..., 
4096) =
2418
lseek(3, 40960, SEEK_SET)               = 40960
read(3, "tions\" value 259\ncomponent \"conn"..., 4096) = 4096
lseek(3, 69632, SEEK_SET)               = 69632
read(3, "\231x>\0004n>\331\263\257>@N\245>\211\222\352>`[\307<\225"..., 
4096) =
2418
ERROR: Import:  Missing data: Reached end-of-file 'data file'  /  Trying 
to read field 'field0'



Marcelo

Rob Ross wrote:

> Hey,
>
> Can you strace the app and see what system call it's trying to use?
>
> Thanks,
>
> Rob
>
> Marcelo Matus wrote:
>
>> Dear PVFS list:
>>
>> We are trying to read a large binary file from a PVFS directory
>> using OpenDx (which uses fgets to read) and we get the
>> following message
>>
>>  ll_pvfs.c, 610): ll_pvfs_file_read failed on 311457
>>  pvfs_map_userbuf: map_user_kiobuf failed
>>
>> while OpenDx reports that the end-of-file has been reached.
>>
>> The  error occurs after reading around 8MB,  and the same
>> file shows no problem when checked using 'md5sum'. pvfs-ping
>> reports no problem, and all the other applications seem
>> to be running fine.
>>
>> OpenDX also opens the same file with no problem after copying it to
>> a local filesystem such as /tmp.
>>
>> The pvfs version is 1.6.3, kernel version Linux 2.4.21-27.0.4.ELsmp,
>> OpenDX 4.3.2.
>>
>> Any ideas?
>>
>>
>> Marcelo
>
> _______________________________________________
> PVFS-users mailing list
> PVFS-users at www.beowulf-underground.org
> http://www.beowulf-underground.org/mailman/listinfo/pvfs-users