[Pvfs2-users] trove_write_callback_fn: I/O error occurred

Sat Sep 3 01:27:03 EDT 2011

hi Michael,

thats nice, i will give it a try.

thanks
Christos
----- Original Message -----
From: "Michael Moore" <mtmoore at omnibond.com>
To: pvfs2-users at beowulf-underground.org
Sent: Friday, September 2, 2011 9:56:25 PM
Subject: Re: [Pvfs2-users] trove_write_callback_fn: I/O error occurred

Christos, 

I committed a change earlier today into CVS Orange-Branch that, in limited testing, resolved the issues I was seeing on the ARM device I have. With help from some other folks here, we tracked it down to alignment issues with memory passed back from Berkeley (namely that it's not aligned). As a miscellaneous hint, to make debugging easier echo 5 into /proc/cpu/alignment so you get a SIGBUS instead of mucked up data when making unaligned accesses. There may be some more cases to track down but with these initial changes you should be able to get up and running. If not, or if you run into more issues, please let us know. 

Sorry for the delay in getting a fix for this issue! 

Thanks, 
Michael 

On Tue, Jul 19, 2011 at 5:47 PM, Kyle Schochenmaier < kschoche at gmail.com > wrote: 

This may be a really dumb question... 
Your data server only has 32MB of ram (assuming this is some NAS or router+attached storage device?) .. is that enough to run your linux kernel as well as then open up all of the necessary buffers, etc for a pvfs2 server? Do we know what a typical tcp-ip footprint is (8MB?) ? 

Kyle Schochenmaier 

On Tue, Jul 19, 2011 at 4:43 PM, Phil Carns < carns at mcs.anl.gov > wrote: 

Can you try running the same pvfs2-cp on the arm machine as well? That may indicate if there is a fundamental problem with the daemon on that node, or if there is an error with the network encoding between x86_64 and arm. 

-Phil 

On 07/12/2011 07:17 AM, Michael Moore wrote: 

I don't have any ideas off-hand, you may be right that it's an architecture issue. Can you re-run the test with EventLogging set to 'bstream, trove' in the server configuration and send those logs? 

Michael 

On Tue, Jul 12, 2011 at 4:03 AM, Christos Filippidis < filippidis at inp.demokritos.gr > wrote: 

Hi, 

I am trying to configure orangefs for testing to the following system: 

Metadata server /client: 
Centos 5 :Linux 2.6.18-194.17.4.el5 #1 SMP Mon Oct 25 15:50:53 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux 

Data server: 
Debian Lenny :Linux 2.6.17.14 #16 PREEMPT Sat Feb 5 12:45:55 UTC 2011 armv5tejl GNU/Linux 

Data server configuration options: 

./configure --prefix=/shares/internal/PUBLIC/orange 

Metadata server /client configuration options: 

./configure --prefix=/state/partition1/orange --with-kernel=/usr/src/kernels/2.6.18-194.17.4.el5-x86_64/ 

The compilation and installation was successful. 
"bin/pvfs2-ping -m /mnt/orange" was also successful. 

But every time i try to "pvfs2-cp" a file i am receiving a "trove_write_callback_fn: I/O error occurred" at the data server side(the same happens with pvfs-2.8.2) 

I tried to reconfigure orangefs with many different options but nothing seams to works. 
I also tried "alt-aio" "null-aio" "directio" but nothing happened. 

It seams that something strange happens with the data server(ARM processor - 200MHZ,32MB RAM)because both orangefs and pvfs2 works fine when i am using as data servers intel or AMD processors. 

Do you have any idea what is wrong ? 
i really need to install orangefs at this ARM processor. 
Thanks in advance 
Christos 

ps: i send you some log files from the data server 

alt-aio 
-------- 
PVFS2 Server version 2.8.4-orangefs starting. 
[E 07/12/2011 07:38:30] trove_write_callback_fn: I/O error occurred 
[E 07/12/2011 07:38:30] handle_io_error: flow proto error cleanup started on 0x154a88: No such file or directory 
[E 07/12/2011 07:38:30] handle_io_error: flow proto 0x154a88 canceled 0 operations, will clean up. 
[E 07/12/2011 07:38:30] handle_io_error: flow proto 0x154a88 error cleanup finished: No such file or directory 

directio 
----------- 
PVFS2 Server version 2.8.4-orangefs starting. 
[E 07/12/2011 07:56:39] dbpf_bstream_direct_write_op_svc: failed to get dspace attr for bstream: (error=-1073742082) 
[E 07/12/2011 07:57:09] job_time_mgr_expire: job time out: cancelling flow operation, job_id: 505. 
[E 07/12/2011 07:57:09] fp_multiqueue_cancel: flow proto cancel called on 0x113f00 
[E 07/12/2011 07:57:09] fp_multiqueue_cancel: I/O error occurred 
[E 07/12/2011 07:57:09] handle_io_error: flow proto error cleanup started on 0x113f00: Operation cancelled (possibly due to timeout) 

bin/pvfs2-ping -m /mnt/orange/ 

(1) Parsing tab file... 

(2) Initializing system interface... 

(3) Initializing each file system found in tab file: /etc/fstab... 

PVFS2 servers: tcp://nas-0-1:3334 
Storage name: pvfs2-fs 
Local mount point: /mnt/orange 
/mnt/orange: Ok 

(4) Searching for /mnt/orange/ in pvfstab... 

PVFS2 servers: tcp://nas-0-1:3334 
Storage name: pvfs2-fs 
Local mount point: /mnt/orange 

meta servers: 
tcp://ikaros:3334 

data servers: 
tcp://nas-0-1:3334 

(5) Verifying that all servers are responding... 

meta servers: 
tcp://ikaros:3334 Ok 

data servers: 
tcp://nas-0-1:3334 Ok 

(6) Verifying that fsid 883136527 is acceptable to all servers... 

Ok; all servers understand fs_id 883136527 

(7) Verifying that root handle is owned by one server... 

Root handle: 1048576 
Ok; root handle is owned by exactly one server. 

============================================================= 

The PVFS2 filesystem at /mnt/orange/ appears to be correctly configured. 

-- 
Christos Filippidis 
Institute of Nuclear Physics 
National Center for Scientific Research "Demokritos" 
Patr. Grigoriou & Neapoleos Str 
153 10 Agia Paraskevi Attikis 
Athens, Greece 
Tel: +30 2106503425 

http://www.inp.demokritos.gr/~filippidisx/ 
_______________________________________________ 
Pvfs2-users mailing list 
Pvfs2-users at beowulf-underground.org 
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users 

_______________________________________________
Pvfs2-users mailing list Pvfs2-users at beowulf-underground.org http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users 

_______________________________________________ 
Pvfs2-users mailing list 
Pvfs2-users at beowulf-underground.org 
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users 

_______________________________________________ 
Pvfs2-users mailing list 
Pvfs2-users at beowulf-underground.org 
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users 

_______________________________________________
Pvfs2-users mailing list
Pvfs2-users at beowulf-underground.org
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users

-- 
Christos Filippidis
Institute of Nuclear Physics
National Center for Scientific Research "Demokritos"
Patr. Grigoriou & Neapoleos Str
153 10 Agia Paraskevi Attikis
Athens, Greece        
Tel:+30 2106503425 

http://www.inp.demokritos.gr/~filippidisx/