[Pvfs2-users] help with pvfs2 and formatdb (used with BLAST)

[Sam Lang]

Hi Randy,

It doesn't look like they actually need to use MAP_SHARED there,  
because they never modify the mapped memory region (they open the file  
with O_RDONLY, and only pass PROT_READ to mmap).  If they were  
modifying the file elsewhere and then expecting the updates to be seen  
in the mapped memory, then they would need MAP_SHARED, but it doesn't  
look like they do that either.  Could you change that MAP_SHARED to  
MAP_PRIVATE, recompile, and run everything on PVFS again?  I think we  
can probably send the blast folks a patch for that, but we want to  
make sure that it works first.

Thanks,
-sam

On Jun 16, 2009, at 8:10 AM, Randall Martin wrote:

> We download the source from http://www.mpiblast.org/Downloads/ 
> Stable.  The
> mmap() call with MAP_SHARED is in mpiblast-1.5.0-pio/ncbi/corelib/ 
> ncbimem.c.
>
> -Randy
>
>> From: Becky Ligon <ligon at clemson.edu>
>> Reply-To: Becky Ligon <ligon at clemson.edu>
>> Date: Mon, 15 Jun 2009 17:49:46 -0400 (EDT)
>> To: Sam Lang <slang at mcs.anl.gov>
>> Cc: Becky Ligon <ligon at clemson.edu>, Phil Carns  
>> <carns at mcs.anl.gov>, Randall
>> Martin <wolf at clemson.edu>, <pvfs2-users at beowulf-underground.org>
>> Subject: Re: [Pvfs2-users] help with pvfs2 and formatdb (used with  
>> BLAST)
>>
>> Sam:
>>
>> Randy will have to answer that question for you, since he is the  
>> one who
>> actually ran the command.  I'll get with him in the morning and  
>> find out.
>>
>> Becky
>>
>>>
>>> Hi Becky,
>>>
>>> Does clemson build formatdb (and blast) from the source code tarball
>>> ncbi?  The reason I ask is that we might be able to make a small
>>> modification (switching from MAP_SHARED to MAP_PRIVATE) to allow  
>>> blast
>>> to work on PVFS deployments.  I can give further info if that's
>>> something you all are willing to try.
>>>
>>> Thanks,
>>> -sam
>>>
>>>
>>> On Jun 11, 2009, at 9:18 AM, Becky Ligon wrote:
>>>
>>>> Phil:
>>>>
>>>> We have a 4Gb input file for the formatdb command.  The problem
>>>> doesn't
>>>> seem to occur until after the command has crunched for several  
>>>> hours.
>>>>
>>>> I'll be happy to share the file with you (and the formatdb command
>>>> that
>>>> Randy is using).  Just let me know the best way to get the file to
>>>> you...I
>>>> can ftp it to you or put it on a machine that you still have access
>>>> to....just let me know!
>>>>
>>>> Becky
>>>>
>>>>
>>>>> Hi Becky,
>>>>>
>>>>> Is there an example input file (either for public download or that
>>>>> you
>>>>> could send to us) that triggers the problem?  I tried this trivial
>>>>> example:
>>>>>
>>>>> http://en.wikipedia.org/wiki/Fasta_format#Header_line
>>>>>
>>>>> ... by just pasting it into a text file and running "formatdb -i
>>>>> <file>".  I also tried just concatenating the same input together
>>>>> many
>>>>> times to get a bigger file, but I didn't have any luck  
>>>>> triggering the
>>>>> problem.
>>>>>
>>>>> -Phil
>>>>>
>>>>> Sam Lang wrote:
>>>>>>
>>>>>> Hi Becky,
>>>>>>
>>>>>> pvfs2_file_write is a kernel module function, so I think that
>>>>>> message
>>>>>> must have come from the kernel log.  There may not be any  
>>>>>> messages
>>>>>> in
>>>>>> the client log (/tmp/pvfs2-client.log), but I was hoping there
>>>>>> would be
>>>>>> something that might point us at the problem (the client daemon  
>>>>>> got
>>>>>> restarted, for example).  Also, the server logs might have
>>>>>> something.
>>>>>> As a last resort, you could try to enable debugging in the kernel
>>>>>> module
>>>>>> to see what happens around the time of another error.
>>>>>>
>>>>>> -sam
>>>>>>
>>>>>> On Jun 9, 2009, at 3:19 PM, Becky Ligon wrote:
>>>>>>
>>>>>>> The second message (pvfs2_file_write) came out of the client log
>>>>>>> (/tmp/pvfs2-client.log).
>>>>>>>
>>>>>>> We are thinking that either the client or server processes are
>>>>>>> running
>>>>>>> out
>>>>>>> of system resources.....maybe file descriptors, for example.
>>>>>>>
>>>>>>> We have now put PVFS on our Palmetto cluster as the primary
>>>>>>> parallel
>>>>>>> file
>>>>>>> system .... no more lustre!  So, now we are trying to iron  
>>>>>>> some of
>>>>>>> these
>>>>>>> problems, which is why I think the problem is really system-
>>>>>>> resource
>>>>>>> related.
>>>>>>>
>>>>>>> I found an old thread from Dec of 2007 where a guy in Beijing  
>>>>>>> had
>>>>>>> the
>>>>>>> same
>>>>>>> problem; unfortunately, there was no response to his problem.
>>>>>>>
>>>>>>> Becky
>>>>>>>
>>>>>>>>
>>>>>>>> Hi Becky,
>>>>>>>> What about the client log?  Anything in there?
>>>>>>>> -sam
>>>>>>>>
>>>>>>>> On Jun 9, 2009, at 10:42 AM, Becky Ligon wrote:
>>>>>>>>
>>>>>>>>> PVFS-USERS:
>>>>>>>>>
>>>>>>>>> When I use formatdb to create a BLAST-usable database, I get  
>>>>>>>>> the
>>>>>>>>> error:
>>>>>>>>>
>>>>>>>>> [formatdb] ERROR:  Failed to create index:  ISAMErrorCode -5.
>>>>>>>>>
>>>>>>>>> The PVFS kernel log shows the following error:
>>>>>>>>>
>>>>>>>>> pvfs2_file_write: error in vectored write to handle xxxx,  
>>>>>>>>> FILE:
>>>>>>>>> xxxx.
>>>>>>>>>
>>>>>>>>> (the x's have values but are just specific to my database).
>>>>>>>>>
>>>>>>>>> I saw a thread in the pvfs2-users archive where someone else  
>>>>>>>>> has
>>>>>>>>> this same
>>>>>>>>> problem, but there were no responses to it.  Am I just  
>>>>>>>>> having a
>>>>>>>>> setup
>>>>>>>>> issue with PVFS?
>>>>>>>>>
>>>>>>>>> We are running PVFS 2.8.
>>>>>>>>>
>>>>>>>>> Any help would be apprecited.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>>
>>>>>>>>> Becky Ligon
>>>>>>>>> --
>>>>>>>>> Becky Ligon
>>>>>>>>> Clemson University
>>>>>>>>> 864-656-3865
>>>>>>>>> _______________________________________________
>>>>>
>>>>
>>>>
>>>> --
>>>> Becky Ligon
>>>> PVFS Developer
>>>> Clemson University
>>>> 864-656-3865
>>>
>>>
>>
>>
>> -- 
>> Becky Ligon
>> PVFS Developer
>> Clemson University
>> 864-656-3865
>