[Pvfs2-users] problem with pvfs2 over diskless cluster

Thu Oct 12 18:46:27 EDT 2006

Hello, Rob,

On Thu, 12 Oct 2006, Rob Ross wrote:

> Thanks for the detailed report. How did you end up fixing the "cannot find 
> db.h" compile errors?

There are two places need to be changed:
1. src/io/trove/trove-dbpf/module.mk:
[root at phoenix trove-dbpf]# diff -u module.mk.oct12 module.mk

--- module.mk.oct12     2006-10-12 02:43:25.000000000 -0600
+++ module.mk   2006-10-12 16:35:32.000000000 -0600
@@ -17,4 +17,4 @@
         $(DIR)/dbpf-sync.c

  # grab trove-ledger.h from handle-mgmt.
-MODCFLAGS_$(DIR) = -I$(srcdir)/src/io/trove/trove-handle-mgmt
+MODCFLAGS_$(DIR) = -I$(srcdir)/src/io/trove/trove-handle-mgmt 
-I/usr/local/BerkeleyDB.4.5/include

2. src/apps/admin/pvfs2-migrate-collection.c
[root at phoenix admin]# diff -u pvfs2-migrate-collection.c.oct12 
pvfs2-migrate-collection.c
--- pvfs2-migrate-collection.c.oct12    2006-10-12 02:50:15.000000000 
-0600
+++ pvfs2-migrate-collection.c  2006-10-12 16:38:05.000000000 -0600
@@ -24,7 +24,7 @@
  #include <dirent.h>
  #include <limits.h>

-#include <db.h>
+#include "/usr/local/BerkeleyDB.4.5/include/db.h"

  #define __PINT_REQPROTO_ENCODE_FUNCS_C
  #include "pvfs2-config.h"

>
> This "DB_THREAD mandates memory allocation flag on DBT data" seems to 
> indicate that the semantics have changed in Berkeley DB, and they're 
> requiring a certain way of returning data that wasn't required before 
> (probably for thread safety)? That's irritating.
>

Is it hard to fix? Anything can I help?

> The "User memory too small for return value" looks like a bug.

Where does the bug live? any quick fixes?

Thanks,

wenbin.

>
> The "Key/data pair already exists" might be normal.
>
> Rob
>
> Wenbin wrote:
>> Thank you, Murali, for your timely response. While I tried your patch, and 
>> paid attention to the output of the make process, I found several places 
>> complained about cannot find 'db.h', including src/io/trove/trove-dbpf/
>> as well as src/apps/admin/. I fixed that, without applying your patch, 
>> since you said it is wrong, I have following results:
>> 
>> [root at phoenix pvfs2]pvfs2-ping -m /mnt/pvfs2/    still works cleanly.
>> [root at phoenix pvfs2]pvfs2-ls /mnt/pvfs2/    produces following in the 
>> server log:
>> 
>> [E 10/12 03:00] TROVE:DBPF:Berkeley DB: DB_THREAD mandates memory 
>> allocation flag on DBT data
>> [E 10/12 03:00] TROVE:DBPF:Berkeley DB: keyval_db->get: Invalid argument
>> [E 10/12 03:00] TROVE:DBPF:Berkeley DB: DB_THREAD mandates memory 
>> allocation flag on DBT data
>> [E 10/12 03:00] TROVE:DBPF:Berkeley DB: keyval_db->get: Invalid argument
>> 
>> [root at phoenix pvfs2]/usr/local/bin/pvfs2-cp 
>> ~wenbin/AMRs/pvfs/mpi-io-test.c /mnt/pvfs2/
>> PVFS_sys_create: File exists
>> Could not open /mnt/pvfs2/
>> 
>> server log shows:
>> [E 10/12 03:04] TROVE:DBPF:Berkeley DB: keyval_db->get: DB_BUFFER_SMALL: 
>> User memory too small for return value
>> [E 10/12 03:04] TROVE:DBPF:Berkeley DB: keyval_db->put: DB_KEYEXIST: 
>> Key/data pair already exists
>> 
>> However, file successfully copied to /mnt/pvfs2/
>> [root at phoenix pvfs2]# /usr/local/bin/pvfs2-ls /mnt/pvfs2/ vimrc
>> backup.sh
>> lost+found
>> mpi-io-test.c
>> 
>> Now server log says:
>> [E 10/12 03:07] TROVE:DBPF:Berkeley DB: DB_THREAD mandates memory 
>> allocation flag on DBT data
>> [E 10/12 03:07] TROVE:DBPF:Berkeley DB: keyval_db->get: Invalid argument
>> [E 10/12 03:07] TROVE:DBPF:Berkeley DB: DB_THREAD mandates memory 
>> allocation flag on DBT data
>> [E 10/12 03:07] TROVE:DBPF:Berkeley DB: keyval_db->get: Invalid argument
>> 
>> [root at phoenix pvfs2]# /usr/local/bin/pvfs2-ls -l /mnt/pvfs2/
>> Failed to get attributes on handle 1048574,1906098859
>> Getattr failure: No such file or directory
>> Failed to get attributes on handle 1048573,1906098859
>> Getattr failure: No such file or directory
>> drwxrwxrwx    1 root     root            4096 2006-10-11 12:36 lost+found
>> Failed to get attributes on handle 1048574,1906098859
>> Getattr failure: No such file or directory
>> 
>> The server log says:
>> [E 10/12 03:14] TROVE:DBPF:Berkeley DB: DB_THREAD mandates memory 
>> allocation flag on DBT data
>> [E 10/12 03:14] TROVE:DBPF:Berkeley DB: keyval_db->get: Invalid argument
>> [E 10/12 03:14] TROVE:DBPF:Berkeley DB: DB_THREAD mandates memory 
>> allocation flag on DBT data
>> [E 10/12 03:14] TROVE:DBPF:Berkeley DB: keyval_db->get: Invalid argument
>> [E 10/12 03:14] TROVE:DBPF:Berkeley DB: DB_THREAD mandates memory 
>> allocation flag on DBT data
>> [E 10/12 03:14] TROVE:DBPF:Berkeley DB: keyval_db->get: Invalid argument
>> 
>> Good thing is that the infinite loop is gone. Is now the situation better? 
>> Where could the problem be?
>> 
>> Thanks,
>> 
>> wenbin.
>> 
>> On Thu, 12 Oct 2006, Murali Vilayannur wrote:
>> 
>>> Hi Wenbin,
>>>> Hello, I tried to install pvfs2, version 1.5.1 on a diskless cluster. 
>>>> The head node is to be both the metadata server as well as the io 
>>>> server. The clients are nfs mounted to the head node. The underlying 
>>>> database is the Berkeley 4.5.20. The check of "pvfs2-ping -m 
>>>> /mnt/pvfs2/" is allright. However, when I tried to do "pvfs2-cp" or 
>>>> "pvfs2-ls", the commands hang. The /tmp/pvfs2-server.log shows 
>>>> following, only 2 lines cut out of many:
>>>> 
>>>> [E 10/11 14:45] TROVE:DBPF:Berkeley DB: keyval_db->get: 
>>>> DB_BUFFER_SMALL: User memory too small for return value
>>>> [E 10/11 14:45] TROVE:DBPF:Berkeley DB: keyval_db->get: 
>>>> DB_BUFFER_SMALL: User memory too small for return value
>>>> ....
>>>> 
>>>> The database is configured with:
>>>> ../dist/configure --enable-pthread_api --disable-libtool-lock 
>>>> --enable-debug
>>>> 
>>>> the pvfs2-1.5.1 is configured with:
>>>> ./configure -with-db=/usr/local/BerkeleyDB.4.5
>>> I dont know how/what is the right way to fix this problem.
>>> Sam will probably find the right fixes for this. It does look like a bdb 
>>> 4.5 specific issue.
>>> 
>>> Attached patch somehow  fixes it for me although it seems (and probably 
>>> is) obviously incorrect. Also, it does look like the server goes off in 
>>> an infinite loop once it hits this error, but that is a separate issue I 
>>> think..
>>> thanks,
>>> Murali
>>> 
>>> 
>> _______________________________________________
>> Pvfs2-users mailing list
>> Pvfs2-users at beowulf-underground.org
>> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
>> 
>
