[Pvfs2-users] Problems with varstrip distribution
Murali Vilayannur
murali.vilayannur at gmail.com
Mon Jan 15 14:22:54 EST 2007
Hi Juan,
Great! Good to know that it works..
I already know about the pvfs2-viewdist problem.
Unfortunately, I dont think it can be fixed cleanly..
The problem is that our kernel-userspace interface currently imposes a
limit of 256 bytes on xattr (key-val pairs). It is really my fault. I
should have made that variable length as well.
For varstrip, length of value in xattr exceeds this limit and hence
you see an error..
That said, I am surprised that it segfaulted though..:(
If this is not too big a deal, I can fix this later..
Thanks,
Murali
> Murali,
>
> Thank you very much for your patch. It seems to work well. For example,
> with 2 I/O servers, and a varstrip configuration "0:512;1:1024", when I
> copy a 6 MB file, 2MB of it are stored in one server, and the remaining
> 4 MB in the other server. The command
>
> setfattr -n "user.pvfs2.dist_params" -v "strips:0:512;1:1024"
>
> also works nicely.
>
> The pvfs2_viewdist command, however, does not work. I get the following
> error:
>
> dist_name = varstrip_dist
> strips = 0:512;1:1024
> Number of datafiles/servers = 2
> Server 0 - (null), handle: 4294967293 (fffffffd.bstream)
> Segmentation fault
>
> Anyway, this error does not worry me. I will take a look at Julian's branch.
>
> Thanks again,
>
> Juan.
>
> Murali Vilayannur wrote:
> > Hi Juan,
> > We had never tested varstrip placement with xattrs... Clearly you just
> > hit upon a bug that I knew was waiting to happen all this while but
> > did not get to fixing it... :(
> > Can you try the attached patch against pvfs-2.6.* and see if it works?
> > You still cannot control which specific I/O node a file will be
> > created since the
> > varstrip placement is with respect to some ordinal numbering of servers
> > which can and is typically random :)
> >
> > Sample invocation of setfattr could be like
> > % setfattr -n "user.pvfs2.dist_params" -v "strips:0:512;1:1024"
> > /mnt/pvfs2/testdir/
> >
> > will take bytes 0-511 on node 0, 511-1535 on node 1 and so on...
> >
> > Note how fragile the parsing logic and requirement on how the distrib
> > params are to be constructed. Very sorry about that.. We really should
> > find a better way for expressing parameters I think...
> >
> > If you really need very specific server location placement, you have
> > to checkout Julian's branch from HEAD (kunkel-hint-branch) and figure
> > out how to invoke PVFS_sys_create() with the hint parameters. NOTE:
> > You cannot use xattrs (either system or vfs) since noone has done that
> > yet with Julian's server-placement hint branch yet...should not be too
> > hard though.
> >
> > let me know if this patch still does not get things working again..
> > thanks,
> > Murali
> >
> > On 1/12/07, Juan Piernas Canovas <juan.piernascanovas at pnl.gov> wrote:
> >> Hi folks,
> >>
> >> I'm testing PVFS2, and one of the things I want to do is to control in
> >> which I/O node a specific file will be stored. With the "simple stripe"
> >> distribution, I can create a directory and specify for it 1 as the
> >> number of datafiles to be used. The problem is that every file created
> >> in that directory is stored in a different I/O node in a round-robin
> >> fashion.
> >>
> >> With the varstrip distribution, you can, in theory at least, indicate
> >> which I/O nodes, and which chunk size in every node, you want to use for
> >> a file. But this distribution does not work for me. I have tried the
> >> following:
> >>
> >> $ mkdir testdir
> >> $ setfattr -n "user.pvfs2.dist_name" -v "varstrip_dist" testdir
> >> $ setfattr -n "user.pvfs2.dist_params" -v "strips:1:512" testdir
> >> $ cp somefile testdir
> >> cp: writing `testdir/somefile': Unknown error 132
> >>
> >> I have also tried to run:
> >>
> >> $ setfattr -n "user.pvfs2.dist_params" -v "1:512" testdir
> >>
> >> but the cp command gets the same error.
> >>
> >> Any clue? Is there any way to specific where I can store a file in
> >> PVFS2?
> >>
> >> Thanks in advance,
> >>
> >> Juan.
> >> _______________________________________________
> >> Pvfs2-users mailing list
> >> Pvfs2-users at beowulf-underground.org
> >> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
> >>
> > ------------------------------------------------------------------------
> >
> > --- src/client/sysint/sys-create.sm.old 2007-01-12 19:32:00.776908000 -0800
> > +++ src/client/sysint/sys-create.sm 2007-01-12 19:44:49.307825000 -0800
> > @@ -15,7 +15,9 @@
> >
> > #include "client-state-machine.h"
> > #include "pvfs2-debug.h"
> > +#include "pvfs2-dist-basic.h"
> > #include "pvfs2-dist-simple-stripe.h"
> > +#include "pvfs2-dist-varstrip.h"
> > #include "job.h"
> > #include "gossip.h"
> > #include "str-utils.h"
> > @@ -1134,34 +1136,50 @@
> > {
> > gossip_debug(GOSSIP_CLIENT_DEBUG, "distribution parameter %s, value %s\n",
> > key[i], val[i]);
> > - /* NOTE: just as in server-config.c when parsing "Param" and
> > - * "Value" fields, we will assume that all values are 64 bit
> > - * integers. The only difference here is that we scan
> > - * directly into a 64 bit integer, rather than converting
> > - * from the int format that dotconf supports.
> > - */
> > - ret = sscanf(val[i], SCANF_lld, &tmp_val);
> > - if(ret != 1)
> > - {
> > - gossip_err("Error: unsupported type for distribution parameter %s, value %s found in directory hints.\n",
> > - key[i], val[i]);
> > - gossip_err("Error: continuing anyway.\n");
> > - }
> > - else
> > - {
> > + if (strncmp(current_dist->dist_name,
> > + PVFS_DIST_SIMPLE_STRIPE_NAME,
> > + PVFS_DIST_SIMPLE_STRIPE_NAME_SIZE) == 0) {
> > + /* NOTE: just as in server-config.c when parsing "Param" and
> > + * "Value" fields, we will assume that all values are 64 bit
> > + * integers. The only difference here is that we scan
> > + * directly into a 64 bit integer, rather than converting
> > + * from the int format that dotconf supports.
> > + */
> > + ret = sscanf(val[i], SCANF_lld, &tmp_val);
> > + if(ret != 1)
> > + {
> > + gossip_err("Error: unsupported type for distribution parameter %s, value %s found in directory hints.\n",
> > + key[i], val[i]);
> > + gossip_err("Error: continuing anyway.\n");
> > + }
> > + else
> > + {
> > + if(current_dist->methods->set_param(current_dist->dist_name,
> > + current_dist->params,
> > + key[i],
> > + &tmp_val))
> > + {
> > +
> > + gossip_err("Error: could not override hinted distribution parameter %s, value %s found in directory hints\n",
> > + key[i],
> > + val[i]);
> > + }
> > + }
> > + } else if (strncmp(current_dist->dist_name,
> > + PVFS_DIST_VARSTRIP_NAME,
> > + PVFS_DIST_VARSTRIP_NAME_SIZE) == 0) {
> > if(current_dist->methods->set_param(current_dist->dist_name,
> > - current_dist->params,
> > - key[i],
> > - &tmp_val))
> > + current_dist->params,
> > + key[i],
> > + val[i]))
> > {
> >
> > gossip_err("Error: could not override hinted distribution parameter %s, value %s found in directory hints\n",
> > - key[i],
> > - val[i]);
> > + key[i], val[i]);
> > }
> > - }
> > - free(key[i]);
> > - free(val[i]);
> > + }
> > + free(key[i]);
> > + free(val[i]);
> > }
> > free(key);
> > free(val);
> > --- src/common/misc/str-utils.c.old 2007-01-12 19:26:49.129773000 -0800
> > +++ src/common/misc/str-utils.c 2007-01-12 19:27:49.448030000 -0800
> > @@ -688,6 +688,7 @@
> > * The given string must be comma separated, and each
> > * segment within the comma regions must be of of
> > * the form key:val.
> > + * val itself can contain colons.
> > * Return the number of such keyval pairs and a
> > * pointer to a double dimensional array of keys and values.
> > * In case of errors, a -ve PVFS error is returned.
> > @@ -697,6 +698,7 @@
> > * NULL - return -PVFS_EINVAL
> > * ab:23 - return nkey as 1, pkey <"ab">, pval <"23">
> > * ab:23,bc:34 - returns nkey as 2, pkey <"ab", "bc">, pval<"23", "34">
> > + * ab:2:3 - returns nkey as 1, pkey <"ab">, pval <"2:3">
> > *
> > */
> > int PINT_split_keyvals(char *string, int *nkey,
> > @@ -780,8 +782,10 @@
> > ptr2 = strdup(val[i]);
> > if (ptr1 == NULL || ptr2 == NULL)
> > break;
> > +#if 0
> > if (strchr(ptr1, ':') || strchr(ptr2, ':'))
> > break;
> > +#endif
> > key[i] = ptr1;
> > val[i] = ptr2;
> > }
> >
>
More information about the Pvfs2-users
mailing list