[PVFS-users] Preparing an IO server under Linux

[Michael Homa]

This is a follow up to my letter on Monday. When you have multiple IO
servers, there is a data partition on each of those machines. You use a
tool like fdisk to create the partition. Then, you use mkfs to create a
filesystem within that partition. The next step is the one that eludes
me. A command is run to prepare the filesystem for use with pvfs. It
creates something like the following in that data-partiion:

000  008  016  024  032  040  048  056  064  072  080  088  096
001  009  017  025  033  041  049  057  065  073  081  089  097
002  010  018  026  034  042  050  058  066  074  082  090  098
003  011  019  027  035  043  051  059  067  075  083  091  099
004  012  020  028  036  044  052  060  068  076  084  092  100
005  013  021  029  037  045  053  061  069  077  085  093  lost+found
006  014  022  030  038  046  054  062  070  078  086  094
007  015  023  031  039  047  055  063  071  079  087  095

The above are all subdirectories. Does anyone recall what command or step
creates these directories?

I've spent several days going through my notes and the pvfs documentation
to no avail. I don't recall what I did to create the directories and I
need to recreate them on a different IO server.

Michael Homa
Academic Computing and Communication Center
University of Illinois at Chicago
email:  mhoma at uic.edu

---------- Forwarded message ----------
Date: Mon, 13 Sep 2004 14:05:13 -0500 (CDT)
From: Michael Homa <mhoma at uic.edu>
To: pvfs-users at www.beowulf-underground.org
Subject: Weird problem - think I forgot a step.

I'm upgrading my 64 node cluster to a more recent kernel (RH 2.4.20-28.7)
and I took the opportunity to rebuild pvfs-1.60. I have three metadata
servers:

  1 metadata server  and four  25GB IO servers /pvfs-scratch3
  1 metadata server  and four  25GB IO servers /pvfs-scratch2
  1 metadata server  and eight 25GB IO servers /pvfs-scratch1

When I start pvfs, as an example, and mount /pvfs-scratch3 I get the
following from df:

Filesystem           1k-blocks      Used Available Use% Mounted on
argo4-4:/pvfs-meta    48710144  48638336     71808 100% /pvfs-scratch3

Notice the available size, that should be closer to 100GB. And the "Used"
is completely wrong since there is nothing in /pvfs-scratch3. When I look
at the IO servers on each of the four machines, I get the correct size:

/dev/hda11            24098424       424  22873852   1% /pvfs-data
/dev/hda11            24098424        20  22874256   1% /pvfs-data
/dev/hda11            24098424       424  22873852   1% /pvfs-data
/dev/hda12            24098424        20  22874256   1% /pvfs-data

There is a step at this point that I think I've forgotten. Two of the data
filesystems contains just subdirectories( which is what it should be); the
other two filesystems have nothing in them. If memory serves, there is a
step to initialize the IO servers which creates the subdirectores.

Could someone refresh my memory?

Michael Homa
Academic Computing and Communication Center
University of Illinois at Chicago
email:  mhoma at uic.edu