Bdtjtk

I have a Dispersed Glusterfs volume comprised of 3x bricks on 3x servers. Recently one of the servers experienced a hard drive failure and dropped out of the cluster. I am trying to replace this brick in the cluster but i cant get it to work.

First up here is the version info:

$ glusterfsd --version

glusterfs 3.13.2

Repository revision: git://git.gluster.org/glusterfs.git

Copyright (c) 2006-2016 Red Hat, Inc. <https://www.gluster.org/>

GlusterFS comes with ABSOLUTELY NO WARRANTY.

It is licensed to you under your choice of the GNU Lesser

General Public License, version 3 or any later version (LGPLv3

or later), or the GNU General Public License, version 2 (GPLv2),

in all cases as published by the Free Software Foundation.

It is running on Ubuntu 18.04.

Here is the existing info:

Volume Name: vol01

Type: Disperse

Volume ID: 061cac4d-1165-4afe-87e0-27b213ea19dc

Status: Started

Snapshot Count: 0

Number of Bricks: 1 x (2 + 1) = 3

Transport-type: tcp

Bricks:

Brick1: srv02:/srv/glusterfs/vol01/brick <-- This is the brick that died

Brick2: srv03:/srv/glusterfs/vol01/brick

Brick3: srv04:/srv/glusterfs/vol01/brick

Options Reconfigured:

nfs.disable: on

transport.address-family: inet

I wish to replace the srv02 brick with a brick from srv05 using the following:

gluster volume replace-brick vol01 srv02:/srv/glusterfs/vol01/brick srv05:/srv/glusterfs/vol01/brick commit force

However when I run this command (as root) I get this error:

volume replace-brick: failed: Pre Validation failed on srv05. brick: srv02:/srv/glusterfs/vol01/brick does not exist in volume: vol01

As far as I know it should work, srv05 is connected:

# gluster peer status

Number of Peers: 3



Hostname: srv04

Uuid: 5bbd6c69-e0a7-491c-b605-d70cb83ebc72

State: Peer in Cluster (Connected)



Hostname: srv02

Uuid: e4e856ba-61df-45eb-83bb-e2d2e799fc8d

State: Peer Rejected (Disconnected)



Hostname: srv05

Uuid: e7d098c1-7bbd-44e1-931f-034da645c6c6

State: Peer in Cluster (Connected)

As you can see srv05 is connected and in the cluster, srv02 is not and disconnected...

All the bricks are the same size on a XFS partitions. The brick on srv05 is empty.

What am I doing wrong? I would prefer not to have to dump the whole FS and rebuild it if possible...

EDIT 2019-01-01:
After following this tutorial here: https://support.rackspace.com/how-to/recover-from-a-failed-server-in-a-glusterfs-array/ to replace the dead server brick (srv02) with the new one.

The server and brick are recognized by the cluster:

# gluster volume status

Status of volume: vol01

Gluster process                             TCP Port  RDMA Port  Online  Pid

------------------------------------------------------------------------------

Brick srv02:/srv/glusterfs/vol01/brick      N/A       N/A        N       N/A

Brick srv03:/srv/glusterfs/vol01/brick      49152     0          Y       21984

Brick srv04:/srv/glusterfs/vol01/brick      49152     0          Y       16681

Self-heal Daemon on localhost               N/A       N/A        Y       2582

Self-heal Daemon on srv04                   N/A       N/A        Y       16703

Self-heal Daemon on srv03                   N/A       N/A        Y       22006

The brick however on the replacement SRV02 is not coming online!

After much searching I found this in the brick log on the new srv02:

[2019-01-01 05:50:05.727791] E [MSGID: 138001] [index.c:2349:init] 0-vol01-index: Failed to find parent dir (/srv/glusterfs/vol01/brick/.glusterfs) of index basepath /srv/glusterfs/vol01/brick/.glusterfs/indices. [No such file or directory]

Not at all sure how to fix this one as its a blank brick that I am looking to bring online and heal!

So in the end I got the brick to come online by the following in the brick volume directory:

# mkdir .glusterfs

# chmod 600 .glusterfs

# cd .glusterfs

# mkdir indices

# chmod 600 indices

# systemctl restart glusterd

The brick came online and the heal process was started with:

# gluster volume heal vol01 full

So far it seams to be functioning just fine.