가상머신에서 네트워크카드를 추가하는것을 배웠다. (http://heni.tistory.com/27)
그냥 add하면 되네!!!! 으악~~~~ㅠ0ㅠ
어제까지의 삽질은 잊고 한번 해본다!
확인해보고자 하는 것은, 골디락스DB에서 glocator, gagent를 구동하고 싶을 때
몇번 PORT를 오픈해놔야 잘 동작하게 할 것인가?
가상 머신을 3개 구동한다.
( heninet1 / heninet2 / heninet3 )
heninet2, heninet3장비는 DB와 gagent를 따로 라인잡아서 띄워야 안전하기 때문에,
머신에 네트워크카드를 두개 세팅해 둔 상태.
heninet1 ) 192.168.0.152
heninet2 ) 192.168.0.160 / 192.168.0.161
heninet3 ) 192.168.0.162 / 192.168.0.166
테스트 환경
골디락스 Release Venus.3.1.4 revision(24256)
CentOS 7.4
1) heninet1(152) : glocator 구동. 방화벽 포트는 42581만 오픈
2) heninet2-1(160) : cserver 구동 (master)
heninet2-2(161) : gagent 구동
방화벽 포트는 10101-10134 / 22581 / 43581(udp)만 오픈
3) heninet3-1(162) : cserver 구동
heninet3-2(166) : gagnet 구동
방화벽 포트는 10101-10134 / 22581 / 43581(udp)만 오픈
시나리오
1) heninet2-1 down -> 클러스터 라인이 끊김.
예상결과 : heninet3-1 인스턴스 내려감. rebalance 가능
2) heninet3-1 down -> 클러스터 라인이 끊김.
예상결과 : heninet3-1 인스턴스 내려감. rebalance 가능
일단 이렇게만.
진짜 42581/udp, 43581/udp만 열어둬도 glocator와 gagent가 잘 작동하는지를 보기위함이다.
잘 동작한다면 골디락스 클러스터 환경에서 네트워크 단절 시, split brain 현상이 방지될꺼임!
heninet2, heninet3 두 장비에 골디락스 클러스터 포트 허용
# firewall-cmd --zone=public --add-port=10101-10134/tcp
클러스터 DB 생성
[heni@heninet2 ~]$ gcreatedb --cluster --member G1N1 --host 192.168.0.160 --port 10101
[heni@heninet3 ~]$ gcreatedb --cluster --member G1N2 --host '192.168.0.162' --port 10101
gSQL> create cluster group G1 cluster member G1N1 host '192.168.0.160' port 10101 ;
Cluster Group created.
gSQL> alter cluster group G1 add cluster member G1N2 host '192.168.0.162' port 10101;
Cluster Group altered.
(이렇게 묶은뒤에 x$cluster_connection으로 확인해보니, 이상하게 166번으로 포트를 잡고있어서
디비 내린 뒤 161, 166번 (gagent붙일) 네트워크 내린 다음 다시 DB올리니까 원하는대로 커넥션 잡혀있다.)
heninet1 머신에 glocator포트 42581/udp 허용
[root@heninet1 ~]# firewall-cmd --zone=public --add-port=42581/tcp
success
heninet2, heninet3 머신에 gagent포트 43581/udp 허용
[root@heninet2 ~]# firewall-cmd --zone=public --add-port=43581/udp
success
[root@heninet3 ~]# firewall-cmd --zone=public --add-port=43581/udp
success
glocator, gagent 구동
[heni@heninet1 trc]$ glocator --status
Copyright (C) 2010 SUNJESOFT Inc. All rights reserved.
Release Venus.3.1.4 revision(24256)
Process ID: 4146
Configuration file:
Unix domain path: /tmp/unix-glocator.42581
Udp listen port: 42581
glocator is running.
[heni@heninet1 trc]$ lsof -i
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
glocator 4146 heni 6u IPv4 47378 0t0 UDP *:42581
[heni@heninet2 ~]$ gagent --status
Copyright (C) 2010 SUNJESOFT Inc. All rights reserved.
Release Venus.3.1.4 revision(24256)
Process ID: 40262
Configuration file: test_gagent.conf
Udp listen port: 43581
gagent is running.
[heni@heninet2 ~]$ cat test_gagent.conf
[AGENT]
HOST = 192.168.0.161
PORT = 43581
LOCATOR_HOST = 192.168.0.152
LOCATOR_PORT = 42581
[heni@heninet2 ~]$ lsof -i
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
gmaster 39698 heni 4u IPv4 94268 0t0 TCP *:10134 (LISTEN)
cdispatch 39711 heni 5u IPv4 94270 0t0 UDP *:10133
cdispatch 39711 heni 7u IPv4 91405 0t0 TCP *:ezmeeting-2 (LISTEN)
cdispatch 39711 heni 9u IPv4 91406 0t0 TCP *:ezproxy-2 (LISTEN)
cdispatch 39711 heni 10u IPv4 91407 0t0 TCP heninet2:51436->192.168.0.162:ezmeeting-2 (CLOSE_WAIT)
cdispatch 39711 heni 11u IPv4 92836 0t0 TCP heninet2:ezmeeting-2->192.168.0.162:57996 (ESTABLISHED)
cdispatch 39711 heni 12u IPv4 92837 0t0 TCP heninet2:51442->192.168.0.162:ezmeeting-2 (ESTABLISHED)
cdispatch 39711 heni 13u IPv4 93936 0t0 TCP heninet2:ezproxy-2->192.168.0.162:48064 (ESTABLISHED)
cdispatch 39711 heni 14u IPv4 93937 0t0 TCP heninet2:35388->192.168.0.162:ezproxy-2 (ESTABLISHED)
gagent 40262 heni 3u IPv4 99899 0t0 UDP *:43581
gagent 40262 heni 5u IPv4 97643 0t0 UDP *:47164
[heni@heninet2 ~]$ netstat -ntuap | grep 'cdispatcher\|gmaster\|gagent'
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp 0 0 0.0.0.0:10101 0.0.0.0:* LISTEN 39711/cdispatcher
tcp 0 0 0.0.0.0:10102 0.0.0.0:* LISTEN 39711/cdispatcher
tcp 0 0 0.0.0.0:10134 0.0.0.0:* LISTEN 39698/gmaster
tcp 0 0 192.168.0.160:35388 192.168.0.162:10102 ESTABLISHED 39711/cdispatcher
tcp 0 0 192.168.0.160:10102 192.168.0.162:48064 ESTABLISHED 39711/cdispatcher
tcp 1 0 192.168.0.160:51436 192.168.0.162:10101 CLOSE_WAIT 39711/cdispatcher
tcp 0 0 192.168.0.160:10101 192.168.0.162:57996 ESTABLISHED 39711/cdispatcher
tcp 0 0 192.168.0.160:51442 192.168.0.162:10101 ESTABLISHED 39711/cdispatcher
udp 0 0 0.0.0.0:43581 0.0.0.0:* 40262/gagent
udp 0 0 0.0.0.0:10133 0.0.0.0:* 39711/cdispatcher
udp 0 0 0.0.0.0:47164 0.0.0.0:* 40262/gagent
[heni@heninet3 ~]$ gagent --status
Copyright (C) 2010 SUNJESOFT Inc. All rights reserved.
Release Venus.3.1.4 revision(24256)
Process ID: 40195
Configuration file: test_gagent.conf
Udp listen port: 43581
gagent is running.
[heni@heninet3 ~]$ cat test_gagent.conf
[AGENT]
HOST = 192.168.0.166
PORT = 43581
LOCATOR_HOST = 192.168.0.152
LOCATOR_PORT = 42581
[heni@heninet3 ~]$ lsof -i
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
gmaster 39639 heni 4u IPv4 93209 0t0 TCP *:10134 (LISTEN)
cdispatch 39652 heni 5u IPv4 92275 0t0 UDP *:10133
cdispatch 39652 heni 7u IPv4 84970 0t0 TCP *:ezmeeting-2 (LISTEN)
cdispatch 39652 heni 9u IPv4 84971 0t0 TCP *:ezproxy-2 (LISTEN)
cdispatch 39652 heni 10u IPv4 86934 0t0 TCP heninet3:57996->192.168.0.160:ezmeeting-2 (ESTABLISHED)
cdispatch 39652 heni 11u IPv4 86935 0t0 TCP heninet3:48064->192.168.0.160:ezproxy-2 (ESTABLISHED)
cdispatch 39652 heni 12u IPv4 93396 0t0 TCP heninet3:ezmeeting-2->192.168.0.160:51442 (ESTABLISHED)
cdispatch 39652 heni 13u IPv4 84990 0t0 TCP heninet3:ezproxy-2->192.168.0.160:35388 (ESTABLISHED)
gagent 40195 heni 4u IPv4 95928 0t0 UDP *:43581
gagent 40195 heni 5u IPv4 99077 0t0 UDP *:59203
[heni@heninet3 ~]$ netstat -ntuap | grep 'cdispatcher\|gmaster\|gagent'
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp 0 0 0.0.0.0:10101 0.0.0.0:* LISTEN 39652/cdispatcher
tcp 0 0 0.0.0.0:10102 0.0.0.0:* LISTEN 39652/cdispatcher
tcp 0 0 0.0.0.0:10134 0.0.0.0:* LISTEN 39639/gmaster
tcp 0 0 192.168.0.162:57996 192.168.0.160:10101 ESTABLISHED 39652/cdispatcher
tcp 0 0 192.168.0.162:10102 192.168.0.160:35388 ESTABLISHED 39652/cdispatcher
tcp 0 0 192.168.0.162:48064 192.168.0.160:10102 ESTABLISHED 39652/cdispatcher
tcp 0 0 192.168.0.162:10101 192.168.0.160:51442 ESTABLISHED 39652/cdispatcher
udp 0 0 0.0.0.0:43581 0.0.0.0:* 40195/gagent
udp 0 0 0.0.0.0:59203 0.0.0.0:* 40195/gagent
udp 0 0 0.0.0.0:10133 0.0.0.0:* 39652/cdispatcher
현재 클러스터 테이블 T1 상태
gSQL> select * from t1;
ID NAME
-- ----
1 a
1 row selected.
trc로그도 잘 올라감을 확인했고,
CLUSTER_SPLIT_BRAIN_RESOLUTION_POLICY = 2 로 설정되어 있다. ( 이 프로퍼티는 0, 1, 2 중 하나를 설정할 수 있음. 나중에 글 적음)
[시나리오1] heninet2-1 down -> 클러스터 라인 단절
예상결과 : heninet3-1 인스턴스 내려감. rebalance 가능
160번 카드 내림 (클러스터 단절)
[root@heninet2 ~]# ifdown ens33
Device 'ens33' successfully disconnected.
결과. 예상대로다.
heninet2 system trc
[2018-01-25 14:05:22.906276 INSTANCE(GOLDILOCKS) THREAD(39698,139847254386432)] [INFORMATION]
[FAILOVER] end offline member
heninet3 system trc
[2018-01-25 14:05:01.829294 INSTANCE(GOLDILOCKS) THREAD(39639,139946915321664)] [INFORMATION]
[EVENT] system shutdown : SUCCESS
이 상태에서 디비 상태 확인한다.
heninet2 (master)
cserver, gagent 잘 살아있음.
gSQL> select * from x$cluster_connection;
ERR-HY000(16354): connection of member 'G1N2' is broken
heninet3 (slave)
cserver는 종료되었으나, gagnet는 살아있음 (db에 붙어있다 -> 공유메모리 dest)
[heni@heninet3 ~]$ gsql --as sysdba
Copyright © 2010 SUNJESOFT Inc. All rights reserved.
Release Venus.3.1.4 revision(24256)
Enter user-name: sys
Enter password:
Connected to an idle instance.
살아있는 마스터디비에서(heninet2) 데이터변경 수행
ID NAME
-- ----
1 a
1 row selected.
gSQL> insert into t1 values (2, 'b');
1 row created.
gSQL> update t1 set name='11' where name='a';
1 row updated.
gSQL> commit;
Commit complete.
gSQL> select * from t1;
ID NAME
-- ----
1 11
2 b
2 rows selected.
rebalance 확인. 성공
gSQL> alter system join database;
ERR-42000(16405): of the total '1' tables in the database, '1' tables need to be rebalanced
System altered.
gSQL> alter database rebalance;
Database altered.
gSQL> select * from t1;
ID NAME
-- ----
1 11
2 b
2 rows selected.
**
종료 된 DB에 붙어있던 gagent를 stop해주어야 dest상태의 공유메모리 제거 됨
[시나리오2] heninet3-1 down (현재 마스터 heninet2)
예상결과 : heninet3-1 인스턴스 내려감. rebalance 가능
[root@heninet3 ~]# ifdown ens33
Device 'ens33' successfully disconnected
결과. 예상대로임. slave가 죽었다!
heninet2 (master)
[2018-01-25 14:32:18.774838 INSTANCE(GOLDILOCKS) THREAD(41288,140027638830848)] [INFORMATION]
[FAILOVER] end offline member
heninet3 (slave)
[2018-01-25 14:32:14.694794 INSTANCE(GOLDILOCKS) THREAD(40951,140565458782016)] [INFORMATION]
[EVENT] system shutdown : SUCCESS
각 노드 DB상태는 시나리오1)과 동일
heninet2에서 데이터 변경 수행
gSQL> select * from t1;
ID NAME
-- ----
1 a
1 row selected.
gSQL> insert into t1 values (2, 'b');
1 row created.
gSQL> update t1 set name='aa' where name='a';
1 row updated.
gSQL> select * from t1;
ID NAME
-- ----
1 aa
2 b
2 rows selected.
gSQL> commit;
Commit complete.
heninet3 디비 구동 후 rebalance
gSQL> startup
Startup success
gSQL> alter system join database;
ERR-42000(16405): of the total '1' tables in the database, '1' tables need to be rebalanced
System altered.
gSQL> alter database rebalance;
Database altered.
gSQL> select * from t1;
ID NAME
-- ----
1 aa
2 b
2 rows selected.
결론. glocator, gagent 포트는 각각 42581, 43581(UDP)만 허용하면 잘 동작한다!
# firewall-cmd --list-all
public (active)
target: default
icmp-block-inversion: no
interfaces: ens34 ens33
sources:
services: ssh dhcpv6-client
ports: 10101-10134/tcp 43581/udp 22581/tcp
protocols:
masquerade: no
forward-ports:
source-ports:
icmp-blocks:
rich rules:
위는 DB와 gagent가 구동되고있는 서버의 포트상태 예시 ( 10101-10134 클러스터 / 43581 gagent / 22581 glsnr )