가상머신에서 네트워크카드를 추가하는것을 배웠다. (http://heni.tistory.com/27)
그냥 add하면 되네!!!! 으악~~~~ㅠ0ㅠ
어제까지의 삽질은 잊고 한번 해본다!



확인해보고자 하는 것은, 골디락스DB에서 glocator, gagent를 구동하고 싶을 때
몇번 PORT를 오픈해놔야 잘 동작하게 할 것인가?


가상 머신을 3개 구동한다.
( heninet1 / heninet2 / heninet3 )

heninet2, heninet3장비는 DB와 gagent를 따로 라인잡아서 띄워야 안전하기 때문에,
머신에 네트워크카드를 두개 세팅해 둔 상태.

heninet1 ) 192.168.0.152
heninet2 ) 192.168.0.160  /  192.168.0.161
heninet3 ) 192.168.0.162  /  192.168.0.166



테스트 환경

골디락스 Release Venus.3.1.4 revision(24256)
CentOS 7.4

1) heninet1(152) : glocator 구동. 방화벽 포트는 42581만 오픈
2) heninet2-1(160) : cserver 구동 (master)
     heninet2-2(161) : gagent 구동
     방화벽 포트는 10101-10134 / 22581 / 43581(udp)만 오픈
3) heninet3-1(162) : cserver 구동
     heninet3-2(166) : gagnet 구동
     방화벽 포트는 10101-10134 / 22581 / 43581(udp)만 오픈

시나리오
1) heninet2-1 down -> 클러스터 라인이 끊김.
예상결과 : heninet3-1 인스턴스 내려감. rebalance 가능

2) heninet3-1 down -> 클러스터 라인이 끊김.
예상결과 : heninet3-1 인스턴스 내려감. rebalance 가능

일단 이렇게만.
진짜 42581/udp, 43581/udp만 열어둬도 glocator와 gagent가 잘 작동하는지를 보기위함이다.
잘 동작한다면 골디락스 클러스터 환경에서 네트워크 단절 시, split brain 현상이 방지될꺼임!



heninet2, heninet3 두 장비에 골디락스 클러스터 포트 허용
# firewall-cmd --zone=public --add-port=10101-10134/tcp

클러스터 DB 생성
[heni@heninet2 ~]$ gcreatedb --cluster --member G1N1 --host 192.168.0.160 --port 10101
[heni@heninet3 ~]$ gcreatedb --cluster --member G1N2 --host '192.168.0.162' --port 10101

gSQL> create cluster group G1 cluster member G1N1 host '192.168.0.160' port 10101 ;
Cluster Group created.

gSQL> alter cluster group G1 add cluster member G1N2 host '192.168.0.162' port 10101;
Cluster Group altered.

(이렇게 묶은뒤에 x$cluster_connection으로 확인해보니, 이상하게 166번으로 포트를 잡고있어서
디비 내린 뒤 161, 166번 (gagent붙일) 네트워크 내린 다음 다시 DB올리니까 원하는대로 커넥션 잡혀있다.)

heninet1 머신에 glocator포트 42581/udp 허용
[root@heninet1 ~]# firewall-cmd --zone=public --add-port=42581/tcp
success

heninet2, heninet3 머신에 gagent포트 43581/udp 허용
[root@heninet2 ~]# firewall-cmd --zone=public --add-port=43581/udp
success

[root@heninet3 ~]# firewall-cmd --zone=public --add-port=43581/udp
success



glocator, gagent 구동
[heni@heninet1 trc]$ glocator --status

Copyright (C) 2010 SUNJESOFT Inc. All rights reserved.
Release Venus.3.1.4 revision(24256)

Process ID: 4146
Configuration file:
Unix domain path: /tmp/unix-glocator.42581
Udp listen port: 42581
glocator is running.
[heni@heninet1 trc]$ lsof -i
COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
glocator 4146 heni    6u  IPv4  47378      0t0  UDP *:42581

[heni@heninet2 ~]$ gagent --status

Copyright (C) 2010 SUNJESOFT Inc. All rights reserved.
Release Venus.3.1.4 revision(24256)

Process ID: 40262
Configuration file: test_gagent.conf
Udp listen port: 43581
gagent is running.
[heni@heninet2 ~]$ cat test_gagent.conf
[AGENT]
HOST = 192.168.0.161
PORT = 43581
LOCATOR_HOST = 192.168.0.152
LOCATOR_PORT = 42581
[heni@heninet2 ~]$ lsof -i
COMMAND     PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
gmaster   39698 heni    4u  IPv4  94268      0t0  TCP *:10134 (LISTEN)
cdispatch 39711 heni    5u  IPv4  94270      0t0  UDP *:10133
cdispatch 39711 heni    7u  IPv4  91405      0t0  TCP *:ezmeeting-2 (LISTEN)
cdispatch 39711 heni    9u  IPv4  91406      0t0  TCP *:ezproxy-2 (LISTEN)
cdispatch 39711 heni   10u  IPv4  91407      0t0  TCP heninet2:51436->192.168.0.162:ezmeeting-2 (CLOSE_WAIT)
cdispatch 39711 heni   11u  IPv4  92836      0t0  TCP heninet2:ezmeeting-2->192.168.0.162:57996 (ESTABLISHED)
cdispatch 39711 heni   12u  IPv4  92837      0t0  TCP heninet2:51442->192.168.0.162:ezmeeting-2 (ESTABLISHED)
cdispatch 39711 heni   13u  IPv4  93936      0t0  TCP heninet2:ezproxy-2->192.168.0.162:48064 (ESTABLISHED)
cdispatch 39711 heni   14u  IPv4  93937      0t0  TCP heninet2:35388->192.168.0.162:ezproxy-2 (ESTABLISHED)
gagent    40262 heni    3u  IPv4  99899      0t0  UDP *:43581
gagent    40262 heni    5u  IPv4  97643      0t0  UDP *:47164
[heni@heninet2 ~]$ netstat -ntuap | grep 'cdispatcher\|gmaster\|gagent'
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp        0      0 0.0.0.0:10101           0.0.0.0:*               LISTEN      39711/cdispatcher   
tcp        0      0 0.0.0.0:10102           0.0.0.0:*               LISTEN      39711/cdispatcher   
tcp        0      0 0.0.0.0:10134           0.0.0.0:*               LISTEN      39698/gmaster       
tcp        0      0 192.168.0.160:35388     192.168.0.162:10102     ESTABLISHED 39711/cdispatcher   
tcp        0      0 192.168.0.160:10102     192.168.0.162:48064     ESTABLISHED 39711/cdispatcher   
tcp        1      0 192.168.0.160:51436     192.168.0.162:10101     CLOSE_WAIT  39711/cdispatcher   
tcp        0      0 192.168.0.160:10101     192.168.0.162:57996     ESTABLISHED 39711/cdispatcher   
tcp        0      0 192.168.0.160:51442     192.168.0.162:10101     ESTABLISHED 39711/cdispatcher   
udp        0      0 0.0.0.0:43581           0.0.0.0:*                           40262/gagent        
udp        0      0 0.0.0.0:10133           0.0.0.0:*                           39711/cdispatcher   
udp        0      0 0.0.0.0:47164           0.0.0.0:*                           40262/gagent        

[heni@heninet3 ~]$ gagent --status

Copyright (C) 2010 SUNJESOFT Inc. All rights reserved.
Release Venus.3.1.4 revision(24256)

Process ID: 40195
Configuration file: test_gagent.conf
Udp listen port: 43581
gagent is running.
[heni@heninet3 ~]$ cat test_gagent.conf
[AGENT]
HOST = 192.168.0.166
PORT = 43581
LOCATOR_HOST = 192.168.0.152
LOCATOR_PORT = 42581
[heni@heninet3 ~]$ lsof -i
COMMAND     PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
gmaster   39639 heni    4u  IPv4  93209      0t0  TCP *:10134 (LISTEN)
cdispatch 39652 heni    5u  IPv4  92275      0t0  UDP *:10133
cdispatch 39652 heni    7u  IPv4  84970      0t0  TCP *:ezmeeting-2 (LISTEN)
cdispatch 39652 heni    9u  IPv4  84971      0t0  TCP *:ezproxy-2 (LISTEN)
cdispatch 39652 heni   10u  IPv4  86934      0t0  TCP heninet3:57996->192.168.0.160:ezmeeting-2 (ESTABLISHED)
cdispatch 39652 heni   11u  IPv4  86935      0t0  TCP heninet3:48064->192.168.0.160:ezproxy-2 (ESTABLISHED)
cdispatch 39652 heni   12u  IPv4  93396      0t0  TCP heninet3:ezmeeting-2->192.168.0.160:51442 (ESTABLISHED)
cdispatch 39652 heni   13u  IPv4  84990      0t0  TCP heninet3:ezproxy-2->192.168.0.160:35388 (ESTABLISHED)
gagent    40195 heni    4u  IPv4  95928      0t0  UDP *:43581
gagent    40195 heni    5u  IPv4  99077      0t0  UDP *:59203
[heni@heninet3 ~]$ netstat -ntuap | grep 'cdispatcher\|gmaster\|gagent'
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp        0      0 0.0.0.0:10101           0.0.0.0:*               LISTEN      39652/cdispatcher   
tcp        0      0 0.0.0.0:10102           0.0.0.0:*               LISTEN      39652/cdispatcher   
tcp        0      0 0.0.0.0:10134           0.0.0.0:*               LISTEN      39639/gmaster       
tcp        0      0 192.168.0.162:57996     192.168.0.160:10101     ESTABLISHED 39652/cdispatcher   
tcp        0      0 192.168.0.162:10102     192.168.0.160:35388     ESTABLISHED 39652/cdispatcher   
tcp        0      0 192.168.0.162:48064     192.168.0.160:10102     ESTABLISHED 39652/cdispatcher   
tcp        0      0 192.168.0.162:10101     192.168.0.160:51442     ESTABLISHED 39652/cdispatcher   
udp        0      0 0.0.0.0:43581           0.0.0.0:*                           40195/gagent        
udp        0      0 0.0.0.0:59203           0.0.0.0:*                           40195/gagent        
udp        0      0 0.0.0.0:10133           0.0.0.0:*                           39652/cdispatcher

현재 클러스터 테이블 T1 상태
gSQL> select * from t1;

ID NAME
-- ----
1  a   

1 row selected.

trc로그도 잘 올라감을 확인했고, 
CLUSTER_SPLIT_BRAIN_RESOLUTION_POLICY = 2 로 설정되어 있다. ( 이 프로퍼티는 0, 1, 2 중 하나를 설정할 수 있음. 나중에 글 적음)



[시나리오1] heninet2-1 down -> 클러스터 라인 단절
예상결과 : heninet3-1 인스턴스 내려감. rebalance 가능

160번 카드 내림 (클러스터 단절)
[root@heninet2 ~]# ifdown ens33
Device 'ens33' successfully disconnected.

결과. 예상대로다.
heninet2 system trc
[2018-01-25 14:05:22.906276 INSTANCE(GOLDILOCKS) THREAD(39698,139847254386432)] [INFORMATION]
[FAILOVER] end offline member

heninet3 system trc
[2018-01-25 14:05:01.829294 INSTANCE(GOLDILOCKS) THREAD(39639,139946915321664)] [INFORMATION]
[EVENT] system shutdown : SUCCESS

이 상태에서 디비 상태 확인한다.

heninet2 (master)
cserver, gagent 잘 살아있음.

gSQL> select * from x$cluster_connection;

ERR-HY000(16354): connection of member 'G1N2' is broken

heninet3 (slave)
cserver는 종료되었으나, gagnet는 살아있음 (db에 붙어있다 -> 공유메모리 dest)

[heni@heninet3 ~]$ gsql --as sysdba

Copyright © 2010 SUNJESOFT Inc. All rights reserved.
Release Venus.3.1.4 revision(24256)

Enter user-name: sys
Enter password:

Connected to an idle instance.


살아있는 마스터디비에서(heninet2) 데이터변경 수행

ID NAME
-- ----
1  a   

1 row selected.

gSQL> insert into t1 values (2, 'b');

1 row created.

gSQL> update t1 set name='11' where name='a';

1 row updated.

gSQL> commit;

Commit complete.

gSQL> select * from t1;

ID NAME
-- ----
1  11  
2  b   

2 rows selected.


rebalance 확인. 성공
gSQL> alter system join database;

ERR-42000(16405): of the total '1' tables in the database, '1' tables need to be rebalanced
System altered.

gSQL> alter database rebalance;

Database altered.

gSQL> select * from t1;

ID NAME
-- ----
1  11  
2  b   

2 rows selected.


**
종료 된 DB에 붙어있던 gagent를 stop해주어야 dest상태의 공유메모리 제거 됨



[시나리오2] heninet3-1 down (현재 마스터 heninet2)
예상결과 : heninet3-1 인스턴스 내려감. rebalance 가능

[root@heninet3 ~]# ifdown ens33
Device 'ens33' successfully disconnected

결과. 예상대로임. slave가 죽었다!

heninet2 (master)
[2018-01-25 14:32:18.774838 INSTANCE(GOLDILOCKS) THREAD(41288,140027638830848)] [INFORMATION]
[FAILOVER] end offline member

heninet3 (slave)
[2018-01-25 14:32:14.694794 INSTANCE(GOLDILOCKS) THREAD(40951,140565458782016)] [INFORMATION]
[EVENT] system shutdown : SUCCESS

각 노드 DB상태는 시나리오1)과 동일

heninet2에서 데이터 변경 수행
gSQL> select * from t1;

ID NAME
-- ----
1  a   

1 row selected.

gSQL> insert into t1 values (2, 'b');

1 row created.

gSQL> update t1 set name='aa' where name='a';

1 row updated.

gSQL> select * from t1;

ID NAME
-- ----
1  aa  
2  b   

2 rows selected.

gSQL> commit;

Commit complete.



heninet3 디비 구동 후 rebalance

gSQL> startup

Startup success

gSQL> alter system join database;

ERR-42000(16405): of the total '1' tables in the database, '1' tables need to be rebalanced
System altered.

gSQL> alter database rebalance;

Database altered.

gSQL> select * from t1;

ID NAME
-- ----
1  aa  
2  b   

2 rows selected.



결론. glocator, gagent 포트는 각각 42581, 43581(UDP)만 허용하면 잘 동작한다!
# firewall-cmd --list-all
public (active)
  target: default
  icmp-block-inversion: no
  interfaces: ens34 ens33
  sources:
  services: ssh dhcpv6-client
  ports: 10101-10134/tcp 43581/udp 22581/tcp
  protocols:
  masquerade: no
  forward-ports:
  source-ports:
  icmp-blocks:
  rich rules:
위는 DB와 gagent가 구동되고있는 서버의 포트상태 예시 ( 10101-10134 클러스터 / 43581 gagent / 22581 glsnr )


+ Recent posts