TroubleShooting
[Tuxedo] 테스트 / Can't send request to service TOUPPER Tperrno = 10
최선을 다하자!
2022. 11. 17. 17:44
테스트의 취지
클라이언트로 부터 request된 Service를 reply해줘야하는 어플리케이션 서버가
reply하지 못하고 계속적으로 죽는 에러가 발생하였을 때,
개발과 운영,
어느 쪽의 문제라는 점이 중요하게 된다.
위와 같은 경우, 즉 해당 에러는 Tuxedo의 문제점이라고 하기는 애매하고,
AP server 프로세스의 문제점일 가능성이 매우 높다는 점을 짐작할 수 있게 된다.
에러 로그를 확인하자.
173306.node1!BBL.55640432.1.0: LIBTUX_CAT:541: WARN: Server GROUP4/402 terminated
173306.node1!BBL.55640432.1.0: LIBTUX_CAT:557: INFO: Server GROUP4/402 being restarted
173307.node1!simpserv4.18940264.1.0: 11-17-2022: Tuxedo Version 12.2.2.0.0, 64-bit
173307.node1!simpserv4.18940264.1.0: LIBTUX_CAT:262: INFO: Standard main starting
173307.node1!simpserv4.18940264.1.0: LIBTUX_CAT:476: WARN: Server 8/402: client process 14287496: lost message
173307.node1!simpserv4.18940264.1.0: LIBTUX_CAT:477: WARN: SERVICE=TOUPPER MSG_ID=0 REASON=server died
173307.node1!simpserv4.18940264.1.0: Welcome to the simple server
173307.node1!restartsrv.20906422.1.-2: 11-17-2022: Tuxedo Version 12.2.2.0.0, 64-bit
173307.node1!restartsrv.20906422.1.-2: server GROUP4/402: CMDTUX_CAT:580: INFO: A server process has restarted: 18940264
서버 프로세스가 죽고 다시 살아난다.
Client가 WSNADDR로 연결한 서버 프로세스에 서비스를 요청했으나 위의 ULOG처럼 서버를 종료시키고 재기동시킨다.
왜 그럴까?
내가 node1 에서 Server 프로세스인 simpserv4.c 를 내가 일부러 건드렸다.. (나중에 내가 이해하기 쉽게 직관적으로 적음)
exit(); 함수를 tpreturn() 함수 전에 넣은 것.
그리고 UBBCONFIG의 *SERVER에서
아래처럼 restart=Y 옵션을 설정.
*SERVERS
DEFAULT: RESTART=Y
REPLYQ=Y
CLOPT="-A"
결론적으로
Server 프로세스는 배정된 WSH를 타고 들어온 Client의 서비스 request를 return하기 전에
exit 함수에 걸리게 됨으로써 서비스는 그대로 죽어버린다.
그리고 환경파일에 설정된 server restart 덕분에 다시 Server 프로세스가 재기동된다.
+
테스트를 위해서 loop 를 걸어놓으면 더 쉽게 테스트가 가능하다.
아래와 같이 쉘스크립트를 만들어 기동 시켰다.
while [ 1 ]
do
echo "=================================================="
./simpcl kill
sleep 1
done
~
# ./A
==================================================
Can't send request to service TOUPPER
Tperrno = 10
==================================================
Can't send request to service TOUPPER
Tperrno = 10
==================================================
Can't send request to service TOUPPER
Tperrno = 10
==================================================
Can't send request to service TOUPPER
Tperrno = 10
==================================================
Can't send request to service TOUPPER
Tperrno = 10
==================================================
Can't send request to service TOUPPER
Tperrno = 10
==================================================
Can't send request to service TOUPPER
Tperrno = 10
==================================================
서버에서 찍히는 Error
175207.node1!BBL.55640432.1.0: LIBTUX_CAT:541: WARN: Server GROUP4/403 terminated
175207.node1!BBL.55640432.1.0: LIBTUX_CAT:557: INFO: Server GROUP4/403 being restarted
175207.node1!simpserv4.15598116.1.0: 11-17-2022: Tuxedo Version 12.2.2.0.0, 64-bit
175207.node1!simpserv4.15598116.1.0: LIBTUX_CAT:262: INFO: Standard main starting
175207.node1!simpserv4.15598116.1.0: LIBTUX_CAT:476: WARN: Server 8/403: client process 29032706: lost message
175207.node1!simpserv4.15598116.1.0: LIBTUX_CAT:477: WARN: SERVICE=TOUPPER MSG_ID=0 REASON=server died
175207.node1!simpserv4.15598116.1.0: Welcome to the simple server
175207.node1!restartsrv.38470000.1.-2: 11-17-2022: Tuxedo Version 12.2.2.0.0, 64-bit
175207.node1!restartsrv.38470000.1.-2: server GROUP4/403: CMDTUX_CAT:580: INFO: A server process has restarted: 15598116
175237.node1!BBL.55640432.1.0: LIBTUX_CAT:541: WARN: Server GROUP4/402 terminated
175237.node1!BBL.55640432.1.0: LIBTUX_CAT:557: INFO: Server GROUP4/402 being restarted
175237.node1!simpserv4.7471764.1.0: 11-17-2022: Tuxedo Version 12.2.2.0.0, 64-bit
175237.node1!simpserv4.7471764.1.0: LIBTUX_CAT:262: INFO: Standard main starting
175237.node1!simpserv4.7471764.1.0: LIBTUX_CAT:476: WARN: Server 8/402: client process 14287496: lost message
175237.node1!simpserv4.7471764.1.0: LIBTUX_CAT:477: WARN: SERVICE=TOUPPER MSG_ID=0 REASON=server died
175237.node1!simpserv4.7471764.1.0: Welcome to the simple server
175237.node1!restartsrv.64684350.1.-2: 11-17-2022: Tuxedo Version 12.2.2.0.0, 64-bit
175237.node1!restartsrv.64684350.1.-2: server GROUP4/402: CMDTUX_CAT:580: INFO: A server process has restarted: 7471764
175307.node1!BBL.55640432.1.0: LIBTUX_CAT:541: WARN: Server GROUP4/401 terminated
175307.node1!BBL.55640432.1.0: LIBTUX_CAT:557: INFO: Server GROUP4/401 being restarted
175307.node1!simpserv4.14221870.1.0: 11-17-2022: Tuxedo Version 12.2.2.0.0, 64-bit
175307.node1!simpserv4.14221870.1.0: LIBTUX_CAT:262: INFO: Standard main starting
175307.node1!simpserv4.14221870.1.0: LIBTUX_CAT:476: WARN: Server 8/401: client process 29032706: lost message
175307.node1!simpserv4.14221870.1.0: LIBTUX_CAT:477: WARN: SERVICE=TOUPPER MSG_ID=0 REASON=server died
175307.node1!simpserv4.14221870.1.0: Welcome to the simple server
175307.node1!restartsrv.36831570.1.-2: 11-17-2022: Tuxedo Version 12.2.2.0.0, 64-bit
175307.node1!restartsrv.36831570.1.-2: server GROUP4/401: CMDTUX_CAT:580: INFO: A server process has restarted: 14221870
175337.node1!BBL.55640432.1.0: LIBTUX_CAT:541: WARN: Server GROUP4/400 terminated
175337.node1!BBL.55640432.1.0: LIBTUX_CAT:557: INFO: Server GROUP4/400 being restarted
175337.node1!simpserv4.15074014.1.0: 11-17-2022: Tuxedo Version 12.2.2.0.0, 64-bit
175337.node1!simpserv4.15074014.1.0: LIBTUX_CAT:262: INFO: Standard main starting
175337.node1!simpserv4.15074014.1.0: LIBTUX_CAT:476: WARN: Server 8/400: client process 14287496: lost message
175337.node1!simpserv4.15074014.1.0: LIBTUX_CAT:477: WARN: SERVICE=TOUPPER MSG_ID=0 REASON=server died
175337.node1!simpserv4.15074014.1.0: Welcome to the simple server
175337.node1!restartsrv.12649134.1.-2: 11-17-2022: Tuxedo Version 12.2.2.0.0, 64-bit
175337.node1!restartsrv.12649134.1.-2: server GROUP4/400: CMDTUX_CAT:580: INFO: A server process has restarted: 15074014
175407.node1!BBL.55640432.1.0: LIBTUX_CAT:541: WARN: Server GROUP4/403 terminated
175407.node1!BBL.55640432.1.0: LIBTUX_CAT:557: INFO: Server GROUP4/403 being restarted
175407.node1!simpserv4.61604136.1.0: 11-17-2022: Tuxedo Version 12.2.2.0.0, 64-bit
175407.node1!simpserv4.61604136.1.0: LIBTUX_CAT:262: INFO: Standard main starting
175407.node1!simpserv4.61604136.1.0: LIBTUX_CAT:476: WARN: Server 8/403: client process 29032706: lost message
175407.node1!simpserv4.61604136.1.0: LIBTUX_CAT:477: WARN: SERVICE=TOUPPER MSG_ID=0 REASON=server died
175407.node1!simpserv4.61604136.1.0: Welcome to the simple server
175407.node1!restartsrv.65929724.1.-2: 11-17-2022: Tuxedo Version 12.2.2.0.0, 64-bit
175407.node1!restartsrv.65929724.1.-2: server GROUP4/403: CMDTUX_CAT:580: INFO: A server process has restarted: 61604136
175437.node1!BBL.55640432.1.0: LIBTUX_CAT:541: WARN: Server GROUP4/402 terminated
175437.node1!BBL.55640432.1.0: LIBTUX_CAT:557: INFO: Server GROUP4/402 being restarted
175437.node1!simpserv4.21168424.1.0: 11-17-2022: Tuxedo Version 12.2.2.0.0, 64-bit
175437.node1!simpserv4.21168424.1.0: LIBTUX_CAT:262: INFO: Standard main starting
175437.node1!simpserv4.21168424.1.0: LIBTUX_CAT:476: WARN: Server 8/402: client process 14287496: lost message
175437.node1!simpserv4.21168424.1.0: LIBTUX_CAT:477: WARN: SERVICE=TOUPPER MSG_ID=0 REASON=server died
175437.node1!simpserv4.21168424.1.0: Welcome to the simple server
175437.node1!restartsrv.57475382.1.-2: 11-17-2022: Tuxedo Version 12.2.2.0.0, 64-bit
175437.node1!restartsrv.57475382.1.-2: server GROUP4/402: CMDTUX_CAT:580: INFO: A server process has restarted: 21168424
LDBAL=N 에 의해 (MSSQ)
round-robin으로 Server Application에 서비스가 수행되는 중
psr
Prog Name Queue Name 2ndQueue Name Grp Name ID RqDone Load Done Current Service
--------- ---------- ---------- -------- -- ------ --------- ---------------
simpserv4 test4 GROUP4 400 15 750 ( IDLE )
WSL 00001.02000 WSLGRP1 2000 0 0 ( IDLE )
BBL 242445 AIX1 0 4 200 ( IDLE )
simpserv4 test4 GROUP4 401 14 700 ( IDLE )
simpserv4 test4 GROUP4 402 14 700 ( IDLE )
simpserv4 test4 GROUP4 403 14 700 TOUPPER