记一次离奇的TNS-12545 TNS-12560 TNS-00515

最近reportDB监听无法随系统自启动，现象比较怪异。因为该服务器上的另一个实例的监听可以正常启动，这个不能自启动实例的监听手动启动又是正常的。因此记下这次离奇暂未找到原因的故障。

1、故障现象

Starting CRON daemondone
Oracle 10g auto start/stop
Startup "USMTHLY" listener.

LSNRCTL for Linux: Version 10.2.0.3.0 - Production on 08-JUL-2014 17:05:49

Copyright (c) 1991, 2006, Oracle.  All rights reserved.

Starting /users/oracle/OraHome10g/bin/tnslsnr: please wait...

TNSLSNR for Linux: Version 10.2.0.3.0 - Production
System parameter file is /users/oracle/OraHome10g/network/admin/listener.ora
Log messages written to /users/oracle/OraHome10g/network/log/listener_usmthly.log
Error listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=10.101.1.106)(PORT=1520)))
TNS-12545: Connect failed because target host or object does not exist
 TNS-12560: TNS:protocol adapter error
  TNS-00515: Connect failed because target host or object does not exist
   Linux Error: 99: Cannot assign requested address

#上面的错误信息了来自系统启动日志，同样出现在该实例的监听器日志里。   

#故障环境
oracle@sysreportDB:/users/oracle> cat /etc/issue

Welcome to SUSE Linux Enterprise Server 10 SP4  (x86_64) - Kernel \r (\l).

oracle@sysreportDB:/users/oracle> sqlplus -v

SQL*Plus: Release 10.2.0.3.0 - Production

#服务器启动后，检查数据库实例，2个实例都已成功启动
oracle@sysreportDB:~> ps -ef | grep pmon | grep -v grep
oracle    4916     1  0 Jul08 ?        00:00:00 ora_pmon_USMTHLY
oracle    5605     1  0 Jul08 ?        00:00:00 ora_pmon_USBO2

oracle@sysreportDB:~> ps -ef | grep lsnr| grep -v grep      #仅仅只有usbo2的监听被启动
oracle    5597     1  0 Jul09 ?        00:00:00 /users/oracle/OraHome10g/bin/tnslsnr listener_USBO2 -inherit

2、故障分析

oracle@sysreportDB:~> oerr tns 12545
12545, 00000, "Connect failed because target host or object does not exist"
// *Cause: The address specified is not valid, or the program being 
// connected to does not exist.
// *Action: Ensure the ADDRESS parameters have been entered correctly; the
// most likely incorrect parameter is the node name.  Ensure that the 
// executable for the server exists (perhaps "oracle" is missing.)
// If the protocol is TCP/IP, edit the TNSNAMES.ORA file to change the
// host name to a numeric IP address and try again.
oracle@sysreportDB:~> oerr tns 12560
12560, 00000, "TNS:protocol adapter error"
// *Cause: A generic protocol adapter error occurred.
// *Action: Check addresses used for proper protocol specification. Before
// reporting this error, look at the error stack and check for lower level
// transport errors. For further details, turn on tracing and reexecute the
// operation. Turn off tracing when the operation is complete.
oracle@sysreportDB:~> oerr tns 00515
00515, 00000, "Connect failed because target host or object does not exist"
// *Cause: The address specified is not valid, or the program being 
// connected to does not exist.
// *Action: Ensure the ADDRESS parameters have been entered correctly; the
// most likely incorrect parameter is the node name.  Ensure that the 
// executable for the server exists (perhaps "oracle" is missing.)

#上面的三个错误号描述信息表明该故障与主机名或IP地址有关，需要检查是否为有效的IP或主机名
#下面查看监听器及TNSNAMES的配置
oracle@sysreportDB:/users/oracle/OraHome10g/network/admin> more listener.ora
SID_LIST_LISTENER_USMTHLY =
   (SID_LIST =
    (SID_DESC =
        (sid_name=USMTHLY)
        (oracle_home = /users/oracle/OraHome10g)
    )
  )

LISTENER_USMTHLY =
  (DESCRIPTION_LIST =
    (DESCRIPTION =
      (ADDRESS_LIST =
        (ADDRESS = (PROTOCOL = TCP)(HOST = 10.101.1.106)(PORT = 1520))
      )
    )
  )

SID_LIST_LISTENER_USBO2 =
   (SID_LIST =
    (SID_DESC =
        (sid_name=USBO2)
        (oracle_home = /users/oracle/OraHome10g)
    )
  )

LISTENER_USBO2 =
  (DESCRIPTION_LIST =
    (DESCRIPTION =
      (ADDRESS_LIST =
        (ADDRESS = (PROTOCOL = TCP)(HOST = 10.101.1.106)(PORT = 1504))
      )
    )
  )

oracle@sysreportDB:/users/oracle/OraHome10g/network/admin> more tnsnames.ora
USMTHLY=
  (DESCRIPTION=
    (ADDRESS=
      (PROTOCOL=TCP)
      (HOST=10.101.1.106)
      (PORT=1520)
    )
    (CONNECT_DATA=
      (SERVICE_NAME=USMTHLY)
    )
  )

USBO2=
  (DESCRIPTION=
    (ADDRESS=
      (PROTOCOL=TCP)
      (HOST=10.101.1.106)
      (PORT=1504)
    )
    (CONNECT_DATA=
      (SERVICE_NAME=USBO2)
    )
  )
#从上面的配置来看，配置无异常。
#下面查看本机host及IP信息，从下面的描述来看，主机host配置无异常
oracle@sysreportDB:~> more /etc/hosts
127.0.0.1       localhost

# special IPv6 addresses
::1             localhost ipv6-localhost ipv6-loopback

fe00::0         ipv6-localnet
ff00::0         ipv6-mcastprefix
ff02::1         ipv6-allnodes
ff02::2         ipv6-allrouters
ff02::3         ipv6-allhosts
10.101.1.106    sysreportDB.2goasp.com sysreportDB  

oracle@sysreportDB:~> hostname
sysreportDB
# Author : Leshami
# Blog   : http://blog.csdn.net/leshami

#下面是其IP信息
oracle@sysreportDB:/var/log> ifconfig
eth0      Link encap:Ethernet  HWaddr 00:0C:29:70:82:63
          inet addr:10.101.1.106  Bcast:10.201.1.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:5763651 errors:0 dropped:0 overruns:0 frame:0
          TX packets:802254 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:8102327746 (7726.9 Mb)  TX bytes:88597013 (84.4 Mb)

lo        Link encap:Local Loopback      #有环路能正常TX,RX
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:92972 errors:0 dropped:0 overruns:0 frame:0
          TX packets:92972 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:4993723 (4.7 Mb)  TX bytes:4993723 (4.7 Mb)

#尝试手动启动监听，手动启动监听无异常    
oracle@sysreportDB:/var/log> lsnrctl start listener_USMTHLY

LSNRCTL for Linux: Version 10.2.0.3.0 - Production on 08-JUL-2014 17:10:49

Copyright (c) 1991, 2006, Oracle.  All rights reserved.

Starting /users/oracle/OraHome10g/bin/tnslsnr: please wait...

TNSLSNR for Linux: Version 10.2.0.3.0 - Production
System parameter file is /users/oracle/OraHome10g/network/admin/listener.ora
Log messages written to /users/oracle/OraHome10g/network/log/listener_usmthly.log
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=sysreportDB.2goasp.com)(PORT=1520)))

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=sysreportDB.2goasp.com)(PORT=1520)))
STATUS of the LISTENER
------------------------
Alias                     listener_USMTHLY
Version                   TNSLSNR for Linux: Version 10.2.0.3.0 - Production
Start Date                08-JUL-2014 17:10:50
Uptime                    0 days 0 hr. 0 min. 0 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /users/oracle/OraHome10g/network/admin/listener.ora
Listener Log File         /users/oracle/OraHome10g/network/log/listener_usmthly.log
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=sysreportDB.2goasp.com)(PORT=1520)))
Services Summary...
Service "USMTHLY" has 1 instance(s).
  Instance "USMTHLY", status UNKNOWN, has 1 handler(s) for this service...
The command completed successfully

#通过上面的测试可以看出，监听器能够正常启动也就是说监听器的配置应该是OK的。

#另外一点发现，系统启动日志里边有127.0.0.2的环路，不知道怎么会有127.0.0.2
Setting up network interfaces:
    lo
    lo        IP address: 127.0.0.1/8
              IP address: 127.0.0.2/8

oracle@sysreportDB:/var/log> ping 127.0.0.2  #环路127.0.0.2 OK
PING 127.0.0.2 (127.0.0.2) 56(84) bytes of data.
64 bytes from 127.0.0.2: icmp_seq=1 ttl=64 time=0.034 ms

3、故障解决

#对于上面的问题实在是百思不得其解，就差使用sqlnet trace。
#下面尝试将监听器配置文件内IP地址使用主机名来代替，竟然成功鸟。
#如下2种方式都可以将监听随系统启动而启动，一个是主机名，一个是full主机名

#LISTENER_USMTHLY =
#  (DESCRIPTION_LIST =
#    (DESCRIPTION =
#      (ADDRESS_LIST =
#        (ADDRESS = (PROTOCOL = TCP)(HOST = sysreportDB)(PORT = 1520))
#      )
#    )
#  )

LISTENER_USMTHLY =
  (DESCRIPTION_LIST =
    (DESCRIPTION =
      (ADDRESS_LIST =
        (ADDRESS = (PROTOCOL = TCP)(HOST = sysreportDB.2goasp.com)(PORT = 1520))
      )
    )
  )

#高兴之余，查看系统日志，汗，还是有错误，这次的错误是TNS-12543，目标主机不可达。 
#怀疑是否hosts文件中是否存在异常字符，于是尝试重建Hosts文件，重建后错误消息依旧如下。
Starting CRON daemondone
Oracle 10g auto start/stop
Starting Oracle10g: Startup "USMTHLY" listener.

LSNRCTL for Linux: Version 10.2.0.3.0 - Production on 09-JUL-2014 17:15:47

Copyright (c) 1991, 2006, Oracle.  All rights reserved.

Starting /users/oracle/OraHome10g/bin/tnslsnr: please wait...

TNSLSNR for Linux: Version 10.2.0.3.0 - Production
System parameter file is /users/oracle/OraHome10g/network/admin/listener.ora
Log messages written to /users/oracle/OraHome10g/network/log/listener_usmthly.log
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=sysreportDB.2goasp.com)(PORT=1520)))

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=sysreportDB)(PORT=1520)))
TNS-12543: TNS:destination host unreachable
 TNS-12560: TNS:protocol adapter error
  TNS-00513: Destination host unreachable
   Linux Error: 101: Network is unreachable

#最终，此题暂时无解，没有找到原因，不影响自启动而已。

4、小结
a、对于TNS-12545，TNS-00515应首先考虑是否使用了合适的IP，主机名，应检查服务器host文件，网络配置等
b、Step a检查完后应考虑是否正确的配置了监听以及tnsnames
c、对于上述该类错误，应同时检查及分析监听器日志
d、对于客户端出现TNS-12545，TNS-00515等，考虑客户端能否将主机转换成对应的IP(客户端tnsnames.ora使用主机名的情形时)
e、有关Oracle网络配置相关可以参考：http://blog.csdn.net/leshami/article/category/828434