故障排除和性能優(yōu)化案例

@@&鍛造凝練IT服務(wù)助推用戶事業(yè)發(fā)展$sqlplus–prelim“/asSQL>oradebughangyzeSQL>oradebughangyzeSQL>oradebugdumpsystemstateSQL>oradebugdumpsystemstate鍛造凝練ITFound76objectswaitingfor /latch:sharedFound22objectswaitingfor /latch:sharedFound11objectswaitingforFound13objectswaitingfor鍛造凝練IT 鍛造凝練IT分析essor):[597]/0/598/1/0xfac2128/553382/LEAF_NW/3/4//493 鍛造凝練ITWedNov1614:26:16StartingORACLEinstancePSP0startedwithpid=3,OSMMANstartedwithpid=4,OS鍛造凝練IT鍛造凝練ITStatPerElapsedTimeCPUTimeBufferDiskParseUserI/OWaitTimeClusterWaitTimeApplicationWaitTime0鍛造凝練IT鍛造凝練IT鍛造凝練ITid=9和id=19的步驟做hashjoin,1行和330萬行表id=9和id=19的步驟做hashjoin,1行和330萬行表鍛造凝練IT鍛造凝練ITcreateindexidx_tb2_idontb2(id)tablespace鍛造凝練IT鍛造凝練IT 群節(jié)點2所在的P595發(fā)生硬件故障，導(dǎo)致節(jié)點2–Sqlplusassysdba故障時刻節(jié)點1–Sqlplus–prelimassysdba”故障時刻節(jié)點1該命需要核查問題原因，以便對系統(tǒng)高可用架構(gòu)提出改進意見鍛造凝練ITORACLERAC/ORACLECRS/IBMHACMP鍛造凝練IT確認ORACLERACTueMay2016:01:30BEISTReconfigurationstarted(oldinc8,newinc10)Listofnodes:0TueMay2016:01:32BEISTSubmittedallGCSremote-cacherequestsFixwriteingcsresourcesReconfigurationcomplete鍛造凝練IT確認ORACLECRS CSSD]2014-05-2016:01:25.885>WARNING:clssnmPollingThread:nodexx2(2)at90%fatal,evictionin1.427secondsseedhbimpd CSSD]2014-05-2016:01:26.885[3343]clssnmPollingThread:nodexx2(2)at90%heartbeatfatal,eviction0.427secondsseedhbimpd CSSD]2014-05-2016:01:27.316[3343]clssnmPollingThread:Evictionstartedfornodexx2(2),flags0x040d,state3,wt4c0seedhbimpd1……… CSSD]2014-05-2016:01:27.316[1801]clssnmDeactivateNode:node2(xx2)left 鍛造凝練IT確認IBMhacmp鍛造凝練IT檢查1節(jié)點crs日志確認節(jié)點2vip2014-05-2016:02:31.357:[CRSEVT][12378]32CAAMonitorHandler::0:Action t)timedoutforora.node2.vip!(timeout=60)2014-05-2016:02:31.357:[CRSAPP][12378]32StartResourceerrorforora.node2.viperrorcode=-22014-05-2016:03:00.787:[CRSEVT][13408]32CAAMonitorHandler::0:Actionoutforora.node1.vip!(timeout=60)`node1`hasexperiencedanunrecoverable2014-05-2016:03:34.719:[CRSRES][12378]32Humaninterventionrequiredtoresumeitsavailability. ysisbecause'ResourceinUNKNOWNstate:2014-05-2016:21:37.316:[CRSEVT][11155]32CAAMonitorHandler::0:Action/oracle/app/oracle/product/10.2.0/crs/bin/racgwrap(check)timedoutforora.node1.vip!鍛造凝練IT節(jié)點1crstimedoutforora.node1.vip!(timeout=60)鍛造凝練IT[node1:root]:/install/nmondata>ls-lrt[node1:root]:/install/nmondata>ls-lrt1May1800:071May1900:071May2000:071May2016:011May2110:25鍛造凝練IT–Kill9被kill-9終止鍛造凝練IT鍛造凝練IT鍛造凝練IT鍛造凝練IT鍛造凝練IT鍛造凝練ITThefilesystemresultisasfollows”的檢測就停止了， PLETE關(guān)鍵字來表示鍛造凝練IT 硬件故障后，導(dǎo)致節(jié)點1無法與節(jié)點2的nfsserver通訊，繼而導(dǎo)鍛造凝練ITNfsmount 當檢查到nfs時，由于nfs以hard/background方式掛載，當nfs 鍛造凝練IT Kill9鍛造凝練IT 差節(jié)點2mount/testfs到節(jié)點1的/testfs,停止節(jié)點2nfs服務(wù)，未能重現(xiàn)。原因在于pwd的輸出為/oracle,首字母o比t要小，因此未檢測到節(jié)點2mount/aa到節(jié)