系統(tǒng)常見(jiàn)故障診斷參考課件

提綱通用方法日志分析方法預(yù)備知識(shí)core分析性能問(wèn)題診斷提綱通用方法1通用方法-了解被診斷系統(tǒng)什么語(yǔ)言開(kāi)發(fā)的架構(gòu)及工作原理日志及其它輸出模塊相互間接口及調(diào)用關(guān)系評(píng)估故障診斷方法可能對(duì)系統(tǒng)帶來(lái)的影響通用方法-了解被診斷系統(tǒng)什么語(yǔ)言開(kāi)發(fā)的2通用方法-正視故障診斷的復(fù)雜性其實(shí)故障診斷是個(gè)模式匹配的過(guò)程首先確認(rèn)這是個(gè)故障一定建立在多方收集的客觀事實(shí)（現(xiàn)象、日志）基礎(chǔ)上充分分析事實(shí)，不要輕易下結(jié)論忽略偶然現(xiàn)象，最好能夠重現(xiàn)故障抓問(wèn)題主要方面，特別是一段時(shí)間沒(méi)有診斷和分析的系統(tǒng)，老問(wèn)題會(huì)帶來(lái)干擾通用方法-正視故障診斷的復(fù)雜性其實(shí)故障診斷是個(gè)模式匹配的過(guò)程3通用方法-工具和方法利用好各種診斷工具詢問(wèn)法粗略了解，要分清事實(shí)和結(jié)論日志分析法利用思維導(dǎo)圖，分析邏輯關(guān)系分段或分環(huán)境進(jìn)行測(cè)試，并記錄結(jié)果做比較敢于懷疑，但要找事實(shí)支持通用方法-工具和方法利用好各種診斷工具4日志分析方法-實(shí)現(xiàn)統(tǒng)計(jì)功能awk(regex,function)grep(regex)sedsorttruniq跟蹤系統(tǒng)和動(dòng)態(tài)庫(kù)函數(shù)調(diào)用：truss/tusc/strace<race查看系統(tǒng)日志：dmesg/errpt日志分析方法-實(shí)現(xiàn)統(tǒng)計(jì)功能awk(regex,functio5日志分析方法-實(shí)踐統(tǒng)計(jì)所有進(jìn)程占用多大內(nèi)存在aix上，統(tǒng)計(jì)代碼段有多大接上，統(tǒng)計(jì)有多少個(gè)不同的段接上，統(tǒng)計(jì)各個(gè)段的大小統(tǒng)計(jì)應(yīng)用日志中，select語(yǔ)句的平均耗時(shí)，及耗時(shí)最大的5條sql跟蹤一下系統(tǒng)調(diào)用自己來(lái)點(diǎn)？日志分析方法-實(shí)踐統(tǒng)計(jì)所有進(jìn)程占用多大內(nèi)存6預(yù)備知識(shí)-動(dòng)態(tài)鏈接庫(kù)由來(lái)和原理優(yōu)缺點(diǎn)加載PIC性能預(yù)備知識(shí)-動(dòng)態(tài)鏈接庫(kù)由來(lái)和原理7預(yù)備知識(shí)-多線程的概念預(yù)備知識(shí)-多線程的概念8預(yù)備知識(shí)-程序內(nèi)存映像預(yù)備知識(shí)-程序內(nèi)存映像9預(yù)備知識(shí)-AIX程序內(nèi)存映像A32-bitapplicationprogramrunningonthesystemhasanaddressspacethatisdividedintothefollowingsegments: 0x00000000to0x0fffffffContainsthekernel. 0x10000000to0x1fffffffContainstheapplicationprogramtext. 0x20000000to0x2fffffffContainstheapplicationprogramdata,theprocessheap,andtheapplicationstack. 0x30000000to0xcfffffffAvailableforusebysharedmemoryormmapservices. 0xd0000000to0xdfffffffContainssharedlibrarytext. 0xe0000000to0xefffffffAvailableforusebysharedmemoryormmapservices. 0xf0000000to0xffffffffContainstheapplicationsharedlibrarydata.ProcessAddressSpacein64-bitApplicationsA64-bitapplicationprogramrunningonthesystemhasanaddressspacethatisdividedintothefollowingsegments: 0x0000000000000000to0x000000000fffffffContainsthekernel. 0x00000000f0000000to0x000000000fffffffReserved. 0x0000000100000000to0x07ffffffffffffffContainstheapplicationprogramtext,applicationprogramdata,theprocessheap,andsharedmemoryormmapservices. 0x0800000000000000to0x08ffffffffffffffPrivatelyloadedobjects. 0x0900000000000000to0x09ffffffffffffffSharedlibrarytextanddata. 0x0f00000000000000to0x0fffffffffffffffApplicationstack.Someitem:Size、ResData、ResText、ResSet預(yù)備知識(shí)-AIX程序內(nèi)存映像A32-bitapplica10預(yù)備知識(shí)-HP-UX程序內(nèi)存映像32bitsapplications預(yù)備知識(shí)-HP-UX程序內(nèi)存映像32bitsapplic11預(yù)備知識(shí)-HP-UX程序內(nèi)存映像64bitsapplications預(yù)備知識(shí)-HP-UX程序內(nèi)存映像64bitsapplic12預(yù)備知識(shí)-Solaris/sparc程序內(nèi)存映像預(yù)備知識(shí)-Solaris/sparc程序內(nèi)存映像13預(yù)備知識(shí)-JVM&J2EE預(yù)備知識(shí)-JVM&J2EE14預(yù)備知識(shí)-CORBA預(yù)備知識(shí)-CORBA15預(yù)備知識(shí)-TuxedoATMI預(yù)備知識(shí)-TuxedoATMI16core分析-core的類型和原因SIGABRT(6)：異常退出，通常程序發(fā)現(xiàn)了錯(cuò)誤并調(diào)用abort，比如C++中沒(méi)有處理的exceptionSIGBUS(7)：內(nèi)存訪問(wèn)時(shí)的硬件故障，通常是地址未對(duì)齊SIGFPE(8)：浮點(diǎn)數(shù)異常，包括除0、溢出等SIGILL(4)：不正確的指令，通常是指針越界導(dǎo)致的ip非法，棧溢出等SIGIOT(6)：執(zhí)行I/O自陷SIGQUIT(3)：退出并產(chǎn)生coreSIGSEGV(11)：錯(cuò)誤的內(nèi)存地址，通常是訪問(wèn)的地址空間不存在，或者沒(méi)有讀寫(xiě)權(quán)限core分析-core的類型和原因SIGABRT(6)：異常17core分析-core文件的備份AIX設(shè)置環(huán)境變量CORE_NAMING=yes，文件名形如core.pid.ddhhmmss每個(gè)用戶的core設(shè)置在/etc/security/userSolaris利用coreadm命令，例如coreadm-icore.%f.%tHP-UX利用coreadm命令，例如coreadm-pcore.%f.%tLinux/proc/sys/kernel/core_pattern/proc/sys/kernel/core_uses_pid=1修改/etc/sysctl.conf，再sysctl-p或echo'core.%e.%p'>/proc/sys/kernel/core_patterncore分析-core文件的備份AIX18core分析-分析步驟查看core文件大小查看limit信息查看core進(jìn)程的環(huán)境變量可先嘗試pstack偶爾要收集系統(tǒng)核心參數(shù)使用調(diào)試工具（gdb/dbx/mdb）對(duì)照core時(shí)的日志編寫(xiě)驗(yàn)證程序core分析-分析步驟查看core文件大小19core分析-我們應(yīng)用的常見(jiàn)core不能分配更多的內(nèi)存（除了limit、32位，還有OS特定的限制）棧溢出（棧空間不夠，遞歸太深）abort，特別是c++的exception頭文件和實(shí)現(xiàn)不一致使用了空指針指針越界/內(nèi)存溢出多線程問(wèn)題重復(fù)freecore分析-我們應(yīng)用的常見(jiàn)core不能分配更多的內(nèi)存（除了20core分析-自己實(shí)踐從前面的類型中自己選，編寫(xiě)程序驗(yàn)證和實(shí)踐環(huán)境：86(AIX)username:lixp61(HP-UXIA64)username:lixp99(Solaris)username:lixpcore分析-自己實(shí)踐從前面的類型中自己選，編寫(xiě)程序驗(yàn)證和實(shí)21core分析-案例1(AIX5.3,32Bits)iom@:/tuxedo/iom/aioss/bin>ls

-l

core-rw-r--r--

1

iom

aigrp

270562700

Dec

30

21:35

core

iom@:/tuxedo/iom/aioss/bin>dbx

unif_exec

coreType

'help'

for

help.[using

memory

image

in

core]Segmentation

fault

in

malloc_common.extend_brk

[/usr/lib/threads/libc.a]

at

0xd0355260

($t1)0xd0355260

(extend_brk+0x20c)

90040004

stw

r0,0x4(r4)(dbx)

(dbx)

wheremalloc_common.extend_brk(internal

error:

assertion

failed

at

line

3650

in

file

frame.c??,

internal

error:

assertion

failed

at

line

3650

in

file

frame.c??,

internal

error:

assertion

failed

at

line

3650

in

file

frame.c??)

at

0xd0355260(dbx)

core分析-案例1(AIX5.3,32Bits)iom@:22core分析-案例2(Solaris10,32bits)current

thread:

t@26=>[1]

__lwp_kill(0x0,

0x6,

0x0,

0x6,

0xffbffeff,

0x0),

at

0xed34d6ac

[2]

raise(0x6,

0x0,

0xed3b5010,

0xed32cd88,

0xffffffff,

0x6),

at

0xed2e69ac

[3]

abort(0x0,

0x1,

0xefb9403c,

0xee4cc,

0xed3b34d8,

0x0),

at

0xed2c2010

[4]

free(0x6fabf0,

0x709b48,

0x0,

0x80,

0x6a,

0x80000000),

at

0xefb80b00

[5]

operator

delete(0x709b48,

0x1aaedf0,

0x1aaedfc,

0xc,

0x14e98,

0xec0ec2fc),

at

0xef95631c

[6]

std::allocator<char>::deallocate(0xea0f4f7a,

0x709b48,

0x8,

0x4f,

0x1aaeb20,

0x0),

at

0xede9ab90

[7]

std::allocator_interface<std::allocator<char>,char>::deallocate(0xea0f4f7a,

0x709b48,

0x8,

0xea0f5334,

0x7ffffc00,

0xe),

at

0xede9a678

[8]

std::basic_string<char,std::char_traits<char>,std::allocator<char>

>::__unLink(0x1aaedf0,

0xea0f5358,

0x1a9ca28,

0x87a100,

0x1a78fc8,

0x8),

at

0xede96d00

[9]

std::basic_string<char,std::char_traits<char>,std::allocator<char>

>::~basic_string(0x1aaedf0,

0x1aaedde,

0x0,

0xffffffff,

0x4f,

0x4f),

at

0xede96050

[10]

SObdCallInfo::~SObdCallInfo(0x1aaedf0,

0x0,

0x0,

0x1aaedd0,

0x0,

0x0),

at

0xede9bbc0

…………

[28]

VISThread::_start(0x10506c8,

0xea0f6000,

0x0,

0x0,

0xef8399b8,

0x1),

at

0xef8399e0

core分析-案例2(Solaris10,32bits)cu23core分析-案例320110901貴州賬務(wù)資料MDBcore問(wèn)題core分析-案例320110901貴州賬務(wù)資料MDB24性能問(wèn)題診斷-常見(jiàn)性能問(wèn)題CPU瓶頸：鎖問(wèn)題、線程切換、邏輯問(wèn)題內(nèi)存瓶頸：換頁(yè)、緩存過(guò)大、邏輯問(wèn)題網(wǎng)絡(luò)瓶頸：網(wǎng)卡、中斷、cpu綁定I/O瓶頸：底層問(wèn)題、目錄操作量大數(shù)據(jù)庫(kù)瓶頸：大數(shù)據(jù)量、索引、鎖……性能問(wèn)題診斷-常見(jiàn)性能問(wèn)題CPU瓶頸：鎖問(wèn)題、線程切換、邏輯25性能問(wèn)題診斷-一般分析過(guò)程CPU瓶頸？?jī)?nèi)存瓶頸？I/O瓶頸？網(wǎng)絡(luò)瓶頸？vmstatpslspssvmonvmstatsarnicepsiostatlslvfilemonfileplaceMoretestnetstatnfsstatnfsonoifconfignetpmonYYYYNNNN性能問(wèn)題診斷-一般分析過(guò)程CPU瓶頸？?jī)?nèi)存瓶頸？I/O瓶頸？26性能問(wèn)題診斷-關(guān)鍵點(diǎn)關(guān)鍵點(diǎn)1：應(yīng)用對(duì)資源的性能/容量要求是什么？有沒(méi)有超過(guò)？關(guān)鍵點(diǎn)2：有沒(méi)有做過(guò)基準(zhǔn)測(cè)試？關(guān)鍵點(diǎn)3：平時(shí)有沒(méi)有收集性能數(shù)據(jù)？性能問(wèn)題診斷-關(guān)鍵點(diǎn)關(guān)鍵點(diǎn)1：應(yīng)用對(duì)資源的性能/容量要求是什27性能問(wèn)題診斷-必讀書(shū)籍AIX通用編程概念：編寫(xiě)并調(diào)試程序.pdf性能管理.pdf網(wǎng)絡(luò)與通信管理.pdfHP-UXcaliper_user_guide.pdfhp-uxlinkerandlibrariesuserguideforhp-ux11.0.pdfJavaTroubleshootingGuideforHP-UXSystems.pdfTunableKernelParameters-HP-UXRelease11iVersion1.6.pdfSolarisSolaris10-Performance-AnalysisusingDynamicTracing(DTrace).pdfS10DeepDive_Performance.pdfDTraceUserGuide.pdfLinkerandLibrariesGuide.pdfMultithreadedProgrammingGuide.pdf性能問(wèn)題診斷-必讀書(shū)籍AIX通用編程概念：編寫(xiě)并調(diào)試程序.p28性能問(wèn)題診斷-AIX工具性能問(wèn)題診斷-AIX工具29性能問(wèn)題診斷-HP-UX工具性能問(wèn)題診斷-HP-UX工具30性能問(wèn)題診斷-Solaris工具性能問(wèn)題診斷-Solaris工具31性能問(wèn)題診斷-vmstat這個(gè)工具可以讓我們看到很多性能問(wèn)題！性能問(wèn)題診斷-vmstat這個(gè)工具可以讓我們看到很多性能問(wèn)題32性能問(wèn)題診斷-contextswitch資料來(lái)源：CandC++ApplicationdevelopmentonAIX.pdf,P115性能問(wèn)題診斷-contextswitch資料來(lái)源：Can33性能問(wèn)題診斷-iostatAIXHP-UXSolaris性能問(wèn)題診斷-iostatAIXHP-UXSolaris34性能問(wèn)題診斷-tcpdumptcpdump[-adeflnNOpqStvx][-c數(shù)量][-F文件名]

[-i網(wǎng)絡(luò)接口][-r文件名][-ssnaplen]

[-T類型][-w文件名][表達(dá)式](1)想要截獲所有的主機(jī)收到的和發(fā)出的所有的數(shù)據(jù)包：

#tcpdumphost(2)想要截獲主機(jī)和主機(jī)或的通信，使用命令：（在命令行中適用括號(hào)時(shí)，一定要

#tcpdumphostand\(or\)(3)如果想要獲取主機(jī)除了和主機(jī)之外所有主機(jī)通信的ip包，使用命令：

#tcpdumpiphostand!(4)如果想要獲取主機(jī)接收或發(fā)出的telnet包，使用如下命令：

#tcpdumptcpport23host進(jìn)程和端口的關(guān)系：pfiles、lsof、truss網(wǎng)卡和ip的關(guān)系：ifconfig性能問(wèn)題診斷-tcpdumptcpdump[-adefl35性能問(wèn)題診斷-netperf長(zhǎng)連接：netperf-tTCP_RR-H20---r6464短連接：netperf-tTCP_CRR-H20其它選項(xiàng)：-c-C-j性能問(wèn)題診斷-netperf長(zhǎng)連接：36性能問(wèn)題診斷-AIX64bitscout0x09000000004922e0_global_lock_common(??,??,??)+0x40c0x09000000000650f0_rec_mutex_lock(??)+0x17c0x0900000000431c5cstd::_Lock::_Wait()(??)+0x2c0x0900000000431bdcstd::_Lockit::_Lockit(int)(??,??)+0x1c0x0900000000431b7cstd::_Lockit::_Lockit(int)(??,??)+0x2c0x09000000057eb110std::basic_ostream<char,std::char_traits<char>>::sentry::sentry(std::basic_ostream<char,std::char_traits<char>>&)(0x144a443a0,0x9001000a01dd780)+0x480x09000000057e5d10std::basic_ostream<char,std::char_traits<char>>&std::operator<<<char,std::char_traits<char>,std::allocator<char>>(std::basic_ostream<char,std::char_traits<char>>&,conststd::basic_string<char,std::char_traits<char>,std::allocator<char>>&)(0x9001000a01dd780,0x144a44580)+0x10c…………0x09000000057e5784COstreamVisitor::visit(constCObdBasicListObject&,bool)(0x144a44a20,0x144a44d18,0x0)+0x3c0x09000000057e90fcoperator<<(std::basic_ostream<char,std::char_traits<char>>&,constCObdBasicListObject&)(0x9001000a01dd780,0x144a44d18)+0x6c0x090000001a8fccf0app_app_wyintf_app_busi4webfuncintf::MBusi4webFuncIntfApp::CIBusi4webFuncIntfImp::get_allProdByBillId(SOBSession*,conststd::basic_string<char,std::char_traits<char>,std::allocator<char>>&,conststd::basic_string<char,std::char_traits<char>,std::allocator<char>>&,app_app_wyintf_app_busi4webfuncintf::MBusi4webFuncIntfAppDef::SOrderInfoList&,ob::CBSErrorMsg&)(0x144a48618,0x144a49f80,0x144a47b50,0x144a47b70,0x144a47b90,0x144a479c0)+0x12d4…………0x090000000615c5f8VISTPoolWorker::begin()(0x12340afb0)+0x5c0x09000000067692a4VISThread::_start(void*)(0x12340afb0)+0x7c0x09000000004934ec_pthread_body(??)+0xdc性能問(wèn)題診斷-AIX64bitscout0x09000037性能問(wèn)題診斷-進(jìn)程占用內(nèi)存大動(dòng)態(tài)庫(kù)減小內(nèi)存占用32位AIX程序使用大數(shù)據(jù)32位還是64位?多線程少進(jìn)程?避免大數(shù)據(jù)性能作為保障定期重啟性能問(wèn)題診斷-進(jìn)程占用內(nèi)存大動(dòng)態(tài)庫(kù)減小內(nèi)存占用38性能問(wèn)題診斷-案例分享partition僵死java的VisiGate問(wèn)題gprs辦理業(yè)務(wù)慢問(wèn)題HP-UX的動(dòng)態(tài)庫(kù)執(zhí)行權(quán)限問(wèn)題Solaris10上ping問(wèn)題Solaris10上“抓”系統(tǒng)重啟IE頁(yè)面連接限制導(dǎo)致的Web白屏計(jì)費(fèi)：追話單慢啊性能問(wèn)題診斷-案例分享partition僵死39性能問(wèn)題診斷-詳細(xì)案例20080828_甘肅中間件主機(jī)CPU占用高的問(wèn)題20081006_遼寧中間件服務(wù)器cache的應(yīng)用報(bào)告20101001CMPak系統(tǒng)響應(yīng)慢的問(wèn)題性能問(wèn)題診斷-詳細(xì)案例20080828_甘肅中間件主機(jī)CPU40性能問(wèn)題診斷-大家交流總結(jié)平時(shí)工作中的案例性能問(wèn)題診斷-大家交流總結(jié)平時(shí)工作中的案例41提綱通用方法日志分析方法預(yù)備知識(shí)core分析性能問(wèn)題診斷提綱通用方法42通用方法-了解被診斷系統(tǒng)什么語(yǔ)言開(kāi)發(fā)的架構(gòu)及工作原理日志及其它輸出模塊相互間接口及調(diào)用關(guān)系評(píng)估故障診斷方法可能對(duì)系統(tǒng)帶來(lái)的影響通用方法-了解被診斷系統(tǒng)什么語(yǔ)言開(kāi)發(fā)的43通用方法-正視故障診斷的復(fù)雜性其實(shí)故障診斷是個(gè)模式匹配的過(guò)程首先確認(rèn)這是個(gè)故障一定建立在多方收集的客觀事實(shí)（現(xiàn)象、日志）基礎(chǔ)上充分分析事實(shí)，不要輕易下結(jié)論忽略偶然現(xiàn)象，最好能夠重現(xiàn)故障抓問(wèn)題主要方面，特別是一段時(shí)間沒(méi)有診斷和分析的系統(tǒng)，老問(wèn)題會(huì)帶來(lái)干擾通用方法-正視故障診斷的復(fù)雜性其實(shí)故障診斷是個(gè)模式匹配的過(guò)程44通用方法-工具和方法利用好各種診斷工具詢問(wèn)法粗略了解，要分清事實(shí)和結(jié)論日志分析法利用思維導(dǎo)圖，分析邏輯關(guān)系分段或分環(huán)境進(jìn)行測(cè)試，并記錄結(jié)果做比較敢于懷疑，但要找事實(shí)支持通用方法-工具和方法利用好各種診斷工具45日志分析方法-實(shí)現(xiàn)統(tǒng)計(jì)功能awk(regex,function)grep(regex)sedsorttruniq跟蹤系統(tǒng)和動(dòng)態(tài)庫(kù)函數(shù)調(diào)用：truss/tusc/strace<race查看系統(tǒng)日志：dmesg/errpt日志分析方法-實(shí)現(xiàn)統(tǒng)計(jì)功能awk(regex,functio46日志分析方法-實(shí)踐統(tǒng)計(jì)所有進(jìn)程占用多大內(nèi)存在aix上，統(tǒng)計(jì)代碼段有多大接上，統(tǒng)計(jì)有多少個(gè)不同的段接上，統(tǒng)計(jì)各個(gè)段的大小統(tǒng)計(jì)應(yīng)用日志中，select語(yǔ)句的平均耗時(shí)，及耗時(shí)最大的5條sql跟蹤一下系統(tǒng)調(diào)用自己來(lái)點(diǎn)？日志分析方法-實(shí)踐統(tǒng)計(jì)所有進(jìn)程占用多大內(nèi)存47預(yù)備知識(shí)-動(dòng)態(tài)鏈接庫(kù)由來(lái)和原理優(yōu)缺點(diǎn)加載PIC性能預(yù)備知識(shí)-動(dòng)態(tài)鏈接庫(kù)由來(lái)和原理48預(yù)備知識(shí)-多線程的概念預(yù)備知識(shí)-多線程的概念49預(yù)備知識(shí)-程序內(nèi)存映像預(yù)備知識(shí)-程序內(nèi)存映像50預(yù)備知識(shí)-AIX程序內(nèi)存映像A32-bitapplicationprogramrunningonthesystemhasanaddressspacethatisdividedintothefollowingsegments: 0x00000000to0x0fffffffContainsthekernel. 0x10000000to0x1fffffffContainstheapplicationprogramtext. 0x20000000to0x2fffffffContainstheapplicationprogramdata,theprocessheap,andtheapplicationstack. 0x30000000to0xcfffffffAvailableforusebysharedmemoryormmapservices. 0xd0000000to0xdfffffffContainssharedlibrarytext. 0xe0000000to0xefffffffAvailableforusebysharedmemoryormmapservices. 0xf0000000to0xffffffffContainstheapplicationsharedlibrarydata.ProcessAddressSpacein64-bitApplicationsA64-bitapplicationprogramrunningonthesystemhasanaddressspacethatisdividedintothefollowingsegments: 0x0000000000000000to0x000000000fffffffContainsthekernel. 0x00000000f0000000to0x000000000fffffffReserved. 0x0000000100000000to0x07ffffffffffffffContainstheapplicationprogramtext,applicationprogramdata,theprocessheap,andsharedmemoryormmapservices. 0x0800000000000000to0x08ffffffffffffffPrivatelyloadedobjects. 0x0900000000000000to0x09ffffffffffffffSharedlibrarytextanddata. 0x0f00000000000000to0x0fffffffffffffffApplicationstack.Someitem:Size、ResData、ResText、ResSet預(yù)備知識(shí)-AIX程序內(nèi)存映像A32-bitapplica51預(yù)備知識(shí)-HP-UX程序內(nèi)存映像32bitsapplications預(yù)備知識(shí)-HP-UX程序內(nèi)存映像32bitsapplic52預(yù)備知識(shí)-HP-UX程序內(nèi)存映像64bitsapplications預(yù)備知識(shí)-HP-UX程序內(nèi)存映像64bitsapplic53預(yù)備知識(shí)-Solaris/sparc程序內(nèi)存映像預(yù)備知識(shí)-Solaris/sparc程序內(nèi)存映像54預(yù)備知識(shí)-JVM&J2EE預(yù)備知識(shí)-JVM&J2EE55預(yù)備知識(shí)-CORBA預(yù)備知識(shí)-CORBA56預(yù)備知識(shí)-TuxedoATMI預(yù)備知識(shí)-TuxedoATMI57core分析-core的類型和原因SIGABRT(6)：異常退出，通常程序發(fā)現(xiàn)了錯(cuò)誤并調(diào)用abort，比如C++中沒(méi)有處理的exceptionSIGBUS(7)：內(nèi)存訪問(wèn)時(shí)的硬件故障，通常是地址未對(duì)齊SIGFPE(8)：浮點(diǎn)數(shù)異常，包括除0、溢出等SIGILL(4)：不正確的指令，通常是指針越界導(dǎo)致的ip非法，棧溢出等SIGIOT(6)：執(zhí)行I/O自陷SIGQUIT(3)：退出并產(chǎn)生coreSIGSEGV(11)：錯(cuò)誤的內(nèi)存地址，通常是訪問(wèn)的地址空間不存在，或者沒(méi)有讀寫(xiě)權(quán)限core分析-core的類型和原因SIGABRT(6)：異常58core分析-core文件的備份AIX設(shè)置環(huán)境變量CORE_NAMING=yes，文件名形如core.pid.ddhhmmss每個(gè)用戶的core設(shè)置在/etc/security/userSolaris利用coreadm命令，例如coreadm-icore.%f.%tHP-UX利用coreadm命令，例如coreadm-pcore.%f.%tLinux/proc/sys/kernel/core_pattern/proc/sys/kernel/core_uses_pid=1修改/etc/sysctl.conf，再sysctl-p或echo'core.%e.%p'>/proc/sys/kernel/core_patterncore分析-core文件的備份AIX59core分析-分析步驟查看core文件大小查看limit信息查看core進(jìn)程的環(huán)境變量可先嘗試pstack偶爾要收集系統(tǒng)核心參數(shù)使用調(diào)試工具（gdb/dbx/mdb）對(duì)照core時(shí)的日志編寫(xiě)驗(yàn)證程序core分析-分析步驟查看core文件大小60core分析-我們應(yīng)用的常見(jiàn)core不能分配更多的內(nèi)存（除了limit、32位，還有OS特定的限制）棧溢出（?？臻g不夠，遞歸太深）abort，特別是c++的exception頭文件和實(shí)現(xiàn)不一致使用了空指針指針越界/內(nèi)存溢出多線程問(wèn)題重復(fù)freecore分析-我們應(yīng)用的常見(jiàn)core不能分配更多的內(nèi)存（除了61core分析-自己實(shí)踐從前面的類型中自己選，編寫(xiě)程序驗(yàn)證和實(shí)踐環(huán)境：86(AIX)username:lixp61(HP-UXIA64)username:lixp99(Solaris)username:lixpcore分析-自己實(shí)踐從前面的類型中自己選，編寫(xiě)程序驗(yàn)證和實(shí)62core分析-案例1(AIX5.3,32Bits)iom@:/tuxedo/iom/aioss/bin>ls

-l

core-rw-r--r--

1

iom

aigrp

270562700

Dec

30

21:35

core

iom@:/tuxedo/iom/aioss/bin>dbx

unif_exec

coreType

'help'

for

help.[using

memory

image

in

core]Segmentation

fault

in

malloc_common.extend_brk

[/usr/lib/threads/libc.a]

at

0xd0355260

($t1)0xd0355260

(extend_brk+0x20c)

90040004

stw

r0,0x4(r4)(dbx)

(dbx)

wheremalloc_common.extend_brk(internal

error:

assertion

failed

at

line

3650

in

file

frame.c??,

internal

error:

assertion

failed

at

line

3650

in

file

frame.c??,

internal

error:

assertion

failed

at

line

3650

in

file

frame.c??)

at

0xd0355260(dbx)

core分析-案例1(AIX5.3,32Bits)iom@:63core分析-案例2(Solaris10,32bits)current

thread:

t@26=>[1]

__lwp_kill(0x0,

0x6,

0x0,

0x6,

0xffbffeff,

0x0),

at

0xed34d6ac

[2]

raise(0x6,

0x0,

0xed3b5010,

0xed32cd88,

0xffffffff,

0x6),

at

0xed2e69ac

[3]

abort(0x0,

0x1,

0xefb9403c,

0xee4cc,

0xed3b34d8,

0x0),

at

0xed2c2010

[4]

free(0x6fabf0,

0x709b48,

0x0,

0x80,

0x6a,

0x80000000),

at

0xefb80b00

[5]

operator

delete(0x709b48,

0x1aaedf0,

0x1aaedfc,

0xc,

0x14e98,

0xec0ec2fc),

at

0xef95631c

[6]

std::allocator<char>::deallocate(0xea0f4f7a,

0x709b48,

0x8,

0x4f,

0x1aaeb20,

0x0),

at

0xede9ab90

[7]

std::allocator_interface<std::allocator<char>,char>::deallocate(0xea0f4f7a,

0x709b48,

0x8,

0xea0f5334,

0x7ffffc00,

0xe),

at

0xede9a678

[8]

std::basic_string<char,std::char_traits<char>,std::allocator<char>

>::__unLink(0x1aaedf0,

0xea0f5358,

0x1a9ca28,

0x87a100,

0x1a78fc8,

0x8),

at

0xede96d00

[9]

std::basic_string<char,std::char_traits<char>,std::allocator<char>

>::~basic_string(0x1aaedf0,

0x1aaedde,

0x0,

0xffffffff,

0x4f,

0x4f),

at

0xede96050

[10]

SObdCallInfo::~SObdCallInfo(0x1aaedf0,

0x0,

0x0,

0x1aaedd0,

0x0,

0x0),

at

0xede9bbc0

…………

[28]

VISThread::_start(0x10506c8,

0xea0f6000,

0x0,

0x0,

0xef8399b8,

0x1),

at

0xef8399e0

core分析-案例2(Solaris10,32bits)cu64core分析-案例320110901貴州賬務(wù)資料MDBcore問(wèn)題core分析-案例320110901貴州賬務(wù)資料MDB65性能問(wèn)題診斷-常見(jiàn)性能問(wèn)題CPU瓶頸：鎖問(wèn)題、線程切換、邏輯問(wèn)題內(nèi)存瓶頸：換頁(yè)、緩存過(guò)大、邏輯問(wèn)題網(wǎng)絡(luò)瓶頸：網(wǎng)卡、中斷、cpu綁定I/O瓶頸：底層問(wèn)題、目錄操作量大數(shù)據(jù)庫(kù)瓶頸：大數(shù)據(jù)量、索引、鎖……性能問(wèn)題診斷-常見(jiàn)性能問(wèn)題CPU瓶頸：鎖問(wèn)題、線程切換、邏輯66性能問(wèn)題診斷-一般分析過(guò)程CPU瓶頸？?jī)?nèi)存瓶頸？I/O瓶頸？網(wǎng)絡(luò)瓶頸？vmstatpslspssvmonvmstatsarnicepsiostatlslvfilemonfileplaceMoretestnetstatnfsstatnfsonoifconfignetpmonYYYYNNNN性能問(wèn)題診斷-一般分析過(guò)程CPU瓶頸？?jī)?nèi)存瓶頸？I/O瓶頸？67性能問(wèn)題診斷-關(guān)鍵點(diǎn)關(guān)鍵點(diǎn)1：應(yīng)用對(duì)資源的性能/容量要求是什么？有沒(méi)有超過(guò)？關(guān)鍵點(diǎn)2：有沒(méi)有做過(guò)基準(zhǔn)測(cè)試？關(guān)鍵點(diǎn)3：平時(shí)有沒(méi)有收集性能數(shù)據(jù)？性能問(wèn)題診斷-關(guān)鍵點(diǎn)關(guān)鍵點(diǎn)1：應(yīng)用對(duì)資源的性能/容量要求是什68性能問(wèn)題診斷-必讀書(shū)籍AIX通用編程概念：編寫(xiě)并調(diào)試程序.pdf性能管理.pdf網(wǎng)絡(luò)與通信管理.pdfHP-UXcaliper_user_guide.pdfhp-uxlinkerandlibrariesuserguideforhp-ux11.0.pdfJavaTroubleshootingGuideforHP-UXSystems.pdfTunableKernelParameters-HP-UXRelease11iVersion1.6.pdfSolarisSolaris10-Performance-AnalysisusingDynamicTracing(DTrace).pdfS10DeepDive_Performance.pdfDTraceUserGuide.pdfLinkerandLibrariesGuide.pdfMultithreadedProgrammingGuide.pdf性能問(wèn)題診斷-必讀書(shū)籍AIX通用編程概念：編寫(xiě)并調(diào)試程序.p69性能問(wèn)題診斷-AIX工具性能問(wèn)題診斷-AIX工具70性能問(wèn)題診斷-HP-UX工具性能問(wèn)題診斷-HP-UX工具71性能問(wèn)題診斷-Solaris工具性能問(wèn)題診斷-Solaris工具72性能問(wèn)題診斷-vmstat這個(gè)工具可以讓我們看到很多性能問(wèn)題！性能問(wèn)題診斷-vmstat這個(gè)工具可以讓我們看到很多性能問(wèn)題73性能問(wèn)題診斷-contextswitch資料來(lái)源：CandC++ApplicationdevelopmentonAIX.pdf,P115性能問(wèn)題診斷-contextswitch資料來(lái)源：Can74性能問(wèn)題診斷-iostatAIXHP-UXSolaris性能問(wèn)題診斷-iostatAIXHP-UXSolaris75性能問(wèn)題診斷-tcpdumptcpdump[-adeflnNOpqStvx][-c數(shù)量][-F文件名]

[-i網(wǎng)絡(luò)接口][-r文件名][-ssnaplen]

[-T類型][-w文件名][表達(dá)式](1)想要截獲所有的主機(jī)收到的和發(fā)出的所有的數(shù)據(jù)包：

#tcpdumphost(2)想要截獲主機(jī)和主機(jī)或的通信，使用命令：（在命令行中適用括號(hào)時(shí)，一定要

#tcpdumphostand\(or\)(3)如果想要獲取主機(jī)除了和主機(jī)之外所有主機(jī)通信的ip包，使用命令：

#tcpdumpiphostand!(4)如果想要獲取主機(jī)接收或發(fā)出的telnet包，使用如下命令：

#tcpdumptcpport23host進(jìn)程和端口的關(guān)系：pfiles、lsof、truss網(wǎng)卡和ip的關(guān)系：ifconfig性能問(wèn)題診斷-tcpdumptcpdump[-adefl76性能問(wèn)題診斷-netperf長(zhǎng)連接：netperf-tTCP_RR-H20---r6464短連接：netperf-tTCP_CRR-H20其它選項(xiàng)：-c-C-j性能問(wèn)題診斷-netperf長(zhǎng)連接：77性能問(wèn)題診斷-AIX64bitscout0x09000000004922e0_global_lock_common(??,??,??)+0x40c0x09000000000650f0_rec_mutex_lock(??)+0x17c0x0900000000431c5cstd::_Lock::_Wait()(??)+0x2c