飛塔防火墻_診斷

1、 Copyright Fortinet Inc. All rights reserved. 14 May 2022FortiGate IITroubleshootingFortiGate 5.2.12目標(biāo)識(shí)別網(wǎng)絡(luò)常規(guī)行為監(jiān)控非正常行為如流量突發(fā)或非典型性協(xié)議Troubleshoot物理和邏輯網(wǎng)絡(luò)接口理解會(huì)話表使用“diagnose debug flow” 來(lái)對(duì)流量流向進(jìn)行排錯(cuò) 對(duì)資源使用問(wèn)題進(jìn)行排錯(cuò), 如當(dāng)防病毒和IPS打開時(shí)高CPU或高內(nèi)存占用測(cè)試沒有保存到flash的OS3在任何問(wèn)題發(fā)生之前定義正常行為(基線):CPU 使用率Memory 使用率流量等級(jí)流量如何走向(流量)使用了哪些協(xié)議和

2、TCP/UDP 端口流量模式和分布Why?如果你知道什么是正常流量, 識(shí)別非正常流量會(huì)更容易NowBaseline (Average)Normal RangeAbnormal4網(wǎng)絡(luò)圖為何需要網(wǎng)絡(luò)圖? 沒有網(wǎng)絡(luò)圖，解釋和分析復(fù)雜網(wǎng)絡(luò)是困難且耗時(shí)的物理圖包含所有物理網(wǎng)絡(luò)接口, 連線和端口對(duì) Layer 1/2/3 的問(wèn)題很有效邏輯圖包含路由器, 邏輯設(shè)備(VDOMs)和UTM對(duì)Layer 3+的問(wèn)題很有效2001:db8:b108port2/24port4/27port/8port35監(jiān)控?cái)?shù)據(jù)流 & 資源使用情況獲取正常的網(wǎng)絡(luò)數(shù)據(jù) 在發(fā)生

3、問(wèn)題之前不正常的行為非常難發(fā)現(xiàn) 除非知道什么是正常的CPU使用率RAM使用率允許通過(guò)的應(yīng)用入和出的帶寬工具SNMPAlert emailLogging / SyslogFortiAnalyzer或者第三方SIEM(system information & event management)Dashboard / get system statusNormalTraffic spikes6SNMPAllowed source of queries7通過(guò)SNMP獲取事件通知 trapDestination觸發(fā)FortiGate t發(fā)送SNMP消息的事件8# get sys statusVersio

4、n: FortiGate-VM64 v5.2.0,build0589,140613 (GA)Virus-DB: 22.00856(2014-09-24 05:33)Extended DB: 1.00000(2012-10-17 15:46)IPS-DB: 5.00549(2014-09-23 00:49)IPS-ETDB: 0.00000(2001-01-01 00:00)Serial-Number: FGVM040000025212Botnet DB: 1.00736(2014-08-24 10:18)License Status: ValidVM Resources: 1 CPU/4 al

5、lowed, 969 MB RAM/6144 MB allowedBIOS version: 04000002Log hard disk: AvailableHostname: STUDENTOperation Mode: NATCurrent virtual domain: rootMax number of virtual domains: 10Virtual domains status: 1 in NAT mode, 0 in TP modeVirtual domain configuration: disableFIPS-CC mode: disableCurrent HA mode

6、: standaloneBranch point: 589Release Version Information: GAFortiOS x86-64: YesSystem time: Thu Oct 9 00:26:54 2014# get sys perf statCPU states: 2% user 15% system 0% nice 83% idleCPU0 states: 2% user 15% system 0% nice 83% idleMemory states: 44% usedAverage network usage: 542 kbps in 1 minute, 105

7、0 kbps in 10 minutes, 512 kbps in 30 minutesAverage sessions: 7 sessions in 1 minute, 5 sessions in 10 minutes, 5 sessions in 30 minutesAverage session setup rate: 0 sessions per second in last 1 minute, 0 sessions per second in last 10 minutes, 0 sessions per second in last 30 minutesVirus caught:

8、0 total in 1 minuteIPS attacks blocked: 0 total in 1 minuteUptime: 0 days, 0 hours, 19 minutes系統(tǒng)信息 & 資源使用情況9# diagnose firewall statistic showgetting traffic statistics.Browsing: 328 packets, 132562 bytesDNS: 797 packets, 127917 bytesE-Mail: 0 packets, 0 bytesFTP: 0 packets, 0 bytesGaming: 0 packets

9、, 0 bytesIM: 0 packets, 0 bytesNewsgroups: 0 packets, 0 bytesP2P: 0 packets, 0 bytesStreaming: 0 packets, 0 bytesTFTP: 0 packets, 0 bytesVoIP: 0 packets, 0 bytesGeneric TCP: 1098554 packets, 817573554 bytesGeneric UDP: 1490 packets, 210976 bytesGeneric ICMP: 0 packets, 0 bytesGeneric IP: 6 packets,

10、192 bytes# diagnose hardware deviceinfo nic port1Name: port1Driver: e1000Version: 5.1.13k2 NAPIFW version: N/ABus: 00:11.0Memory: 0 xfeb80000 - 0 xfeba0000Base address: 0 x1400Interrupt: 18Hwaddr: 00:0c:29:95:8c:faPermanent Hwaddr:00:0c:29:95:8c:faState: upLink: upMtu: 1500Supported: auto 10half 10f

11、ull 100half 100full 1000fullAdvertised: auto 10half 10full 100half 100full 1000fullSpeed: 1000fullAuto: enabledRx packets: 136154Rx bytes: 10901815Rx compressed: 0Rx dropped: 0Rx errors: 0 Rx Length err: 0 Rx Buf overflow: 0 Rx Crc err: 0 Rx Frame err: 0 Rx Fifo overrun: 0 Rx Missed packets: 0Tx pac

12、kets: 1611Tx bytes: 257565.Multicasts: 0Collisions: 0帶寬利用率，系統(tǒng)崩潰和錯(cuò)誤10其他工具CLI get system status get system performance status diagnose sys top diagnose sys top-summary diagnose hardware sysinfo memory diagnose hardware sysinfo shm diagnose netlink device list diagnose hardware deviceinfo nic port1 dia

13、gnose firewall statistics show .DashboardSNMP trapsAlert emailLogs11# diagnose hardware deviceinfo nic port1Description :FortiASIC NP6 AdapterDriver Name :FortiASIC Unified NPU DriverName :np6_2PCI Slot :8d:00.0irq :58Board :FGT3700DSN :NP6KR44613000276Major ID :2Minor ID :0lif id :0lif oid :156netd

14、ev oid :156netdev flags :1203Current_HWaddr :08:5b:0e:4a:2e:e4Permanent_HWaddr:08:5b:0e:4a:2e:e4phy name :np6_2_0bank_id :255phy_addr :0 x20lane :0sw_port :51sw_np_port (cat)vid_phy6 :0 x000 x000 x0b0 x000 x000 x00vid_fwd6 :0 x000 x000 x000 x000 x000 x00oid_fwd6 :0 x000 x000 x000 xcc0 x000 x00= Link

15、 Status =Admin :upnetdev status :downautonego_setting:1link_setting :1link_speed :40000link_duplex :1Speed :0Duplex :Fulllink_status :Downrx_link_status :0int_phy_link :0local_fault :0local_warning :0remote_fault :0= Counters =Rx Pkts :0Rx Bytes :0Tx Pkts :0Tx Bytes :0Host Rx Pkts :0Host Rx Bytes :0

16、Host Rx dropped :0Host Tx Pkts :4Host Tx Bytes :198Host Tx dropped :0sw_rx_pkts :0sw_rx_bytes :0sw_tx_pkts :0sw_tx_bytes :0sw_np_rx_pkts :4sw_np_rx_bytes :272sw_np_tx_pkts :0sw_np_tx_bytes :0物理層/數(shù)據(jù)鏈路層的Troubleshooting12網(wǎng)絡(luò)層的Troubleshooting：路由# execute ping-options ?data-size 定義數(shù)據(jù)包的大小,以bytes為單位df-bit 在

17、IP頭里設(shè)置 DF 位interval 兩個(gè)ping直接的間隔時(shí)間,以秒為單位pattern 十六進(jìn)制格式, e.g. 00ffaabbrepeat-count 重復(fù)ping多少次source auto | timeout 定義多少秒后timeouttos IP的服務(wù)類型ttl 存活時(shí)間 time-to-live.validate-reply 有效的reply數(shù)據(jù).view-settings 查看ping的當(dāng)前設(shè)置# execute ping # execute traceroute | 13網(wǎng)絡(luò)層的Troubleshooting：會(huì)話1.清空之前的過(guò)濾條件# diagnose sys ses

18、sion filter clear2.設(shè)置過(guò)濾條件# diagnose sys session filter ?dport destination portdst destination IP addresspolicy policy idsport source portsrc source ip address3.列出所有匹配過(guò)濾條件的會(huì)話# diagnose sys session list4.清空所有匹配過(guò)濾條件的會(huì)話# diagnose sys session clear14會(huì)話表：TCPsession info: proto=6 proto_state=65 duration=3

19、expire=9 timeout=3600 flags=00000000 sockflag=00000000 sockport=443 av_idx=9 use=5origin-shaper=guarantee-100kbps prio=2 guarantee 12800Bps max 134217728Bps traffic 13895Bpsreply-shaper=guarantee-100kbps prio=2 guarantee 12800Bps max 134217728Bps traffic 13895Bpsper_ip_shaper=ha_id=0 policy_dir=0 tu

20、nnel=/state=redir local may_dirty ndr npu nlb os rs statistic(bytes/packets/allow_err): org=864/8/1 reply=2384/7/1 tuples=3orgin-sink: org pre-post, reply pre-post dev=7-6/6-7 gwy=/hook=post dir=org act=snat 10:57999-9:443(6:57999)hook=pre dir=rep

21、ly act=dnat 9:443-6:57999(10:57999)hook=post dir=reply act=noop 9:443-10:57999(:0)pos/(before,after) 0/(0,0), 0/(0,0)misc=0 policy_id=1 id_policy_id=0 auth_info=0 chk_client_info=0 vd=0serial=0008b037 tos=ff/ff ips_view=1 app_list=2000 ap

22、p=24534dd_type=0 dd_mode=0per_ip_bandwidth meter: addr=10, bps=4872npu_state=00000000npu info: flag=0 x00/0 x00, offload=0/0, ips_offload=0/0, epid=0/0, ipid=0/0, vlan=0/0連連接狀接狀態(tài)態(tài)流量整形流量整形數(shù)據(jù)包數(shù)據(jù)包統(tǒng)計(jì)統(tǒng)計(jì)剩余剩余TTLNAT硬件加速硬件加速協(xié)議協(xié)議目的端口目的端口會(huì)會(huì)話處話處理理15會(huì)話表：協(xié)議proto=6服務(wù)代碼在IP頭中常見的代碼1 = ICMP6 = TCP17 = UDP13

23、2 = SCTPIPv4 Header16傳輸層的Troubleshooting: TCP狀態(tài)proto_state=05總是兩位數(shù)字第一位數(shù)字 = 客戶端的會(huì)話狀態(tài)（沒有代理則是0）第二位數(shù)字 = 服務(wù)器端的會(huì)話狀態(tài)StateValueExpiry Timer (default)NONE010 sESTABLISHED13600 sSYN_SENT2120 sSYN & SYN/ACK360 sFIN_WAIT4120 sTIME_WAIT5120 sCLOSE610 sCLOSE_WAIT7120 sLAST_ACK830 sLISTEN9120 s17傳輸層的Troubleshootin

24、g: UDP & ICMP 狀態(tài)雖然UDP 是一個(gè)無(wú)狀態(tài)協(xié)議, FortiGate 仍會(huì)有兩個(gè)不同的 “proto_state” 值:State Value未看到UDP回應(yīng) 00看到UDP回應(yīng)01ICMP 無(wú)狀態(tài)proto_state 一直標(biāo)記為0018傳輸層的Troubleshooting: SCTP狀態(tài)StateValueExpiry Timer (default)SCTP_S_NONE060 sSCTP_S_ESTABLISHED13600 sSCTP_S_CLOSED210 sSCTP_S_COOKIE_WAIT35 sSCTP_S_COOKIE_ECHOED410 sSCTP_S_S

25、HUTDOWN_SENT530 sSCTP_S_SHUTDOWN_RECD630 sSCTP_S_SHUTDOWN_ACK_SENT73 sSCTP_S_MAX8n/a19會(huì)話表：會(huì)話處理標(biāo)識(shí)state=log shape may_dirty 并不通用 如果會(huì)話被卸載到ASIC芯片上，則不一定代表是現(xiàn)在的狀態(tài)hardware accelerationFlagMeaninglogSession is being loggedlocalSession is to/from local stackextSession is created by a firewall session helperma

26、y_dirtySession is created by traffic hitting a policy. ndrSession will be checked by IPS signaturendsSession will be checked by IPS anomalybrSession is being bridged (TP mode)npuSession is possible to be offloaded to NPUwccpSession is handled by WCCPnpdSession cannot be offloaded to NPUdirtyNext pac

27、ket in original direction will be revalidated against policyredirSession is being processed by an application layer proxyauthedSession was successfully authenticatedauthSession is requires (or required) authenticationsrc-visSession is being scanned for device detection purposes20會(huì)話表: 連接自動(dòng)刪除會(huì)話超時(shí)expir

28、e=89 timeout=3600不活躍的會(huì)話當(dāng)兩個(gè)值都為0時(shí)TCP連接被拆除FIN, FIN/ACK, ACKTCP連接超時(shí)tcp-halfclose-timer: FIN WAIT and CLOSE WAITtcp-half-open-timer: SYN SENT and SYN & SYN/ACKtcp-timewait-timer: TIME WAITudp-idle-timer21高級(jí)抓包選項(xiàng)#diag sniffer packet 當(dāng)抓取了這個(gè)數(shù)目的報(bào)文時(shí)自動(dòng)停止抓包 修改時(shí)間戳的格式a 絕對(duì)UTC時(shí)間l 當(dāng)?shù)貢r(shí)間22高級(jí)抓包選項(xiàng)：輸出 # diag sniff packet

29、any icmp 4 interfaces=any filters=icmp 2.101199 wan2 in 10 - : icmp: echo request 2.101400 wan1 out 6 - : icmp: echo request 2.123325 wan1 in - 6: icmp: echo reply 2.123500 wan2 out - 10: icmp: echo reply 4 packets received by

30、 filter 0 packets dropped by kernel # diag sniff packet any icmp 4 3 l interfaces=any filters=icmp 2014-11-14 10:28:19.769989 wan2 in 10 - : icmp: echo request 2014-11-14 10:28:19.770143 wan1 out 6 - : icmp: echo request 2014-11-14 10:28:19.792325 wan1 in

31、- 6: icmp: echo reply 3 packets received by filter 0 packets dropped by kernel報(bào)文數(shù)量時(shí)間戳23診斷系統(tǒng)反應(yīng)過(guò)慢高高CPU使用率使用率高內(nèi)存使用率高內(nèi)存使用率上一個(gè)開啟的功能是什么？每次開啟一個(gè)功能快速診斷CPU使用率有多高，為什么？# get system performance status# diagnose sys top 124高CPU占用率的Troubleshooting: get sys perf stat# get system performance statusCPU sta

32、tes: 4% user 13% system 0% nice 83% idleCPU0 states: 3% user 13% system 0% nice 84% idleCPU1 states: 5% user 13% system 0% nice 82% idleCPU2 states: 2% user 13% system 0% nice 85% idleCPU3 states: 6% user 13% system 0% nice 81% idleMemory states: 19% usedAverage network usage: 12740 kbps in 1 minute

33、, 3573 kbps in 10 minutes, 1077 kbps in 30 minutesAverage sessions: 118 sessions in 1 minute, 11 sessions in 10 minutes, 40 sessions in 30 minutesAverage session setup rate: 11 sessions per second in last 1 minute, 0 sessions per second in last 10 minutes, 1 sessions per second in last 30 minutesVir

34、us caught: 3 total in 1 minuteIPS attacks blocked: 64 total in 1 minuteUptime: 60 days, 9 hours, 58 minutesCPU使用率網(wǎng)絡(luò)使用率內(nèi)存使用率25高CPU使用率：臨時(shí)bypass一些檢查進(jìn)程n可以暫時(shí)bypass一些檢查進(jìn)程# diagnose test application ipsmonitor 5全局bypass更容易,之后可以調(diào)整策略來(lái)確認(rèn)問(wèn)題無(wú)需檢查的任務(wù)繼續(xù)運(yùn)行n CPU使用率在bypass這些進(jìn)程后是否降低了？n恢復(fù)某個(gè)檢查進(jìn)程：# diagnose test applicat

35、ion ipsmonitor 526內(nèi)存診斷進(jìn)程的內(nèi)存使用率# get system performance status# diag sys top-summaryFortiOS的內(nèi)存使用率，不是某一個(gè)進(jìn)程的# diagnose hardware sysinfo mem# diagnose hardware sysinfo slab27# diagnose sys top-summary CPU | 38.4% Mem | 54.0% 1009M/1841M Processes: 20 (running=1 sleeping=86) PID RSS CPU% MEM% FDS TIME+ N

36、AME * 72 32M 34.2 1.7 11 00:03.39 httpclid x5 95 11M 1.9 0.6 20 53:07.83 cw_wtpd 40 23M 1.2 1.3 24 03:02.60 httpsd x5 1173 27M 0.0 1.5 10 00:02.82 pyfcgid x4 36 10M 0.0 0.5 88 00:47.75 zebos_launcher x12 37 9M 0.0 0.5 9 00:00.23 uploadd 38 15M 0.0 0.8 41 01:52.19 miglogd 39 9M 0.0 0.5 5 00:01.41 kmi

37、glogd 46 25M 0.0 1.4 821 01:47.98 proxyd x6 47 10M 0.0 0.5 7 00:00.12 wad_diskd 51 12M 0.0 0.7 16 00:02.72 scanunitd x3 53 61M 0.0 3.3 16 00:15.14 ipsmonitor x2 57 9M 0.0 0.5 7 00:00.13 merged_daemons 69 13M 0.0 0.7 18 00:34.20 urlfilter 在diag sys top 中RAM復(fù)雜 交叉的進(jìn)程會(huì)使條目眾多 交叉的進(jìn)程中共享數(shù)據(jù)# diagnose sys topRun Time: 11 days, 3 hours and 29 minutes0U, 0S, 10I; 500T, 345F, 78KF thttp 48 S 0.0 4.4 httpsd 74 S 0.0 3.4 httpsd 54 S 0.0 3.4 cmdbsvr 23 S 0.0 3.4 httpsd 18618 S 0.0 2.9 httpsd 18645 S 0.0 2.9 httpsd 18643 S