ibm存儲(chǔ)服務(wù)器高端bluelite l3spectrum scale overview

1、IBM Spectrum Scale介紹存儲(chǔ)方案專家級(jí)課程課程安排 Spectrum Scale GPFS-FPO Elastic Storage ServerIBM Spectrum ScaleUnleash New Storage Economics on a Global Scale3我們可以提供高性能、高可擴(kuò)展性的非結(jié)構(gòu)化數(shù)據(jù)管理方案技術(shù)基于Elastic Storage (General Parallel File System - GPFS)- 支持YB級(jí)別數(shù)據(jù)核心功能全局命名空間支持跨多站點(diǎn)的單一文件系統(tǒng)高性能數(shù)據(jù)訪問滿足從部門級(jí)到全球多站點(diǎn)的需求自動(dòng)分層，數(shù)據(jù)生命周期管理從Fl

2、ash到磁帶企業(yè)級(jí)方案：數(shù)據(jù)安全，高可用，高可靠，高可擴(kuò)展開放：符合POSIX，與OpenStack和Hadoop整合價(jià)值消除數(shù)據(jù)瓶頸，極大提高性能增強(qiáng)跨地理區(qū)域間的協(xié)作消除重復(fù)數(shù)據(jù)，大大降低成本通過單一文件系統(tǒng)實(shí)現(xiàn)多應(yīng)用間的共享通過將數(shù)據(jù)分別放在最適合的存儲(chǔ)上提高方案的整體性價(jià)比（從閃存到磁帶庫(kù)）案例- 提高50%性能在IO密集型的半導(dǎo)體模擬場(chǎng)景- 硬件故障時(shí)保持?jǐn)?shù)據(jù)持續(xù)可用 - 分析超過20PB數(shù)據(jù)- 風(fēng)預(yù)報(bào)數(shù)據(jù)響應(yīng)時(shí)間減少97%-幫助基因科學(xué)計(jì)算提高230倍性能4IBM Spectrum Scale（GPFS）：軟件定義存儲(chǔ)改變存儲(chǔ)生態(tài)CinderSwiftHadoop Connecto

3、rNFS 單一軟件定義存儲(chǔ)方案，支持所有應(yīng)用類型Linear capacity & performance scale outPOSIXEnterprise storage on standard hardwareSingle Name SpaceTechnical ComputingBig Data & AnalyticsCloudBlock ObjectFileIBM General Parallel File System (GPFS) GPFS能解決如下客戶痛點(diǎn)擴(kuò)展性：當(dāng)前最大規(guī)模8192節(jié)點(diǎn)性能：最大實(shí)測(cè)吞吐率405GB/s可靠性：無單一故障點(diǎn)、在線軟件升級(jí)、在線硬件更換、多分拷貝、

4、遠(yuǎn)程容災(zāi)存儲(chǔ)及數(shù)據(jù)管理：簡(jiǎn)單有效地管理數(shù)據(jù)，自動(dòng)化信息生命周期管理工具可以管理數(shù)以十億記的文件，數(shù)十PB級(jí)別的數(shù)據(jù)（一個(gè)文件系統(tǒng)內(nèi)）Windows、Linux、AIX全面支持IBM、HP、EMC等等多個(gè)存儲(chǔ)產(chǎn)品支持15年的市場(chǎng)歷史，研發(fā)至今已經(jīng)超過20年全球數(shù)以萬記的用戶，數(shù)十萬個(gè)GPFS系統(tǒng)久經(jīng)行業(yè)驗(yàn)證, 風(fēng)險(xiǎn)最小可擴(kuò)展、高性能、高可靠 的軟件定義存儲(chǔ)解決方案為什么選擇GPFS誰(shuí)需要GPFS- 凡是多節(jié)點(diǎn)共享數(shù)據(jù)的，都需要GPFSBI分析金融電信企業(yè)ERP交易所Technical Computing計(jì)算中心生命科學(xué)石油勘探芯片設(shè)計(jì)互聯(lián)網(wǎng)應(yīng)用電子郵件Web服務(wù)器集群在線數(shù)據(jù)存儲(chǔ) 數(shù)字媒體動(dòng)畫渲

5、染廣告設(shè)計(jì)視頻監(jiān)控?cái)?shù)據(jù)中心 用戶文件存儲(chǔ) 多數(shù)據(jù)中心整合 容災(zāi)系統(tǒng)大數(shù)據(jù)智能發(fā)電機(jī)械制造航空航天天氣預(yù)報(bào)IBM General Parallel File System (GPFS) History and evolution2006/2009200520021998HPCGPFS General File ServingStandardsPortable operating system interface (POSIX) semantics-Large blockDirectory and Small file perfData managementVirtual Tape Server

6、(VTS)Linux Clusters (Multiple architectures)IBM AIX Loose ClustersGPFS 2.1-2.3HPCResearchVisualizationDigital MediaSeismicWeather explorationLife sciences32 bit /64 bitInter-op (IBM AIX & Linux)GPFS MulticlusterGPFS over wide area networks (WAN)Large scale clustersthousands of nodesGPFS 3.1-3.32010F

7、irst called GPFSGPFS 3.4Enhanced Windows cluster support- Homogenous Windows ServerPerformance and scaling improvementsEnhanced migration and diagnostics support 2012 2014Ease of administrationMultiple-networks/ RDMADistributed Token ManagementWindows 2008 Multiple NSD serversNFS v4 SupportSmall fil

8、e performanceInformation lifecycle management (ILM)Storage PoolsFile setsPolicy EngineGPFS 3.5Active File ManagementGPFS Storage ServerFile Placement optimizerGPFS 4.1Native Encryption and Secure EraseFlash Local Read Only Cache4K Disk SectorAFM Parallel TransferNFS Data MigrationGPFS是什么?IBM集群并行文件系統(tǒng)

9、，所有節(jié)點(diǎn)訪問相同數(shù)據(jù)能提供標(biāo)準(zhǔn)的POSIX訪問接口支持AIX，Linux，以及Windows性能和容量無縫擴(kuò)展能力GPFSSANSDSSDSSDSF1 紅牛車隊(duì)實(shí)時(shí)數(shù)據(jù)分析極高IO密集型高性能計(jì)算Vestas加速跨中心大數(shù)據(jù)計(jì)算從1周左右加速至數(shù)個(gè)小時(shí)為跨大洲的電影制作建設(shè)私有云系統(tǒng)提供多媒體數(shù)據(jù)存儲(chǔ)服務(wù)GPFS軟件定義存儲(chǔ) Software Defined Storage美國(guó)國(guó)家大氣研究中心11P 在線存儲(chǔ)14P 離線存儲(chǔ)GPFS統(tǒng)一解決方案Infrastructure Matters低延遲 全局統(tǒng)一數(shù)據(jù)訪問線性Scale-Out性能增長(zhǎng)基于成熟軟、硬件產(chǎn)品構(gòu)建企業(yè)級(jí)存儲(chǔ)服務(wù)GPFS 集群

10、架構(gòu)GPFS Highlights高擴(kuò)展性(實(shí)施案例)單文件系統(tǒng)容量 18Peta單集群節(jié)點(diǎn) 9620最大實(shí)施案例帶寬 405GB/s高帶寬支持全局Stripe大BlockSize(最大支持16M)多節(jié)點(diǎn)并行讀寫高可用基于Log機(jī)制保證數(shù)據(jù)一致性GPFS Native Raid或副本機(jī)制在線管理(在線增加刪除Disk和Node)單一命名空間，標(biāo)準(zhǔn)Posix接口支持分布式鎖機(jī)制所有節(jié)點(diǎn)都可成為Meta Server集中/虛擬化彈性存儲(chǔ)（分布式架構(gòu)）V1V2V3V4V5V5.VnCCC12滿足大數(shù)據(jù)時(shí)代存儲(chǔ)需求Systems of Record交易型/流程型的工作模式控制數(shù)據(jù)增長(zhǎng)速度通過虛擬化提高

11、存儲(chǔ)使用效率/通過集中提高數(shù)據(jù)可管理性傳統(tǒng)應(yīng)用架構(gòu)參與型/分析型快速迭代，爆炸式數(shù)據(jù)增長(zhǎng)彈性數(shù)據(jù)壓力 Systems of Engagement新一代應(yīng)用架構(gòu)Integrated Storage Management and Data Protection（集成數(shù)據(jù)管理及保護(hù)）13Filer 1Filer 2Filer 3Filer 4GPFS提供全局的資源共享，負(fù)載均衡能力with automatedstorage tieringGlobal NamespaceWhy?把多個(gè)數(shù)據(jù)集整合入一個(gè)命名空間允許多個(gè)職能部門分享部分和全部數(shù)據(jù)幫助建立安全、有效、可靠的數(shù)據(jù)分享機(jī)制跨集群數(shù)據(jù)共享-優(yōu)化的

12、企業(yè)內(nèi)合作LANSANSANGPFSLocal area network (LAN)SANStorage area network (SAN)General Parallel File System (GPFS)網(wǎng)絡(luò)共享磁盤協(xié)議跨部門的統(tǒng)一命名空間快照 在線的數(shù)據(jù)保護(hù)典型用途文件系統(tǒng)備份大大減少文件系統(tǒng)備份、恢復(fù)時(shí)間在線訪問舊版本文件節(jié)省存儲(chǔ)空間，只保存被修改的數(shù)據(jù)遠(yuǎn)程系統(tǒng)災(zāi)備 和操作系統(tǒng)提供的遠(yuǎn)程復(fù)制命令結(jié)合（如rsync），將鏡像以增量方式復(fù)制至遠(yuǎn)端服務(wù)器按時(shí)點(diǎn)生成只讀文件系統(tǒng)有效使用存儲(chǔ)空間快速自動(dòng)生成copyinodeblockcopydata blockGPFS通過以下3個(gè)抽象功能支

13、持信息生命周期管理：Storage pool，一組存儲(chǔ)Fileset，一組文件Policy，將文件與存儲(chǔ)聯(lián)系起來的規(guī)則Policy 類型 移動(dòng), 比如，把數(shù)據(jù)庫(kù)文件置于RAID1存儲(chǔ)上，把其他數(shù)據(jù)置于RAID6存儲(chǔ)上 合并, 比如，把30天前的項(xiàng)目文件移至SATA盤，把60天前的項(xiàng)目文件移至磁帶上 刪除, 比如，把7天前的臨時(shí)文件刪除GPFS-FPO Manager NodeCluster managerLock managerQuota managerAllocation managerPolicy managerSystem PoolData PoolsGPFS NodesNetworkS

14、SDPoolSASPoolSATAPoolGPFSPlacement PolicyApplicationGPFSPlacement PolicyApplicationGPFSPlacement PolicyApplicationGPFS-FPOPlacement PolicyApplicationPosixDisk PoolsExternal(Tape) Pools:HPSS, TSM,MAID, VTL,信息生命周期管理 (ILM) . . .分層存儲(chǔ) 存儲(chǔ)虛擬化/home/appl/data/web/important_big_spreadsheet.xls/home/appl/data/

15、web/big_architecture_drawing.ppt/home/appl/data/web/unstructured_big_video.mpg/home/appl/data/web/home/appl/data/web/important_big_spreadsheet.xls/home/appl/data/web/big_architecture_drawing.ppt/home/appl/data/web/unstructured_big_video.mpgPolicy EngineVirtualized Global NamespacePool 1: FlashSystem

16、sPool 2: Solid State DrivesPool 3: Nearline SAS DrivesGPFS NodesGPFS NodesGPFS NodesStorage ControllersStorage ControllersStorage ControllersLogical PhysicalGPFS備份 離線的數(shù)據(jù)保護(hù)GPFS 掃描引擎快速發(fā)現(xiàn)文件系統(tǒng)增量，通知 磁帶庫(kù) 進(jìn)行備份避免遍歷文件系統(tǒng)以查找文件系統(tǒng)增量多個(gè)訪問節(jié)點(diǎn)可以并行地向 磁帶庫(kù) 服務(wù)器備份數(shù)據(jù)，提高備份的性能減少了備份所需的時(shí)間備份服務(wù)器掃描引擎所提供的結(jié)果GPFS-FPO ClusterTSM 所支持的

17、任何設(shè)備，包括： ProtectTier de-dup Virtual Tape Library TapeLTFS數(shù)據(jù)一個(gè)完整的GPFS文件系統(tǒng)備份流程備份GPFS集群信息，如節(jié)點(diǎn)IP，節(jié)點(diǎn)名，等等。備份GPFS磁盤（或RAID）配置信息備份GPFS文件系統(tǒng)配置信息，如塊大小，復(fù)制個(gè)數(shù)等等備份GPFS文件內(nèi)容到備份系統(tǒng)（以增量方式向磁帶復(fù)制文件系統(tǒng)中的文件內(nèi)容）備份GPFS元數(shù)據(jù)信息GPFS AFM本地集群的一個(gè)Fileset 可以同遠(yuǎn)程集群的一個(gè)或多Fileset聯(lián)系在一起客戶服務(wù)器以本地速度訪問數(shù)據(jù)數(shù)據(jù)以本地速度寫數(shù)據(jù)到緩存，更新的數(shù)據(jù)將在后臺(tái)異步的寫回到Home端。如果網(wǎng)絡(luò)中斷，緩存的數(shù)

18、據(jù)仍舊可以被持續(xù)訪問，新數(shù)據(jù)會(huì)在網(wǎng)絡(luò)重連后自動(dòng)刷回到HomepNFS/NFS over the WANGW NodesGPFS layerCache Cluster Site1(GPFS)Cache Cluster Site 2(GPFS)Home Cluster Site(GPFS/NFS)GPFS layerPull on cache missPush on writeX22LROC 本地磁盤緩存 最開始，數(shù)據(jù)主要來自SAS存儲(chǔ)，客戶端IOPS為5000隨著數(shù)據(jù)量緩存比例上升，IOPS增加到32000，同時(shí)，存儲(chǔ)系統(tǒng)訪問量降低95% 5,000 IOPS 10K RPM SAS Drive

19、s 32,000 IOPS Flash SSD 6x利用兩塊200GB的SSD，緩存一個(gè)SAN存儲(chǔ)設(shè)備，其中包含48塊300GB的10K SAS磁盤GPFS FPOGPFS and Big Data Map Reduce File Placement Optimizer (GPFS-FPO)Shared Nothing ArchitectureSupport for both POSIX and non POSIX interfaceSupport for map reduce applicationsChunksData LocalityEfficient configurable repl

20、icationFast recoveryGPFS ClusterShared Nothing PoolShared Disk Pool2425GPFS-FPO : A Scalable File-system for Shared Nothing ArchitecturesKey Ideas:LocalityWrite AffinityChunksPipelined ReplicationDistributed recoveryLocalDisk Node LocalDisk Node LocalDisk Node LocalDisk Node GPFS common file system

21、namespace 26Chunks: allows applications to define own logical block sizeWorkloads comprised of small and large blocksSmall blocks (eg: 512KB)Large blocks (eg: 128MB) Solution: Multiple block sizes in the same file systemNew allocation schemeBlock group factor for block sizeEffective block size(chunk

22、s) = block size * block group factorFile blocks are laid out based on “effective block size” or chunksApplication “chunk”FS blockNew allocation policy(Metablock factor, Block Size)OptimumChunk Size:128MB27Write Affinity: Allow applications to dictate layoutWide striping with multiple replicas:No loc

23、ality of data in a particular nodeBad for commodity architectures due to excessive network trafficWrite Affinity for a file or a set of files on a given nodeAll relevant files will be allocated on the node, near a node or in the same rackConfigurability of replicasEach replica can be configured for

24、either wide striping or write affinityN1FileN2N3Read File 1Wide striped layout: Read over local disk + networkN1N2N3N4Read File 1Write affinity layout:Read over only local diskN1N2N3N4Read File 1Write affinity and wide striping with replication: Read over only local disk, recover from wide striped r

25、eplicaParallelRecoveryObservationData is suited for GPFS-FPOMetadata is suited for regular GPFSMake allocation type a storage pool propertyMetadata pool: no write affinityData pools: write affinityGPFS-FPO: distributed metadataNo single point of failure but too many metadata nodesIncreases probabili

26、ty of any 3 metadata nodes going down loss of dataAssumes replication=3Typical metadata profile in GPFS-FPOOne or two metadata nodes per rack (failure domain)Random access patternsHybrid allocation: treat metadata and data differentlyMetadata:No write affinityRack 1Rack 2Rack 3Data:Write Affinity29L

27、ocality Awareness: Make applications aware where data is locatedScheduler assigns tasks to nodes where data chunks resideneeds to know the location of each chunkGPFS API maps each chunk to its node locationMap is compacted to optimize its size N1D1N1:D1N1Task Engine: Location MapSchedulerRead Locati

28、on MapSchedule F(D1) on N1F(D1)GPFSConnectorpass locality infoApplication30Pipelined Replication: Efficient replication of dataDisk 1Disk 2Disk 3Disk 4Disk 5Node 1Node 2Node 3Node 4Node 5N1ClientN2N3B0B1B2WriteWrite PipelineHigher Degree of replicationPipelining for lower latencySingle source does n

29、ot utilize the bandwidth effectively Able to saturate 1 Gbps Ethernet link31Fast Recovery: Recover quickly from disk failuresPolicy-based recoveryRestripe on a disk failureAlternatively, rebuild the disk when the disk is replacedFast recoveryIncremental recoveryKeep track of changes during failure a

30、nd recover what is needed Distributed restripeRestripe load is spread out over all the nodesQuickly figure out what blocks are needed to be recovered when a node failsN1N1N1FileN2N3Well replicated fileN2 rebooted:Writes go to N1 and N3N2 comes back and catches up by copying missed writes from N1 and

31、 N3 in parallelN2N3N2N3New writeCopy Deltas Elastic Storage ServerElastic Storage Server 協(xié)作創(chuàng)新的完美風(fēng)暴大容量，高IO吞吐率高達(dá)12GB/s的IO吞吐率1PB的磁盤容量（可用容量近700TB，視RAID方式而定）GNR技術(shù)內(nèi)置世界領(lǐng)先的De-cluster技術(shù)，大幅減少磁盤重建所需時(shí)間，提高系統(tǒng)可用性加強(qiáng)的容錯(cuò)性能，可以實(shí)現(xiàn)2路或3路的容錯(cuò)方案，并可共存于一個(gè)陣列中端對(duì)端的檢錯(cuò)功能，完全避免磁盤”無聲數(shù)據(jù)失效“豐富而完善的異步磁盤與系統(tǒng)診斷功能多路徑設(shè)計(jì)，無單點(diǎn)失效內(nèi)置IBM GPFS軟件20年來被各行

32、業(yè)及核心機(jī)構(gòu)廣泛驗(yàn)證的文件系統(tǒng)無限擴(kuò)展的并行文件系統(tǒng)集高性能與高可靠性于一身多樣化的信息管理Elastic Storage Server的定位Focus: Ease of UseReliabilityIBM Data Management LeadershipFocus:Managed Building BlockGovernmentHigh EndResearchPetroleumFinancial ServicesSmaller InstallationsHigher End UniversityMedia/Ent.CAEBio/Life ScienceIBM Tape, Tivoli S

33、torage Manager, and HPSSDirect Attached(DS3500 + V3700)SONASDCS3700DCS3700+Elastic Storage ServerGPFSFocus:Raw PerformanceI/O BandwidthElastic Storage: 405 GB/Second of Performance at Argonne National Labs!應(yīng)對(duì)超大規(guī)模數(shù)據(jù)存儲(chǔ)需求 IBM Elastic Storage Server(ESS)Uses GPFS Native RAIDNo controllers; powerLinux se

34、rvers with attached JBODsolJBOD Disk EnclosuresPower Linux ServerPower Linux ServerClientsRDFDR IB10/40 GbEMigrate RAID and Disk Management to Commodity File Servers!olRDCustom DedicatedDisk ControllersJBOD Disk EnclosuresNSD File Server 1NSD File Server 2ClientsGPFS Native RAIDGPFS Native RAIDFile/

35、Data Servers我們?nèi)绾巫龅?！ 減少數(shù)據(jù)轉(zhuǎn)發(fā)37Model GL4Analytics and Cloud4 Enclosures, 20U232 NL-SAS, 2 SSD10+ GB/SecModel GL6PetaScale Storage6 Enclosures, 28U348 NL-SAS, 2 SSD12+ GB/sec Elastic Storage Server GL ModelsModel GL2Analytics Focused2 Enclosures, 12U116 NL-SAS, 2 SSD5+ GB/Sec5146 MachineTypePower S822L

36、 Servers 20 Cores Each1818-80e Expansion ChassisRed Hat 7Graphical User InterfaceManagement Server and HMCElastic Storage SoftwareElastic Storage Native RAIDxCat or Platform Cluster Mgr. Opt.10 Gb, 40 Gb Enet, FDR InifinibandFrom 116 to 348 Spinning Disk3 Years MaitenanceBuilding Block approach to G

37、rowthHigh Capacity Storage for Analytics and Cloud ServingUses 4U, 60 Drive Storage Enclosures2TB or 4 TB DrivesA Client-Ready Petabyte in Single Rack!Elastic Storage Server GS ModelsModel GS124 SSD6 GB/SecModel GS246 SAS + 2 SSD or 48 SSD Drives2 GB/Sec SAS 12 GB/Sec SSDModel GS494 SAS + 2 SSD or 9

38、6 SSD Drives5 GB/Sec SAS16 GB/Sec SSDModel GS6142 SAS + 2 SSD 7 GB/SecSmaller Configurations for High Velocity Ingest or Lower Cost Entry PointUses 2U, 24 Drive Storage Enclosures400, 800 GB SSD Drives or 1.2 TB SAS DrivesHighest “Performance per U” Delivered to ClientsDeployable alone or as part of

39、 an ESS Configuration as “Platinum” tier5146 MachineTypePower S822L - 20 Cores EachPower Expansion ChassisRed Hat 7Graphical User InterfaceManagement Server and HMCElastic Storage SoftwareElastic Storage Native RAIDxCat or Platform Cluster Mgr. Opt.10 Gb, 40 Gb Enet, FDR Inifiniband3 Years MaintenanceBuilding Block approach t