大數(shù)據(jù)分析存儲解決方案 課件

從企業(yè)數(shù)據(jù)向大數(shù)據(jù)的擴展TraditionalApproachStructured,analytical,logicalSystemsofRecordNewApproach

Creative,holisticthought,intuitionSystemsOfEngagementMultimediaSystemsofInsight

EnterpriseIntegration

andContextAccumulationStructured

Repeatable

LinearUnstructured

Exploratory

DynamicDataWarehouseWebLogsSocialDataTextData:

emailsSensordata:

imagesRFIDInternalAppDataTransactionDataMainframeDataOLTPSystemDataHadoopand

StreamsTraditionalSourcesNewSourcesERP

data具備洞悉能力的系統(tǒng)SystemsofInsight從企業(yè)數(shù)據(jù)向大數(shù)據(jù)的擴展TraditionalApproa對新式基礎架構的需求在可靠和安全的環(huán)境中處理關鍵業(yè)務應用存取和處理海量數(shù)據(jù)——包括結(jié)構化和非結(jié)構化數(shù)據(jù)速度及時響應隨時可能出現(xiàn)的商業(yè)機會，這就需要靈活、實時性的基礎架構ThedynamicsofSoRandSoE:通過負載及資源部署的優(yōu)化，來增強靈活性和效益通過采用包括基于開放標準的技術等新技術來改善ITeconomicsSystemofRecord（SoR）SystemsofEngagement（SoE）對的決策對的地方對的時間點BigData&Analytics對新式基礎架構的需求在可靠和安全的環(huán)境中處理關鍵業(yè)務應用Sy大數(shù)據(jù)分析的新型架構解決方案IBMBigData&AnalyticsInfrastructureDataZoneApplicationZone大數(shù)據(jù)分析的新型架構解決方案IBMBigData&A4SmartMeteringGridOperations電網(wǎng)管理FieldService外勤現(xiàn)場服務ResourcePlanning資源規(guī)劃CustomerService/CustomerOperations實現(xiàn)真正的有效的法規(guī)遵從及時發(fā)現(xiàn)能源損耗問題、以及偷電和欺詐行為提高客戶滿意度電量使用預測更為精確電網(wǎng)運維優(yōu)化減少停電次數(shù)和時間案例:SmartMetering智慧電力計費

大數(shù)據(jù)分析應用可以帶來真正的業(yè)務價值法規(guī)遵從4SmartMeteringGridOperations案例:用大數(shù)據(jù)分析來加強

SmartMetering數(shù)據(jù)分析的高可用性，以確保隨時了解用戶喜好跨應用的TB級的數(shù)據(jù)需求–通用虛擬化存儲平臺實時收集、存儲并分析數(shù)據(jù)，最快可達50,000datapoints/sec歷史用電狀態(tài)數(shù)據(jù)的復雜查詢處理數(shù)據(jù)在加載到數(shù)據(jù)倉庫前的清洗、驗證，這些數(shù)據(jù)可能來自很多的用戶、收費系統(tǒng)或斷電保護系統(tǒng)關系掌控

構建和維護電網(wǎng)的唯一試圖對整個企業(yè)的結(jié)構化和非結(jié)構化數(shù)據(jù)t做全局導覽Navigation，從中發(fā)現(xiàn)Discover價值分析用戶用電情況，偵測偷電、改表等行為預測哪些用戶適合于哪些分時時段電價或需求/響應服務分時時段電價的實時定價或

提供及時的需求/響應服務案例:用大數(shù)據(jù)分析來加強SmartMetering數(shù)IBMBigData&AnalyticsReferenceArchitectureBigDataPlatformCapabilitiesInformationIngestReal-timeAnalyticsWarehouse&DataMartsAnalyticAppliancesAllDataSourcesAdvancedAnalytics/

NewInsightsNew/

EnhancedApplicationsCognitive認知LearnDynamically?Prescriptive規(guī)范BestOutcomes?Predictive預測WhatCouldHappen?Descriptive

描述WhatHasHappened?ExplorationandDiscoveryWhatDoYouHave?StreamingDataTextDataApplicationsDataTimeSeriesGeoSpatialRelationalSocialNetworkVideo&ImageAutomatedProcessCaseManagementAnalyticApplicationsWatsonCloudServicesISVSolutionsAlertsIBMBigData&AnalyticsReferNewInfrastructureLeveragesDataTypesDatain

MotionDataat

RestDatain

ManyFormsInformationIngestionandOperationalInformationDecision

ManagementBIandPredictiveAnalyticsNavigation

andDiscoveryIntelligence

AnalysisRawDataStructuredDataTextAnalyticsDataMiningEntityAnalyticsMachineLearningLandingArea,AnalyticsZoneandArchiveVideo/AudioNetwork/SensorEntityAnalyticsPredictiveReal-timeAnalyticsExploration,IntegratedWarehouse,andMartZonesDiscoveryDeepReflectionOperationalPredictive

StreamProcessingDataIntegrationMasterDataStreamsInformationGovernance,SecurityandBusinessContinuityBigInsightsStreamsWarehouseNewInfrastructureLeveragesD大數(shù)據(jù)分析存儲解決方案課件InfoSphereBigInsightsHadoop-based低延遲分析，針對多樣化的、海量靜態(tài)數(shù)據(jù)Data-At-RestNetezzaHighCapacityAppliance基于結(jié)構化數(shù)據(jù)的可查詢歸檔Netezza1000基于結(jié)構化數(shù)據(jù)的

BI+定制化分析DataSmartAnalyticsSystem基于結(jié)構化數(shù)據(jù)的運營分析InformixTimeseriesTime-structuredanalyticsInfoSphereWarehouse基于結(jié)構化數(shù)據(jù)的大容量數(shù)據(jù)分析InfoSphereStreams低延遲流數(shù)據(jù)分析Velocity,Variety&VolumeData-In-MotionMPPDataWarehouseStreamComputingInformationIntegrationHadoopInfoSphereInformationServer海量數(shù)據(jù)集成和轉(zhuǎn)化ApacheHadoop:跨服務器集群的大數(shù)據(jù)集分布式處理開放系統(tǒng)框架，采用的是一種簡單化編程模型IBMBigDataPlatform大數(shù)據(jù)平臺InfoSphereBigInsightsNetezzaWhat:一種開源軟件，將數(shù)據(jù)計算分布到整個集群的常見商用服務器和存儲上Why:傳統(tǒng)的計算架構是一種沿縱向擴展模式，通過更快的SAN、大容量內(nèi)存和多級緩存將數(shù)據(jù)加載到CPU上，成本比較高。What:Hadoop把大數(shù)據(jù)集合拆分區(qū)劃為小數(shù)據(jù)集合，再把小數(shù)據(jù)集合分發(fā)到多臺普通服務器上，是一種橫向擴展模式。Why:Scalable,Flexible,CostEffective,FaultTolerentComponents:MapReduce,HDFSWhatisHadoop?What:一種開源軟件，將數(shù)據(jù)計算分布到整個集群的常見商用NameNode(Metadatastore)NodesHDFSClusterOperatingSystemNodesElasticStorage-SNCClusterKernelLevelIBMValueforHadoop!HDFS把數(shù)據(jù)分散存儲在多個存儲節(jié)點Node上HDFS設計時就假設存儲節(jié)點有失效的可能，所以HDFS會把一份數(shù)據(jù)復制3份以上，分散存儲在多個節(jié)點上，從而實現(xiàn)系統(tǒng)整體上的可靠性HDFS文件系統(tǒng)是由服務器節(jié)點集群組成的，每臺服務器依照HDFS的特有block協(xié)議支持網(wǎng)絡化block數(shù)據(jù)HDFSNameNode有發(fā)生單點故障的危險IBM在改善文件系統(tǒng)的性能同時消除了單點故障

——ElasticStorage-SNC(availableasbetacode)Hadoop說明,MapReduce,HDFSNameNode(Metadatastore)NodesHadoopStackWhatdoesitlooklike?HadoopStackWhatdoesitlook典型Hadoop存儲的PainPoints在選擇HDFS的組件（如軟件、服務器、網(wǎng)絡和存儲等）時很難選對在從測試環(huán)境遷移到生產(chǎn)環(huán)境時，需要做的調(diào)優(yōu)和調(diào)整工作太繁復了長期持續(xù)不斷的運維保障過于繁重，比如老要更換失效組件（尤其是硬盤），這使得保證期望的SLA非常難CPU和存儲去耦本來用戶的CPU和內(nèi)存已經(jīng)滿足計算需求，但為了存儲容量需要安裝更多的硬盤不得不買更多的、不必要的CPU和內(nèi)存Storageoptionsavailablehavecleargaps本地存儲的利用率低(~25%)，每次需要擴容的時候就要添加更多的服務器，而一旦硬盤失效后需要重建，服務器越多，失效的幾率越高，性能也就越差典型Hadoop存儲的PainPoints在選擇HDFS的IBMStorageforHadoop傳統(tǒng)的Hadoop集群使用的是服務器內(nèi)置硬盤存儲。如果用作測試或科學研究還好，可作為業(yè)務運行的存儲就要采用企業(yè)存儲Hadoop集群要負責數(shù)據(jù)保護和復制重建（就是copy）失效的數(shù)據(jù)集到不同節(jié)點上——嚴重影響CPU性能，無法實現(xiàn)企業(yè)級的RASReplicatedata–問題同上擴展的時候同時增加處理器/網(wǎng)絡/存儲，無法做到物盡其用（nowaytoseparatethese3evenifexcesscapacityexistinginone(e.g.NeededmorestoragebuthadtoaddComputeandNetwork)）使用外部存儲可以將存儲負載和Hadoop計算節(jié)點分離，同時還獲得了企業(yè)存儲的好處。SellthevalueofXIV,V7000,SVC,etc.用戶一般會隨HadoopFileSystem部署；采用ElasticStorage可以有很多好處14IBMStorageforHadoop14數(shù)據(jù)加速ExperiencetheinstantresultsthatcomefromIBMFlashSystemDriveasmuchas45X

fasteranalyticsresultsoncertainworkloads數(shù)據(jù)負載的多樣性和靈活性XIVdeliverspredictableperformancethatscaleslinearlywithouthotspotsdeliveringinsightsfromanalyticsfasterwithtuning-freedatadistributionScale-out,parallelprocessingofElasticStoragesoftwareandintegrationwithFlashSystemdramaticallyacceleratesperformanceofAnalyticsclustersVirtualStorageCenterwithSVCautomaticallyoptimizesdatawarehouseperformanceandcostacrossFlashandDiskMainframeDataEnvironmentsIntegrationwithDB2&specialtyanalytics“engines”leveragingDS8870delivers4x

reductioninbatchtimeswithnewHighPerformanceFlashEnclosuresHighspeedencryptiononeverydrivetypesecuresdata數(shù)據(jù)保護和保留

LTFSEEw/tapeprovidesreducedTCObyupto90%overdiskforlongtermretentionofdataatrestwithalargeopenformattaperepositoryReducetheamountofdatatobestoredbyupto25timeswithProtecTIERde-duplication12x更快IBMFlashSystemincreasedSPLUNK&SASapplicationefficiencytoperformbusinessanalytics20x改善

inactionablesupplychainanalytics,4xreductioninbatchtimes,virtualizationforplug&play6x時間節(jié)省“GPFSallowsustomovethemetadatafromthedisktotheFlashSystemonline.Oncewedidthat,thebackupswerereduceddowntoaboutanhour.”

2hrsbecomes2minutes失效切換時間大幅縮短MappingCharacteristicstoIBMStorageProducts數(shù)據(jù)加速12x更快20x改善6x時間節(jié)省2hrsStorageInfrastructure需求適用于所有的5種應用場景

OptimizedMulti-TemperatureWarehouse優(yōu)化的多級存儲庫

AllFlashFlashSystemHybridDS8000EasyTierXIV+SSDCachingStorwizeEasyTierFlashSystemSolution(VSC+FlashSystem)PureSystemsPureFlex(XIVorStorwizew/EasyTier)PureDataforTransactions(Storwize)PureDataforAnalytics(Netezza)StorageInfrastructure需求適用于所有Midrange&EntryTier0AccelerationSmarterStorageIntegratedSystemsEnterpriseOfferingsXIVzEnterpriseSolutionsforAnalyticswithDS8000PureDataSystemforOperationalAnalyticswithStorwizePureFlexSystemwithStorwizeDS8000SmartAnalyticsSystemswithDS3xxxOpen&ExtensibleStorwizefamilyFlashSystemfamilyIBMSmarterStorage的設計就是支持大數(shù)據(jù)分析

高效和優(yōu)化數(shù)據(jù)基礎架構MidrangeSmarterStorageIntegrIBMFlashSystem：為大數(shù)據(jù)分析應用設計的，讓應用和數(shù)據(jù)實現(xiàn)極速IBMFlashSystem的

極速性能

讓實時業(yè)務決策成為可能適合于模塊化數(shù)據(jù)存儲結(jié)構的Hadoop系統(tǒng)。某些或所有數(shù)據(jù)可以保存到Flash閃存上，其他可以保存到XIVIBMFlashSystem：為大數(shù)據(jù)分析應用設計的，讓應IBMXIV:OptimizeddataworkloaddiversityforBigData&AnalyticsIBMXIV的高性能無須人工干預配置，且適用于各種各樣的存儲負載IBMXIV的效率

高的異乎尋常，而且簡單性業(yè)內(nèi)最高，內(nèi)置友好界面IBMXIV

的彈性是企業(yè)級的，完全保證了數(shù)據(jù)的可用性和業(yè)務連續(xù)性IBMXIV:OptimizeddataworkloXIV:為Analytics而生

無與倫比的性能可擴展的網(wǎng)格存儲架構任意時間支持任意讀寫負載板上的閃存Flash

無與倫比的可靠性精致的數(shù)據(jù)分布無雙的磁盤重建時間企業(yè)級的可用性

無與倫比的簡易性簡單的規(guī)劃、供給和靈活性上線后零維護零調(diào)優(yōu)“XIV最吸引我們的地方就是其超強的性能…we正是由于XIV為我們的精細復雜的分析應用提供了一致的高性能，使得我們能夠為我們的用戶帶來更多的價值?！盭IV:為Analytics而生無與倫比的性能可擴展SAS

和XIV網(wǎng)格架構——完美的結(jié)合大規(guī)模并行計算

保持持續(xù)地最佳性能BalancedPerformance性能均衡

常年零調(diào)整UnprecedentedScalability史無前例的擴展性

配合添加SAS節(jié)點和XIV模塊即可SAS和XIV網(wǎng)格架構——完美的結(jié)合大規(guī)模并行計算IBMSVC:OptimizeddataworkloadflexibilityforBigData&AnalyticsIBMSVC通過如下功能在IBM大數(shù)據(jù)產(chǎn)品線上增加了靈活性：完整和數(shù)據(jù)虛擬化和數(shù)據(jù)移動性高級集群和復制多路鏡像，readpreferredoptionRealTimeCompression實時壓縮EasyTierHotExtentcachingStorwizeV7000/UIBMSVCIBMSVC:Optimizeddataworklo設計原則Real-TimeCompression實時壓縮是設計來做：作用于

ActivePrimaryData專用的壓縮平臺PlatformhandlesALLheavyliftingassociatedwithcompression不會影響性能Wemodifyacompressedfilein-placeefficiently不會改變用戶應用Usersnoradminsneedtochangeanything處理流程不變壓縮是在線完成，不是事后壓縮業(yè)界標準壓縮算法所采用的壓縮算法已經(jīng)使用了幾十年StorwizeV7000/UIBMSVC設計原則Real-TimeCompression實時壓縮是24流處理計算&IBMFlashSystems24流處理計算&IBMFlashSystemsData:是擁有還是保存?或是是分析和開始行動!DatainDataat25Data:是擁有還是保存?或是是分析和開始行動!DaInfoSphereStreams:大數(shù)據(jù)流分析為分析動態(tài)數(shù)據(jù)而建多并發(fā)輸入數(shù)據(jù)流大規(guī)?？蓴U展Massivescalability分析和處理的數(shù)據(jù)多樣化Structured,unstructured,video,audioAdvancedanalyticoperators自適應實時分析WithDataWarehousesWithHadoopSystemsInfoSphereStreams:大數(shù)據(jù)流分析為分析動Currentfactfinding當前數(shù)據(jù)查詢分許流動中的數(shù)據(jù)——在數(shù)據(jù)落盤前低延遲模式,pushmodel數(shù)據(jù)驅(qū)動——真正的數(shù)據(jù)分析Historicalfactfinding歷史數(shù)據(jù)查詢查找和分析存儲在磁盤上的數(shù)據(jù)信息批處理模式,pullmodel查詢驅(qū)動:submitsqueriestostaticdataTraditionalComputingStreamComputing流數(shù)據(jù)計算代表著計算模式的變遷Real-timeAnalyticsCurrentfactfinding當前數(shù)據(jù)查詢HistRealTimeAnalytics實時分析

想象一下你如何用防火栓喝水來自多個多樣輸入源的大量數(shù)據(jù)直接處理和過濾數(shù)據(jù)，而不必存儲僅保存有價值的數(shù)據(jù)僅關聯(lián)對數(shù)據(jù)最感興趣的用戶隨著數(shù)據(jù)信息的產(chǎn)生采取行動RealTimeAnalytics實時分析

想象一下你如AdaptiveAnalytics自適應分析

DatainMotionandDataatRest的集成1.DataIngest數(shù)據(jù)集成，數(shù)據(jù)挖掘，機器學習，

統(tǒng)計建模實時和歷史數(shù)據(jù)洞察力的可視化3.AdaptiveAnalyticsModel數(shù)據(jù)收取，

在線分析準備，模式校驗Data2.Bootstrap/EnrichControlflowInfoSphereBigInsights,Database&WarehouseInfoSphereStreamsAdaptiveAnalytics自適應分析

Datai

AdaptiveReal-TimeAnalytics自適應實時分析來自多個多樣輸入源的大量數(shù)據(jù)過去、現(xiàn)在和未來全方位綜合性視圖實時分析，低延時結(jié)果Fullcontextfordeepanalysis深度分析的完整的上下文跨datainmotionanddataatrest的常用數(shù)據(jù)分析自適應-隨機而變當發(fā)現(xiàn)非預期行為時，自適應當識別出新數(shù)據(jù)意義時深度分析之開始沒有意識到的數(shù)據(jù)意義，隨后才可能意識到自適應——在開始沒有意識到的，隨后可以找出數(shù)據(jù)模式

AdaptiveReal-TimeAnalyticsStockmarketImpactofweatheronsecuritiespricesAnalyzemarketdataatultra-lowlatenciesMomentumCalculatorFraudpreventionDetectingmulti-partyfraudRealtimefraudpreventione-ScienceSpaceweatherpredictionDetectionoftransienteventsSynchrotronatomicresearchGenomicResearchTransportationIntelligenttrafficmanagementAutomotiveTelematicsEnergy&UtilitiesTransactivecontrolPhasorMonitoringUnitDownholesensormonitoringNaturalSystemsWildfiremanagementWatermanagementOtherManufacturingTextAnalysisERPforCommoditiesReal-timemultimodalsurveillanceSituationalawarenessCybersecuritydetectionLawEnforcement,

Defense&CyberSecurityHealth&LifeSciencesICUmonitoringEpidemicearlywarningsystemRemotehealthcaremonitoringTelephonyCDRprocessingSocialanalysisChurnpredictionGeomapping如何使用InfoSphereStreams?StockmarketFraudpreventione-加快數(shù)據(jù)流入分析系統(tǒng)的速度向交易方向加速。。。一個高效和靈活的基礎架構顯然可以加快流速，并平衡不同數(shù)據(jù)分析的需求CoresSCMStorageNetworkCoresSCMStorageNetworkCoresSCMStorageNetworkCoresSCMStorageNetwork++預測分析

數(shù)據(jù)倉庫文本分析HadoopWorkloads優(yōu)化敏感性分析加快流速價值時間“觸發(fā)事件”數(shù)據(jù)完備交易Insight預見獲取數(shù)據(jù)時間分析數(shù)據(jù)時間行動時間加快數(shù)據(jù)流入分析系統(tǒng)的速度向交易方向加速。。。一個高效和靈活大數(shù)據(jù)分析的新式基礎架構解決方案IBMBigData&AnalyticsInfrastructureDataZoneApplicationZone大數(shù)據(jù)分析的新式基礎架構解決方案IBMBigData&Experiencereal-timeanalyticalinsightswithupto50xbetterperformancethanenterprisedisksystemsusingIBMFlashCore?technologyPreserveandprotectinfrastructurecontinuitywhilescalingtoover2petabyteofeffectiveall-flashcapacityunderasingleintegrateinterfaceDeliveragilityanddataeconomicswith4xgreatercapacityinlessrackspacethancompetitiveall-flashproductsSynchronizedandComplimentarytoOverarchingStorageMessaging-Acceleratetimetoinsightsthrough"datawithoutborders."IBMinnovationfreesdatawithagileandsimpletousestoragesolutionsdeliveringsuperiordataeconomicsIBMFlashSystemCoreLaunchMessagingDriveacompleteparadigmshiftinEnterpriseStoragewiththeallnewIBMFlashSystemFamilyExperiencereal-timeanalyticaIBMFlashSystemFamily

2015ThemeTimetoinsight.Timetovalue.Timetomarket.IBMFlashSystem,it’sabouttime.FlashRealized!IBMFlashSystemFamily

2015ThIBMFlashSystemV9000

FoundationalPillarsIBMFlashCore?TechnologyistheDNAoftheFlashSystemFamilyScalablePerformanceEnduringEconomicsAgileIntegrationIBMFlashSystemV9000

FoundatiIntroducingtheNewIBMFlashSystemFamilyOfferingsIBMFlashSystem900ExtremePerformance:Delivers100microsecondresponsetimesMacroEfficiency:Lowestlatencyofferingwith>40%greatercapacityatalowercostpercapacityEnterpriseReliability:IBMenhancedMicronMLCflashtechnologywithFlashWearGuaranteePoweredbyIBMFlashCore?TechnologyIBMFlashSystemV9000ScalablePerformance:Growcapacityandperformancewithupto2.2PBscalingcapabilityEnduringEconomics:NextgenerationflashmediawithlowercostpercapacityAgileIntegration:FullyintegratedsystemmanagementtosimplifymanagementandimproveworkforceproductivityunderasinglenamespaceIntroducingtheNewIBMFlashSFlashSystem900IntroducingIBMFlashSystem900,thenextgenerationinourlowestlatencyofferingIBMMicroLatency?withupto1.1millionIOPS40%greatercapacityata10%lowercostpercapacityIBMFlashCore?technology,oursecretsauceTechnicalcollaborationwithMicronTechnology,ourflashchipsupplierIBMenhancedflashtechnologyMLCNANDflashofferingwithFlashWearGuaranteeVAAIUNMAPandVASAsupportwithIBMSISforimprovedcloudstorageperformanceandefficiencyMinimumlatency

Write90μsRead155μsMaximumIOPS4KBRead(100%,random)1,100,00Read/write(70%/30%,random)800,000Write(100%,random)600,000Maximumbandwidth256KBRead(100%,sequential)10GB/sWrite(100%,sequential)4.5GB/sPerformanceat-a-glanceIBMMicroLatencymoduletype1.2TB2.9TB5.7TBModulesquantity4681012681012681012RAID5capacity(TB)9.61211.617.423.229.022.834.245.657.0RawCapacity(TB)7.110.714.217.821.426.335.143.952.752.770.387.9105.5FlashSystem900IntroducingIBMIBMintroducesafullyintegrated,fullymanaged,fullfunctionall-flashstoragesystemFlashSystemV9000Scalableall-flasharchitecturewithfullsetofadvanceddatafeaturesPerformsatupto2.5MIOPSwithIBMMicroLatency,scalableto19.2GB/sScalesto456TBusableandupto2.28PBeffectivecapacityinonly34UUpto57TBusableandupto285TBeffectivecapacityinonly6UNewlicensingstructuretosimplifyorderingandplanningforExternalDataVirtualization,FlashCopy,MetroMirror,andReal-timeCompressionScalablePerformanceAgileIntegrationEnduringEconomicsPoweredbyFlashCore?TechnologyIBMintroducesafullyintegra從企業(yè)數(shù)據(jù)向大數(shù)據(jù)的擴展TraditionalApproachStructured,analytical,logicalSystemsofRecordNewApproach

Creative,holisticthought,intuitionSystemsOfEngagementMultimediaSystemsofInsight

EnterpriseIntegration

andContextAccumulationStructured

Repeatable

LinearUnstructured

Exploratory

DynamicDataWarehouseWebLogsSocialDataTextData:

emailsSensordata:

imagesRFIDInternalAppDataTransactionDataMainframeDataOLTPSystemDataHadoopand

StreamsTraditionalSourcesNewSourcesERP

data具備洞悉能力的系統(tǒng)SystemsofInsight從企業(yè)數(shù)據(jù)向大數(shù)據(jù)的擴展TraditionalApproa對新式基礎架構的需求在可靠和安全的環(huán)境中處理關鍵業(yè)務應用存取和處理海量數(shù)據(jù)——包括結(jié)構化和非結(jié)構化數(shù)據(jù)速度及時響應隨時可能出現(xiàn)的商業(yè)機會，這就需要靈活、實時性的基礎架構ThedynamicsofSoRandSoE:通過負載及資源部署的優(yōu)化，來增強靈活性和效益通過采用包括基于開放標準的技術等新技術來改善ITeconomicsSystemofRecord（SoR）SystemsofEngagement（SoE）對的決策對的地方對的時間點BigData&Analytics對新式基礎架構的需求在可靠和安全的環(huán)境中處理關鍵業(yè)務應用Sy大數(shù)據(jù)分析的新型架構解決方案IBMBigData&AnalyticsInfrastructureDataZoneApplicationZone大數(shù)據(jù)分析的新型架構解決方案IBMBigData&A43SmartMeteringGridOperations電網(wǎng)管理FieldService外勤現(xiàn)場服務ResourcePlanning資源規(guī)劃CustomerService/CustomerOperations實現(xiàn)真正的有效的法規(guī)遵從及時發(fā)現(xiàn)能源損耗問題、以及偷電和欺詐行為提高客戶滿意度電量使用預測更為精確電網(wǎng)運維優(yōu)化減少停電次數(shù)和時間案例:SmartMetering智慧電力計費

大數(shù)據(jù)分析應用可以帶來真正的業(yè)務價值法規(guī)遵從4SmartMeteringGridOperations案例:用大數(shù)據(jù)分析來加強

SmartMetering數(shù)據(jù)分析的高可用性，以確保隨時了解用戶喜好跨應用的TB級的數(shù)據(jù)需求–通用虛擬化存儲平臺實時收集、存儲并分析數(shù)據(jù)，最快可達50,000datapoints/sec歷史用電狀態(tài)數(shù)據(jù)的復雜查詢處理數(shù)據(jù)在加載到數(shù)據(jù)倉庫前的清洗、驗證，這些數(shù)據(jù)可能來自很多的用戶、收費系統(tǒng)或斷電保護系統(tǒng)關系掌控

構建和維護電網(wǎng)的唯一試圖對整個企業(yè)的結(jié)構化和非結(jié)構化數(shù)據(jù)t做全局導覽Navigation，從中發(fā)現(xiàn)Discover價值分析用戶用電情況，偵測偷電、改表等行為預測哪些用戶適合于哪些分時時段電價或需求/響應服務分時時段電價的實時定價或

提供及時的需求/響應服務案例:用大數(shù)據(jù)分析來加強SmartMetering數(shù)IBMBigData&AnalyticsReferenceArchitectureBigDataPlatformCapabilitiesInformationIngestReal-timeAnalyticsWarehouse&DataMartsAnalyticAppliancesAllDataSourcesAdvancedAnalytics/

NewInsightsNew/

EnhancedApplicationsCognitive認知LearnDynamically?Prescriptive規(guī)范BestOutcomes?Predictive預測WhatCouldHappen?Descriptive

描述WhatHasHappened?ExplorationandDiscoveryWhatDoYouHave?StreamingDataTextDataApplicationsDataTimeSeriesGeoSpatialRelationalSocialNetworkVideo&ImageAutomatedProcessCaseManagementAnalyticApplicationsWatsonCloudServicesISVSolutionsAlertsIBMBigData&AnalyticsReferNewInfrastructureLeveragesDataTypesDatain

MotionDataat

RestDatain

ManyFormsInformationIngestionandOperationalInformationDecision

ManagementBIandPredictiveAnalyticsNavigation

andDiscoveryIntelligence

AnalysisRawDataStructuredDataTextAnalyticsDataMiningEntityAnalyticsMachineLearningLandingArea,AnalyticsZoneandArchiveVideo/AudioNetwork/SensorEntityAnalyticsPredictiveReal-timeAnalyticsExploration,IntegratedWarehouse,andMartZonesDiscoveryDeepReflectionOperationalPredictive

StreamProcessingDataIntegrationMasterDataStreamsInformationGovernance,SecurityandBusinessContinuityBigInsightsStreamsWarehouseNewInfrastructureLeveragesD大數(shù)據(jù)分析存儲解決方案課件InfoSphereBigInsightsHadoop-based低延遲分析，針對多樣化的、海量靜態(tài)數(shù)據(jù)Data-At-RestNetezzaHighCapacityAppliance基于結(jié)構化數(shù)據(jù)的可查詢歸檔Netezza1000基于結(jié)構化數(shù)據(jù)的

BI+定制化分析DataSmartAnalyticsSystem基于結(jié)構化數(shù)據(jù)的運營分析InformixTimeseriesTime-structuredanalyticsInfoSphereWarehouse基于結(jié)構化數(shù)據(jù)的大容量數(shù)據(jù)分析InfoSphereStreams低延遲流數(shù)據(jù)分析Velocity,Variety&VolumeData-In-MotionMPPDataWarehouseStreamComputingInformationIntegrationHadoopInfoSphereInformationServer海量數(shù)據(jù)集成和轉(zhuǎn)化ApacheHadoop:跨服務器集群的大數(shù)據(jù)集分布式處理開放系統(tǒng)框架，采用的是一種簡單化編程模型IBMBigDataPlatform大數(shù)據(jù)平臺InfoSphereBigInsightsNetezzaWhat:一種開源軟件，將數(shù)據(jù)計算分布到整個集群的常見商用服務器和存儲上Why:傳統(tǒng)的計算架構是一種沿縱向擴展模式，通過更快的SAN、大容量內(nèi)存和多級緩存將數(shù)據(jù)加載到CPU上，成本比較高。What:Hadoop把大數(shù)據(jù)集合拆分區(qū)劃為小數(shù)據(jù)集合，再把小數(shù)據(jù)集合分發(fā)到多臺普通服務器上，是一種橫向擴展模式。Why:Scalable,Flexible,CostEffective,FaultTolerentComponents:MapReduce,HDFSWhatisHadoop?What:一種開源軟件，將數(shù)據(jù)計算分布到整個集群的常見商用NameNode(Metadatastore)NodesHDFSClusterOperatingSystemNodesElasticStorage-SNCClusterKernelLevelIBMValueforHadoop!HDFS把數(shù)據(jù)分散存儲在多個存儲節(jié)點Node上HDFS設計時就假設存儲節(jié)點有失效的可能，所以HDFS會把一份數(shù)據(jù)復制3份以上，分散存儲在多個節(jié)點上，從而實現(xiàn)系統(tǒng)整體上的可靠性HDFS文件系統(tǒng)是由服務器節(jié)點集群組成的，每臺服務器依照HDFS的特有block協(xié)議支持網(wǎng)絡化block數(shù)據(jù)HDFSNameNode有發(fā)生單點故障的危險IBM在改善文件系統(tǒng)的性能同時消除了單點故障

——ElasticStorage-SNC(availableasbetacode)Hadoop說明,MapReduce,HDFSNameNode(Metadatastore)NodesHadoopStackWhatdoesitlooklike?HadoopStackWhatdoesitlook典型Hadoop存儲的PainPoints在選擇HDFS的組件（如軟件、服務器、網(wǎng)絡和存儲等）時很難選對在從測試環(huán)境遷移到生產(chǎn)環(huán)境時，需要做的調(diào)優(yōu)和調(diào)整工作太繁復了長期持續(xù)不斷的運維保障過于繁重，比如老要更換失效組件（尤其是硬盤），這使得保證期望的SLA非常難CPU和存儲去耦本來用戶的CPU和內(nèi)存已經(jīng)滿足計算需求，但為了存儲容量需要安裝更多的硬盤不得不買更多的、不必要的CPU和內(nèi)存Storageoptionsavailablehavecleargaps本地存儲的利用率低(~25%)，每次需要擴容的時候就要添加更多的服務器，而一旦硬盤失效后需要重建，服務器越多，失效的幾率越高，性能也就越差典型Hadoop存儲的PainPoints在選擇HDFS的IBMStorageforHadoop傳統(tǒng)的Hadoop集群使用的是服務器內(nèi)置硬盤存儲。如果用作測試或科學研究還好，可作為業(yè)務運行的存儲就要采用企業(yè)存儲Hadoop集群要負責數(shù)據(jù)保護和復制重建（就是copy）失效的數(shù)據(jù)集到不同節(jié)點上——嚴重影響CPU性能，無法實現(xiàn)企業(yè)級的RASReplicatedata–問題同上擴展的時候同時增加處理器/網(wǎng)絡/存儲，無法做到物盡其用（nowaytoseparatethese3evenifexcesscapacityexistinginone(e.g.NeededmorestoragebuthadtoaddComputeandNetwork)）使用外部存儲可以將存儲負載和Hadoop計算節(jié)點分離，同時還獲得了企業(yè)存儲的好處。SellthevalueofXIV,V7000,SVC,etc.用戶一般會隨HadoopFileSystem部署；采用ElasticStorage可以有很多好處53IBMStorageforHadoop14數(shù)據(jù)加速ExperiencetheinstantresultsthatcomefromIBMFlashSystemDriveasmuchas45X

fasteranalyticsresultsoncertainworkloads數(shù)據(jù)負載的多樣性和靈活性XIVdeliverspredictableperformancethatscaleslinearlywithouthotspotsdeliveringinsightsfromanalyticsfasterwithtuning-freedatadistributionScale-out,parallelprocessingofElasticStoragesoftwareandintegrationwithFlashSystemdramaticallyacceleratesperformanceofAnalyticsclustersVirtualStorageCenterwithSVCautomaticallyoptimizesdatawarehouseperformanceandcostacrossFlashandDiskMainframeDataEnvironmentsIntegrationwithDB2&specialtyanalytics“engines”leveragingDS8870delivers4x

reductioninbatchtimeswithnewHighPerformanceFlashEnclosuresHighspeedencryptiononeverydrivetypesecuresdata數(shù)據(jù)保護和保留

LTFSEEw/tapeprovidesreducedTCObyupto90%overdiskforlongtermretentionofdataatrestwithalargeopenformattaperepositoryReducetheamountofdatatobestoredbyupto25timeswithProtecTIERde-duplication12x更快IBMFlashSystemincreasedSPLUNK&SASapplicationefficiencytoperformbusinessanalytics20x改善

inactionablesupplychainanalytics,4xreductioninbatchtimes,virtualizationforplug&play6x時間節(jié)省“GPFSallowsustomovethemetadatafromthedisktotheFlashSystemonline.Oncewedidthat,thebackupswerereduceddowntoaboutanhour.”

2hrsbecomes2minutes失效切換時間大幅縮短MappingCharacteristicstoIBMStorageProducts數(shù)據(jù)加速12x更快20x改善6x時間節(jié)省2hrsStorageInfrastructure需求適用于所有的5種應用場景

OptimizedMulti-TemperatureWarehouse優(yōu)化的多級存儲庫

AllFlashFlashSystemHybridDS8000EasyTierXIV+SSDCachingStorwizeEasyTierFlashSystemSolution(VSC+FlashSystem)PureSystemsPureFlex(XIVorStorwizew/EasyTier)PureDataforTransactions(Storwize)PureDataforAnalytics(Netezza)StorageInfrastructure需求適用于所有Midrange&EntryTier0AccelerationSmarterStorageIntegratedSystemsEnterpriseOfferingsXIVzEnterpriseSolutionsforAnalyticswithDS8000PureDataSystemforOperationalAnalyticswithStorwizePureFlexSystemwithStorwizeDS8000SmartAnalyticsSystemswithDS3xxxOpen&ExtensibleStorwizefamilyFlashSystemfamilyIBMSmarterStorage的設計就是支持大數(shù)據(jù)分析

高效和優(yōu)化數(shù)據(jù)基礎架構MidrangeSmarterStorageIntegrIBMFlashSystem：為大數(shù)據(jù)分析應用設計的，讓應用和數(shù)據(jù)實現(xiàn)極速IBMFlashSystem的

極速性能

讓實時業(yè)務決策成為可能適合于模塊化數(shù)據(jù)存儲結(jié)構的Hadoop系統(tǒng)。某些或所有數(shù)據(jù)可以保存到Flash閃存上，其他可以保存到XIVIBMFlashSystem：為大數(shù)據(jù)分析應用設計的，讓應IBMXIV:OptimizeddataworkloaddiversityforBigData&AnalyticsIBMXIV的高性能無須人工干預配置，且適用于各種各樣的存儲負載IBMXIV的效率

高的異乎尋常，而且簡單性業(yè)內(nèi)最高，內(nèi)置友好界面IBMXIV

的彈性是企業(yè)級的，完全保證了數(shù)據(jù)的可用性和業(yè)務連續(xù)性IBMXIV:OptimizeddataworkloXIV:為Analytics而生

無與倫比的性能可擴展的網(wǎng)格存儲架構任意時間支持任意讀寫負載板上的閃存Flash

無與倫比的可靠性精致的數(shù)據(jù)分布無雙的磁盤重建時間企業(yè)級的可用性

無與倫比的簡易性簡單的規(guī)劃、供給和靈活性上線后零維護零調(diào)優(yōu)“XIV最吸引我們的地方就是其超強的性能…we正是由于XIV為我們的精細復雜的分析應用提供了一致的高性能，使得我們能夠為我們的用戶帶來更多的價值?！盭IV:為Analytics而生無與倫比的性能可擴展SAS

和XIV網(wǎng)格架構——完美的結(jié)合大規(guī)模并行計算

保持持續(xù)地最佳性能BalancedPerformance性能均衡

常年零調(diào)整UnprecedentedScalability史無前例的擴展性

配合添加SAS節(jié)點和XIV模塊即可SAS和XIV網(wǎng)格架構——完美的結(jié)合大規(guī)模并行計算IBMSVC:OptimizeddataworkloadflexibilityforBigData&AnalyticsIBMSVC通過如下功能在IBM大數(shù)據(jù)產(chǎn)品線上增加了靈活性：完整和數(shù)據(jù)虛擬化和數(shù)據(jù)移動性高級集群和復制多路鏡像，readpreferredoptionRealTimeCompression實時壓縮EasyTierHotExtentcachingStorwizeV7000/UIBMSVCIBMSVC:Optimizeddataworklo設計原則Real-TimeCompression實時壓縮是設計來做：作用于

ActivePrimaryData專用的壓縮平臺PlatformhandlesALLheavyliftingassociatedwithcompression不會影響性能Wemodifyacompressedfilein-placeefficiently不會改變用戶應用Usersnoradminsneedtochangeanything處理流程不變壓縮是在線完成，不是事后壓縮業(yè)界標準壓縮算法所采用的壓縮算法已經(jīng)使用了幾十年StorwizeV7000/UIBMSVC設計原則Real-TimeCompression實時壓縮是63流處理計算&IBMFlashSystems24流處理計算&IBMFlashSystemsData:是擁有還是保存?或是是分析和開始行動!DatainDataat64Data:是擁有還是保存?或是是分析和開始行動!DaInfoSphereStreams:大數(shù)據(jù)流分析為分析動態(tài)數(shù)據(jù)而建多并發(fā)輸入數(shù)據(jù)流大規(guī)?？蓴U展Massivescalability分析和處理的數(shù)據(jù)多樣化Structured,unstructured,video,audioAdvancedanalyticoperators自適應實時分析WithDataWarehousesWithHadoopSystemsInfoSphereStreams:大數(shù)據(jù)流分析為分析動Currentfactfinding當前數(shù)據(jù)查詢分許流動中的數(shù)據(jù)——在數(shù)據(jù)落盤前低延遲模式,pushmodel數(shù)據(jù)驅(qū)動——真正的數(shù)據(jù)分析Historicalfactfinding歷史數(shù)據(jù)查詢查找和分析存儲在磁盤上的數(shù)據(jù)信息批處理模式,pullmodel查詢驅(qū)動:submitsqueriestostaticdataTraditionalComputingStreamComputing流數(shù)據(jù)計算代表著計算模式的變遷Real-timeAnalyticsCurrentfactfinding當前數(shù)據(jù)查詢HistRealTimeAnalytics實時分析

想象一下你如何用防火栓喝水來自多個多樣輸入源的大量數(shù)據(jù)直接處理和過濾數(shù)據(jù)，而不必存儲僅保存有價值的數(shù)據(jù)僅關聯(lián)對數(shù)據(jù)最感興趣的用戶隨著數(shù)據(jù)信息的產(chǎn)生采取行動RealTimeAnalytics實時分析

想象一下你如AdaptiveAnalytics自適應分析

DatainMotionandDataatRest的集成1.DataIngest數(shù)據(jù)集成，數(shù)據(jù)挖掘，機器學習，

統(tǒng)計建模實時和歷史數(shù)據(jù)洞察力的可視化3.AdaptiveAnalyticsModel數(shù)據(jù)收取，

在線分析準備，模式校驗Data2.Bootstrap/EnrichControlflowInfoSphereBigInsights,Database&WarehouseInfoSphereStreamsAdaptiveAnalytics自適應分析

Datai

AdaptiveReal-TimeAnalytics自適應實時分析來自多個多樣輸入源的大量數(shù)據(jù)過去、現(xiàn)在和未來全方位綜合性視圖實時分析，低延時結(jié)果Fullcontextfordeepanalysis深度分析的完整的上下文跨datainmotionanddataatrest的常用數(shù)據(jù)分析自適應-隨機而變當發(fā)現(xiàn)非預期行為時，自適應當識別出新數(shù)據(jù)意義時深度分析之開始沒有意識到的數(shù)據(jù)意義，隨后才可能意識到自適應——在開始沒有意識到的，隨后可以找出數(shù)據(jù)模式

AdaptiveReal-TimeAnalyticsStockmarketImpactofweatheronsecuritiespricesAnalyzemarketdataatultra-lowlatenciesMomentumCalculatorFraudpreventionDetectingmulti-partyfraudRealtimefraudpreventione-ScienceSpaceweatherpredictionDetectionoftransienteventsSynchrotronatomicresearchGenomicResearchTransportationIntelligenttrafficmanagementAutomotiveTelematicsEnergy&UtilitiesTransactivecontrolPhasorMonitoringUnitDownholesensormonitoringNaturalSystemsWildfiremanagementWatermanagementOtherManufacturingTextAnalysisERPforCommoditiesReal-timemultimodalsurveillanceSituationalawarenessCybersecuritydetectionLawEnforcement,

Defense&CyberSecurityHealth&LifeSciencesICUmonitoringEpidemicearlywarningsystemRemotehealthcaremonitoringTelephonyCDRprocessingSocialanalysisChurnpredictionGeomapping如何使用InfoSphereStreams?StockmarketFraudpreventione-加快數(shù)據(jù)流入分析系統(tǒng)的速度向交易方向加速。。。一個高效和靈活的基礎架構顯然可以加快流速，并平衡不同數(shù)據(jù)分析的需求CoresSCMStorageNetworkCoresSCMStorageNetworkCoresSCMStorageNetworkCoresSCMStorageNetwork++預測分析

數(shù)據(jù)倉庫文本分析HadoopWorkloads優(yōu)化敏感性分析加快流速價值時間“觸發(fā)事件”數(shù)據(jù)完備交易Insight預