并行數(shù)據(jù)庫(kù)技術(shù)分析及未來展望

并行數(shù)據(jù)庫(kù)技術(shù)分析及未來展望Agenda并行數(shù)據(jù)庫(kù)技術(shù)要點(diǎn)分析未來展望Paralleldatabasesystemseekstoimproveperformancethroughparallelizationofvariousoperationssuchasloadingdata,buildingindexes,andevaluatingqueriesbyusingmultipleCPUsandDisksinParallel.[1]MotivationParallelmachinesarebecomingquitecommonandaffordablePricesofmicroprocessors,memoryanddiskshavedroppedsharplyDatabasesaregrowingincreasinglylargelargevolumesoftransactiondataarecollectedandstoredforlateranalysis.multimediaobjectslikeimagesareincreasinglystoredindatabases并行數(shù)據(jù)庫(kù)

ImprovesResponseTime.

INTERQUERYPARALLELISM

Itispossibletoprocessanumberoftransactionsinparallelwitheachother.

ImprovesThroughput.

INTRAQUERYPARALLELISMItispossibletoprocess‘sub-tasks’ofatransaction

inparallelwitheachother.并行數(shù)據(jù)庫(kù)–目標(biāo)(1)

Speed-UpAddingmoreresourcesresultsinproportionallylessrunningtimeforafixedamountofdata.10secondstoscanaDBof10,000recordsusing1CPU1secondtoscanaDBof10,000recordsusing10CPUs

Scale-Up(transaction/data)Ifresourcesareincreasedinproportiontoanincreaseindata/problemsize,theoveralltimeshouldremainconstant1secondtoscanaDBof1,000recordsusing1CPU

1secondtoscanaDBof10,000recordsusing10CPUs并行數(shù)據(jù)庫(kù)–目標(biāo)(2)Shared-MemoryandShared-DiskArchitecturesShared-Memory:allprocessorsshareacommonmainmemoryandsecondarymemoryLoadbalancingisrelativelyeasytoachieve,butsufferfrommemoryandbuscontentionShared-Disk:allprocessors,eachofwhichhasitsownlocalmainmemory,sharethedisks并行數(shù)據(jù)庫(kù)-體系結(jié)構(gòu)(1)Shared-NothingArchitectureEachprocessorhasitsownlocalmainmemoryanddisksLoadbalancingbecomesdifficult并行數(shù)據(jù)庫(kù)-體系結(jié)構(gòu)(2)Shared-SomethingArchitectureAmixtureofshared-memoryandshared-nothingarchitecturesEachnodeisashared-memoryarchitectureconnectedtoaninterconnectionnetworkalashared-nothingarchitecture并行數(shù)據(jù)庫(kù)-體系結(jié)構(gòu)(3)元數(shù)據(jù)管理查詢分析和執(zhí)行器分布式事務(wù)分布式鎖并發(fā)控制副本數(shù)據(jù)同步并行數(shù)據(jù)庫(kù)–功能模塊Agenda架構(gòu)特點(diǎn)技術(shù)要點(diǎn)分析未來展望2/5/2023111.ParallelDB/D.S.Jagli111.ParallelDB/D.S.Jagli技術(shù)要點(diǎn)–層次結(jié)構(gòu)MasterServer:外部App訪問入口集群元數(shù)據(jù)管理SQLParser生成執(zhí)行計(jì)劃和任務(wù)調(diào)度2/5/2023121.ParallelDB/D.S.Jagli12Range

HashRoundRobinA...EF...JK...NO...ST...ZA...EF...JK...NO...ST...ZA...EF...JK...NO...ST...ZGoodforequijoins,exact-matchqueries,andrangequeriesGoodforequijoins,exactmatchqueriesGoodtospreadload1.ParallelDB/D.S.Jagli技術(shù)要點(diǎn)–數(shù)據(jù)分區(qū)2/5/2023131.ParallelDB/D.S.Jagli131.ParallelDB/D.S.Jagli技術(shù)要點(diǎn)–存儲(chǔ)和索引(1)列存儲(chǔ)(columnstore)數(shù)據(jù)按列存儲(chǔ)，每一列單獨(dú)存放只訪問查詢涉及的列，大量降低I/O數(shù)據(jù)類型一致，方便壓縮數(shù)據(jù)包建索引，數(shù)據(jù)即索引行存儲(chǔ)(rowstore)數(shù)據(jù)是按行存儲(chǔ)的 沒有索引的查詢使用大量I/O建立索引和物化視圖需要花費(fèi)大量時(shí)間和資源面對(duì)查詢的需求，數(shù)據(jù)庫(kù)必須被大量膨脹才能滿足性能要求。技術(shù)要點(diǎn)–存儲(chǔ)和索引(2)混合存儲(chǔ)(hybridstore)在大尺度上數(shù)據(jù)是按行存儲(chǔ)，在小尺度上數(shù)據(jù)是按列存儲(chǔ)在大尺度上以行作為任務(wù)切分，在小尺度上以列進(jìn)行I/O優(yōu)化技術(shù)要點(diǎn)–存儲(chǔ)和索引(3)模糊索引精確索引

B+樹

LSM

COLA3.列存索引(Projection）

技術(shù)要點(diǎn)–查詢引擎技術(shù)要點(diǎn)–分布式事務(wù)(1)技術(shù)要點(diǎn)–分布式事務(wù)(2)技術(shù)要點(diǎn)–任務(wù)調(diào)度與資源管理技術(shù)要點(diǎn)–高可用技術(shù)要點(diǎn)–集群擴(kuò)展數(shù)據(jù)庫(kù)對(duì)一致性的天然要求數(shù)據(jù)庫(kù)元數(shù)據(jù)管理數(shù)據(jù)分布的再平衡技術(shù)要點(diǎn)–備份和災(zāi)難恢復(fù)在線增量備份。支持HDFS作為備份存儲(chǔ)介質(zhì)，HDFS上的備份數(shù)據(jù)可通過Hadoop平臺(tái)直接進(jìn)行分析。以異地災(zāi)備替代常規(guī)備份操作。異地?cái)?shù)據(jù)中心之間實(shí)現(xiàn)同步/異步操作。備份和恢復(fù)兩個(gè)過程，以及異地災(zāi)備中的主從集群可實(shí)現(xiàn)異構(gòu)恢復(fù)。技術(shù)要點(diǎn)–SSD應(yīng)用場(chǎng)景數(shù)據(jù)量大，熱點(diǎn)數(shù)據(jù)無法全部在緩存中。用戶訪問I/O量大，造成系統(tǒng)響應(yīng)慢系統(tǒng)整體壓力低，QPS值無法提高系統(tǒng)切換后，預(yù)熱時(shí)間長(zhǎng)等于故障Agenda并行數(shù)據(jù)庫(kù)技術(shù)要點(diǎn)分析未來展望未來展望–并行數(shù)據(jù)庫(kù)在大數(shù)據(jù)基礎(chǔ)設(shè)置中的定位(1)適應(yīng)已有的數(shù)據(jù)庫(kù)生態(tài)，平滑替代無法跟上快速增長(zhǎng)的數(shù)據(jù)量的傳統(tǒng)EDW，連接BI工具。EDW工具的復(fù)雜性和對(duì)復(fù)雜SQL的支持程度，超過了目前Hadoop的能力。IngestLandingandAnalyticsSandboxZoneIndexes,facetsHive/HBaseColStoresDocumentsInVarietyofFormatsAnalyticsMapReduceIngestionandReal-timeAnalyticZoneDataSinksFilter,TransformIngestCorrelate,ClassifyExtract,AnnotateWarehousingZoneEnterpriseWarehouseDataMartsQueryEnginesCubesDescriptive,PredictiveModelsModelsWidgetsDiscovery,VisualizerSearchAnalyticsandReportingZoneMetadataandGovernanceZoneConnectors未來展望–并行數(shù)據(jù)庫(kù)與Hadoop未來展望–并行數(shù)據(jù)庫(kù)與Hadoop集成(SQL-on-HadoopGoogle,Mega，Spanner和F1向我們展現(xiàn)了跨數(shù)據(jù)中心數(shù)據(jù)庫(kù)的可能。Cloudera開源了大數(shù)據(jù)查詢分析引擎。ImpalaHortonworks開源了StingerFackbook開源了Presto類似Pregel，UCBerkeleyAMPLAB實(shí)驗(yàn)室開發(fā)了Spark圖計(jì)算框架，并以Spark為核心開源了大數(shù)據(jù)查詢分析引擎Shark。EMC的PivotalHAWQHadapt的HybridStorageEngine未來展望–并行數(shù)據(jù)庫(kù)與Hadoop集成(Connector)未來展望–并行數(shù)據(jù)庫(kù)與Hadoop集成(Hybrid)未來展望–IT基礎(chǔ)設(shè)施的變化新的IT基礎(chǔ)設(shè)置，如企業(yè)云環(huán)境，一體機(jī)，存儲(chǔ)平臺(tái)SDS(SoftwareDefineStorage)都解決了傳統(tǒng)MPP數(shù)據(jù)庫(kù)中的存儲(chǔ)高可用問題。新一代的MPP數(shù)據(jù)庫(kù)，將不再需要在MPP內(nèi)部進(jìn)行存儲(chǔ)層副本的管理和高可用機(jī)制，這將極大簡(jiǎn)化MPP實(shí)現(xiàn)的復(fù)雜度。MPP數(shù)據(jù)庫(kù)將更傾向于具有事務(wù)能力的SQLSuperComputer，以及通過對(duì)存儲(chǔ)層的適配，達(dá)到最高性能。

Figure5.Tenant-relatedfactorsandhowtheyaffect“

isolatedversusshared"dataarchitecturedecisions未來展望–IT基礎(chǔ)設(shè)施的變化（多租戶）未來展望–并行數(shù)據(jù)庫(kù)技術(shù)的發(fā)展實(shí)現(xiàn)跨不同地理位置數(shù)據(jù)中心數(shù)據(jù)庫(kù)。例如,Google的Mega加強(qiáng)數(shù)據(jù)庫(kù)內(nèi)部的資源管理機(jī)制和任務(wù)的優(yōu)先級(jí)調(diào)用實(shí)現(xiàn)完整的事務(wù)和并發(fā)控制能力。擴(kuò)展支持應(yīng)用類型的范圍，逐漸模糊NOSQL與SQL的不同。例如，Vertica中的FlexEngine支持類似MongoDB可動(dòng)態(tài)調(diào)整的基于非循環(huán)有向圖的具有容錯(cuò)