超大規(guī)模實時數(shù)倉架構(gòu)挑戰(zhàn)與實踐解析課件

超大規(guī)模實時數(shù)倉架構(gòu)挑戰(zhàn)與實踐解析技術(shù)創(chuàng)新

變革未來超大規(guī)模實時數(shù)倉架構(gòu)挑戰(zhàn)與實踐解析技術(shù)創(chuàng)新變革未來1數(shù)據(jù)庫發(fā)展RDBMS[SQL+OLTP

]Data

warehouseData

Cube[ETL+OLAP

]GraphVectorTextStructured

DataHeterogeneous

DataRDBMSNoSQL/NewSQLDB[Multi-Model+HTAP

]Structured

DataStructuredDataTime

SeriesSpatialData數(shù)據(jù)庫發(fā)展RDBMSDatawarehouseData2阿里巴巴

－

數(shù)據(jù)庫阿里巴巴－數(shù)據(jù)庫3OLAP-

OLTPOn-LineTransaction

ProcessingShort-lived

tr(ns()tions(Typi)(llySm(lld(t(())ess

footprint(OftentimeRepetitive

oper(tionsOn-Linenal-tical

ProcessingLon-runnin-

queriesComplexjoin

oper(tionsExplor(toryqueriesth(tm(y())ess(l(r-e(mountofd(t(OLAP-OLTPOn-LineTransaction4數(shù)據(jù)庫發(fā)展

－

架構(gòu)數(shù)據(jù)庫發(fā)展－架構(gòu)5數(shù)據(jù)庫發(fā)展

－

核心技術(shù)組件數(shù)據(jù)庫發(fā)展－核心技術(shù)組件6數(shù)據(jù)庫發(fā)展

－

更高的應用訴求數(shù)據(jù)庫發(fā)展－更高的應用訴求7數(shù)據(jù)庫發(fā)展—技術(shù)挑戰(zhàn)數(shù)據(jù)庫發(fā)展—技術(shù)挑戰(zhàn)8整體架構(gòu)支持任意條件組合的高并發(fā)低延時查詢支持大吞吐數(shù)據(jù)寫入兼容MySQL生態(tài)支持高可用（在線升級/擴縮容，單點故障應用透明）整體架構(gòu)支持任意條件組合的高并發(fā)低延時查詢9執(zhí)行調(diào)度WorkloadPerQueryFairSchedulerPerTask

WeightScheduler場景混合負載10-20XPerformanceImprove執(zhí)行調(diào)度Workload場景混合負載10-20XPerfor10內(nèi)存管理內(nèi)存調(diào)度：統(tǒng)一內(nèi)存結(jié)構(gòu)，定長塊內(nèi)存池化,

Binary

Process，彈性分配，支持落盤內(nèi)存管理內(nèi)存調(diào)度：統(tǒng)一內(nèi)存結(jié)構(gòu)，定長塊內(nèi)存池化,Binar11自研優(yōu)化器自研優(yōu)化器12GPU引擎JIT代碼生成和IR層優(yōu)化支持CPU和GPU運行時加速庫Snappy解壓高速排序高并發(fā)顯存管理CPU-GPU堆外內(nèi)存?zhèn)鬏擲SD至顯存直傳Compute

NodeGPU引擎JIT代碼生成和IR層優(yōu)化ComputeNode13GPU引擎

－

性能GPU引擎－性能14SQL

OptimizationMemory

optParameterizationMonitoringTuningAuto-adminDiagnosisRestoreAnomaly

DetectionSecure

proxyProtectionPatchIdentificationThreat

DetectionAlertElasticitySchedulingResource

predictionBackup

RestoreOperationsMonitoringHA、DisasterRecoveryUpgradeExpansionandcontractionWholestage

OptResourcemanagementDatabase

Self-DrivingMetadataTaskmanageDatacollectionSecureReliableCost-effectiveEfficientML

Algorithms自動化運維SQLOptimizationMemoryoptPara15新硬件RDMANVM3DX

PointOpen-Channel

SSDAPPSFile

systemFT

LOpenChannel10

LibraryNVMe

DriverOpenChannelFirmwareSSD

ControllerGPU/FPGA新硬件RDMANVMOpen-ChannelSSDAPPS16Forrester-

ContendersGartner–Niche

Player業(yè)界認可Forrester-ContendersGartner17標準測試

-

TPC-DSRealistictable

scalingRealworlddata

contentNon-uniform

distributionsComplexrelationships(Factto

Fact)標準測試-TPC-DSRealistictables18TPC-DS

R&port &

VLDBVLDB2019Analyt