開源AI和大數(shù)據(jù)存儲編排平臺

1、Alluxio - 開源AI和大數(shù)據(jù)存儲編排平臺顧榮Alluxio PMC & Maintainer提綱Alluxio項目&系統(tǒng)簡介Alluxio 2.0新特性概覽Alluxio未來發(fā)展趨勢快覽總結(jié)數(shù)據(jù)處理的四大趨勢驅(qū)動了新型基礎(chǔ)架構(gòu)的需求Separation of Compute & StorageHybrid Multi cloud environmentsSelf-service data across the enterpriseRiseof the object storeData Ecosystem - BetaData Ecosystem 1.0COMPUTESTORAGESTO

2、RAGECOMPUTE大數(shù)據(jù)之路與企業(yè)創(chuàng)新的選擇同置(Co-located )Co-located compute & HDFS on the same clusterMR / Hive HDFSHiveHDFS分散(Disaggregated)Disaggregated compute & HDFS on the same cluster混合云化部署HDFSBurst HDFS data in the cloud,public or private支持更多計算框架Support Presto, Spark and other computes without app changes向?qū)ο蟠鎯?/p>

3、過渡Enable & accelerate big data onobject stores技術(shù)轉(zhuǎn)變中的挑戰(zhàn)Accessing data over WAN too slowCopying data to compute cloud time consuming and complexUsing another storage system like S3 means expensive application changesUsing S3 via HDFS connector leads to extremely low performance混合云部署HDFSCopying data to

4、 multiple compute clouds time consuming and error proneMigrating applications for new storage systems is complex & time consumingStoring and managing multiple copies of the data becomes expensive支持更多計算框架Object stores performance for big data workloads can be very poorNo native support for popular fr

5、ameworksExpensive metadata operations reduce performance even moreNo support for hybrid environments directly向?qū)ο蟠鎯^渡12/2/197計算與存儲實現(xiàn)獨立可擴展性FUSE Compatible File SystemHadoop Compatible File SystemNative Key-Value InterfaceNative File SystemUnifying Data at Memory SpeedGlusterFS InterfaceAmazon S3 Inter

6、faceSwift InterfaceHDFS InterfaceAlluxio: a Virtual Distributed File System (VDFS)Java File APIHDFS InterfaceS3 InterfaceREST APIHDFS DriverS3 DriverSwift DriverNFS DriverFUSE Interface12/2/19計算與存儲實現(xiàn)獨立可擴展性Master-WorkerMaster管理全部元數(shù)據(jù)監(jiān)控各個Worker狀態(tài)Worker管理本地MEM、SSD和HDDClient向用戶和應(yīng)用提供訪問接口向Master和Worker發(fā)送請求

7、Under File System一般用于備份9Under File SystemMasterClient齒侈尺Worker1MEM SSD HDDnode 1Worker3MEM SSD HDDnode 3Worker2MEM SSD HDDnode 2Alluxio系統(tǒng)內(nèi)部整體架構(gòu)Alluxio數(shù)據(jù)編排賦能的幾類場景hybrid cloud environmentsSame instanceOn premise/ containerAccelerate big data frameworksBurst big data workloads inon the public cloudSame

8、 instance/ containerDramatically speed-up big data on object stores on premise高級使用場景Enable big data on object stores across single or multiple cloudsOrchestrate data frameworks on the public cloudAlluxio的核心創(chuàng)新數(shù)據(jù)伸縮性 Data Elasticity with a unified namespaceAbstract data silos & storage systems to indep

9、endently scale data on-demand with computeRun Spark, Hive, Presto, ML workloads on your data located anywhereAccelerate big data workloads with transparent tiered local data數(shù)據(jù)可訪問性 Data Accessibility for popular APIs & API translation數(shù)據(jù)本地性 Data Locality with Intelligent Multi-tiering基于智能多層緩存實現(xiàn)數(shù)據(jù)本地性Lo

10、cal performance from remote data using multi-tier storage通過提供流行APIs和API轉(zhuǎn)換實現(xiàn)數(shù)據(jù)可訪問性Convert from Client-side Interface to native Storage Interface通過統(tǒng)一命名空間實現(xiàn)數(shù)據(jù)可伸縮性Enables effective data management across different Under StoreUses Mounting with Transparent Naming統(tǒng)一命名空間（Unified Namespace）Transparent acces

11、s to understorage makes all enterprise data available locallySUPPORTSHDFSNFSOpenStackCephAmazon S3AzureGoogle CloudIT OPS FRIENDLYStorage mounted into Alluxio by central ITSecurity in Alluxio mirrors source dataAuthentication throughLDAP/ADWireline encryptionHDFS #1Object StoreNFSHDFS #2100+ Known P

12、roduction DeploymentsConsumerTravel & TransportationTelco & MediaTechnologyFinancial ServicesRetail & EntertainmentData & Analytics ServicesIncredible Open Source Momentum with growing community1000+ contributors & growing4278+ Git StarsApache 2.0 LicensedHundreds of thousands of downloadsGithub: /A

13、lluxio/alluxio Join the conversation on Slack /slackFinding high-fit use-casesExample First ProjectsEnterprise Storage & Big Data TeamsVirtual Data LakesGradual transition to low cost storageUnify hybrid-cloud storageMachine Learning & Data Science TeamsAccelerate trainingImprove productivityCompute

14、 ZoneStandalone or managed with Mesos or YarnStorage in Different Availability ZoneEither on-prem or cloudTensorflowPrestoHDFSSparkAlluxio is installed with or near compute to unify data stores, stage remote data, and improve system performance.19Alluxio適用場景分析Alluxio適用場景分析2021Alluxio 2.x新特性介紹支持超大規(guī)模數(shù)

15、據(jù)工作負載支持超過10億+個文件2.0引入了分層元數(shù)據(jù)管理(tiered metadata management)這一新選項，以支持包含 超過10億個文件的單群集部署。我們現(xiàn)在默認使用RocksDB進行堆外存儲。熱數(shù)據(jù)的元數(shù)據(jù)繼續(xù)存儲在堆內(nèi)的進程內(nèi)存中，而其余元數(shù)據(jù)由Alluxio在進程內(nèi)存外 進行管理。alluxio.master.metastore可以配置為僅使用堆內(nèi)存儲。高度分布式數(shù)據(jù)服務(wù)2.0引入了Alluxio作業(yè)服務(wù)(Job Service)，這是一種分布式集群服務(wù)，可以實現(xiàn)復(fù)制、 持久化、跨存儲移動和分布式加載等數(shù)據(jù)操作，從而實現(xiàn)高性能和大規(guī)模擴展。支持超大規(guī)模數(shù)據(jù)工作負載自適應(yīng)

16、副本以增強數(shù)據(jù)本地性該功能為Alluxio配置一定數(shù)量范圍的自動管理的存儲數(shù)據(jù)副本數(shù)。alluxio.user.file.replication.max和alluxio.user.file.replication.min可用于指定該 范圍。內(nèi)嵌式日志以達到高可用性2.0設(shè)計了一種稱為內(nèi)嵌式日志(embedded journal)的面向文件/對象元數(shù)據(jù)的新 容錯和高可用模式。內(nèi)嵌式日志使用RAFT共識算法，并且實現(xiàn)方面獨立于任何其他外部存儲系統(tǒng)。這 對于抽象對象存儲特別有用。Alluxio 2.x新特性介紹支持超大規(guī)模數(shù)據(jù)工作負載自適應(yīng)副本以增強數(shù)據(jù)本地性Alluxio MasterAlluxi

17、o WorkerUnder StoreAlluxio WorkerAlluxio WorkerAlluxio WorkerApplicationAlluxio ClientBlock-1Block-1ApplicationAlluxio ClientBlock-1ApplicationAlluxio ClientApplicationAlluxio ClientBlock-1Block-1SetReplicaMax(2)Alluxio 2.x新特性介紹支持超大規(guī)模數(shù)據(jù)工作負載內(nèi)嵌式日志以達到高可用性 Alluxio 1.x HA依賴ZK/HDFS組件Alluxio HA運行模式Zookeepe

18、r: 負責選擇leader masterHDFS: 負責存儲日志文件，并在多個 masters直接共享存在的問題日志存儲的選擇受限依賴于第三方組件，服務(wù)的調(diào)試恢復(fù) 都比較困難。HDFS集群本身的不穩(wěn)定，會使得 Alluxio集群維護成本變大Standby MasterLeading MasterStandby MasterShared Storagewrite journalread journalAlluxio 2.x新特性介紹支持超大規(guī)模數(shù)據(jù)工作負載內(nèi)嵌式日志以達到高可用性Alluxio 2.x去除了ZK/HDFS依賴在Alluxio三個Master內(nèi)部利用RAFT算 法達成共識（ Con

19、sensus）狀態(tài)只有Leading master提交狀態(tài)變化， Standby masters保持同步優(yōu)勢可以采用本地磁盤存儲日志（Master 節(jié)點間作副本）挑戰(zhàn)性能調(diào)優(yōu)Standby MasterLeading MasterStandby MasterRaftState ChangeState ChangeState ChangeAlluxio 2.x新特性介紹更好的存儲抽象，實現(xiàn)完全獨立和彈性的計算支持跨不同版本的HDFS集群數(shù)據(jù)的爆炸式增長導(dǎo)致企業(yè)通常會擁有許多數(shù)據(jù)倉庫，包括采用跨不同版本的 多個Hadoop集群。目前，跨這些集群的統(tǒng)一訪問非常困難。使用Alluxio 2.0， 用戶

20、可以使用Alluxio連接到多個多種版本的HDFS集群，并實現(xiàn)統(tǒng)一的數(shù)據(jù)訪 問。與Hadoop主動同步該新功能是與HDFS iNotify進行對接集成，可對存儲在Hadoop中的文件所發(fā)生 的任何數(shù)據(jù)和元數(shù)據(jù)更改進行更新，允許通過Alluxio訪問數(shù)據(jù)的應(yīng)用程序能 夠主動接收最新更新。Alluxio 2.x新特性介紹Alluxio 2.x新特性介紹對機器學(xué)習(xí)、數(shù)據(jù)查詢等系統(tǒng)更強的支撐支持在任意存儲上運行機器學(xué)習(xí)和深度學(xué)習(xí)工作負載機器學(xué)習(xí)和深度學(xué)習(xí)框架往往需要從Hadoop或?qū)ο蟠鎯χ刑崛〈笠?guī)模數(shù)據(jù)，這通常是手動且 非常耗時的過程。Alluxio的FUSE功能支持POSIX兼容的API，因此通過

21、Alluxio，TensorFlow、Caffe等框架以及 其他基于Python的模型可以使用傳統(tǒng)文件系統(tǒng)的訪問方式直接訪問任何存儲系統(tǒng)中的數(shù)據(jù)。與結(jié)構(gòu)化數(shù)據(jù)管理與查詢系統(tǒng)進行深度整合在Alluxio層面提供Catalog Service，提供了對結(jié)構(gòu)化數(shù)據(jù)的抽象，添加Hive MetaStore到 Alluxio中就像掛載一個文件系統(tǒng)。Alluxio感知文件和對象的數(shù)據(jù)存儲結(jié)構(gòu)和模式(schema)，從而更好地提供服務(wù)，提供了 Alluxio Data Transformation服務(wù)，例如：自動將CSV格式的文件轉(zhuǎn)成Parquet格式將很多小的表文件整合成大文件，減少查詢耗時等Alluxi

22、o Catalog Service (Target 2.1)Serve Metadata of Tables (like Hive Meta Store)Highly Efficient by using Apache Iceberg (e.g., no slow dir listing)Speed up query planning, independent of speeding up by caching files in Alluxio File SystemAlluxio Connector for Presto (Target 2.1)Presto connects to Alluxio d