從統(tǒng)一數(shù)據(jù)治理到下一代企業(yè)級(jí)+Data+Agent+的實(shí)踐與創(chuàng)新

......2018年的33ZB，增長(zhǎng)到2025年的175ZB。Q數(shù)據(jù)貫穿于A(yíng)I應(yīng)用的整個(gè)鏈路數(shù)據(jù)治理–AI應(yīng)用中不可或缺的部分https://www.llamaindex.ai/blog/introducing-llamacloud-and-llamaparse-af8cedf9006b?GDPR、CCPA等要求CONFLUENTCONFLUENT"----------h、---------"新業(yè)務(wù)和LLM需要一種統(tǒng)一的方式來(lái)訪(fǎng)問(wèn)所有數(shù)據(jù)。Technical&BusinessDataConnecttotheDataThatMattersMost.ConnecttotheDataThatMattersMost.Automaticallyfind,classify,andmapallofyourdata-everywhere.allofyourdata-everywhere.Automateend-to-enddatarightsrequestsandreporting.Automateend-to-enddatarightsrequestsandreporting.Enrichtechnicalmetadatawithbusinessandoperationalmetadataforfullvisibility.AutomaticallyclassifymoretypesofdatainAutomaticallyclassifymoretypesofdatainmoreplaces.IPDataSimplifyandautomatedatalifecyclemanagementfromcollectiontodestruction.DataScience/AIScience/AIStreamingAnalyticsWarehouseBusinessIntelligenDataOrchestrationDataOrchestrationAAB組織壁壘技術(shù)壁壘安全壁壘管理壁壘trino鄉(xiāng)stor數(shù)據(jù)智能調(diào)優(yōu)統(tǒng)一權(quán)限管控統(tǒng)一數(shù)據(jù)管控面，旨在幫助用戶(hù)快速構(gòu)建和管理湖倉(cāng)架構(gòu)，實(shí)現(xiàn)全方位湖倉(cāng)管理數(shù)據(jù)智能調(diào)優(yōu)統(tǒng)一權(quán)限管控統(tǒng)一數(shù)據(jù)/元數(shù)據(jù)管理視圖從元數(shù)據(jù)層面上達(dá)到SSOT↓頻等Metastore數(shù)據(jù)倉(cāng)庫(kù)實(shí)時(shí)消息數(shù)據(jù)湖Next-GenDataCatalogistheCoreinNewOpenDataArchitecture/ Tabulardata…}….}Non-tabulardata…}….}TechnologyTechnologyCommunicationCommunicationDataDataternetofternetofthingsAutomatioAutomationNetworkingNetworkingDataDataEngineerDataIngestionDataIngestionSparkSparkHDFSHDFSClientS3S3SDKutomatioutomationCopyCopyStructuredUnstructuredTraining/Checkpoint/Training/Checkpoint/Metrics/ModelData統(tǒng)一元數(shù)據(jù)治理前統(tǒng)一元數(shù)據(jù)治理后數(shù)據(jù)導(dǎo)出容易造成信息泄漏；數(shù)據(jù)使用無(wú)法跟蹤統(tǒng)一數(shù)據(jù)查找和訪(fǎng)問(wèn)；統(tǒng)一安全模型和數(shù)據(jù)訪(fǎng)問(wèn)DataScientistAIEngineerDataScientistAIEngineerTechnologyDataEngineerDataIngestionTensorflowCommunicationSparkTensorflowPytorchPytorchHDFSClientETLETLDataS3SDKternetofthingsGravitino!WriteDataReadDataAWriteDataGravitinoACLUnifiedAccessControlGravitinoDatasetUnstructuredDataScientistAIEngineerDataScientistAIEngineerTechnologyDataEngineerDataIngestionTensorflowCommunicationSparkTensorflowPytorchPytorchHDFSClientETLETLDataS3SDKternetofthingsGravitino!WriteDataReadDataAWriteDataGravitinoACLUnifiedAccessControlGravitinoDatasetUnstructuredDataUnstructuredDataStructuredDataDataDataNetworking生成式大語(yǔ)言模型（LLM）的功能相結(jié)合。由決策引擎選擇RAG方法；使用多種數(shù)據(jù)源/知識(shí)庫(kù)結(jié)合關(guān)鍵字和語(yǔ)義檢索的結(jié)果；檢索前后進(jìn)行一定預(yù)處理；基于向量的檢索；易于實(shí)現(xiàn)，有限的數(shù)據(jù)，缺少對(duì)查詢(xún)和結(jié)果進(jìn)行校驗(yàn)KB等）?框架（LangChain，LlamaIndex?基礎(chǔ)模型/Embedding模型?不能提供針對(duì)某個(gè)用戶(hù)的準(zhǔn)確信息在Retrieval前：在Retrieval前：-TextToSQL（或其它查詢(xún)語(yǔ)法）-改寫(xiě)問(wèn)題（通過(guò)LLM）-對(duì)查詢(xún)結(jié)果進(jìn)行re-ranking--提供更多的上下文信息和一定的推理??更進(jìn)一步，在Retrieval前進(jìn)行查詢(xún)路由：-在不同KB之間選擇-在不同retrieve方式之間選擇-使用LLM判斷在RAG框架中為每個(gè)數(shù)據(jù)庫(kù)、每種數(shù)據(jù)源開(kāi)發(fā)連接器/reader、獲取其描述信息、獲得prompt模版、進(jìn)行NL2SQL/QL、查詢(xún)數(shù)據(jù)、輸入給LLMn每個(gè)數(shù)據(jù)源/存儲(chǔ)都需流程打通、產(chǎn)品打通非結(jié)構(gòu)化數(shù)據(jù)非結(jié)構(gòu)化數(shù)據(jù)資產(chǎn)地圖與資產(chǎn)治理傳統(tǒng)機(jī)器學(xué)習(xí)大模型預(yù)訓(xùn)練/微調(diào)支持設(shè)置TTL與TTV數(shù)據(jù)處理流程A流程數(shù)據(jù)加工spark/Flink數(shù)據(jù)分析OLAP數(shù)據(jù)探查數(shù)據(jù)加工spark/Flink數(shù)據(jù)分析OLAP數(shù)據(jù)探查Notebook模型訓(xùn)練LLM模型部署數(shù)據(jù)集成集成引擎根據(jù)血緣推薦TTL、TTV管理前降本40%管理后!降本40%管理后特征分析特征分析pytorchpandas模型訓(xùn)練pytorchpandas模型訓(xùn)練模型部署提取特征sparkModel數(shù)據(jù)落倉(cāng)模型部署提取特征sparkModel數(shù)據(jù)落倉(cāng)TaloslcebergFileset模型校驗(yàn)TaloslcebergFileset模型校驗(yàn)數(shù)據(jù)分析數(shù)據(jù)分析python微調(diào)/預(yù)訓(xùn)練Dockersparkpython用戶(hù)問(wèn)答語(yǔ)料數(shù)據(jù)選取微調(diào)數(shù)據(jù)微調(diào)/預(yù)訓(xùn)練Dockersparkpython用戶(hù)問(wèn)答語(yǔ)料數(shù)據(jù)選取微調(diào)數(shù)據(jù)FilesetlcebergHive/lceberglcebergpythonpython部署/在線(xiàn)推理校驗(yàn)/離線(xiàn)推理Docker測(cè)試數(shù)據(jù)部署/在線(xiàn)推理校驗(yàn)/離線(xiàn)推理Docker測(cè)試數(shù)據(jù)Fileset實(shí)踐2:某互聯(lián)網(wǎng)社交平臺(tái)業(yè)務(wù)側(cè)耦合度高：元數(shù)據(jù)使用方調(diào)用異構(gòu)數(shù)據(jù)源方式多種多樣數(shù)據(jù)治理能力有限：無(wú)法提供統(tǒng)一的審計(jì)、權(quán)限管理、TTL能力半結(jié)構(gòu)化/非結(jié)構(gòu)化數(shù)據(jù)源缺乏管理跨源數(shù)據(jù)Schema維護(hù)成本高OneMeta：集成Gravitino提供定制化接口：dropPartitiosByFilter/loadFileDetail/loadFiles......提供定制化catalog實(shí)現(xiàn)：BiliIcebergCatalog/BiliKafkaCatalog/BiliDatabusCatalog......降低代碼的侵入性，便于同步社區(qū)最新代碼解耦業(yè)務(wù)方復(fù)雜依賴(lài)，降低元數(shù)據(jù)使用成本解決由于引擎間差異、數(shù)據(jù)源差異造成的元數(shù)據(jù)不一致問(wèn)題解決由于HiveMetaStore造成的性能瓶頸Fileset文件治理主要流程：1.數(shù)據(jù)治理平臺(tái)制定治理策略TTL