OPPO數(shù)據(jù)湖加速大模型訓(xùn)練2024

DataFun

#

2024OPPO數(shù)據(jù)湖加速大模型訓(xùn)練OPPO數(shù)據(jù)湖平臺(tái)介紹Glacier–自研湖倉(cāng)引擎流批一體大模型任務(wù)加速大模型任務(wù)接入數(shù)據(jù)平臺(tái)

Rust加速大模型任務(wù)技術(shù)規(guī)劃目錄CONTENTDataFun

#

202401OPPO數(shù)據(jù)湖平臺(tái)介紹OPPO大數(shù)據(jù)架構(gòu)自研數(shù)據(jù)湖產(chǎn)品-Glacier管理和優(yōu)化開(kāi)源數(shù)據(jù)湖表秒級(jí)寫(xiě)入索引增強(qiáng)非結(jié)構(gòu)化數(shù)據(jù)優(yōu)化南天門(mén)DAM數(shù)據(jù)產(chǎn)品接入層LivyHiveserverFlinkGateway計(jì)算引擎SparkHivePrestoFlink元數(shù)據(jù)HMS存儲(chǔ)HDFSData

FormaticebergCubeFSS3kafkaHudiPaimonGlacier數(shù)據(jù)入湖CDC入湖采集服務(wù)入湖高性能&可靠性客戶(hù)端單表支持100+客戶(hù)端同時(shí)提交數(shù)據(jù)秒級(jí)延遲基于alluxio的流文件低延遲的流式讀取和數(shù)據(jù) 分析正確性保障元數(shù)據(jù)管理-GMS元數(shù)據(jù)定時(shí)任務(wù)生命周期管理數(shù)據(jù)治理流批一體任務(wù)SQL兼容執(zhí)行計(jì)劃區(qū)分流批一體維表DataFun

#

202402大模型任務(wù)加速數(shù)據(jù)湖&大模型平臺(tái)化管理任務(wù)&數(shù)據(jù)計(jì)算提效，存儲(chǔ)降本Spark-ML庫(kù)資源調(diào)度對(duì)接訓(xùn)練平臺(tái)PySpark7z數(shù)據(jù)入湖內(nèi)存優(yōu)化（Pickler）并行度優(yōu)化長(zhǎng)文本切分/壓縮Remote

Shuffle在線建數(shù)據(jù)湖表并寫(xiě)入數(shù)據(jù)數(shù)據(jù)加速分類(lèi)數(shù)據(jù)索引去重結(jié)果檢索緩存加速訓(xùn)練任務(wù)優(yōu)化MiniHash，Kmeans任務(wù)優(yōu)化全局精確去重實(shí)現(xiàn)數(shù)據(jù)向量化(Rust改造)Tokenizer:Input：RDD<String>Output:

RDD<Vector<Int>>數(shù)據(jù)精確去重任務(wù)優(yōu)化任務(wù)Rust化改造提升效率內(nèi)存安全環(huán)境簡(jiǎn)單豐富的開(kāi)源庫(kù)Rust版本，內(nèi)存效率更高Tokenizer開(kāi)源Rust實(shí)現(xiàn)資源調(diào)度跨集群任務(wù)