FFA2024分論壇-流式湖倉(cāng) 合輯-部分1

數(shù)據(jù)湖解決的問(wèn)題流批一體的演進(jìn)112233+批批批批批批流流流流流流批批批批批批流流流流流流批批批批流流流流流流…);批批批批流流流流流流 Fallback Fallback流流流流T當(dāng)天（1-5分鐘可見(jiàn)延遲）中間層D T當(dāng)天（1-5分鐘可見(jiàn)延遲）數(shù)據(jù)湖AI方向的探索}spark=SparkSession.buipaimon_table="my_catalodf=spark.read.format(”paimon").load(paimondefconvert_image_to_pdf(image_patpdf_path=new_path(imagfilter_df=df.filter(.filter(col("name").endsWith("filter_df.rdd.forea模型預(yù)測(cè)oss://..oss://..oss://..oss://..THANKYOU核心策略高l基于Paimon構(gòu)建湖倉(cāng)公共層，統(tǒng)一口THANKYOU業(yè)務(wù)應(yīng)用推薦策略實(shí)時(shí)數(shù)倉(cāng)數(shù)據(jù)源...報(bào)表分析數(shù)據(jù)存儲(chǔ)業(yè)務(wù)應(yīng)用推薦策略實(shí)時(shí)數(shù)倉(cāng)數(shù)據(jù)源...報(bào)表分析數(shù)據(jù)存儲(chǔ)實(shí)時(shí)數(shù)倉(cāng)數(shù)據(jù)存儲(chǔ)（u_id:11,r_id:99,status:true,ts:01）（u_id:11,r_id:99,status:false,ts:02）selectrid,+I（u_id:11,r_idselectrid,+I（u_id:11,r_id:99,satus:true,ts:01）-U（u_id:11,r_id:99,satus:true,ts:01）sum(status),...fromdwd_db_talelast_value(status...fromdwd_db_tale...+U（u_id:11,r_id:99,satus:false,ts:02）...fromdwd_db_talerid,createtabledwd_db_tale(statusint,...)selectcount(distinctuid)fromdwd_db_talewheredtbetween{startTime}and{endTime}LookupChangelogLookupChangelogAggregationEngineAggregationEnginecreatetabledwd_db_tale(statusboolean...)partitionedby(datestring)withtblproperties('merge-engine'='deduplicate','changelog-producer'='lookup','primary_key'='uid,rid,date...)+I（u_id:11,r_id:99,satus:1,ts:01）-U（u_id:11,r_id:99,satus:1,ts:01）+U（u_id:11,r_id:99,satus:0,ts:02）createtabledws_db_tale(statusbigint...)withtblproperties('merge-engine'='aggregation','fields.status.aggregate-function'='sum','primary_key'='rid',...)00精準(zhǔn)一次vid,dt,vid,mint,sumvid,dt,vid,mint,sum(value),last_value(value,ts),.........fromdwd_db_taledwd_db_talefromdwd_db_taledwd_db_tale...vid,createtabledwd_db_tale(value...bigintvid,dt,sum(value),value...bigintvid,dt,sum(value),...dwd_db_taledwd_db_tale)否是基于session集群可擴(kuò)展、低成本的方案探索探索TB級(jí)的維表高性能同步方案探索TB級(jí)的維表高性能同步方案PartialUpdate、TagBranch等能力落地實(shí)踐THANKYOU流批鏈路統(tǒng)一THANKYOU余StreamingupsertDatalakeStreamingupsertDatalakePartialupdateSchemaevolutionAggregationSchemaevolutionAggregationl雙流join狀態(tài)大影響穩(wěn)定性文件系統(tǒng)上的總數(shù)據(jù)量最新快照引用的數(shù)據(jù)量可擴(kuò)展性優(yōu)化（shufflehashjoin）THANKYOU分行機(jī)構(gòu)員工總數(shù)分行機(jī)構(gòu)2022年5月，經(jīng)中國(guó)銀保監(jiān)會(huì)批準(zhǔn)，正式吸收合并洛陽(yáng)銀行、平頂山銀行及焦作中旅銀行。先后榮獲“年度十佳城市商業(yè)銀行”“鐵馬十佳銀營(yíng)業(yè)網(wǎng)點(diǎn)資產(chǎn)總額行”“最佳上市公司”“年度卓越城商行”等稱營(yíng)業(yè)網(wǎng)點(diǎn)資產(chǎn)總額a）實(shí)時(shí)抽取數(shù)據(jù)a）實(shí)時(shí)監(jiān)控協(xié)同交易b）實(shí)時(shí)報(bào)表大屏展示c）實(shí)時(shí)海量數(shù)據(jù)存儲(chǔ)d）面向業(yè)務(wù)人員開(kāi)發(fā)b）與行內(nèi)資源兼容交易流水表賬戶表賬戶表賬戶表賬戶表......事實(shí)表/事件表/流水表維度表/賬戶表/屬性表基于事實(shí)表的實(shí)時(shí)增量計(jì)算基于維度表的實(shí)時(shí)全量計(jì)算主要是INSERT操作主要是UPDATE操作秒級(jí)秒級(jí)、分鐘級(jí)增量計(jì)算主要是LookupJoins主要是RegularJoins緩存中間數(shù)據(jù)量小緩存中間數(shù)據(jù)量大計(jì)算邏輯相對(duì)簡(jiǎn)單計(jì)算邏輯相對(duì)復(fù)雜存儲(chǔ)選擇按照日期分區(qū)存儲(chǔ)選擇單分區(qū)多分桶業(yè)務(wù)系統(tǒng)實(shí)時(shí)采集業(yè)務(wù)系統(tǒng)實(shí)時(shí)采集實(shí)時(shí)計(jì)算&實(shí)時(shí)存儲(chǔ);meta對(duì)公業(yè)務(wù)運(yùn)營(yíng)業(yè)務(wù)零售業(yè)務(wù)風(fēng)險(xiǎn)業(yè)務(wù)對(duì)公業(yè)務(wù)運(yùn)營(yíng)業(yè)務(wù)零售業(yè)務(wù)風(fēng)險(xiǎn)業(yè)務(wù)changelog流公共匯總changelog流貼源數(shù)據(jù)貼源數(shù)據(jù)changelog流增量數(shù)據(jù)tinsert、update、deletechangelog流增量數(shù)據(jù)全量加增量'changelog-producer.r實(shí)時(shí)采集實(shí)時(shí)計(jì)算Paimon維度表小rp實(shí)時(shí)采集實(shí)時(shí)計(jì)算Paimon維度表信貸系統(tǒng)實(shí)時(shí)批量查詢換機(jī)流量數(shù)據(jù)實(shí)時(shí)批量查詢換機(jī)流量數(shù)據(jù)海量數(shù)據(jù)實(shí)時(shí)?湖a）提煉適配場(chǎng)景：首先篩選需要業(yè)務(wù)人員進(jìn)行規(guī)則開(kāi)發(fā)的場(chǎng)景進(jìn)行適配；b）拖拽開(kāi)發(fā)測(cè)試：低代碼的拖拽式DAG頁(yè)面，達(dá)到設(shè)計(jì)即開(kāi)發(fā)，預(yù)校驗(yàn)，模擬測(cè)試；c）規(guī)則動(dòng)態(tài)更新：運(yùn)行時(shí)可實(shí)時(shí)更新算子規(guī)則，更新既上線，高效快速迭代；d）關(guān)聯(lián)系統(tǒng)對(duì)接：DAG底層采用JSON+自研DSL表達(dá)式表述，便于其它系統(tǒng)對(duì)接。THANKYOU 架構(gòu)簡(jiǎn)單，大量場(chǎng)景適用作業(yè)夜間運(yùn)行延遲較高：天級(jí)/小時(shí)級(jí)更新成本較高延遲低：秒級(jí)成本較高，場(chǎng)景有限?主鍵表支持大規(guī)模的更新寫入?更新性能高效?更新方式靈活?完整變更日志生成?分鐘級(jí)延遲?非主鍵表支持大規(guī)模流批處理?小文件自動(dòng)合并?PB級(jí)大規(guī)模數(shù)據(jù)管理?支持時(shí)間旅行（數(shù)據(jù)版本回溯）?支持?jǐn)?shù)據(jù)分支切換?包含詳細(xì)信息的系統(tǒng)表》porIS 字節(jié)、汽車之家、喜馬20222023-2024202420242024CREATECATALOG`my-catalog`WITH(ordersorders_syncMaxComputeSQLCREATECATALOG`dw`WITH(ASTABLEd.orders;Value12Value12Value12Value12Value11102120?‘deletion-vectors.enabled’=‘true’啟用DV計(jì)算數(shù)據(jù)不需要經(jīng)過(guò)合并，節(jié)省大量CPU開(kāi)銷支持非主鍵過(guò)濾條件的下推Value11Value11Value1Value1Value1Value11Value1110 86420 86420INT,00`user_id`BIGINT,00selectuser_id,max(sell_nick_name)fromorders400400`user_id`BIGINT,0THANKYOU實(shí)時(shí)數(shù)倉(cāng)實(shí)時(shí)數(shù)倉(cāng)近實(shí)時(shí)數(shù)倉(cāng)近實(shí)時(shí)數(shù)倉(cāng)湖倉(cāng)探索階段湖倉(cāng)探索階段質(zhì)量監(jiān)控告警平臺(tái)數(shù)據(jù)開(kāi)發(fā)管理運(yùn)維平臺(tái)質(zhì)量監(jiān)控告警平臺(tái)數(shù)據(jù)開(kāi)發(fā)管理運(yùn)維平臺(tái)..................數(shù)倉(cāng)建設(shè)個(gè)數(shù)倉(cāng)建設(shè)個(gè)個(gè)個(gè)離線T+1維度拓寬場(chǎng)景離線T+1維度拓寬場(chǎng)景Reaj業(yè)務(wù)域內(nèi)實(shí)時(shí)拓寬場(chǎng)景業(yè)務(wù)域內(nèi)實(shí)時(shí)拓寬場(chǎng)景Reaj業(yè)務(wù)過(guò)程業(yè)務(wù)過(guò)程分分?jǐn)?shù)倉(cāng)建設(shè)--數(shù)倉(cāng)建設(shè)------>joinjoinReajjoinjoin!多流join