2019年idea版大數(shù)據(jù)項(xiàng)目之推薦系統(tǒng)5電影設(shè)計(jì)

電影推薦系統(tǒng)設(shè)計(jì)講師：武晟然項(xiàng)目框架數(shù)據(jù)源解析統(tǒng)計(jì)推薦模塊離線推薦模塊實(shí)時(shí)推薦模塊基于內(nèi)容的推薦模塊主要內(nèi)容項(xiàng)目框架大數(shù)據(jù)處理流程系統(tǒng)模塊設(shè)計(jì)項(xiàng)目系統(tǒng)架構(gòu)項(xiàng)目數(shù)據(jù)流圖數(shù)據(jù)應(yīng)用數(shù)據(jù)計(jì)算數(shù)據(jù)存儲(chǔ)數(shù)據(jù)采集數(shù)據(jù)源日志數(shù)據(jù)關(guān)系數(shù)據(jù)圖片視頻KafkaSqoopFlumeScribeETL工具結(jié)構(gòu)化數(shù)據(jù)半結(jié)構(gòu)化數(shù)據(jù)非結(jié)構(gòu)化數(shù)據(jù)HDFSMapReduce可視化EchartsD3HBaseCassandraGreenPlumOracleSparkFlinkStormMahout業(yè)務(wù)應(yīng)用BI分析Tableau數(shù)據(jù)生命周期大數(shù)據(jù)處理流程我們的目標(biāo)電影推薦網(wǎng)站實(shí)時(shí)推薦服務(wù)基于模型的推薦基于內(nèi)容的推薦協(xié)同過濾的推薦離線推薦服務(wù)實(shí)時(shí)個(gè)性化推薦離線個(gè)性化推薦統(tǒng)計(jì)推薦相似性推薦評(píng)分標(biāo)簽檢索離線統(tǒng)計(jì)服務(wù)內(nèi)容檢索服務(wù)系統(tǒng)模塊設(shè)計(jì)離線在線近線項(xiàng)目系統(tǒng)架構(gòu)log系統(tǒng)數(shù)據(jù)流圖數(shù)據(jù)源解析電影信息用戶評(píng)分信息電影標(biāo)簽信息movies.csvratings.csvtags.csv電影ID(MID)電影名稱(NAME)電影描述(DESCRI)電影時(shí)長(TIMELONG)發(fā)行時(shí)間(ISSUE)拍攝時(shí)間(SHOOT)電影語言(LANGUAGE)電影類別(DIRECTOR)電影演員(ACTORS)電影導(dǎo)演(DIRECTOR)1ToyStory-81minutesMarch2020011995EnglishAdventure|Animation|…|FantasyTomHanks|…|JimJohnLasseter…………………………30ShanghaiTriad-108minutesDecember1220001995ChineseCrime|DramaGongLi|…|LiBao-TianZhangYimou電影信息用戶ID(UID)電影ID(MID)電影評(píng)分(SCORE)評(píng)分時(shí)間(TIMESTAMP)67158164106511196367159023.51064245507…………671595251063502716用戶評(píng)分信息用戶ID(UID)電影ID(MID)電影標(biāo)簽(TAG)標(biāo)簽時(shí)間(TIMESTAMP)15339sandra'boring'bullock1138537770151955dentist1193435061…………15100365uganda1425876220電影標(biāo)簽信息主要數(shù)據(jù)模型統(tǒng)計(jì)推薦模塊歷史熱門電影統(tǒng)計(jì)近期熱門電影統(tǒng)計(jì)電影平均評(píng)分統(tǒng)計(jì)各類別Top10優(yōu)質(zhì)電影統(tǒng)計(jì)統(tǒng)計(jì)推薦模塊SparkSession.readSparkSession.writemongo-spark-connector統(tǒng)計(jì)推薦模塊歷史熱門電影統(tǒng)計(jì)統(tǒng)計(jì)所有歷史數(shù)據(jù)中每個(gè)電影的評(píng)分?jǐn)?shù)selectmid,count(mid)ascountfromratingsgroupbymid

RateMoreMoviesRateMoreMovies數(shù)據(jù)結(jié)構(gòu)：mid，count近期熱門電影統(tǒng)計(jì)統(tǒng)計(jì)每月的電影評(píng)分個(gè)數(shù)，就代表了電影近期的熱門度selectmid,score,changeDate(timestamp)asyearmonthfromratings

ratingOfMonthselectmid,count(mid)ascount,yearmonthfromratingOfMonthgroupbyyearmonth,midorderbyyearmonthdesc,countdesc RateMoreRecentlyMovieschangDate：UDF函數(shù)，使用SimpleDateFormat對(duì)Date進(jìn)行格式轉(zhuǎn)化，轉(zhuǎn)化格式為“yyyyMM”RateMoreRecentlyMovies數(shù)據(jù)結(jié)構(gòu)：mid，count，yearmonth電影平均評(píng)分統(tǒng)計(jì)selectmid,avg(score)asavgfromratingsgroupbymid AverageMoviesAverageMovies數(shù)據(jù)結(jié)構(gòu)：mid，avg各類別Top10評(píng)分電影統(tǒng)計(jì)selecta.mid,genres,if(isnull(b.avg),0,b.avg)scorefrommoviesaleftjoinaverageMoviesbona.mid=b.mid movieWithScorespark.sql("select*from(select"+"mid,"+"gen,"+"score,"+"row_number()over(partitionbygenorderbyscoredesc)rank"+"from"+"(selectmid,score,explode(splitGe(genres))genfrommovieWithScore)

genresMovies)rankGenresMovies"+"whererank<=10")splitGe：UDF函數(shù)，按照‘|’字符對(duì)字符串進(jìn)行切分離線推薦模塊用ALS算法訓(xùn)練隱語義模型計(jì)算用戶推薦矩陣計(jì)算電影相似度矩陣離線推薦模塊DataSet[MovieRating]SparkSession.readRDD[Rating(uid,mid,score)]mapvalmodel=ALS.train(trainData,rank,iterations,lambda)RMSE

均方根誤差：均方誤差的算術(shù)平方根，預(yù)測值與真實(shí)值之間的誤差參數(shù)調(diào)整

可以通過均方根誤差，來多次調(diào)整參數(shù)值，選擇RMSE最小的一組參數(shù)值rank，iterations，lambdaALS推薦模型訓(xùn)練ALSuserRDD:RDD[Int]movieRDD:RDD[Int]userMovies:RDD[(uid,mid)]predictRating:RDD[Rating(uid,mid,predict)]userGroupRatings:RDD[(uid,Seq[Rating])]userRecs:RDD[(uid,Seq[(mid,score)])]笛卡爾積model.predict(userMovies)groupByKeysortBy(‘score’).take(20)SparkSession.write計(jì)算用戶推薦矩陣movieFeatures:RDD[mid,DoubleMatrix]movieFeatures:RDD[mid,DoubleMatrix]movieSim:RDD[(mid,(mid,consinSim))]modelmoviesSimGroup:RDD[(mid,Seq(mid,consinSim))]笛卡爾積Filter(consinSim>0.6).groupByKductFeaturesMatrix（N，M）=Matrix（N，K）×

Matrix（K，M）

計(jì)算電影相似度矩陣

movieRecs:RDD[(uid,Seq[(mid,score)])]SparkSession.writespark-redis存儲(chǔ)電影相似度矩陣實(shí)時(shí)推薦模塊實(shí)時(shí)推薦架構(gòu)實(shí)時(shí)推薦優(yōu)先級(jí)計(jì)算基于模型的實(shí)時(shí)推薦模塊log計(jì)算速度要快結(jié)果可以不是特別精確有預(yù)先設(shè)計(jì)好的推薦模型uid,mid,score,timestamp基于模型的實(shí)時(shí)推薦模塊基本原理：用戶最近一段時(shí)間的口味是相似的備選電影推薦優(yōu)先級(jí):A5.0B4.0C1.0D4.0X?Y?Z?用戶最近k次評(píng)分備選電影Sim(A,X)Sim(B,X)Sim(C,X)X的推薦優(yōu)先級(jí)分?jǐn)?shù)為：？=(sim(A,X)*5+sim(B,X)*4+sim(C,X)*1)/3+lg2–lg1推薦優(yōu)先級(jí)計(jì)算KRatings:Seq[(mid,score)]Movies【候選電影】:Seq[Int]計(jì)算分?jǐn)?shù)：((sim(A,X)*5+sim(B,X)*4+sim(C,X))/3獲取和K次評(píng)分的每個(gè)電影的相似度Score(X)+log2–log1和前面的實(shí)時(shí)結(jié)果合并更新加上偏移項(xiàng)uid,mid,score,timestampRedis電影相似度矩陣Redis用戶最近K