推薦系統(tǒng)案例-第二天3mllib與

MLlib與MLlib與推薦系主要內1SparkMLlib1SparkMLlib2MLlib推薦算法3MLlib推薦算法4總主要內1SparkMLLib1SparkMLLib2MLlib推薦算法3MLlib推薦算法4總SparkSparkMLlibMLLib是基于Spark引擎實現(xiàn)的機器學良好的擴展性和容充分利用了Spark擴展性屬于Spark生態(tài)系統(tǒng)重要組成實現(xiàn)了大部分常用的數(shù)據(jù)挖聚類分類推薦

MLlib分類算(summarystatisticsstratifiedsamplingrandomdatagenerationClassificationandregression分類一般針對離散型數(shù)linearmodels(SVMs,logisticregressionlinearregressionnaive decisiontreesensemblesoftrees(RandomForestsandGradient-BoostedTreesMLlib分類算 alternatingleastsquaresALS)交替最小二乘法(ALS singular position 奇異值分principal ysis(PCA)主成分分stochasticgradientdescent隨機梯度下limited-memoryBFGS(L-BFGS)短時 的BFGS(擬 DeveloperApi在未來的發(fā)布種可能會被修改主要內1SparkMLlib1SparkMLlib2MLlib推薦算法3MLlib推薦算法4總MLlib協(xié)同過濾算MLlibMLlib基于ALS（alternatingleastsquares）的協(xié)同numBlocks:計算并行度(若為-1表示自動化配置Rank：模型中隱含影響因子，默認是Iterations：迭代次數(shù)，默認是Lambda：ALS中正則化參implicitPrefs：是否使用顯式反饋變量或使用隱式反饋數(shù)據(jù)的變Alpha：ALS中的一個參數(shù)，作用于隱式反饋變量，控制基本的信心主要內1SparkMLlib1SparkMLlib2MLlib推薦算法3MLlib推薦算法4總MLlibMLlibALS加載數(shù)據(jù)將數(shù)據(jù)集解析成ALS要求的格將數(shù)據(jù)集分割成兩部分：訓練集和測試運行ALS，產(chǎn)生并評估模將最終模型用于推MLlibMLlibALS推薦步驟1valvalconf=newvalsc=newvalvalmovieLensHomeDir=valdataSet=sc.textFile(newFile(movieLensHomeDir,MLlibMLlibALS推薦步驟2valvalrating=dataSet.map{linevalfields=//format:(timestamp%10,Rating(userId,movieId,}MLlibMLlibALS推薦步驟3valnumPartitions=valtraining=ratings.filter(x=>x._1<valvalidation=ratings.filter(x=>x._1>=6&&x._1<valtest=ratings.filter(x=>x._1>=valnumTraining=training.count()valnumTest=test.count()MLlibALSMLlibALS推薦步驟4：運行defdeftrain(ratings:RDD[Rating],rank:Int,iterations:Int,lambda::MatrixFactorizationModel=你的代}MLlibALSMLlibALS推薦步驟4(2)：運行ranks=List(8,vallambdas=List(1.0,valnumIters=List(10,varbestModel:Option[MatrixFactorizationModel]=NonevarbestValidationRmse=Double.MaxValuevarbestRank=varbestLambda=-1.0varbestNumIter=-1//findbestmodelherevaltestRmse=computeRmse(bestModel.get,test,println("Thebestmodelwastrainedwithrank="+bestRank+"andlambda="++",andnumIter="+bestNumIter+",anditsRMSEonthetestsetis"+testRmse+MLlibALS推薦步驟4(2)：運行for(rank<-ranks;lambda<-lambdas;numIter<-numIters)valmodel=ALS.train(training,rank,numIter,valvalidationRmse=computeRmse(model,validation,numValidation)println("RMSE(validation)="+validationRmse+"forthemodeltrainedwithrank=+rank+",lambda="+lambda+",andnumIter="+numIter+".")if(validationRmse<bestValidationRmse){bestModel=Some(model)bestRank=rankbestLambda=bestNumIter=}}MLlibALS推薦步驟MLlibALS推薦步驟5：使用模 //...你的代}valcandida