中文自然語言處理-商品評論情感判別

1、中文自然語言處理一商品評論情感判別1、數(shù)據(jù)集下載fromsklearn.model_selectionimporttrain_test_splitfromgensim.models.word2vecimportWord2Vecimportnumpyasnpimportpandasaspdimportjiebafromsklearn.externalsimportjoblibfromsklearn.svmimportSVC2、載入數(shù)據(jù)，做預(yù)處理(分詞)，切分訓(xùn)練集與測試集#載入數(shù)據(jù)，做預(yù)處理(分詞)，切分訓(xùn)練集與測試集defload_file_and_preprocessing():neg=pd

2、.read_excel(chinese_data/neg.xls,header=None,index=None)pos=pd.read_excel(chinese_data/pos.xls,header=None,index=None)cw=lambdax:list(jieba.cut(x)poswords=pos0.apply(cw)negwords=neg0.apply(cw)#use1forpositivesentiment,0fornegativey=np.concatenate(np.ones(len(pos),np.zeros(len(neg)#訓(xùn)練集：測試集=8:2x_train

3、,x_test,y_train,y_test=train_test_split(np.concatenate(poswords,negwords),y,test_size=0.2)#NumPy提供了多種文件操作函數(shù)方便存取數(shù)組內(nèi)容(npy格式以二進(jìn)制存儲數(shù)據(jù)的)np.save(pre_data/y_train.npy,y_train)np.save(pre_data/y_test.npy,y_test)returnx_train,x_test3、計算訓(xùn)練集和測試集每條評論數(shù)據(jù)的向量并存入文件#對每個句子的所有詞向量取均值，來生成一個句子的/ectordefbuild_sentence_vect

4、or(text,size,w2v_model):vec=np.zeros(size).reshape(1,size)count=0forwordintext:try:vec+=w2v_modelword.reshape(1,size)count+=1exceptKeyError:continueifcount!=0:vec/=countreturnvec#計算詞向量defget_train_vecs(x_train,x_test):n_dim=300#詞向量維度#試用Word2Vec建立詞向量模型w2v_model=Word2Vec(size=n_dim,window=5,sg=0,hs=0,

5、negative=5,min_count=10)w2v_model.build_vocab(x_train)#準(zhǔn)備模型詞匯表#在評論訓(xùn)練集上建模w2v_model.train(x_train,total_examples=w2v_model.corpus_count,epochs=w2v_model.iter)#川練詞向量#訓(xùn)練集評論向量集合train_vecs=np.concatenate(build_sentence_vector(z,n_dim,w2v_model)forzinx_train)np.save(pre_data/train_vecs.npy,train_vecs)#將訓(xùn)練集

6、保存到文件中print(train_vecs.shape)#輸出訓(xùn)練集的維度#在測試集上訓(xùn)練w2v_model.train(x_test,total_examples=w2v_model.corpus_count,epochs=w2v_model.iter)w2v_model.save(pre_data/w2v_model/w2v_model.pkl)test_vecs=np.concatenate(build_sentence_vector(z,n_dim,w2v_model)forzinx_test)np.save(pre_data/test_vecs.npy,test_vecs)prin

7、t(test_vecs.shape)4、獲得訓(xùn)練集向量和標(biāo)簽，測試集向量和標(biāo)簽#獲得訓(xùn)練集向量和標(biāo)簽，測試集向量和標(biāo)簽defget_data():train_vecs=np.1oad(pre_data/train_vecs.npy)y_train=np.load(pre_data/y_train.npy)test_vecs=np.1oad(pre_data/test_vecs.npy)y_test=np.load(pre_data/y_test.npy)returntrain_vecs,y_train,test_vecs,y_test5、訓(xùn)練SVM模型#訓(xùn)練SVM模型defsvm_train(

8、train_vecs,y_train,test_vecs,y_test):c1f=SVC(kerne1=rbf,verbose=True)c1f.fit(train_vecs,y_train)#艮據(jù)給定的訓(xùn)練數(shù)據(jù)擬合SVM模型job1ib.dump(c1f,pre_data/svm_mode1/mode1.pk1)#保存訓(xùn)練好的SVM模型print(c1f.score(test_vecs,y_test)#輸出測試數(shù)據(jù)的平均準(zhǔn)確度6、構(gòu)建待遇測句子的向量#構(gòu)建待遇測句子的向量defget_predict_vecs(words):n_dim=300w2v_mode1=Word2Vec.load(p

9、re_data/w2v_mode1/w2v_mode1.pk1)train_vecs=bui1d_sentence_vector(words,n_dim,w2v_mode1)returntrain_vecs7、對單個句子進(jìn)行情感判斷#對單個句子進(jìn)行情感判斷defsvm_predict(string):words=jieba.lcut(string)words_vecs=get_predict_vecs(words)clf=joblib.load(pre_data/svm_model/model.pkl)result=clf.predict(words_vecs)ifint(resultO)=1:print(string,positive)else:print(string,negative)if_name_=_main_:#x_train,x_test=loadile_and_preprocessing()#get_train_vecs(x_train,x_test)#train_vecs,y_train,test_vecs,y_test=get_data()#svm_train(train_vecs,y_train,test_v