Python爬蟲程序設(shè)計(jì)KC26課件_第1頁
Python爬蟲程序設(shè)計(jì)KC26課件_第2頁
Python爬蟲程序設(shè)計(jì)KC26課件_第3頁
Python爬蟲程序設(shè)計(jì)KC26課件_第4頁
Python爬蟲程序設(shè)計(jì)KC26課件_第5頁
已閱讀5頁,還剩41頁未讀, 繼續(xù)免費(fèi)閱讀

下載本文檔

版權(quán)說明:本文檔由用戶提供并上傳,收益歸屬內(nèi)容提供方,若內(nèi)容存在侵權(quán),請(qǐng)進(jìn)行舉報(bào)或認(rèn)領(lǐng)

文檔簡介

2.6.1項(xiàng)目簡介2.6.1項(xiàng)目簡介2.6.1項(xiàng)目簡介在中國天氣網(wǎng)()中輸入一個(gè)城市的名稱,例如輸入深圳,那么會(huì)轉(zhuǎn)到地址/weather1d/101280601.shtml的網(wǎng)頁顯示深圳的天氣預(yù)報(bào),其中101280601是深圳的代碼,每個(gè)城市或者地區(qū)都有一個(gè)代碼。如圖2-6-1、2-6-2所示。2.6.1項(xiàng)目簡介Python爬蟲程序設(shè)計(jì)KC26課件Python爬蟲程序設(shè)計(jì)KC26課件2.6.2HTML代碼分析2.6.2HTML代碼分析2.6.2HTML代碼分析用Chrome瀏覽器瀏覽網(wǎng)站,鼠標(biāo)指向7天天氣預(yù)報(bào)的今天位置,點(diǎn)擊右鍵彈出菜單,選擇“檢查”就可以打開這個(gè)位置對(duì)應(yīng)的HTML代碼,如圖2-6-3所示。2.6.2HTML代碼分析選擇<ulclass="tclearfix">元素,點(diǎn)擊右鍵彈出菜單選擇"EditasHTML",就可以進(jìn)入編輯狀態(tài),復(fù)制整個(gè)HTML,結(jié)果如下:<ulclass="tclearfix"><liclass="on"><h1>5日(今天)</h1><bigclass="png40d01"></big><bigclass="png40n01"></big><ptitle="多云"class="wea">多云</p><pclass="tem"><span>32</span>/<i>28℃</i></p><pclass="win"><em><spantitle="無持續(xù)風(fēng)向"class=""></span><spantitle="無持續(xù)風(fēng)向"class=""></span></em>選擇<ulclass="tclearfix">元素,點(diǎn)擊<i>微風(fēng)</i></p><divclass="slid"></div></li><li><h1>6日(明天)</h1><bigclass="png40d01"></big><bigclass="png40n01"></big><ptitle="多云"class="wea">多云</p><pclass="tem"><span>32</span>/<i>27℃</i></p><pclass="win"><em><spantitle="無持續(xù)風(fēng)向"class=""></span><spantitle="無持續(xù)風(fēng)向"class=""></span></em><i>微風(fēng)</i></p><divclass="slid"></div></li><li><i>微風(fēng)</i><h1>7日(后天)</h1><bigclass="png40d01"></big><bigclass="png40n01"></big><ptitle="多云"class="wea">多云</p><pclass="tem"><span>32</span>/<i>27℃</i></p><pclass="win"><em><spantitle="無持續(xù)風(fēng)向"class=""></span><spantitle="無持續(xù)風(fēng)向"class=""></span></em><i>微風(fēng)</i></p><divclass="slid"></div></li><li><h1>8日(周四)</h1><bigclass="png40d01"></big><bigclass="png40n01"></big><ptitle="多云"class="wea">多云</p><pclass="tem"><span>32</span>/<i>27℃</i></p><pclass="win"><em><spantitle="無持續(xù)風(fēng)向"class=""></span><spantitle="無持續(xù)風(fēng)向"class=""></span></em><i>微風(fēng)</i></p><divclass="slid"></div></li><li><h1>7日(后天)</h1><h1>9日(周五)</h1><bigclass="png40d01"></big><bigclass="png40n01"></big><ptitle="多云"class="wea">多云</p><pclass="tem"><span>33</span>/<i>27℃</i></p><pclass="win"><em><spantitle="無持續(xù)風(fēng)向"class=""></span><spantitle="無持續(xù)風(fēng)向"class=""></span></em><i>微風(fēng)</i></p><divclass="slid"></div></li><li><h1>9日(周五)</h1><h1>10日(周六)</h1><bigclass="png40d01"></big><bigclass="png40n01"></big><ptitle="多云"class="wea">多云</p><pclass="tem"><span>33</span>/<i>27℃</i></p><pclass="win"><em><spantitle="無持續(xù)風(fēng)向"class=""></span><spantitle="無持續(xù)風(fēng)向"class=""></span></em><i>微風(fēng)</i></p><divclass="slid"></div></li><li><h1>10日(周六)</h1><h1>11日(周日)</h1><bigclass="png40d01"></big><bigclass="png40n07"></big><ptitle="多云轉(zhuǎn)小雨"class="wea">多云轉(zhuǎn)小雨</p><pclass="tem"><span>33</span>/<i>26℃</i></p><pclass="win"><em><spantitle="無持續(xù)風(fēng)向"class=""></span><spantitle="無持續(xù)風(fēng)向"class=""></span></em><i>微風(fēng)</i></p><divclass="slid"></div></li></ul><h1>11日(周日)</h1>2.6.3爬取天氣預(yù)報(bào)數(shù)據(jù)2.6.3爬取天氣預(yù)報(bào)數(shù)據(jù)2.6.3爬取天氣預(yù)報(bào)數(shù)據(jù)通過分析HTML代碼,我們可以編寫爬取的程序爬取深圳7天的天氣預(yù)報(bào)數(shù)據(jù):

frombs4importBeautifulSoupfrombs4importUnicodeDammitimporturllib.request

url="/weather/101280601.shtml"try:headers={"User-Agent":"Mozilla/5.0(Windows;U;WindowsNT6.0x64;en-US;rv:1.9pre)Gecko/2008072421Minefield/3.0.2pre"}req=urllib.request.Request(url,headers=headers)data=urllib.request.urlopen(req)data=data.read()dammit=UnicodeDammit(data,["utf-8","gbk"])data=dammit.unicode_markupsoup=BeautifulSoup(data,"lxml")lis=soup.select("ul[class='tclearfix']li")forliinlis:try:date=li.select('h1')[0].textweather=li.select('p[class="wea"]')[0].texttemp=li.select('p[class="tem"]span')[0].text+"/"+li.select('p[class="tem"]i')[0].textprint(date,weather,temp)exceptExceptionaserr:print(err)exceptExceptionaserr:print(err)2.6.3爬取天氣預(yù)報(bào)數(shù)據(jù)程序爬取結(jié)果:5日(今天)多云32/28℃6日(明天)多云32/27℃7日(后天)多云32/27℃8日(周四)多云32/27℃9日(周五)多云33/27℃10日(周六)多云33/27℃11日(周日)多云轉(zhuǎn)小雨33/26℃由此可見爬取的數(shù)據(jù)與我們直接從網(wǎng)站看到的是一樣的。

程序爬取結(jié)果:2.6.4爬取與存儲(chǔ)天氣預(yù)報(bào)數(shù)據(jù)2.6.4爬取與存儲(chǔ)天氣預(yù)報(bào)數(shù)據(jù)2.6.4爬取與存儲(chǔ)天氣預(yù)報(bào)數(shù)據(jù)我們可以獲取北京、上海、廣州、深圳等城市的代碼,爬取這些城市的天氣預(yù)報(bào)數(shù)據(jù),并存儲(chǔ)到sqllite數(shù)據(jù)庫weathers.db中,存儲(chǔ)的數(shù)據(jù)表weathers是:createtableweathers(wCityvarchar(16),wDatevarchar(16),wWeathervarchar(64),wTempvarchar(32),constraintpk_weatherprimarykey(wCity,wDate))"編寫程序依次爬取各個(gè)城市的天氣預(yù)報(bào)數(shù)據(jù)存儲(chǔ)在數(shù)據(jù)庫中,程序如下:2.6.4爬取與存儲(chǔ)天氣預(yù)報(bào)數(shù)據(jù)frombs4importBeautifulSoupfrombs4importUnicodeDammitimporturllib.requestimportsqlite3

classWeatherDB:defopenDB(self):self.con=sqlite3.connect("weathers.db")self.cursor=self.con.cursor()try:self.cursor.execute("createtableweathers(wCityvarchar(16),wDatevarchar(16),wWeathervarchar(64),wTempvarchar(32),constraintpk_weatherprimarykey(wCity,wDate))")except:self.cursor.execute("deletefromweathers")

defcloseDB(self):mit()self.con.close()frombs4importBeautifulSoupdefinsert(self,city,date,weather,temp):try:self.cursor.execute("insertintoweathers(wCity,wDate,wWeather,wTemp)values(?,?,?,?)",(city,date,weather,temp))exceptExceptionaserr:print(err)

defshow(self):self.cursor.execute("select*fromweathers")rows=self.cursor.fetchall()print("%-16s%-16s%-32s%-16s"%("city","date","weather","temp"))forrowinrows:print("%-16s%-16s%-32s%-16s"%(row[0],row[1],row[2],row[3]))

classWeatherForecast:def__init__(self):self.headers={"User-Agent":"Mozilla/5.0(Windows;U;WindowsNT6.0x64;en-US;rv:1.9pre)Gecko/2008072421Minefield/3.0.2pre"}self.cityCode={"北京":"101010100","上海":"101020100","廣州":"101280101","深圳":"101280601"}definsert(self,city,date,weatdefforecastCity(self,city):ifcitynotinself.cityCode.keys():print(city+"codecannotbefound")return

url="/weather/"+self.cityCode[city]+".shtml"try:req=urllib.request.Request(url,headers=self.headers)data=urllib.request.urlopen(req)data=data.read()dammit=UnicodeDammit(data,["utf-8","gbk"])data=dammit.unicode_markupsoup=BeautifulSoup(data,"lxml")lis=soup.select("ul[class='tclearfix']li")defforecastCity(self,city):forliinlis:try:date=li.select('h1')[0].textweather=li.select('p[class="wea"]')[0].texttemp=li.select('p[class="tem"]span')[0].text+"/"+li.select('p[class="tem"]i')[0].textprint(city,date,weather,temp)self.db.insert(city,date,weather,temp)exceptExceptionaserr:print(err)exceptExceptionaserr:print(err)forliinlis:defprocess(self,cities):self.db=WeatherDB()self.db.openDB()

forcityincities:self.forecastCity(city)

#self.db.show()self.db.closeDB()

ws=WeatherForecast()cess(["北京","上海","廣州","深圳"])print("completed")defprocess(self,cities):北京7日(今天)晴間多云,北部山區(qū)有陣雨或雷陣雨轉(zhuǎn)晴轉(zhuǎn)多云31℃/17℃北京8日(明天)多云轉(zhuǎn)晴,北部地區(qū)有分散陣雨或雷陣雨轉(zhuǎn)晴34℃/20℃北京9日(后天)晴轉(zhuǎn)多云36℃/22℃北京10日(周六)陰轉(zhuǎn)陣雨30℃/19℃北京11日(周日)陣雨27℃/18℃北京12日(周一)陰轉(zhuǎn)晴28℃/20℃北京13日(周二)晴32℃/21℃上海7日(今天)多云30/21℃上海8日(明天)多云轉(zhuǎn)陰32/23℃上海9日(后天)陣雨32/24℃上海10日(周六)中雨27/22℃上海11日(周日)小雨轉(zhuǎn)多云29/22℃上海12日(周一)多云30/22℃上海13日(周二)多云轉(zhuǎn)陰30/21℃廣州7日(今天)多云35/27℃廣州8日(明天)多云35/28℃廣州9日(后天)多云35/28℃廣州10日(周六)多云35/28℃廣州11日(周日)多云35/28℃廣州12日(周一)雷陣雨35/27℃廣州13日(周二)雷陣雨轉(zhuǎn)大雨33/24℃深圳7日(今天)陣雨轉(zhuǎn)多云34/28℃深圳8日(明天)晴34/28℃深圳9日(后天)晴34/28℃深圳10日(周六)晴轉(zhuǎn)陣雨34/28℃深圳11日(周日)陣雨33/27℃深圳12日(周一)陣雨32/27℃深圳13日(周二)陣雨轉(zhuǎn)中雨32/25℃北京7日(今天)晴間多云,北部山區(qū)有陣雨或雷陣雨轉(zhuǎn)晴轉(zhuǎn)多2.6.1項(xiàng)目簡介2.6.1項(xiàng)目簡介2.6.1項(xiàng)目簡介在中國天氣網(wǎng)()中輸入一個(gè)城市的名稱,例如輸入深圳,那么會(huì)轉(zhuǎn)到地址/weather1d/101280601.shtml的網(wǎng)頁顯示深圳的天氣預(yù)報(bào),其中101280601是深圳的代碼,每個(gè)城市或者地區(qū)都有一個(gè)代碼。如圖2-6-1、2-6-2所示。2.6.1項(xiàng)目簡介Python爬蟲程序設(shè)計(jì)KC26課件Python爬蟲程序設(shè)計(jì)KC26課件2.6.2HTML代碼分析2.6.2HTML代碼分析2.6.2HTML代碼分析用Chrome瀏覽器瀏覽網(wǎng)站,鼠標(biāo)指向7天天氣預(yù)報(bào)的今天位置,點(diǎn)擊右鍵彈出菜單,選擇“檢查”就可以打開這個(gè)位置對(duì)應(yīng)的HTML代碼,如圖2-6-3所示。2.6.2HTML代碼分析選擇<ulclass="tclearfix">元素,點(diǎn)擊右鍵彈出菜單選擇"EditasHTML",就可以進(jìn)入編輯狀態(tài),復(fù)制整個(gè)HTML,結(jié)果如下:<ulclass="tclearfix"><liclass="on"><h1>5日(今天)</h1><bigclass="png40d01"></big><bigclass="png40n01"></big><ptitle="多云"class="wea">多云</p><pclass="tem"><span>32</span>/<i>28℃</i></p><pclass="win"><em><spantitle="無持續(xù)風(fēng)向"class=""></span><spantitle="無持續(xù)風(fēng)向"class=""></span></em>選擇<ulclass="tclearfix">元素,點(diǎn)擊<i>微風(fēng)</i></p><divclass="slid"></div></li><li><h1>6日(明天)</h1><bigclass="png40d01"></big><bigclass="png40n01"></big><ptitle="多云"class="wea">多云</p><pclass="tem"><span>32</span>/<i>27℃</i></p><pclass="win"><em><spantitle="無持續(xù)風(fēng)向"class=""></span><spantitle="無持續(xù)風(fēng)向"class=""></span></em><i>微風(fēng)</i></p><divclass="slid"></div></li><li><i>微風(fēng)</i><h1>7日(后天)</h1><bigclass="png40d01"></big><bigclass="png40n01"></big><ptitle="多云"class="wea">多云</p><pclass="tem"><span>32</span>/<i>27℃</i></p><pclass="win"><em><spantitle="無持續(xù)風(fēng)向"class=""></span><spantitle="無持續(xù)風(fēng)向"class=""></span></em><i>微風(fēng)</i></p><divclass="slid"></div></li><li><h1>8日(周四)</h1><bigclass="png40d01"></big><bigclass="png40n01"></big><ptitle="多云"class="wea">多云</p><pclass="tem"><span>32</span>/<i>27℃</i></p><pclass="win"><em><spantitle="無持續(xù)風(fēng)向"class=""></span><spantitle="無持續(xù)風(fēng)向"class=""></span></em><i>微風(fēng)</i></p><divclass="slid"></div></li><li><h1>7日(后天)</h1><h1>9日(周五)</h1><bigclass="png40d01"></big><bigclass="png40n01"></big><ptitle="多云"class="wea">多云</p><pclass="tem"><span>33</span>/<i>27℃</i></p><pclass="win"><em><spantitle="無持續(xù)風(fēng)向"class=""></span><spantitle="無持續(xù)風(fēng)向"class=""></span></em><i>微風(fēng)</i></p><divclass="slid"></div></li><li><h1>9日(周五)</h1><h1>10日(周六)</h1><bigclass="png40d01"></big><bigclass="png40n01"></big><ptitle="多云"class="wea">多云</p><pclass="tem"><span>33</span>/<i>27℃</i></p><pclass="win"><em><spantitle="無持續(xù)風(fēng)向"class=""></span><spantitle="無持續(xù)風(fēng)向"class=""></span></em><i>微風(fēng)</i></p><divclass="slid"></div></li><li><h1>10日(周六)</h1><h1>11日(周日)</h1><bigclass="png40d01"></big><bigclass="png40n07"></big><ptitle="多云轉(zhuǎn)小雨"class="wea">多云轉(zhuǎn)小雨</p><pclass="tem"><span>33</span>/<i>26℃</i></p><pclass="win"><em><spantitle="無持續(xù)風(fēng)向"class=""></span><spantitle="無持續(xù)風(fēng)向"class=""></span></em><i>微風(fēng)</i></p><divclass="slid"></div></li></ul><h1>11日(周日)</h1>2.6.3爬取天氣預(yù)報(bào)數(shù)據(jù)2.6.3爬取天氣預(yù)報(bào)數(shù)據(jù)2.6.3爬取天氣預(yù)報(bào)數(shù)據(jù)通過分析HTML代碼,我們可以編寫爬取的程序爬取深圳7天的天氣預(yù)報(bào)數(shù)據(jù):

frombs4importBeautifulSoupfrombs4importUnicodeDammitimporturllib.request

url="/weather/101280601.shtml"try:headers={"User-Agent":"Mozilla/5.0(Windows;U;WindowsNT6.0x64;en-US;rv:1.9pre)Gecko/2008072421Minefield/3.0.2pre"}req=urllib.request.Request(url,headers=headers)data=urllib.request.urlopen(req)data=data.read()dammit=UnicodeDammit(data,["utf-8","gbk"])data=dammit.unicode_markupsoup=BeautifulSoup(data,"lxml")lis=soup.select("ul[class='tclearfix']li")forliinlis:try:date=li.select('h1')[0].textweather=li.select('p[class="wea"]')[0].texttemp=li.select('p[class="tem"]span')[0].text+"/"+li.select('p[class="tem"]i')[0].textprint(date,weather,temp)exceptExceptionaserr:print(err)exceptExceptionaserr:print(err)2.6.3爬取天氣預(yù)報(bào)數(shù)據(jù)程序爬取結(jié)果:5日(今天)多云32/28℃6日(明天)多云32/27℃7日(后天)多云32/27℃8日(周四)多云32/27℃9日(周五)多云33/27℃10日(周六)多云33/27℃11日(周日)多云轉(zhuǎn)小雨33/26℃由此可見爬取的數(shù)據(jù)與我們直接從網(wǎng)站看到的是一樣的。

程序爬取結(jié)果:2.6.4爬取與存儲(chǔ)天氣預(yù)報(bào)數(shù)據(jù)2.6.4爬取與存儲(chǔ)天氣預(yù)報(bào)數(shù)據(jù)2.6.4爬取與存儲(chǔ)天氣預(yù)報(bào)數(shù)據(jù)我們可以獲取北京、上海、廣州、深圳等城市的代碼,爬取這些城市的天氣預(yù)報(bào)數(shù)據(jù),并存儲(chǔ)到sqllite數(shù)據(jù)庫weathers.db中,存儲(chǔ)的數(shù)據(jù)表weathers是:createtableweathers(wCityvarchar(16),wDatevarchar(16),wWeathervarchar(64),wTempvarchar(32),constraintpk_weatherprimarykey(wCity,wDate))"編寫程序依次爬取各個(gè)城市的天氣預(yù)報(bào)數(shù)據(jù)存儲(chǔ)在數(shù)據(jù)庫中,程序如下:2.6.4爬取與存儲(chǔ)天氣預(yù)報(bào)數(shù)據(jù)frombs4importBeautifulSoupfrombs4importUnicodeDammitimporturllib.requestimportsqlite3

classWeatherDB:defopenDB(self):self.con=sqlite3.connect("weathers.db")self.cursor=self.con.cursor()try:self.cursor.execute("createtableweathers(wCityvarchar(16),wDatevarchar(16),wWeathervarchar(64),wTempvarchar(32),constraintpk_weatherprimarykey(wCity,wDate))")except:self.cursor.execute("deletefromweathers")

defcloseDB(self):mit()self.con.close()frombs4importBeautifulSoupdefinsert(self,city,date,weather,temp):try:self.cursor.execute("insertintoweathers(wCity,wDate,wWeather,wTemp)values(?,?,?,?)",(city,date,weather,temp))exceptExceptionaserr:print(err)

defshow(self):self.cursor.execute("select*fromweathers")rows=self.cursor.fetchall()print("%-16s%-16s%-32s%-16s"%("city","date","weather","temp"))forrowinrows:print("%-16s%-16s%-32s%-16s"%(row[0],row[1],row[2],row[3]))

classWeatherForecast:def__init__(self):self.headers={"User-Agent":"Mozilla/5.0(Windows;U;WindowsNT6.0x64;en-US;rv:1.9pre)Gecko/2008072421Minefield/3.0.2pre"}self.cityCode={"北京":"101010100","上海":"101020100","廣州":"101280101","深圳":"101280601"}definsert(self,city,date,weatdefforecastCity(self,city):ifcitynotinself.cityCode.keys():print(city+"codecannotbefound")return

url="/weather/"+self.cityCode[city]+".shtml"try:req=urllib.request.Request(url,headers=self.headers)data=urllib.request.urlopen(req)data=data.read()dammit=UnicodeDammit(data,["utf-8","gbk"])data=dammit.unicode_markupsoup=BeautifulSoup(data,"lxml")lis=soup.select("ul[class='tclearfix']li")defforecastCity(self,city):forliin

溫馨提示

  • 1. 本站所有資源如無特殊說明,都需要本地電腦安裝OFFICE2007和PDF閱讀器。圖紙軟件為CAD,CAXA,PROE,UG,SolidWorks等.壓縮文件請(qǐng)下載最新的WinRAR軟件解壓。
  • 2. 本站的文檔不包含任何第三方提供的附件圖紙等,如果需要附件,請(qǐng)聯(lián)系上傳者。文件的所有權(quán)益歸上傳用戶所有。
  • 3. 本站RAR壓縮包中若帶圖紙,網(wǎng)頁內(nèi)容里面會(huì)有圖紙預(yù)覽,若沒有圖紙預(yù)覽就沒有圖紙。
  • 4. 未經(jīng)權(quán)益所有人同意不得將文件中的內(nèi)容挪作商業(yè)或盈利用途。
  • 5. 人人文庫網(wǎng)僅提供信息存儲(chǔ)空間,僅對(duì)用戶上傳內(nèi)容的表現(xiàn)方式做保護(hù)處理,對(duì)用戶上傳分享的文檔內(nèi)容本身不做任何修改或編輯,并不能對(duì)任何下載內(nèi)容負(fù)責(zé)。
  • 6. 下載文件中如有侵權(quán)或不適當(dāng)內(nèi)容,請(qǐng)與我們聯(lián)系,我們立即糾正。
  • 7. 本站不保證下載資源的準(zhǔn)確性、安全性和完整性, 同時(shí)也不承擔(dān)用戶因使用這些下載資源對(duì)自己和他人造成任何形式的傷害或損失。

最新文檔

評(píng)論

0/150

提交評(píng)論