數(shù)據(jù)挖掘原語和挖掘語言.ppt

1、挖掘原語，語言和體系結(jié)構(gòu),數(shù)據(jù)挖掘原語 數(shù)據(jù)挖掘語言 數(shù)據(jù)挖掘系統(tǒng)體系結(jié)構(gòu) 總結(jié),數(shù)據(jù)挖掘原語劃分,挖掘相關(guān)的數(shù)據(jù) 挖掘的知識類型 背景知識 模式的興趣度量 結(jié)果的表示與可視化,任務(wù)相關(guān)數(shù)據(jù),數(shù)據(jù)庫（或數(shù)據(jù)倉庫）名稱 例如：AllElectronics_db 數(shù)據(jù)庫表（或數(shù)據(jù)倉庫的立方體） 例如：表item,customer,purchase,items_sold 數(shù)據(jù)選擇條件 例如：選取本年度加拿大地區(qū)購買商品的數(shù)據(jù) 選取條件可能在概念上層次高于DB/DW的數(shù)據(jù) 如：”type=home entertainment”，DB/DW中數(shù)據(jù)tv,cd player,vcr 有關(guān)的屬性（或維） 例如

2、：item表的name,price屬性;customer表的income,age屬性。 系統(tǒng)應(yīng)具備自動選取相關(guān)屬性的機制，比如通過評估各屬性與特定操作的相關(guān)程度。 數(shù)據(jù)分組的標準 例如：根據(jù)日期進行分組,挖掘的知識類型,描述(characterization) 區(qū)別分析(discrimination) 關(guān)聯(lián)(association) 分類/預(yù)測(classification/prediction) 聚類(clustering),例: 用戶如果想發(fā)掘AllElectronics數(shù)據(jù)庫中用戶的購買習(xí)慣，可能會選擇下面關(guān)聯(lián)規(guī)則： P(X:customer,W)Q(X,Y)=buys(X,Z) X是c

3、ustomer表的主鍵，P,Q是謂詞變量(在相關(guān)數(shù)據(jù)中定義)，W,Y,Z是目標變量。可能的挖掘結(jié)果是： age(X,”3039”) income (X,”40k49k”) = buys(X,”VCR”)2.2%,60% accupation(X,”student”)age(X,”2029”)=buys(X,”computer”)1.4%,70%,背景知識：概念層次,概念層次 模式層次(schema hierarchy) 例：Streetcityprovince_or_statecountry 集合-分組層次(set-grouping hierarchy) 例： young,middle_age

4、d,seniorall(age) 20-39 = young, 40-59 = middle_aged 基于操作層次(operation-derived hierarchy) 包括信息解碼，復(fù)雜數(shù)據(jù)對象的信息提取，數(shù)據(jù)聚類，數(shù)據(jù)分布分析算法等 例： email address: login-name department university country 基于規(guī)則層次(rule-based hierarchy) 例： low_profit_margin (X) = price(X, P1) and cost (X, P2) and (P1 - P2) $50 用戶對數(shù)據(jù)間關(guān)系的預(yù)測 可以用

5、于評價挖掘模式的興趣度量,模式興趣度量,簡潔性(simplicity) 如：(關(guān)聯(lián)) 規(guī)則長度, (決策) 決策樹規(guī)模大小 確定性(certainty) 如：confidence, P(A|B) = n(A and B)/ n (B), classification reliability or accuracy ( also known as rule reliability , rule strength, rule quality, certainty factor, discriminating weight )等. 有用性(utility) 如：support (associatio

6、n),s(A=B)=n(A nd B)/n(all), noise threshold (description) 新穎程度(novelty) 如：not previously known, surprising (used to remove redundant rules, e.g., Canada vs. Vancouver rule implication support ratio,結(jié)果模式的可視化,挖掘系統(tǒng)應(yīng)能夠用多種形式來顯示發(fā)掘出來的模式 如：規(guī)則，表，報表，圖表，圖，決策數(shù)和立方體 挖掘系統(tǒng)應(yīng)能夠支持挖掘結(jié)果的多種操作 如：drill-down , roll-up , sli

7、cing , dicing ,rotation,挖掘原語，語言和體系結(jié)構(gòu),數(shù)據(jù)挖掘原語 數(shù)據(jù)挖掘語言 數(shù)據(jù)挖掘系統(tǒng)體系結(jié)構(gòu) 總結(jié),DMQL一個數(shù)據(jù)挖掘語言,動機 為了能提供交互式數(shù)據(jù)挖掘能力 通過提供一個類似SQL的語言 希望能像SQL語言一樣成為挖掘標準語言 希望成為系統(tǒng)開發(fā)和演化(evolution)的基礎(chǔ) 希望促進信息交換，技術(shù)轉(zhuǎn)移，商業(yè)化并獲得廣泛承認 設(shè)計 DMQL 在前面介紹的挖掘原語基礎(chǔ)之上進行設(shè)計,任務(wù)相關(guān)數(shù)據(jù)的語法表示,use database , or use data warehouse from where in relevance to order by group

8、by having ,任務(wù)相關(guān)數(shù)據(jù)語法表示（續(xù)）,例：如果挖掘AllElectronics的加拿大顧客經(jīng)常購買的商品之間的關(guān)聯(lián)，針對顧客不同收入和年齡，并且數(shù)據(jù)用購買日期進行分組。挖掘相關(guān)數(shù)據(jù)可以寫成： use database AllElectronics_db in relevance to I.name , I.price , C.income , C.age from customer C , item I , purchase P , item_sold S where I.item_ID=S.item_ID and S.trans_ID=P.trans_ID and P.cust_

9、ID=C.cust_ID group by P.date,挖掘知識類型的語法,:= | | | | := Mine characterization as analyze 例：mine characteristics as customerPurchasing analyze count% :=Mine comparison as for where versus where analyze 例：mine comparison as purchaseGroups for bigSpenders where avg(I.price) $100 analyze count versus budge

10、tSpenders where avg(I.price),$100,挖掘知識類型的語法(續(xù)),:=mine association as matching 例：mine associations as buyingHabits matching P(X:customer,W)Q(X,Y)=buys(X,Z) :=mine classification as analyze 例： mine classification as classifyingCustomerCreditRating analyze credit_info :=Mine prediction as analyze set a

11、ttribute_or_dimention_i= 例：mine prediction as predictItemPrice analyze price set category = “TV” and brand=“SONY”,概念層次語法,語法： Use hierarchy for 不同概念層次采用不同定義方式 模式概念層次 define hierarchy time_hierarchy on date as date,month quarter,year 集合-分組概念層次 define hierarchy age_hierarchy for age on customer as leve

12、l1: young, middle_aged, senior level0: all level2: 20, ., 39 level1: young level2: 40, ., 59 level1: middle_aged level2: 60, ., 89 level1: senior,概念層次語法（續(xù)）,基于操作概念模式(operation-derived hierarchies) define hierarchy age_hierarchy for age on customer as age_category(1), ., age_category(5) := cluster(def

13、ault, age, 5) $50) and (price - cost) $250,興趣度量語法,語法： with threshold = threshold_value 例: with support threshold = 0.05 with confidence threshold = 0.7,挖掘知識表示語法,用戶指定顯示方法 display as 為在不同概念層次上觀察結(jié)果： Multilevel_Manipulation := roll up on | drill down on | add | drop ,一個完整的DMQL語句,use database AllElectron

14、ics_db use hierarchy location_hierarchy for B.address mine characteristics as customerPurchasing analyze count% in relevance to C.age , I.type , I.place_made from customer C, item I , purchases P , items_sold S , works_at W , branch B where I.item_ID = S.item_ID and S.trans_ID = P.trans_ID and P.cus

15、t_ID = C.cust_ID and P.method_paid = AmEx and P.empl_ID = W.empl_ID and W.branch_ID = B.branch_ID and B.address = Canada and I.price = 100 with noise threshold = 0.05 display as table,其它數(shù)據(jù)挖掘語言,關(guān)聯(lián)規(guī)則語言 MSQL (Imielinski & Virmani99) MineRule (Meo Psaila and Ceri96) Query flocks 基于Datalog 語法 (Tsur et al

16、98) OLEDB for DM (Microsoft2000) 和 OLE DB, OLE DB for OLAP一起致力于DB,DW,DM的標準化 到2000年3月止，已經(jīng)完成了predictive modeling( classification & Prediction ), clustering,還未包含 characterization, discrimination , association modeling 等。 CRISP-DM (CRoss-Industry Standard Process for Data Mining) 是一個國際性項目，包含數(shù)據(jù)庫公司，數(shù)據(jù)倉庫公司

17、，用戶公司(user companies) 目的在于提供有效數(shù)據(jù)挖掘的平臺和過程結(jié)構(gòu)(process structure) 強調(diào)運用數(shù)據(jù)挖掘技術(shù)來解決商業(yè)問題,挖掘原語，語言和體系結(jié)構(gòu),數(shù)據(jù)挖掘原語 數(shù)據(jù)挖掘語言 數(shù)據(jù)挖掘系統(tǒng)體系結(jié)構(gòu) 總結(jié),數(shù)據(jù)挖掘系統(tǒng)體系結(jié)構(gòu),數(shù)據(jù)挖掘系統(tǒng)與 DB/DW 系統(tǒng)的耦合程度 零耦合用文件作為數(shù)據(jù)源和存放結(jié)果數(shù)據(jù),不推薦 松散耦合 用DB/DW作數(shù)據(jù)源，查詢結(jié)果寫入文件或DB/DW；但不使用DB/DW的提供的數(shù)據(jù)結(jié)構(gòu)和查詢優(yōu)化方法。 半緊耦合提升挖掘系統(tǒng)性能 部分挖掘原語在DB/DW中實現(xiàn)，如sorting, indexing, aggregation , histogram analysis, multiway join, precomputation of some statistic functions such as count ,sum