深度相機(jī)、微軟kinect及其應(yīng)用

USTCSummerSchool2012AdvancesinComputerGraphics（計(jì)算機(jī)圖形學(xué)前沿進(jìn)展）深度相機(jī)、微軟Kinect及其應(yīng)用童晶2012-6-281/50介紹的主要內(nèi)容：三維掃描技術(shù)介紹深度相機(jī)的原理、優(yōu)勢(shì)（TOF，Kinect）深度相機(jī)的應(yīng)用研究三維數(shù)據(jù)重建人機(jī)交互與用戶(hù)跟蹤我們的工作：三維發(fā)型掃描重建（TOF）三維人體重建（Kinect）總結(jié)2/50三維掃描技術(shù)介紹3DScanningGeometryDynamics(Motion,Deformation,etc.)Rendering(IlluminationModel)DigitalEmilyProject3/50三維掃描的應(yīng)用:娛樂(lè)業(yè)和消費(fèi)電子4/50三維掃描的應(yīng)用:歷史遺跡保護(hù)5/50三維掃描的應(yīng)用:醫(yī)學(xué)圖像和手術(shù)規(guī)劃6/50三維掃描的應(yīng)用：機(jī)器人(交互與導(dǎo)航)7/50三維掃描的應(yīng)用:檢測(cè)和逆向工程8/50三維激光掃描原理精度高速度慢、不易操作、成本高9/50The3DScanningPipelineAssignTexture(BRDF,etc.)Align3DPointCloudsReconstruct3DPointCloud(s)DataCaptureExtractIsosurface(PolyhedralMesh)10/50三維掃描設(shè)備laserscanstructuredlight$240,000$50,00011/50深度數(shù)據(jù)獲取方法LaserscanningsystemMulti-viewsystemTime-of-flightsensor(TOF)$4,000MicrosoftKinect$15012/50深度相機(jī)原理（TimeofFlight）3DsceneSensor13/50深度相機(jī)原理（Kinect）當(dāng)激光穿透毛玻璃后形成隨機(jī)衍射斑點(diǎn)，這些散斑（laserspeckle）具有高度的隨機(jī)性，而且會(huì)隨著距離的不同變換圖案。空間中任意兩處散斑圖案都不同Lightcoding打出了一個(gè)具有三維縱深的“體編碼”，只要看物體表面的散斑圖案，就可以知道這個(gè)物體在什么位置14/50捕獲深度數(shù)據(jù)特點(diǎn)Time-of-flightsensor(TOF)Upto54FPS高速拍照低分辨率176*144噪聲大15/50深度數(shù)據(jù)預(yù)處理利用單幀深度數(shù)據(jù)進(jìn)行去噪(高斯平滑濾波、雙邊濾波)利用高分辨率彩色圖像的信息來(lái)提高低分辨率深度數(shù)據(jù)的質(zhì)量利用多幀數(shù)據(jù)的冗余信息16/50數(shù)據(jù)捕獲與剛體配準(zhǔn)固定TOF相機(jī)，待測(cè)物體旋轉(zhuǎn)ICP對(duì)齊17/50數(shù)據(jù)捕獲與剛體配準(zhǔn)固定TOF相機(jī)，待測(cè)物體旋轉(zhuǎn)ICP對(duì)齊18/50靜態(tài)場(chǎng)景重建數(shù)據(jù)預(yù)處理分割、去噪序列剛體對(duì)齊19/50全局剛體對(duì)齊20/50靜態(tài)場(chǎng)景重建數(shù)據(jù)預(yù)處理分割、去噪序列剛體對(duì)齊全局剛體對(duì)齊網(wǎng)格重建21/50靜態(tài)場(chǎng)景重建KinectFusion:Realtime3DReconstructionandInteractionUsingaMovingDepthCameraReconstructionof3DModelsusingKinectandRGBD-SLAM22/50動(dòng)態(tài)場(chǎng)景重建算法23/50動(dòng)態(tài)場(chǎng)景重建算法24/50Templatebasedmethod25/50三維場(chǎng)景重建—應(yīng)用三維掃描KinectFusion-PCL-ReconstructMe（三維人體掃描、三維物體掃描、生成雕塑）街頭快速人像雕塑人機(jī)交互飛行機(jī)器人、盲人導(dǎo)航、輔助倒車(chē)增強(qiáng)現(xiàn)實(shí)Kinect魔術(shù)三維重建表情跟蹤如何看到看不見(jiàn)的人26/50Kinect捕獲數(shù)據(jù)深度數(shù)據(jù)彩色數(shù)據(jù)骨架、手勢(shì)人臉跟蹤聲音KinectSDK1.5現(xiàn)場(chǎng)測(cè)試Kinect

for

PC201227/50骨架跟蹤算法？28/50骨架跟蹤算法1Accurate3DPoseEstimationFromaSingleDepthImage.

iccv201129/50骨架跟蹤算法2Real-TimeHumanPoseRecognitioninPartsfromSingleDepthImages.CVPR2011(bestpaper)Fromansingleinputdepthimage,aper-pixelbodypartdistributionisinferred.Localmodesofthissignalareestimatedforthe3Dlocationsofbodyjoints.30/50人手跟蹤算法（測(cè)試）31/50跟蹤算法—應(yīng)用游戲（Natal宣傳視頻）Avatar輔助外科手術(shù)、醫(yī)學(xué)教育監(jiān)控虛擬試衣（1,2）、虛擬試鞋機(jī)器人控制（1，2）、自動(dòng)購(gòu)物車(chē)手勢(shì)交互（1，2）視點(diǎn)相關(guān)的三維顯示、全息桌面將舞蹈轉(zhuǎn)換為音樂(lè)……32/50我們的工作如何利用深度相機(jī)獲得的低分辨率、高噪聲、高幀率的數(shù)據(jù)，生成較高質(zhì)量的三維人體模型3DBodyScanningwithHairstyleusingoneTime-of-FlightCamera.CASA2011(CAVW)Scanning3DFullHumanBodiesusingKinects.IEEEVirtualReality2012(TVCG)/50HairdatacapturedbyTOFcamerapicturestakenwithsimilarviewsoneframeofrawTOFdata34/50NoisecharacterofTOFdatarandomnoisesystematicbiasgroundtruthmeasureddata35/50Howtoreducethenoisereducerandomnoise?reducesystematicbias?oneframeofrawTOFdatabytemporalaverageusinganotherview36/50EnergyfunctiontorefineaveragemeshesAverageMeshesEnergyfunctiontobeminimized:37/5038/50Rawdatagotfromtime-of-flightcameraReconstructedmesh39/50Genereratehairfibers40/50ScanstaticobjectswithnormalmaterialmodelgotbyourmethodcolorImagerawTOFdatalaserscanmodelgotby[Cui,etal,CVPR2010]41/50ScanfullbodywithoneKinect42/50Scanbodyatcloserdistanceupmiddown43/50Interferenceproblemwithoutinterferencewithinterference44/50Thesetupofoursystem45/50Rigidalignment46/50Resultsafterglobalnon-rigidalignment47/50