數據結構本科教學資料-hdf數據結構的研究

1、hdf數據結構研究1. hdf概念hdf, hierarchical data format,分層數據格式，美國國家高級計算應用中心ncsa (national center for supercomputing applications)為了滿足各領域研究需求而研制 的一種能高效存儲和分發(fā)科學數據的新型數據格式，是一種分層式數據管理結構，被地球觀 測系統(tǒng)數據和信息系統(tǒng)核心系統(tǒng)所選用作為標準數據格式。它的表現(xiàn)形式是一種多目標的文 件格式，目的是為了在分布式環(huán)境中共享科學數據。ndf文件格式的優(yōu)勢在于：可移植性強(獨立于操作平臺)；屬于超文本文件；可以存 儲并處理大數據量；一個文件集可以管理多種

2、類型的數據結構；具有可擴展性。由于hdf的 諸多優(yōu)點，這種格式已經被廣泛用于目前國外各種衛(wèi)星傳感器的標準數據格式。在影像數據 庫多源數據管理屮，hdf格式發(fā)揮了很好的作用，利用hdf數據結構建立遠程圖彖工程，并與 數據庫進行交互；可以進行遠程圖像處理；遠程影像解譯，統(tǒng)計分析；影像運算、信息挖掘、 影像分類；綜合處理影像、矢量、高程數據，三維可視顯示等。2. hdf數據結構hdf文件有六種主要數據類型，如圖：1柵格圖像raster image：數據模式提供一種靈活方式存儲、描述柵格圖像數據，包括 8bit柵格圖像。2. 調色板palette：也叫作彩色查對表，它提供圖像的色譜。3. 科學數據集s

3、cientific data set：用來存儲和描述多維科學數據陳列。4. hdf注釋annotation：是文字串，用來描述hdf文件或hdf數據目標。5. vdata：是一個框架，用于存儲和描述數據表。6. vgroup：是用來把相關數據目標聯(lián)系起來。一個vgroup可以含有其它vgroup,以及數據目標。任一個hdf目標均可以包扌舌進某個vgroup屮。palctlcvdata (table)vgroup(group of hdf data structures)而在hdf5中，建立一個新的hdf文件后，會有5種數據格式顯示:hdf4中只有前三種，但nasa下來的數據再hdfview 里

4、面顯示的是版本4.o group駐 dataset亀 irnayetable芬 datatype 邂 link3. 對 于 nasa 的 merra300.prod.assim.inst6_3d_ana_nv.2010o531.hdf進行的格式研究。其中有基于sd (science data)和gd (vgroup id)兩種方式對hdf文件進行處理,例 如下圖，eosgrid就是一個vgroup,而ps就是一個sd的datafieldo利用sd方式進行讀寫1) sd方式讀収的變量，是整個hdf文件的變量，沒有層次結構的顯示，下面是輸出的和hdfview比對的結果。function hdf_s

5、d_vardir, hdfidvarnames =''hdf_sd_fileinfo, hdfid, nvars, ngatts if (nvars gt 0) then beginvarnames = strarr(nvars)for index 二 ol, nvars - il do begin varid = hdf_sd_sclcct (hdfid, index) hdf_sd_getinfo, varid, name=name hdf sd endaccess, varid varnnmes index = nameendforendifreturn, varname

6、sendidl> pp=hdf_sd_vardir(hdfid)idl> print, ppft merra300ins|6_3cf egsgroi|科丹理 delpa毎03送:heigws毬 tutw qj grid mnbjtew>tnnrieosorid fdrri&rrsion) ：3 ydm eosgrid :dmemion) 昌| h?(dimemwg(dimens?cn>ps delp t u v qv 03 xdim:eosgrid ydim:eosgrid height:eosgrid time:eosgrid xdim ydimheight

7、time2) sd方式讀取的屬性，當變量輸入為空吋，顯示的是整個hdf文件的屬性。下面是輸出 的和hdfview比對的結果。function hdf_sd_attdir, hdf 11), varnameif (n_elements(hdfid) eq 0) then $message, ' hdfid is undefincd'attnames =if (varname eq ' ') then beginhdf_sd_fileinfo, hdfid, nvars, nattsendif else beginindex = hdf_sd_nametoindex

8、(hdfid, varname)varicl = hdf_sd_select (hdf id, index)hdf_sd getinfo, varid, natts=nattsendelseif (natts gt 0) then beginattnames 二 strarr(natts)for index = 0l, natts-1 do beginif (varname eq '') then beginhdf_sd_attrinfo, hdfid, index,name=nameendif else beginhdf_sd_attrinfo, varid, index,n

9、ame=nameendelseattnamesindex = nameendforendifif (varname ne " ) then hdf_sd_endaccess, varidreturn, attnamesendidl> kk二hdf_sd_attdir(hdfid,'')idl> print, kkhdfeosversion structmetadata0 missingvalue conventions title history institutionsource references commont contact ks nstep p

10、top pint ak bk coremetadata 0archivedmetadata. 0i general j attributesnumber of attributes = 19addnamevaluetypearray sizehdfeosversio nhdfeos v2.148-bit character12structmetadata.ogroup=swathstructure.8-bit character32000missing value9.9999999e1432-bit floating-point1 |conven tionscf-1.08-bit charac

11、ter6titlemerra reanalysis. geo8-bit character28historyfile written by cfio8-bit character20 |institutio nglobal modeling and ass.8-bit character94sourceglobal modeling and ass.8-bit character54|refere nceshttp:/gmao.gsfc. nasa.go.8-bit character41commentgeos-5.2.08-bit character10contacthttp:/gmao.g

12、sfc. nasa.gow8-bit character26ks4032-bit integer1 |nstep1576032-bit integer1 |ptop1.032-bit floating-point1pint15039.30332-bit floating-point1 |ak1.0,2.0000002,3.27000.32-bit floating-point73bk0.0. 0.0. 0.0, 0.0, 0.0, 0.0.32-bit floating-point73coremetadata.ogroup = inventorym.8-bit character5859arc

13、hivedmetadata.ogroup = archivedme.8-bit character5733) sd的一些概念(即上例打開文件所用的hdfid)科學數據集即sd (science dataset)的td號和已在hdf文件結構屮定義的地球物理參 數名的關系，然后讀取數據i1df_sd_getdata到內存中，關閉數據文件hdf_sd_endaccess, hdf_endo每打開一個hdf文件將會分配個sdjd,應該是相當于臨時的內存號之類的。(每次的id 值不一樣)idl> hdf id=hdf_sd_start (' merra300. prod, assim. i

14、ns16_3d_ana_nv. 20100531. hdf')% loaded dlm: hdf.idl> print, hdfid393216idl> hdfid=hdf_sd_start (，merra300. prod, assim. inst6_3d_ana_nv. 20100531. hdf')idl> print, hdfid17170433idl> hdfid二hdf sd start mltrra300. prod, assim. inst6 3d ana nv. 20100531. hdf')idl> print, hdf

15、id339476504) sd方式將已有hdf文件的數據寫入單個vgroup的hdf文件。以讀取 merra300. prod, assim. inst6_3d_ana_nv. 20100531. hdf 的 eosgrid (vgroup)下的 ps dataficld 為例：fid=e0s_gd_0pen(，merra300. prod, assim. inst6_3d_ana_nv. 20100531. hdf', /read) grid二 eos_gd_attach(fid,' e0sgrid，)r7 = eos_gd_fieldinfo(grid, * ps，, ra

16、nk, dims, numbertype, dimlist)r6 二 eos_gd_readfield(grid, ps，, data) k二dims pp=fltarr(dims) pp=data hdfid=hdf_sd_start c 20100531. hdf', /create) varid=hdf_sd_create(hdfid,' psd', k, /dfnt_fl0at32) dimid=hdf_sd dimgetid(varid, 0)hdf sd dimset, di mid, name 二'xdim' dimid=hdf_sd_di

17、mgetid(varid,1) hdf_sd_dimset, dimid, name=，ydim" dimid=hdf_sd_dimgetid(varid, 2) hdf sd dimset, dimid, name=，time， hdf_sd_adddata, varid, datahdf_sd_attrset, varid,' qym",' idl examples' hdf_sd_attrset, hdfid,' creation_data，, systime() hdf_sd_endaccess, varidhdf sd end,hd

18、fid此程序是將ps中的數據寫入到新建的20100531. hdf文件的psd dataficld中,在hdfvicw中的效果:愆 tableview ps /eosgrid/data fields/ cadocuments and settingsxadrr "me |沏|i 4 i吩|聊|table圃炸o°4 gtableview psd 廠 cadocuments and settingsadministratort20100531 t0123401234067433.3767433.3767433.3767433.3767433.37067433 3767433 3

19、767433 3767433.3767433.37166409.37664253766441 3766473.3766489.37166409.37664253766441.3766473.3766489.37265241.36765273.36765289 36765321.36765337.367265241.36765273 36765289 36765321.36765337.367364441 36764441 36764441 36764457 36764457.367364441.36764441.36764441.36764457.36764457.367464393.3676

20、43933676439336764393.36764409.367464393.36764393 36764393.36764393.36764409.367564841.36764857 36764905.36764937.36765001.367564841.36764857 36764905.36764937.36765001.367665369.36765401.3676543336765497.36765577.37665369.36765401.36765433.36765497.36765577.37766521.3766569.3766601.3766681.3766777.3

21、7766521.3766569.3766601.3766681.3766777.37868297.3768505.3768665.3768825.3768905.37868297.37685053768665.3768825.3768905.37969257.376986537706653772329.3773773.37969257.37698653770665.3772329.3773773.371073073.3773097.3774313.3776931.3779853.371073073.3773097.3774313.3776931.3779853.371190393.378968

22、937895293790985.3792521.371190393.3789689 3789529 3790985.3792521.371298681.3798681 3798681 3798681.3798713.371298681.3798681 3798681.3798681.3798713.371398713.3798713 3798745.3798777.3798777.371398713.3798713 3798745.3798777.3798777.371498777 3798777 3798777 3798809 3798809.371498777.3798777.379877

23、7.3798809.3798809.371598745.3798777.3798777.3798809.3798809.371598745.3798777.3798777.3798809.3798809.371698713.3798745.3798745.3798745.3798745.371698713.3798745 3798745.3798745.3798745.371798649.379864937986493798649.3798681.371798649.3798649.3798649.3798649.3798681.371898553.3798553.3798553.379855

24、3.3798553.371898553.3798553.3798553.3798553.3798553.371998457.3798457.3798457.3798457.3798457.371998457.37984573798457.3798457.3798457.372098393.3798393.3798361.3798361.3798361.372098393.3798393.3798361.3798361.3798361.372198329.37983293798329 3798297.3798297.372198329.37983293798329.3798297.3798297

25、.372298297.37982973798297 3798265.3798265.372298297 3798297 3798297 3798265.3798265.372398297.3798297 3798265 3798265.3798233.372398297.37982973798265.3798265.3798233.372498265.3798233.3798233 3798201.3798201.372498265.3798233.3798233.3798201.3798201.372598233 3798233 3798201 3798201 3798201.3725982

26、33.37982333798201.3798201.3798201.372698265.379823337982333798233.3798201.372698265.3798233.3798233.3798233.3798201.372798265.3798265 37982333798233.3798201.372798265.3798265.3798233.3798233.3798201.372898297.3798265.37982333798233.3798201.372898297.3798265.3798233.3798233.3798201.372998265.3798265.

27、3798233.3798201.3798169.372998265.3798265.3798233.3798201.3798169.373098265.3798233.3798201.3798169.3798137.373098265.37982333798201.3798169.3798137.37訕 imageview - ps - /eosorid/data fields/ c:documents and setting$administratormmerra300.prod.as$im ins16_3d_ana.timage炸10廠4r-注意：1.在hdfview中口己添加datafi

28、eld在idl程序中識別不了。 2. hdfview屮不能刪去元素。q2owouva 啲 se.pcudooevalr山左國日a633s 蝴r-ji151關左isooe0礎sp o)匚 iq ®a以圖形方式打開:qeosceeoa利用gd方式進行讀寫1 )多個vgroup(rtl gridid標識)打開方法的探究1. 讀取vgroup相關信息a. 相關信息：result = eos_gd_gridinfo(g刃d/d, xdimsize, ydunsize, upleft, lowrig/it) this function returns the number of rows, co

29、lumns and the location, in meters, of the upper left and lower right corners of the grid image.b. vgroup 數 fl: result = eos_gd_inqgrid( filename, gridlist , length=variable)c. 直接從 filename 和 gridname 讀取相關信息：result = eos_gd_query( filename, gridnanie, info)2. 讀取某個vgroup下某個屬性的相關東西a. 值：result = eos_gd_

30、readattr(ghd/£>, attmatne, dcitbuf)值將 datbuf 中，returnssucceed(o)訐 successful and fail(-l) otherwise.b .相關信息：result = eos_gd_attrinfo(g/7d/d, attrname, numbertype, count) attribute name (string). a named variable that will contain the number type (long) of an attribute. returns succeed (0) if

31、 successful and fail (-1) otherwise.c.屬性數目：result = eos_gd_inqattrs( gridld, attrlist , lengthw加曲5 )3. 讀取某個vgixnip下dataset的相關信息a. 相關信息：result = eos_gd_fieldinfo(g/7w/£), fieldname, rank, dims, numbertype, dbnlist) this function retrieves information on a specific data field.b. dataset 數 h : res

32、ult = eos_gd_inqfields(g/7t/£>, fieldlist, rank, numbertype)c. 值：result = eos_gd_readfield( gridid, fieldname, buffer eqge=array ,start way , str ide way)4. 下血以打開 merra300.prod.assim.inst6_3d_ana_nv.20100531 .hdf 分析相關 datafields 和attributes 為例pro analysishdf；根據文件名和vgroup名獲取信息rl 二 eos_gd_quer

33、y(' merra300. prod, assim. inst6_3d_ana_nv. 20100531. hdf , $'e0sgrid，, info)print,' info二'，info；根據文件名獲取vgroup個數信息(好像有些識別不了)r2 二 eos_gd_inqgridmerra300. prod, assiin. inst6_3d_ana_nv. 20100531. hdf , gridlist , length=length)print,gridlist, length；打開一個確定的vgroup;open the filefid=e0s_g

34、d_0pen(，merra300. prod, assim. inst6_3d_ana_nv. 20100531. hdf', /read);get the gridgrid二 eos_gd_attach(fid,' eosgrii)')；讀取屬性數目r3 = eos_gd_inqattrs( grid, attrlist, length=lengthl)print, attri ist, lengthl;讀取_fv_u屬性相關信息和值rd = e0s_gd_attrinf0(grid, * _fv_u，, numbertype, count)r5 = eos_gd_r

35、eadattr(grid,'_fv_u', datbuf)print, numbertype, count, datbuf;讀取datafield數目r6 = eos_gd_inqfields(grid, fieldlist, rank, numbertype) print,fieldlist, rank, numbertype;讀取名為ps的datafield值和相關信息r7 = eos_gd_fieldinfo(grid,' ps', rank, dims, numbertype, dimlist) r6 = eos gd readfield(grid,&#

36、39; ps', data) print, rank, dims, numbertype, diml istend運行結果：info= missing_value time, height472ps, delp, t, u, v, qv, 03, xdim, ydimjleight, time344444141111lr"lrr0000055666600.000000000.000000000.000000000.000000000.000000000.000000000.000000000.000000000.000000000.000000000.000000000.00

37、0000000.000000000-11.8000000e-1. 8000000e+00890000000.54036112-1-111-120eosgrid7missingvalue1354 1. 00000c+0150. 0000000. 0000000. 000000ps, delp, t, u, v, qv, 03, xdim, ydim, height, time3444444111155555556666354036145xdim, ydim, time其中綠色字體標識的是：直接從filename和gridname讀収相關信息：result = eos_

38、gd_query( filename, gridname, info)中 info 的值，參數解釋如下：fieldidl data typedescriptionattributesstring arrayarray of attribute namesdimension_namesstring arraynames of dimensionsdimension_sizeslong arraysizes of dimensionsfield_namesstring arraynames of fieldsfield_rankslong arrayranks (dimensions) of fi

39、eldsfield_t ypeslong arrayidl types of fieldsgctp_projectionlonggctp projection codegctp_projection_parmdouble arraygctp projection parametersgctp_spheroidlonggctp spheroid codegctp_zonelonggctp zone code (for utm projecti on)image_lowrightdouble2location of lower right corner (meters)image_upleftdo

40、uble2location of upper left corner (meters)image_x_dimlongnumber of columns in grid imageimage_y_dimlongnumber of rows in grid imagenum_attributeslongnumber of attributesnum_dimslongnumber of dimensionsnum_idx_mapslongnumber of indexed dimension mapping entriesnum_mapslongnumber of dimension mapping

41、 entriesnum_fieldslongnumber of fieldsnum_geo_fieldslongnumber of geolocation field entriesorigin_codelongorigin codepix_reg_codelongpixel registration code附：eosgrid在hdfvicw中結構圖coo>0®>0 ch 山 qos2nq_m/p_o aell® gahs2)多個vgroup下的添加hdf文件現(xiàn)在有問題;f ile=f ilepath (j ctscan. dat', subdir=，

42、examp les/data");openr, lun, file, /get lun;data=bytarr(256, 256);readu, lun, data;free_lun, lunfid = e0s_gd_0pen(gdq. hdf，/create)help, fidupleftpt=fltarr(2)lowrightpt=fltarr(2)upleftpt0=10584. 50041dupleftpt1=3322395. 95445dlowrightpt0=813931.10959dlowrightptl=214162. 53278dxdimsize=120ydimsi

43、ze=200grid = eos_gd_create(fid,' who', xdimsize, ydimsize, upleftpt, lowrightpt)help, gridstatus 二 eos_gd_detach(grid)gridl = eos_gd_attach(fid,'who')si = eos_gd_defdim(gridl, "ydint, 2)s2 二 eos_gd_defdim(gridl, xdin)，2)print, si, s2f32 = 3. 14s4 = eos_gd_writeattr(gridl, "

44、scalarfloat", f32)help,s4s3 = eos_gd_deffield(gridl, "temperature", "ydim, xdim，z, 5)help,s3;temperatureindgen (2,2);s2 二 eos_gd_writefield(grid, temperature, temperature)r6=eos_gd_detach (gridl)help,r6r7二eos_gd _close(fid)help,r7參考網址：/fonims/index.phphttd：/hdfco

45、sorg/fonjms/aichivc/indcxdhp?t339html問題：1.定義的vgroup顯示成功了，但是在hdfview屮看不了()2. eos_gd_deffield總是定義不成功()3.3寫入和讀出無格式二進制文件，并利用idl的一個demo實現(xiàn)3維可視化1注意這里是用writeu進行寫入，寫入的是二進制，如果用printf寫入，那就是ascii碼。 pro becomedathdf id=hdf_ sd start c merra300. prod, assim. inst6 3d ana nv. 20100531. hdf')index = hdf_sd_name

46、toindex(hdfid, ' ps')if (index it 0) then $message, ' sds was not found: ' + varnameselect and read the sosvarid = hdf_sd_select(hdfid, index)hdf_sd_getdata, varid, datahdf_sd_endaccess, varidhelp,data;data=dist(10, 100);data=lindgen(10, 100)openw, lun,' today. dat/ , /get_lunwri

47、teu, lun, datafree_lun, lunend2注意writeu對應readu, prints應readf,分別是二進制和ascii的寫入和讀取 pro testdata_file = filepathc today. datj, $ subdirectory二'examples，, ' data');import elevation datadem data = fltarr(540, 361, 4)openr, unit, data_filc, /get_lunhelp,unitreadu, unit, dem_datafree lun, unit;i

48、n crease size of data for visibi li ty.;dem_data = congrid(dcm_data, 12& 12& /interp) ;loadct,1;tvscl, dem_data;dem data=median(dem data, 4);shade surf, dem datahelp,dem data;z=dem_data 0:255, 0:255, 1;shade_surf, zprint, dem data0:3, 0:3, 0:3;print, dem dataend注意：1. 由于writeu是以byte為單位寫入，所以最多只能8個字符，也就是說如果出現(xiàn) 6334782.234將會出現(xiàn)截斷：6334782.,所以最好在writeu之前進行數據的處理。2. 如果要保留精度的話也對以通過printf和readf方式來實現(xiàn)，在寫入和讀出時定義for