Opened 6 years ago

Closed 6 years ago

#1734 closed defect (fixed)

WCST_Import error from Numpy with metadata "auto"

Reported by: Bang Pham Huu Owned by: Bang Pham Huu
Priority: major Milestone: 9.6
Component: wcst_import Version: development
Keywords: Cc: Dimitar Misev, Vlad Merticariu
Complexity: Medium

Description

From PML's netCDF ingredient with setting metadata: "auto"

      "coverage": {
        "crs": "EPSG/0/4326@OGC/0/AnsiDate",
        "metadata": {
          "type": "xml",
          "global": "auto"
	},

which throws error from WCST_Import.sh

 File "/home/rasdaman/rasdaman_community/rasdaman/applications/wcst_import/wcst_import.py", line 130, in main
    reg.run_recipe(session)
  File "/home/rasdaman/rasdaman_community/rasdaman/applications/wcst_import/master/recipe/recipe_registry.py", line 72, in run_recipe
    recipe.describe()
  File "/home/rasdaman/rasdaman_community/rasdaman/applications/wcst_import/recipes/general_coverage/recipe.py", line 140, in describe
    importer = self._get_importer()
  File "/home/rasdaman/rasdaman_community/rasdaman/applications/wcst_import/recipes/general_coverage/recipe.py", line 461, in _get_importer
    self.importer = Importer(self._get_coverage(), self.options['wms_import'],
  File "/home/rasdaman/rasdaman_community/rasdaman/applications/wcst_import/recipes/general_coverage/recipe.py", line 381, in _get_coverage
    coverage = self._get_netcdf_coverage(recipe_type)
  File "/home/rasdaman/rasdaman_community/rasdaman/applications/wcst_import/recipes/general_coverage/recipe.py", line 423, in _get_netcdf_coverage
    self.options['tiling'], self._netcdf_global_metadata_fields(),
  File "/home/rasdaman/rasdaman_community/rasdaman/applications/wcst_import/recipes/general_coverage/recipe.py", line 282, in _netcdf_global_metadata_fields
    metadata_dict = escape_metadata_dict(self.__parse_netcdf_global_metadata())
  File "/home/rasdaman/rasdaman_community/rasdaman/applications/wcst_import/util/string_util.py", line 56, in escape_metadata_dict
    metadata_dict[key] = escape(value)
  File "/usr/lib64/python2.7/xml/sax/saxutils.py", line 32, in escape
    data = data.replace("&", "&")
AttributeError: 'numpy.float32' object has no attribute 'replace'

and global attributes of this netCDF file is:

// global attributes:
		:Conventions = "CF-1.6" ;
		:Metadata_Conventions = "Unidata Dataset Discovery v1.0" ;
		:cdm_data_type = "Grid" ;
		:comment = "See summary attribute" ;
		:creator_email = "rsghelp@pml.ac.uk" ;
		:creator_name = "Plymouth Marine Laboratory" ;
		:creator_url = "http://rsg.pml.ac.uk" ;
		:geospatial_lat_max = 90.f ;
		:geospatial_lat_min = -90.f ;
		:geospatial_lat_resolution = ".04166666666666666666" ;
		:geospatial_lat_units = "decimal degrees north" ;
		:geospatial_lon_max = 180.f ;
		:geospatial_lon_min = -180.f ;
		:geospatial_lon_resolution = ".04166666666666666666" ;
		:geospatial_lon_units = "decimal degrees east" ;
		:geospatial_vertical_max = 0.f ;
		:geospatial_vertical_min = 0.f ;
		:institution = "Plymouth Marine Laboratory" ;
		:keywords = "satellite,observation,ocean,ocean colour" ;
		:keywords_vocabulary = "none" ;
		:license = "ESA CCI Data Policy: free and open access" ;
		:naming_authority = "uk.ac.pml" ;
		:number_of_optical_water_types = "14" ;
		:platform = "Orbview-2/SeaStar,Aqua,Envisat" ;
		:processing_level = "Level-3" ;
		:product_version = "2.0" ;
		:project = "Climate Change Initiative - European Space Agency" ;
		:references = "http://www.esa-oceancolour-cci.org/" ;
		:sensor = "SeaWiFS,MODIS,MERIS" ;
		:source = "SeaWiFS R2010.0, MODISA R2013.1, MERIS L1B 3rd reprocessing + OCL corrections" ;
		:spatial_resolution = "4km nominal at equator" ;
		:standard_name_vocabulary = "NetCDF Climate and Forecast (CF) Metadata Conventions Version 1.6" ;
		:title = "ESA CCI Ocean Colour Product" ;
		:number_of_files_composited = 31 ;
		:summary = "Data products generated by the Ocean Colour component of the European Space Agency\'s Climate Change Initiative project.  These files are monthly composites of merged sensor (MERIS, MODIS, SeaWiFS) products, where MODIS and MERIS wereband-shifted and bias-corrected to SeaWiFS bands and values.  MODIS and SeaWiFS RRS products were derived from 4km resolution level-3 binned standard NASA products; MERIS was derived from a 4km level-3 binning of the output of the HYGEOS POLYMER algorithm.  Derived products, such as Kd and chlorophyll were computed with the standard SeaDAS (SeaDAS v6.4-OCCCI) algorithms.  QAA IOPs were derived using the standard SeaDAS (SeaDAS v6.4-OCCCI) algorithm, with a modified backscatter table to match that used in the bandshifting.  Uncertainty estimates were added using Tim Moore\'s fuzzy water classification and uncertainty estimation algorithm." ;
		:tracking_id = "4d11baa9-933d-4484-822e-c2d92c826653" ;
		:id = "ESACCI-OC-L3S-OC_PRODUCTS-MERGED-1M_MONTHLY_4km_GEO_PML_OC4v6_QAA-200007-fv2.0.nc" ;
		:creation_date = "Thu Apr 23 15:38:25 2015" ;
		:date_created = "Thu Apr 23 15:38:25 2015" ;
		:time_coverage_resolution = "P1M" ;
		:time_coverage_duration = "P1M" ;
		:start_date = "01-JUL-2000 00:00:00.000000" ;
		:stop_date = "31-JUL-2000 23:59:00.000000" ;
		:time_coverage_start = "200007010000Z" ;
		:time_coverage_end = "200007312359Z" ;
		:history = "Fri Apr 24 18:46:45 2015: ncks --variable lat,lon --exclude -C --overwrite --chunk_dimension lat,256 --chunk_dimension lon,256 /data/caspian2/scratch/cci/v2.0-production/stage11-composites/geographic/monthly/chlor_a/2000/ESACCI-OC-L3S-CHLOR_A-MERGED-1M_MONTHLY_4km_GEO_PML_OC4v6-200007-fv2.0.nc ESACCI-OC-L3S-CHLOR_A-MERGED-1M_MONTHLY_4km_GEO_PML_OC4v6-200007-fv2.0.nc\n",
			"Source data were: ESACCI-OC-L3S-OC_PRODUCTS-MERGED-1D_DAILY_4km_GEO_PML_OC4v6_QAA-20000701-fv2.0.nc, ESACCI-OC-L3S-OC_PRODUCTS-MERGED-1D_DAILY_4km_GEO_PML_OC4v6_QAA-20000702-fv2.0.nc, ESACCI-OC-L3S-OC_PRODUCTS-MERGED-1D_DAILY_4km_GEO_PML_OC4v6_QAA-20000703-fv2.0.nc, ESACCI-OC-L3S-OC_PRODUCTS-MERGED-1D_DAILY_4km_GEO_PML_OC4v6_QAA-20000704-fv2.0.nc, ESACCI-OC-L3S-OC_PRODUCTS-MERGED-1D_DAILY_4km_GEO_PML_OC4v6_QAA-20000705-fv2.0.nc, ESACCI-OC-L3S-OC_PRODUCTS-MERGED-1D_DAILY_4km_GEO_PML_OC4v6_QAA-20000706-fv2.0.nc, ESACCI-OC-L3S-OC_PRODUCTS-MERGED-1D_DAILY_4km_GEO_PML_OC4v6_QAA-20000707-fv2.0.nc, ESACCI-OC-L3S-OC_PRODUCTS-MERGED-1D_DAILY_4km_GEO_PML_OC4v6_QAA-20000708-fv2.0.nc, ESACCI-OC-L3S-OC_PRODUCTS-MERGED-1D_DAILY_4km_GEO_PML_OC4v6_QAA-20000709-fv2.0.nc, ESACCI-OC-L3S-OC_PRODUCTS-MERGED-1D_DAILY_4km_GEO_PML_OC4v6_QAA-20000710-fv2.0.nc, ESACCI-OC-L3S-OC_PRODUCTS-MERGED-1D_DAILY_4km_GEO_PML_OC4v6_QAA-20000711-fv2.0.nc, ESACCI-OC-L3S-OC_PRODUCTS-MERGED-1D_DAILY_4km_GEO_PML_OC4v6_QAA-20000712-fv2.0.nc, ESACCI-OC-L3S-OC_PRODUCTS-MERGED-1D_DAILY_4km_GEO_PML_OC4v6_QAA-20000713-fv2.0.nc, ESACCI-OC-L3S-OC_PRODUCTS-MERGED-1D_DAILY_4km_GEO_PML_OC4v6_QAA-20000714-fv2.0.nc, ESACCI-OC-L3S-OC_PRODUCTS-MERGED-1D_DAILY_4km_GEO_PML_OC4v6_QAA-20000715-fv2.0.nc, ESACCI-OC-L3S-OC_PRODUCTS-MERGED-1D_DAILY_4km_GEO_PML_OC4v6_QAA-20000716-fv2.0.nc, ESACCI-OC-L3S-OC_PRODUCTS-MERGED-1D_DAILY_4km_GEO_PML_OC4v6_QAA-20000717-fv2.0.nc, ESACCI-OC-L3S-OC_PRODUCTS-MERGED-1D_DAILY_4km_GEO_PML_OC4v6_QAA-20000718-fv2.0.nc, ESACCI-OC-L3S-OC_PRODUCTS-MERGED-1D_DAILY_4km_GEO_PML_OC4v6_QAA-20000719-fv2.0.nc, ESACCI-OC-L3S-OC_PRODUCTS-MERGED-1D_DAILY_4km_GEO_PML_OC4v6_QAA-20000720-fv2.0.nc, ESACCI-OC-L3S-OC_PRODUCTS-MERGED-1D_DAILY_4km_GEO_PML_OC4v6_QAA-20000721-fv2.0.nc, ESACCI-OC-L3S-OC_PRODUCTS-MERGED-1D_DAILY_4km_GEO_PML_OC4v6_QAA-20000722-fv2.0.nc, ESACCI-OC-L3S-OC_PRODUCTS-MERGED-1D_DAILY_4km_GEO_PML_OC4v6_QAA-20000723-fv2.0.nc, ESACCI-OC-L3S-OC_PRODUCTS-MERGED-1D_DAILY_4km_GEO_PML_OC4v6_QAA-20000724-fv2.0.nc, ESACCI-OC-L3S-OC_PRODUCTS-MERGED-1D_DAILY_4km_GEO_PML_OC4v6_QAA-20000725-fv2.0.nc, ESACCI-OC-L3S-OC_PRODUCTS-MERGED-1D_DAILY_4km_GEO_PML_OC4v6_QAA-20000726-fv2.0.nc, ESACCI-OC-L3S-OC_PRODUCTS-MERGED-1D_DAILY_4km_GEO_PML_OC4v6_QAA-20000727-fv2.0.nc, ESACCI-OC-L3S-OC_PRODUCTS-MERGED-1D_DAILY_4km_GEO_PML_OC4v6_QAA-20000728-fv2.0.nc, ESACCI-OC-L3S-OC_PRODUCTS-MERGED-1D_DAILY_4km_GEO_PML_OC4v6_QAA-20000729-fv2.0.nc, ESACCI-OC-L3S-OC_PRODUCTS-MERGED-1D_DAILY_4km_GEO_PML_OC4v6_QAA-20000730-fv2.0.nc, ESACCI-OC-L3S-OC_PRODUCTS-MERGED-1D_DAILY_4km_GEO_PML_OC4v6_QAA-20000731-fv2.0.nc; netcdf_compositor_cci composites  Rrs_412, Rrs_443, Rrs_490, Rrs_510, Rrs_555, Rrs_670, atot_412, atot_443, atot_490, atot_510, atot_555, atot_670, chlor_a, aph_412, aph_443, aph_490, aph_510, aph_555, aph_670, adg_412, adg_443, adg_490, adg_510, adg_555, adg_670, bbp_412, bbp_443, bbp_490, bbp_510, bbp_555, bbp_670, kd_490, water_class1, water_class2, water_class3, water_class4, water_class5, water_class6, water_class7, water_class8, water_class9, water_class10, water_class11, water_class12, water_class13, water_class14, Rrs_412_bias, Rrs_443_bias, Rrs_490_bias, Rrs_510_bias, Rrs_555_bias, Rrs_670_bias, aph_412_bias, aph_443_bias, aph_490_bias, aph_510_bias, aph_555_bias, aph_670_bias, adg_412_bias, adg_443_bias, adg_490_bias, adg_510_bias, adg_555_bias, adg_670_bias, kd_490_bias, chlor_a_log10_bias with --mean,  chlor_a_log10_rmsd, Rrs_412_rmsd, Rrs_443_rmsd, Rrs_490_rmsd, Rrs_510_rmsd, Rrs_555_rmsd, Rrs_670_rmsd, aph_412_rmsd, aph_443_rmsd, aph_490_rmsd, aph_510_rmsd, aph_555_rmsd, aph_670_rmsd, adg_412_rmsd, adg_443_rmsd, adg_490_rmsd, adg_510_rmsd, adg_555_rmsd, adg_670_rmsd, kd_490_rmsd with --root-mean-square, and  total_nobs, MODISA_nobs, MERIS_nobs, SeaWiFS_nobs - with --total\n",
			"1429804840 Subsetted from standardised_geo/ESACCI-OC-L3S-OC_PRODUCTS-MERGED-1M_MONTHLY_4km_GEO_PML_OC4v6_QAA-200007-fv2.0.nc to only include variables MERIS_nobs_sum,MODISA_nobs_sum,SeaWiFS_nobs_sum,chlor_a,chlor_a_log10_bias,chlor_a_log10_rmsd,crs,lat,lon,time,total_nobs_sum" ;
		:netcdf_file_type = "NETCDF4_CLASSIC" ;
		:NCO = "4.3.0" ;

Reference: http://earthserver.eu/trac2/ticket/259

Change History (1)

comment:1 by Bang Pham Huu, 6 years ago

Resolution: fixed
Status: newclosed
Note: See TracTickets for help on using tickets.