Opened 6 years ago

Closed 6 years ago

#1860 closed enhancement (fixed)

WCST_Import no analyze file if it was added to trackfile .resume.json

Reported by: Bang Pham Huu Owned by: Bang Pham Huu
Priority: major Milestone: 9.7
Component: wcst_import Version: development
Keywords: Cc: Dimitar Misev, Vlad Merticariu
Complexity: Medium

Description

WCST_Import analyzes input files to create slice to import. However, if the file was already imported before and added to the trackfile .resume.json, this file will not be imported and the time to process is useless (especially when importing large of files and one wants to run new files to import and old imported files still being analyzed by image drivers).

Also, if "mock": true, then input files shouldn't be added to trackfile .resume.json as they are not imported (dry test).

Change History (2)

comment:1 by Bang Pham Huu, 6 years ago

This problem when importing data via NFS is clearly when files needed to copy to local machine then it analyzes for just gdalinfo and then it checks it is in .resume file already (waisted!!!).

Another problem is for NFS, it can take a really long time to fetch a file and it throws exception, but later try again with gdalinfo it can show the result. So, in wcst_import/util/gdal_util.py, this one can wait for 10 seconds and retry before throwing an exception.

            self.gdal_dataset = gdal.Open(str(self.gdal_file_path).encode('utf8'))
            if self.gdal_dataset is None:
                raise RuntimeException("The file at path " + gdal_file_path + " is not a valid GDAL decodable file.")
Version 0, edited 6 years ago by Bang Pham Huu (next)

comment:2 by Bang Pham Huu, 6 years ago

Resolution: fixed
Status: newclosed
Note: See TracTickets for help on using tickets.