Opened 4 years ago

Closed 4 years ago

#2258 closed defect (fixed)

improve netcdf import (analysis) speed

Reported by: Dimitar Misev Owned by: Dimitar Misev
Priority: major Milestone: 10.0
Component: wcst_import Version: 9.8
Keywords: Cc: Bang Pham Huu
Complexity: Medium

Description

Analysing netcdf files can be very slow if expressions are used in the ingredients file, e.g.:

Analyzing file (1/20): processed:2017090100.m01.nc ...
Elapsed time: 13.337 s.

it is much slower (13.3s) than the rasdaman import (1.14s)

[2020-03-14 14:11:35] 1/20 - file 'processed:2017090100.m01.nc' - grid domains [0,0,0:24,0:5,0:420,0:460] of size 71.12 MB; 
Total time to ingest file 1.14 s @ 62.39 MB/s.
Progress: [##----------------------------] 1/20 5.00% 

The problem is that numpy array is converted into python list and then operations like min/max are done on the list. This is slow, such operations should be performed directly on numpy arrays.

Change History (1)

comment:1 by Dimitar Misev, 4 years ago

Resolution: fixed
Status: assignedclosed
Note: See TracTickets for help on using tickets.