Occassional error on rasimport

[James Passmore]

When running a batch import using python I get occasional errors like:

ERROR - RasdamanHelper2::updateImage, l. 685: image update failed: RasManager Error: Write transaction in progress, please retry again later.

ERROR - rimport::main, l. 1371: RasManager Error: Write transaction in progress, please retry again later.

The batch process is using python subprocess like…

p3 = subprocess.Popen(rasimport,shell=True,stdout=subprocess.PIPE,stderr=subprocess.PIPE)
p3.wait()

Where rasimport is something like:

"time rasimport -d " + str(tile) + "/" +  str(subtile) + " -coll pga_cir -t RGBImage:RGBSet -s jpg -oid 6145"

It appears that rasimport is reporting it is finished, when in fact the processing is not complete.

[Dimitar Misev]

It could as well be that some error during the import happened which didn't clear the transaction properly. You could try restarting the server and retrying the update? Ultimately for best stability and control I'd use rasql rather than rasimport.

[James Passmore]

The problem with retrying the update is that this batch import ran for about 5 days and ingested over 1Tb of data.

[Dimitar Misev]

Some error happened during the import. Did you check the logs of rasdaman and return code of rasimport?

[James Passmore]

I tried to check the logs (they are very big), but didn't spot anything unusual though it is possible I missed something; what string should I search for?

[Dimitar Misev]

"Error", but most likely there was some segfault or error in postgres which is not even logged, so it will be hard to figure out. I assume rasimport is not prepared for handling such unexpected issues and then ends up with inconsistent transaction state.

[James Passmore]

Replying to dmisev:

"Error", but most likely there was some segfault or error in postgres which is not even logged, so it will be hard to figure out. I assume rasimport is not prepared for handling such unexpected issues and then ends up with inconsistent transaction state.

I tried "Error" but didn't find anything

[Marcus Sen]

Tested with v9.0.0 and v9.0.1 and same problem remains although intermittent and not repeatable when running same set of loading commands.

[Marcus Sen]

I've been running imports with v9.0.3 and v9.0.4 for a while now and although the intermittent nature of the problem makes it difficult to be certain I think it has been long enough to decide that it doesn't seem to be repeating. Will close for now and re-open if it re-occurs.