Opened 11 years ago

Closed 11 years ago

#333 closed defect (fixed)

2Gb file limit

Reported by: Heinrich Stamerjohanns Owned by: Heinrich Stamerjohanns
Priority: minor Milestone: 9.0
Component: rasserver Version: 8.4
Keywords: Cc:
Complexity: Medium

Description (last modified by Heinrich Stamerjohanns)

There seems to be a file limit of 2 Gb for files to import.

bash-4.1$ rasql —user rasadmin —passwd rasadmin —mdddomain '[0:819,0:819,0:819]' —mddtype FloatCube -f float_d3s820.dat -q 'insert into fs3s1000 values $1'
rasql: rasdaman query tool v1.0, rasdaman v8.4.0rc.1 — generated on 28.02.2013 15:10:32.
opening database RASBASE at localhost:7001...ok
fetching type information for FloatCube from database, using readonly transaction…ok
reading file float_d3s820.dat…terminate called after throwing an instance of 'RasqlError'
Aborted

Data below 2Gb (0:799, 0:799, 0:799] is imported ok.
This limit is not dependent on memory size, as the tested machine has 16 Gb of Ram.

Change History (11)

in reply to:  description comment:1 by Heinrich Stamerjohanns, 11 years ago

Description: modified (diff)
Priority: major → minor

Replying to hstamerjohanns:

There seems to be a file limit of 2 Gb for files to import.

bash-4.1$ rasql —user rasadmin —passwd rasadmin —mdddomain '[0:819,0:819,0:819]' —mddtype FloatCube -f float_d3s820.dat -q 'insert into fs3s1000 values $1'
rasql: rasdaman query tool v1.0, rasdaman v8.4.0rc.1 — generated on 28.02.2013 15:10:32.
opening database RASBASE at localhost:7001...ok
fetching type information for FloatCube from database, using readonly transaction…ok
reading file float_d3s820.dat…terminate called after throwing an instance of 'RasqlError'
Aborted

Data below 2Gb (0:799, 0:799, 0:799] is imported ok.
This limit is dependent on memory size, as the tested machine has
16 Gb of Ram.

comment:2 by Dimitar Misev, 11 years ago

Most likely some int overflow, we just need to find the variable and change it to long.

comment:3 by Dimitar Misev, 11 years ago

Maybe you could compile with —debug-enabled, it will give a better idea of where in the code the error is thrown.

comment:4 by Heinrich Stamerjohanns, 11 years ago

Yes there is an 'int size' in applications/rasql/rasql.cc, investigating…

comment:5 by Heinrich Stamerjohanns, 11 years ago

the cleanest way would be to change r_Range from int to long or better int64_t. This has
storage consequences, so this needs to be discussed and aimed for 9.0.

comment:6 by Dimitar Misev, 11 years ago

Milestone: → 9.0
Owner: changed from Dimitar Misev to Heinrich Stamerjohanns
Status: new → assigned

comment:7 by Heinrich Stamerjohanns, 11 years ago

I have just imported a 4Gb file. The issue has been fixed by the 'Expand r_Range to long long' fix.

comment:8 by Heinrich Stamerjohanns, 11 years ago

Status: assigned → accepted

comment:9 by Heinrich Stamerjohanns, 11 years ago

By the way, this fix only affects 64-bit systems. Due to Unix limitations, 32-bit systems are still limited
to 2 Gb. (but who is still using 32-bit Linux anyway?)

comment:10 by Heinrich Stamerjohanns, 11 years ago

The fix needs further testing, do not use it yet.

comment:11 by Dimitar Misev, 11 years ago

Resolution: → fixed
Status: accepted → closed
Note: See TracTickets for help on using tickets.