Opened 6 years ago

Closed 6 years ago

#1708 closed defect (fixed)

Rasql_Round float number should have a consistent length for fractional part

Reported by: Bang Pham Huu Owned by: bbell
Priority: major Milestone: 9.6
Component: rasql Version: development
Keywords: Cc: Dimitar Misev
Complexity: Medium

Description

This query:

select stddev_samp(<[0:5] 28,29,30,31,32,33>)
Result element 1: 1.87083 (5 digits after .)
number before round: 1.87082869339

and this query:

rasql -q 'select arctan(-10294012870517d)' --out string
Result element 1: -1.5708 (4 digits after .)
number before round: -1.57079632679

it should have either 4 or 5 digits length for fractional part, not both.

Change History (4)

comment:1 by Dimitar Misev, 6 years ago

We could add an option to rasql to control the output formatting.

comment:2 by bbell, 6 years ago

First we should find out why it even bothers to round in the first place. Last I checked, we removed the base precision enforcement in csv/json output; perhaps it is still present in our string output?

comment:3 by Dimitar Misev, 6 years ago

This is not in csv, it's in rasql.cc, which prints scalar results.
See browser:applications/rasql/rasql.cc#L541, for float/double it just does cout << scalar. We should make it consistent with the CSV/JSON output?

comment:4 by Bang Pham Huu, 6 years ago

Resolution: fixed
Status: newclosed
Note: See TracTickets for help on using tickets.