Use floor when converting degrees to integer

[drinckes]

The encoding and decoding implementations use integers rather than floats to avoid the build up of precision losses through repeated division or multiplication, and to improve the computation speed.

The initial conversion of degrees to integer for latitude is:

latVal := int64(math.Round(latDegrees * finalLatPrecision))
latVal += latMax * finalLatPrecision

(finalLatPrecision is 2.5e7, and is the number of divisions in one degree of latitude that a maximum length OLC code can represent.)

The conversion results in an integer value that will be used to compute the code. Conversion from the integer to the code and back is stable (I mean there are no issues with precision loss).

Ideally we would have an integer that when converted back to a range contains the original latitude value. By range, I mean using the conversion value and adding 1/finalLatPrecision.

Unfortunately using math.Round() means that the resulting integer can be 1 higher than it should be, and after converting back to a range it does not contain the original latitude.

(Everything also applies to longitude.)

The impact is small because we are talking about being out by 1/2.5e7 or 1/8.192e6 degrees, or about half a centimetre, but this won't be helping the test cases.

[drinckes]

I wrote (in golang) some code that for a billion latitudes, computed the integer value using both round and floor methods, and then converted the integer value back and checked if the original latitude is contained by that range.

• Using rounding, it was contained 499950126 times (~50%, which makes sense if you think what rounding does)
• Using floor, it was contained 999998537 times. (My guess is the 1500 that weren't are down to floating point precision issues.)

So I think the implementations need to be changed to use floor, not round.

[drinckes]

Also - the Go implementation has untyped values for many of the constants and they should change to be explicitly typed.

[drinckes]

Just in case you wanted more bad rounding news, Python uses yet another approach to rounding called "round ties to even", which is, umm, unexpected:

>>> round(3.5)
4
>>> round(4.5)
4

[drinckes]

Based on tests it appears that it should actually be a floor operation, not round, because the value used for encoding should never be higher than the passed latitude or longitude, or there is the risk that the decoded value will not contain the original coordinates.

[drinckes]

Ah ha. For latitude, we can use floor on the initial conversion of val * 2.5e7, but we shouldn't for longitude until AFTER we've normalised the value.

UPDATE: I tried and it looks like the two approaches are the same.