Banjo

52 thoughts
last posted Nov. 9, 2015, 7:13 p.m.

5 earlier thoughts

0

Math: Integer vs. IEEE vs. FixedPoint vs. BigInteger vs . BigDecimal vs. Rational

Recent experience programming games in lua reveal that it is all too easy to try and compare IEEE numbers incorrectly because they're some small fraction off of expectations. Plain Math.floor() isn't the right way to round to the nearest integer, you have to use Math.floor(0.5 + ...) which is hard to remember and error-prone.

There should be a sane way to compare numbers within a tolerance, or specify an integer comparison versus a floating point one.

For maximum specificity people wouldn't use standard operators but instead specify exactly what kind of math they want to use with each operation. e.g. z = int32.add(x,y). But this seems very wordy while z = x + y is somewhat ambiguous.

Languages with multiple number types seem to like to stick with the widest of any type used in the operation, but no wider. Adding two ints gives an int - even if there's information lost to overflow.

Languages like LISP don't allow overflow when using the standard operators, they'll promote the type to a larger size as necessary. This is actually very nice although there is a cost to performance it's not much worse than Lua in this regard. They probably provide functions to perform fixed-format operations as well.

I don't know the right solution for this, but if one could use the standard operators and get the right results it would be a huge win.

46 later thoughts