-
Welcome to Smogon! Take a moment to read the Introduction to Smogon for a run-down on everything Smogon, and make sure you take some time to read the global rules.
-
Congrats to the winners of the 2023 Smog Awards!
Done Floating point imprecision in /math
- Thread starter Geysers
- Start date
As someone who admittedly uses /math for various purposes such as homework (I find it much handier than the Google calculator for simple operations), this is something I intended to report at some point but I was beaten to it.
I'm sure many tech people are aware of why this happens, but for those who aren't, it's due to numbers being stored in binary. Because of this, all integers are able to be represented perfectly, and other decimals that can be expressed as a fraction of some power of 2 such as 0.25 (1/4) and 1.328125 (85/64) can also be represented perfectly. However, a number such as 0.1 can't be expressed perfectly because it would have infinite decimals in binary (1/16+1/32+1/256+1/512+1/4096+1/8192...). I read this article to write this explanation and it explains pretty well what is going on with decimals. Also, this is not just a problem with inputs containing a decimal point, but also fraction inputs; for example, 10/3 returns 3.3333333333333335, which has an unusual 5 at the end where a 3 is expected. Additionally, a reasonable human would be able to compute 1/7+1/7+1/7+1/7+1/7+1/7+1/7 as 1, but /math will return 0.9999999999999998 instead.
Originally, I was going to propose doing some complex function to convert non-integer values to integer, which could fix problems such as the repetitive 1/7 addition or the operation posted by the OP, but after finding out that very simple operations such as 10/3 are affected, it doesn't seem like imperfection can be made perfect. Instead, I think limiting the amount of sigfigs to be displayed, probably 10 max (with some means of hiding or avoiding trailing zeros), is the best solution. This is what calculators in real life implement as far as I can tell, and it shouldn't have any drawbacks because digits beyond the 10th sigfig, and honestly way before that, are very unlikely to be useful.
I'm sure many tech people are aware of why this happens, but for those who aren't, it's due to numbers being stored in binary. Because of this, all integers are able to be represented perfectly, and other decimals that can be expressed as a fraction of some power of 2 such as 0.25 (1/4) and 1.328125 (85/64) can also be represented perfectly. However, a number such as 0.1 can't be expressed perfectly because it would have infinite decimals in binary (1/16+1/32+1/256+1/512+1/4096+1/8192...). I read this article to write this explanation and it explains pretty well what is going on with decimals. Also, this is not just a problem with inputs containing a decimal point, but also fraction inputs; for example, 10/3 returns 3.3333333333333335, which has an unusual 5 at the end where a 3 is expected. Additionally, a reasonable human would be able to compute 1/7+1/7+1/7+1/7+1/7+1/7+1/7 as 1, but /math will return 0.9999999999999998 instead.
Originally, I was going to propose doing some complex function to convert non-integer values to integer, which could fix problems such as the repetitive 1/7 addition or the operation posted by the OP, but after finding out that very simple operations such as 10/3 are affected, it doesn't seem like imperfection can be made perfect. Instead, I think limiting the amount of sigfigs to be displayed, probably 10 max (with some means of hiding or avoiding trailing zeros), is the best solution. This is what calculators in real life implement as far as I can tell, and it shouldn't have any drawbacks because digits beyond the 10th sigfig, and honestly way before that, are very unlikely to be useful.
The best implementation to solve this is have /math calculate at least 2-3 decimal places (8-10 bits) more than the significant maximum number of digits displayed, this should reduce the ammount of floating point errors by several orders of magnitude. Another approach would be to have common real numbers such as 0.1 stored in symbolic form rather than float.
dot-comfey fixed this in https://github.com/smogon/pokemon-showdown/pull/10244 by limiting the amount of sig figs a result can have to 15.