I think the biggest question is what problems will be addressed by introducing the exact term order. Four issues were brought up initially:
integer + float conflation
ordered data structures
maps key order
glitch at zero
But the current proposal would address only one of them: ordered data structures.
It does not address integer + float confusion because, if anyone wants to compare them, I doubt they would expect 1 :< 1.0 or 0 :> -0.0 as actual results. The proposed order makes sense from a perspective of total ordering but does not address the integer + float conflation. IMO, a reasonable integer/float comparison means either as-is today or strongly typed (i.e. comparing integers and floats raises).
The proposal does not address maps key order either because the proposed ordering would be slower than the current maps key order implementation (plus Erlang/OTP 26 has a different ordering for atom keys which cannot be generalized because it is node specific - it only works for maps because we are responsible for their serialization/deserialization).
Finally, assuming we introduce :< and >:, it does not address the glitch at zero either, because =:= needs to remain backwards compatible. Or perhaps we can change the behaviour of =:= towards -0.0, but then it means the discussion is orthogonal to the one here.
Assuming the analysis above is correct, then I believe we should choose a solution that will be:
the most efficient for ordered data structures
does not guarantee a specific order, only that distinct terms have an order (similar to map keys ordering)
and does not introduce specific operators (as this is mostly for those implementing data structures)
Then separately discuss if the behaviour of =:= -0.0 should be changed or not (there are arguments on both sides). If no change, then we should at least introduce a erlang:sign/1 function (perhaps even a guard).
EDIT: in my opinion, we should not document the map term order at all. It is 100% an implementation detail as much as its hashed counterpart, and the same should apply to other ordered data structures. If the user of data structure wants to impose a order, then the data structure should also hold the comparison function that provides said user-specific order.
My concern with promoting this pattern is that someone can still write when C >:= 0 and run into the same pitfalls when using map key ordering. So, in order to be safe, I would still advocate to check for the type when doing ordered comparisons in guards.
I note that the Erlang comparison operators were copied from Prolog (although for some unknown reason the meaning of == and =:= was swapped and the corresponding inequalities also, something I
continue to find confusing).
I further note that < > and so on are arithmetic comparisons in Prolog Term comparison is a total order and written using @, so @< @=< @> @>= and I really do not see why there’s any need
to invent yet another set of names for these operations.
What is intuitive varies from person to person. I try to avoid using the word “intuitive”, but if I would use it I would use it like this:
I find the map key order very intuitive.
There are programming languages that don’t allow mixing of integers and floats without explicit conversions. In Erlang, there are operators such as div and band that only works on integers. So even though some Erlang operators, such as + allow mixing, integers and floats are distinct types that behave differently, and an Erlang programmer needs to understand the distinction.
I find the map key order intuitive, because you first order the values by type, then by value if the types are the same.
I find the exact term order less intuitive, because you have to:
Order the values by type, except that integers and floats should be considered to be the type “number”.
If types are the same, order by value.
For the “number” type, if the values are equal, order by the real types.
The map key order needs less complex code than the exact term order, and having smaller code with fewer possible branches can indirectly result in faster code. It is probable that the JIT can completely inline the new operators with map key order, with no need for a call to a fallback function. (The current operators handle the small integer case inline, and call a helper code fragment for everything else.)
As far as I understand (not having done an actual implementation), implementing the exact term order in the JIT will give no opportunities for new optimizations compared to the current operators.
I agree.
It only has to attract users for appropriate use cases, which is mainly in implementation of ordered data structures. For that use case, a BIF works just as well as an operator.
Using exact term order operators for range checks of characters or integers in guards has no advantage over using the current operators.
I agree. I think that we should introduce a BIF that does the comparison with an undocumented total term order (for simplicity of implementation, it will probably be the map key order). That will be a minimal implementation cost. Introducing new operators for the exact term order will be a lot work that does not seem to be worth the effort.
the comparison with an undocumented total term order
What does this mean, you can’t do an term_to_binary() round trip and be sure that it is ordered the same on future versions of erlang?
Also see @uwiger answer above, he needs of a total order above in nifs as well, wouldn’t it be preferable if they could be the same inside Erlang and in external libraries?
Yes. Or rather implementing a total term order that can be used for ordered data structures.
The other 3 “issues” are things to consider more than issues to be solved.
I do not know in which way “integer + float conflation” is an issue to solve, more than that in the coerced term order it is the integer + float conflation that makes it not a total term order. That we will get in different ways surprising properties of a total term order is to be expected since a total term order cannot have some floats equal to integers. The coerced term order with its established operators is still useful in many cases, and a new term order does not need to aim at replacing it.
It does not seem reasonable to use a different order for maps’ keys in a new total term order. I had that as a question at first, but no longer think it is an issue that needs solving.
The “glitch at zero” is a detail that might influence what is a reasonable total term order, i.e for the proposed exact term order it could affect if integers should be less than floats, the other way around, or closer to zero. I don’t think it is longer possible to change that -0.0 and 0.0 are equal. It is according to the IEEE floating point standard, and therefore they cannot be distinguished in Erlang, other than in an external format such as when converting to a binary or printing in io:format/*. Sure, we could bring this up again, but given the earlier design decisions such as to throw an exception instead of allowing +inf and -inf, which makes sign preserving calculations with infinity and zero impossible, I cannot see any reason change the =:= and =/= operators into differing between -0.0 and 0.0, and it would be dangerously incompatible to do so.
If -0.0 was =/= to 0.0, then I think it would be reasonable for the exact term order to have an integer closer to zero than a float, because that would be symmetric around 0. But when -0.0 is =:= to 0.0 it is more reasonable to have an integer less than a float. So I do not think this is a completely orthogonal issue.
Furthermore I think that introducing something in Erlang that can tell the difference between -0.0 and 0.0 would be bad since then the notion of them being equal gets broken. Their difference is only interesting in external floating point formats, and so it should be.
I guess that since Erlang did not implement unification there was never a need for a \= operator (inverse of =). That opened for using 2-character /= and == operators to fits graphically in with =< and friends. I actually think the Prolog arithmetic operators =:= and =\= look a bit odd besides the others such as =<, but I am probably Erlang biased.
Since Erlang by that has stepped away from Prolog, and already in the Erlang Book named the =:= and =/= operator as the exact comparison operators, I think extending the coerce comparison operators with a : to mark their exactness is a good looking solution.
Fun fact: I just found that in SWI Prolog there is a “standard order” that is just as the here proposed exact term order . There is also an ISO Prolog standard term order that is just as the maps keys order. For both orders Prolog has chosen to have float less than integer, though.
Apologies if this is too loosely related, but what I find confusing is that we have the same operator for numerical comparison and term comparison. So if I accidentally compare {ok, 2} > 3 instead of 2 > 3, I won’t get an error, but a hard-to-find bug. Static analysis won’t help it too. Possibly that could be taken into consideration if there are any changes in that matter.
Sure, but it is still important (and I don’t think that you say that it isn’t). Then you have to try to reason about what makes it intuitive for most users and perhaps especially for new users.
If we had not ever allowed any mixed integer and float operations, and existing comparison of integers and floats had not compared them as numbers, i.e., number had not even been a concept in the language, then I would agree with you.
However, my claim is since the language works the way it does today, extending it with exact term order is more intuitive for most users and especially new users. There is of course no way to be completely sure, but I think my arguments for why that is (which I wont repeat) are quite strong.
Then, of course, there could be reasons for introducing a less intuitive solution if those reasons are good/important enough. I don’t think that I’ve seen arguments for such reasons that that I think is important enough though.
I think this would be a really bad idea. Then users implementing their own erl_interface/jinterface like libraries for interaction with Erlang would have to rely on an undocumented feature in order to support total term order.
Agreed. That part of my suggestion was a bad idea. The order should be documented.
I still think a BIF would be an easy way to add useful functionality for implementing ordered data structures.
For the use case of implementing an ordered data structure, it is not clear to me why it would be important to have a total term order that is “intuitive”. For me, it seems sufficient to KNOW that the term order is total. Why would I need to understand exactly how terms are ordered? I just need to know that they are ordered in some defined total order, and I would also want that the comparison function is as efficient as possible. Would anyone care to clarify why it is important to have an “intuitive” term order when implementing an ordered data structure?
I would say that it in general always is important to design things as intuitive as possible to make it as easy as possible for users to understand and remember how things work.
As I said before though, there might be reasons to not go that route, but then I think that in that case it is important to have good/important reasons for it.
And yes opinions about what is intuitive and what is a good/important reason will differ between users, but I guess that this is what we are discussing.
If all you want is to have a total term order for implementing an ordered data structure, then I agree that it is (almost 100 %) enough to know that there is a defined total order that will not change even through serialization and deserialization.
But to motivate new operators, completing the set of comparison operators, I think it is attractive that our new first and only total term order is “intuitive” (and harmonizes with the original intention, as I perceive it, this is subjective), for instance being a minimal extension of the existing coerce term order.
Prolog has apparently (partly) chosen this way since the default “standard order” in SWI Prolog is just as the proposed exact term order, and, of course, there is also an ISO Prolog standard that has chosen the standard order to be just as the maps key order.
In this case I think that the exact term order is a better fit for completing the set of exact comparison operators.
And, regarding efficiency; it is only when comparing an integer with a float that type conversion comes into play, and that is already considered bad practice. So it is only bad practice code that will suffer any performance wise…
One mechanism we could use to validate is to check which other programming languages have comparisons between integers and floats with similar semantics to the proposed ones. As far as I know, I am not aware of any language where integers and floats are comparable but 1 < 1.0 is true.
Personally, I can’t recall someone saying comparison between integers and floats in Erlang/Elixir to be confusing, especially because it is common place in most dynamic languages (JS, Ruby, Python, Clojure, etc). Instead, the confusion arises from two areas (in my experience):
Creation of new data structures (which is done by a very small part of the community).
I find this can be tackled by a BIF. The order is not relevant, as long as it exists. I think intuition here is not a goal because there is nothing intuitive about ports being ordered before pids. I would prioritize performance and, as you duly noted, this order should be documented.
Comparison is done by structure instead of semantically. For example, if you have tuples, you can order the fields in a way that < and > works - such as date and times - but that’s not possible for maps and also not possible for all data types (for example {decimal, Base, Exponent} has several ways to represent 10).
If we were starting again - I know we cannot - I would perhaps do:
Introduce erlang:total_compare(Left, Right) - total comparison for data types, performance focused
< and > only works for integers and floats with either the semantics we have today or by not coercing. It should raise for all other data types. This would force users to use proper comparison functions whenever they need semantic behaviour (and if you don’t care about semantics, you use total_compare/2 above)
EDIT: if we assume the proposed order is intuitive, then it is all good, but what happens if the proposed order isn’t deemed intuitive in 5 years? The risk is having two set of comparison operators that behave in slightly non-intuitive ways, and I personally find that risk to be too high. We will push those decisions to everyone writing and reviewing code: on guards and conditionals, should I use < or :<?
In SWI prolog 1.0 @< 1 and 1 @< 1.1, where @< is their “standard order” less than operator. They have chosen float to be less than integer when arithmetically equal.
Thank you! There is probably a law somewhere that says “if you say there isn’t an example, someone will give you an example”.
I should have known better!
A better phrasing would be that this is not a common behaviour in the majority of programming languages. I don’t like justifying design decisions based on “majority” but since we are mentioning “intuitive” and “newcomers”, it is hard to deny comparisons with other languages play a large role on what is deemed “normal”.
I don’t say anything at all about the coerced term order, neither good nor bad. It is there and is what it is. I’m focusing on where we are today and where the addition of a total term order would take us. Once introduced, users most likely need to understand the relationship between coerced and total term order. The argument I’m making is that I think using map key order as total term order will be more confusing for most users than using exact term order.
When would that be useful? When would I want to use one of the new exact term operators instead of one of the current operators (except in the implementation of an ordered data structure)? What useful property would the new operators have that the old ones don’t have?
There are some additional costs. I have already mentioned the indirect cost of larger code size. There is also an additional cost for exact term order in a comparison when something being compared is not a small integer. When that happens, it is needed to check whether it is a bignum and also whether it is a float. For the map key order, there is no need to check for a float (if not an integer, it must be a type that is greater than an integer). All that probably means that the JIT can probably inline the entire comparison operator without any call to a fallback helper function. For the exact term order operator, we will still need a fallback helper function.
The new exact term operators would not be any more efficient than the current ones. If anything, they will be less efficient.
So again, I ask, what is the use case for the new exact term operators?
I think that is very good idea that would eliminate the possibility for many old mistakes while not introducing the possibility for a lot of new mistakes. Taking @raimo’s example:
foo(V) when 2 =:< V, V =:< 5 -> in;
foo(V) when V :< 0; 7 :< V -> out;
foo(V) when is_integer(V) -> neither.
The guard for the middle clause would now immediately fail if V is not an integer, which is the expected behavior.
They also make much sense from an implementer’s perspective. If one of the values being compared is a literal integer, the JIT can inline the code for the comparison with no need for a call to a fallback.
First off, -0.0 =:= 0.0 coming out true is a BUG.
It’s simply wrong behaviour.
It is also a clear and blatant BUG that 0.0 = -0.0
succeeds.
Erlang CAN distinguish these two numbers and DOES
distinguish them. (Most obviously in term_to_binary/1
and float_to_list/1, but there are others.)
It makes NO sense to appeal to the IEEE standard for
floating-point arithmetic in a programming language
that goes out of its way NOT to conform to that standard.
Not just omitting IEEE operations such as copysign(-,-)
– and there is no shortage of other missing operations –
but also in omitting IEEE-mandated required values.
The facth that arithmetic comparison unifies 0.0, -0.0,
and 0 is no reason why “exact equality” should do so,
and certainly we’d have to regard any so-called total
order on terms that made (or, right now, makes)
T1 =:= T2,
F(T1) =/= F(T2)
come out true when F is a pure function, as badly broken.
As missing half the POINT of “equality”. As making it
harder to write correct code, for no advantage.
Arithmetic comparison SHOULD identify 0, 0.0, and -0.0
Exact comparison and matching should NOT.
Consider this.
f(X, X) → X.
f(0.0, -0.0).
Will the answer be 0.0 or -0.0? Can you tell which without
trying it? Is it obvious to you that it depends on the
order of the arguments? Is clause matching supposed to
depend on the order of the arguments?
Oh! That should mean that maps cannot be used in keys in distributed algorithms since they compare differently on different nodes. Would this mean that we need a comparison operation that throws a badarg when comparing maps?
I am starting to wonder if we should make this change for OTP-26.0 since it apparently changes to a term order that is not consistent over serialization+deserialization such as node restarts (through storage) nor between nodes.
The use case is, as stated, to get a total term order, that for example can be used when creating ordered data structures.
I simply think that the exact term order is a better augmentation of the coerce term order since it only creates a difference for terms that today are equal, and that this is more important than performance.
Yes. I also think it is a bug and that the behaviour is incorrect.
I have been in several discussions about this and it so far has always ended in that this behaviour is as old as floating points in Erlang. Aparently there is lots of code out there that does:
case foo() of
0.0 -> this_wont_work;
F when is_float(F) -> bar(4711 / F)
end
or even
baz(F) when is_float(F) ->
if F =:= 0.0 -> this_wont_work;
true -> bar(4711 / F)
end.
since there has been a notion or a fact that the =:= operator is optimized better than the == operator if the compiler can figure out that the argument is a float.
So changing this behaviour would break lots of existing code in a subtle way.
The original design of floating points in Erlang avoided +Inf, -Inf, and NaN, and thereby removed the need to distinguish between -0.0 and 0.0, and so conflated them into 0.0 and made them both == and =:=. Blaming on IEEE’s requirement of comparing them equal is just a bad excuse, and declaring that float_to_list/1 and term_to_binary'1 creates an external format that is not “in the Erlang language” is also a strech and more than a half lie.
And I still cannot see how we could fix this bug in a safe way…
