It seems to me that you're assuming a Platonist mathematics, in which the fact that we use the word "infinity" implies there is an actual Infinity that "exists" out there somewhere. This is mistaken. Infinity is a concept used in analysis. Think of it not as a number, but as a symbol. Real analysis is a logic, not arithmetic; and the term "infinity" in it doesn't need to fit into the category of "number" or "operator"; it just needs to be defined in a way that doesn't render the logic inconsistent. When you then apply the theorems you proved in analysis on values which include "infinity" terms, "infinity" there isn't an atomic symbol with a referent out there somewhere; it's more like a macro, which would expand into the entire body of constraints imposed by its use according to the logic of analysis.
It seems to me that you're assuming a Platonist mathematics, in which the fact that we use the word "infinity" implies there is an actual Infinity that "exists" out there somewhere. This is mistaken. Infinity is a concept used in analysis. Think of it not as a number, but as a symbol. Real analysis is a logic, not arithmetic; and the term "infinity" in it doesn't need to fit into the category of "number" or "operator"; it just needs to be defined in a way that doesn't render the logic inconsistent. When you then apply the theorems you proved in analysis on values which include "infinity" terms, "infinity" there isn't an atomic symbol with a referent out there somewhere; it's more like a macro, which would expand into the entire body of constraints imposed by its use according to the logic of analysis.
At least, that sounds more plausible to me.