One of the best now arguments for the existence, or alleged existence of infinity in mathematics, outside of set theory, comes from number theory. While infinity is used in the theory of limits and hence the calculus, that symbol is one of convenience, and can be eliminated if necessary, in terms such as “increases without bonds,” what the ancient Greek philosophers would call the “potential infinite,” by contrast to the actual infinite. Strict finitists who would say that there are only a finite number of say natural numbers, 0, 1, 2, 3, … n … such as made by finitists in the following papers: 
  https://www.math.uni-hamburg.de/home/loewe/HiPhI/Slides/bendegem.pdf
  http://www.jstor.org/stable/pdf/2272346.pdf?refreqid=excelsior%3A6e4ad837e563fca884ba1b46e6069c68 
  //blog.alor.org/file://uofa/users$/users0/a1066120/1468-1449-1-PB.pdf 
  https://projecteuclid.org/download/pdf_1/euclid.ndjfl/1093634481 
  https://www.jstor.org/stable/pdf/2273760.pdf?refreqid=excelsior%3A5a69fb898236f1f5f11fb5b134fa9c78 
  https://onlinelibrary.wiley.com/doi/epdf/10.1111/j.1746-8361.1955.tb01332.x

     The conventional Platonistic mathematician will argue that given a hypothetical “last” number L, simply consider L + 1, to obtain a larger number. The question-begging assumption is made that L + 1 actually denotes, but we ignore that. The argument assumes that numbers exist in some non-material realm of abstract essence, as has been done from Plato to Gödel. That view is subject to metaphysical difficulties that mathematicians seldom consider, such as how is mathematical knowledge possible if the entities in question have primarily a non-material existence, since causal interaction with the brain is ruled out:
  http://www.columbia.edu/~jc4345/benacerraf%20with%20bib.pdf 

     If a Platonist account of mathematical entities is rejected, and there are many good reasons for doing so:
  https://www.calstatela.edu/sites/default/files/groups/Department%20of%20Philosophy/realism_and_anti-realism_in_mathematics.pdf

     Mathematics may be more closely related to human social practices. Numbers may not be infinite, nor could chains of reasoning processes, since the human brain is finite, and of finite information processing capacity. Further, there will be physical limits of the possible representation of a number. A number written on a hypothetical medium, at the smallest possible length permitted by quantum mechanics, expanding the entire length of the known  physical universe (assumed to be finite, about 93 billion light years), would not be computable in principle. Suppose that the number is composed of the digits of the decimal expansion of  a series of 99999…. and after each digit, the raised to a power symbol “^” is added.

     We can then suppose that the number is not merely written in a linear fashion on our hypothetical quantum mechanical medium, but extends in 3D space, or if string theory is correct, into N-dimensional space. Thus, the entire universe is full up with one step ladder  number. And, it is not possible to physically or conceptually add 1 to this number, because there is no space in the universe to even hypothetically add any more symbols. We could say that this number, Ʊ is the largest natural number, even if there could be a Platonist conception of a larger number, because no bigger number could be represented. Likewise, a step ladder number comprising the entire universe could be constructed to refute the standard proof that there is an infinity of prime numbers. Note that the largest prime number found to date is:

 277,232,917-1 with 23,249,425 digits by Pace, Woltman, Kurowski, Blosser & GIMPS (26 Dec 2017). By our conception, that is a very small number.

     A natural number is defined to be prime if the only divisors of that number are 1 and the number itself, so that the first few primes are 2, 3, 5, 7, 11, 13, 17, 19 … Note that an odd number is not necessarily prime, 9 being an example. The even number 2 is prime, but that is the only one, because bigger numbers will be divisible by 2. There are good reasons why the number 1 is not defined as prime, or composite (not prime), not fitting the definition of either concept:
  https://www.quora.com/Why-is-1-neither-prime-nor-composite
  http://mathforum.org/library/drmath/view/57036.html 
  https://primes.utm.edu/notes/faq/one.html
  https://cs.uwaterloo.ca/journals/JIS/VOL15/Caldwell1/cald5.html

     Euclid (died 285 BC), in his Elements, has been attributed the proof that there are an infinite number of primes. Actually, Euclid in book IX, proposition 20, argued this:

Proposition 20
Prime numbers are more than any assigned multitude of prime numbers.
Let A, B, and C be the assigned prime numbers.
I say that there are more prime numbers than A, B, and C.
Take the least number DE measured by A, B, and C. Add the unit DF to DE.
Then EF is either prime or not.
First, let it be prime. Then the prime numbers A, B, C, and EF have been found which are more than A, B, and C.

VII.31
Next, let EF not be prime. Therefore it is measured by some prime number. Let it be measured by the prime number G.
I say that G is not the same with any of the numbers A, B, and C.
If possible, let it be so. Now A, B, and C measure DE, therefore G also measures DE. But it also measures EF. Therefore G, being a number, measures the remainder, the unit DF, which is absurd.
Therefore G is not the same with any one of the numbers A, B, and C. And by hypothesis it is prime. Therefore the prime numbers A, B, C, and G have been found which are more than the assigned multitude of A, B, and C.
Therefore, prime numbers are more than any assigned multitude of prime numbers.
https://mathcs.clarku.edu/~djoyce/java/elements/bookIX/propIX20.html

     At no point did Euclid use the notion of infinity, which the Greek mathematicians were generally opposed to, following Aristotle in holding only to the “potential infinite.”
  https://link.springer.com/chapter/10.1007/978-1-4899-0007-4_4 
  http://www.math.tamu.edu/~dallen/masters/infinity/content2.htm

     A very good article discussing Euclid’s proof and some flawed interpretations of it is by M. Hardy and C. Woodgold, “Prime Simplicity,” The Mathematical Intelligencer, (2009):
  https://www.researchgate.net/publication/226338654_Prime_Simplicity
  https://link.springer.com/content/pdf/10.1007%2Fs00283-012-9322-z.pdf

Prime numbers being more than any assigned multitude of prime numbers means that there is no greatest prime. As we will see, it will require further argument to justify the claim that there are a literal infinity of primes. Further, Euclid did not present the standard proof by contradiction or reductio, but showed that given any list of primes, one can show that a new prime can be added to the list. That does not show, as all the textbooks state, that there is an infinity of primes. What is established is the weaker thesis that there is no largest prime. But, I doubt whether, if Platonism is rejected, that the proof which requires listing the finite list of primes could even get started if we consider that we are dealing with super-large, but still finite numbers like Ʊ, where there is no way (except in the Platonistic imagination) of representing Ʊ + 1. Regardless of this, even on Platonistic assumptions, the best that the Euclidian argument shows is that there is no largest prime, and it does not follow from this that there is an infinity of primes. Indeed, set theory does not prove that there is an infinite set of objects, but adds an axiom of infinity to get the desired result, something recognised by Zermelo in 1908. It was also added as an axiom by Russell and Whitehead in Principia Mathematica. However, it was pointed out by F. Ramsey, “The Foundations of mathematics,” Proc. London Math Soc, vol. 25, 1926, pp. 338-384, that the axiom of infinity needs to be taken as a primitive and cannot be proven. I will argue in other papers, that it can be refuted though. The critical analysis continues in “God and Mathematics 4: A Refutation of the Power Set Axiom.”