Saunders Mac Lane quotes:

. . . in one sense a foundation is a security blanket: If you meticulously follow the rules laid down, no paradoxes or contradictions will arise. In reality there is now no guarantee of this sort of security . . .

. . . the membership relation for sets can often be replaced by the composition operation for functions. This leads to an alternative foundation for Mathematics upon categories -- specifically, on the category of all functions. Now much of Mathematics is dynamic, in that it deals with morphisms of an object into another object of the same kind. Such morphisms (like functions) form categories, and so the approach via categories fits well with the objective of organizing and understanding Mathematics. That, in truth, should be the goal of a proper philosophy of Mathematics.

... the notions category and functor were not formulated or put in print until the idea of a natural transformation was also at hand.

The standard "foundation" for mathematics starts with sets and their elements. It is possible to start differently, by axiomatising not elements of sets but functions between sets. This can be done by using the language of categories and universal constructions.