It is hard to demonstrate that mathematics is the best, most truthful way to describe the universe. It is, though. It is the best way to describe relationships between things, which is the only sensible approach to describing the universe, to describing
cause and effect. As was said by Crispy on another thread, best not to ask what matter is, better to ask what matter
does. This applies to everything – we cannot say what the universe is, merely what it does.
This article by Hawking begins to describe how Godel's theorem can be applied to other systems, and boils Godel down into its essence, that of self-reference:
Penrose examines this in far greater detail, and also considers the similarity between this reasoning and Turing's proof that no Turing machine could ever solve the stopping problem. It has similar reasoning. For instance, consider the Goldbach conjecture that every even number greater than 2 is the sum of two prime numbers. You set a machine to try out each even number in turn and to stop when it comes across one that is not the sum of two primes. But what if the conjecture is true? You then need another algorithm that will decide whether or not this algorithm will ever stop. And such a 'stopping algorithm' does not exist. (You can prove that it doesn't exist, but I'm not going to go into that; you'll have to take my word for it for the sake of brevity.) You need something from outside the set of algorithms to decide upon the validity of the algorithms – sets of algorithms cannot justify themselves on their own.
Sorry if this is an incomplete explanation, but I hope it is pointing in the direction of my thought. TBH if you don't accept my initial statement about mathematics, none of the rest follows at all.