Warren Buffett’s Berkshire Hathaway has returned a staggering 3,641,614% since its inception in 1965.

Those results speak for themselves. By comparison, the S&P 500 has returned 30,209% in the same time frame. A single dollar invested in Berkshire Hathaway in 1965 would have turned into $36,714, while the same dollar invested into the S&P 500 would have returned just $303.

But not even Buffett is immune to the law of large numbers. The bigger something gets, the harder it is for it to keep growing exponentially. Berkshire Hathaway has managed to double its share price in twice one year exactly twice — and both instances were back in the 1970s.

As Buffett put it over a decade ago, “The highest rates of return I ever achieved were in the 1950s. I killed the Dow. You ought to see the numbers. But I was investing peanuts back then. I think I could make you 50% a year on $1 million. No, I know I could. I guarantee it.”

Why Buffett Envies You

But it’s more than the law of large numbers working against the Oracle of Omaha today. Buffett — and every other billionaire investor and institutional player — is essentially banned from investing in the most explosive opportunities in any meaningful way.

Let’s say Buffett wanted to invest in a small-cap company valued at $1 million. He could legally invest a few thousand dollars and hopefully, watch it soar to $100,000 or more.

But that would be a pittance for Berkshire Hathaway, which is a juggernaut valued at hundreds of billions of dollars today. A home run on a small investment won’t move the needle for Buffett.

His other option would be to invest far more — say $500,000. But then he would own so much of the company that he would need to file a 13D Schedule form with the Securities & Exchange Commission and take on the headaches that come with being what is legally known as a “beneficial owner.”

Buffett will continue to collect hundreds of millions of dollars per year in dividends from the vast amount of shares he owns in household names like Coca-Cola Co. (NYSE: KO), Apple Inc. (NASDAQ: AAPL) and Bank of America Corp. (NYSE: BAC).