Who was mainly responsible for the growth of the Roman Empire?

Because your question is framed in terms of the Roman Empire, I would say that the Roman Empire had grown the most at the death of Augustus in AD14.  The addition of Egypt after the Battle of Actium was a significant addition to the Roman Empire's lands, and ensured that Rome would dominate the Mediterranean.

But, the Roman Empire did not begin until the reign of Augustus.  Prior to that, Rome was a Republic, ruled primarily by a Senate.  This period would include the time when Julius Caesar made his conquests of Gaul, etc.  If your question is intended to include the pre-Augustan period of the Roman Republic, then I would agree that Julius Caesar would be the one most responsible for increasing the size of Rome.  If you mean only the imperial period, then I would say Augustus.