Why was it a pity that Bilbo didn't kill Gollum? How does Gandalf reply to this statement and what does Frodo learn from Gandalf's comments?

Though Frodo meant this in a negative sense, Gandalf spins his comment to a higher, more literal level. Pity was what stayed Bilbo's hand, a deeper sense of morality. Frodo believes that much evil could have been prevented by killing Gollum, yet Gandalf views  Gollum's death as the loss of opportunity to do good (as the end of the novel proves).

When Frodo says that Gollum deserves death, Gandalf replies, "Deserves death? Many that live deserve death, and many who die deserve life. Can you give it to them? Then do not be too quick to deal out death in punishment, for even the very wise cannot see all ends."

Perhaps on a deeper level, Tolkien is making a statement about capital punishment, but in this particular case, he is speaking more about a transcendent Cause in the realm of Middle Earth. As a staunch Roman Catholic, Tolkien had a deep belief in Grace, in which God works in human life beyond what humans are capable of themselves. Premature death, like Frodo's wish for Gollum, does not give way to Grace, to what God (or Iluvatar, as Tolkien names God in "The Silmarillion") can do through an individual life. Bilbo's show of pity by not killing Gollum connected to this higher purpose, this grace.

This answers the question of why Tolkien chose to destroy the Ring in the manner in which he did. It was not by Frodo's effort, though his effort got him to the point where Grace could take over. This is Tolkien's point: that human action can take one only so far. If Frodo had intentionally destroyed the Ring after all, it would have left no room for Grace.

And this is what Gandalf meant by his reply to Frodo's regret that Bilbo did not kill Gollum. Without Gollum, the Ring would not have been destroyed but would have been taken by Frodo, creating a new Dark Lord. Grace stepped in in the unlikely form of Gollum and prevented this from happening.

From this Frodo learns that human (or hobbit) effort is valuable, but not indispensible. There are more forces at work in Middle Earth than just what is seen. As Bilbo was "meant" to find the Ring, so Bilbo was "meant" to let Gollum live. Gandalf's message is that, in order to destroy evil, the forces of good will need to show a "passive activity," in which they step back and look at the larger picture. There is a limit to what the inhabitants of Middle Earth can do, but still they must do what they can. As Elrond says, "The road must be trod, but it will be very hard. And neither strength nor wisdom will carry us far upon it. This quest may be attempted by the weak with as much hope as the strong. Yet such is oft the course of deeds that move the wheels of the world; small hands do them because they must, while the eyes of the great are elsewhere."