How does Dr. Roylott's plan back fire and what is his motive for the crimes? Also What would have happened to Helen if she didn't consult Holmes

Dr. Roylott's plan backfires because he is unsuccessful in killing Helen in the same way that he killed her twin sister, Julia, by allowing a deadly snake to crawl into a vent, enter her room and bite her during the night.  Julia was bitten as she slept in her bed which was nailed to the floor so that the snake would easily land on the bed as it crawled down the fake bell cord.

Julia's murder was only the first part of Dr. Roylott's plan, he killed her so that he did not have to give her the inheritance that her mother put aside for both her daughters which they were to receive when they each married.  Julia was engaged to be married when she "died suddenly" of unknown causes.

Helen becomes very suspicious, after she becomes engaged, and her stepfather, Dr. Roylott asks that she begin sleeping in Julia's old bedroom.

"Their client, Miss Stoner, is in "a pitiable state of agitation, her face all drawn and gray, with restless, frightened eyes, like those of some hunted animal." In spite of her miserable appearance Holmes sees in her the courage to fulfill a daring plan to trap the villainous Roylott."  

Once Helen begins sleeping in her dead sister's room, she starts to have strange experiences in this room.

"Because during the last few nights I have always, about three in the morning, heard a low, clear whistle. I am a light sleeper, and it has awakened me. I cannot tell where it came from-perhaps from the next room, perhaps from the lawn. I thought that I would just ask you whether you had heard it." (Conan Doyle)

Dr. Roylott does not get the opportunity to go through with his plan because Helen seeks out the help of Sherlock Holmes because she is afraid for her life.  She wants Holmes help in solving the mystery of the death of her sister.   Once Sherlock Holmes gets involved in the case, he wisely determines that there is a motive of money involved regarding Dr. Roylott and the twins, Julia, the dead sister and Helen.  Holmes ends up figuring out exactly how Julia Stoner was killed because he and Dr. Watson spend the night in Julia's old room, sitting in the dark listening. Holmes uncovers the truth, and Dr. Roylott is actually killed by his own snake.  Had Sherlock Holmes not gotten involved with the case, Helen Stoner would have died under mysterious circumstances just like Julia did two years earlier.