Absolutely safe. Having moved there in 1986 as a child I have grown up in the area and since bought a unit and now a house for my family. The area contains a lot of parks and in particular is very safe for walking at most hours of the day which we wnjoy the most. My parents continue to live there as do we. Crime is very very uncommon. Car break ins are rare (norwood nearby has issues) and house breakins we have not had or had any friends experience in our street. Having lived in Melbourne and the Middle East - it has been a pleasure to return home and not to worry about our children playing.
Toorak Gardens is a very prestigious and incredibly safe suburb. You can walk around at night without worrying about personal security. In addition it has lovely wide streets, and is a very green suburb with a lot of trees. I would live there in a heart beat if I could afford to!
I grew up in Rose Park (next door to Toorak) and we had one break in and ongoing issues with drug syringes in the street and garden. It will depend on the location but, to be honest, you're no safer or worse off than any other area. Is it a great neighbourhood? Absolutely! Would I sleep with my doors and windows open? Never.