After searching a lot, I chanced upon Diogo Gomes's blog on Using svn+ssh with RapidSVN
Diogo suggests using Plink.exe but the Putty session started hanging and there was the problem of the Plink window opening up, each time RapidSVN would try to connect to the repository.
A simpler solution worked for me, using TortoisePlink.exe. This solution also takes care of the password caching by passing the username and password directly to the TortoisePlink through a environment variable. If you do not want to pass the credentials, simply avoid passing the parameters in the environment variable.
So here goes...
Step 1 : Add TortoisePlink.exe path to the Environment Variable
* Please note the "\\" in the path and the parameter for username is a small "el".
Value: C:\\TortoisePlink.exe -l [username]-pw [password]
Step 2 :The same path needs to be added to the Subversion config file. You can locate this file in Application Data/Subversion/config
# ssh = $SVN_SSH C:/TortoisePlink.exe* Please note the "/" in the path.
Two simple configurations and you are set!