Hi Gaurav,
Thanks for the clarification!
Since you did not open the dialog from your own custom code, closing it might be a bit trickier. I've played around with it a little bit, but the only thing I found was the kind of hacky
document.getElementsByTagName("dialog")[0].close();
It worked for me in a setup similar to yours (note: I tested this on Innovator 12.0). My onSearchDialog method simply contained
alert("hi");
document.getElementsByTagName("dialog")[0].close();
and when clicking "Pick related", it opened the dialog, alerted "hi" and then closed the search dialog.
Obsiously this approach is not very robust, but maybe it can serve as a starting point for you? Maybe you can find a way to uniquely identify the dialog you opened in your setup (document.getElementsByTagName("dialog")[0] makes me feel kind of uneasy), but I couldn't find anything else so far.
Maybe someone else knows a more solid approach.
Cheers,
C