In order for a Process to be able to interact with Internet Explorer or a Custom Dialog, you will need to modify some of the Internet Explorer settings. By default, Internet Explorer comes with security settings enabled that do not allow an external application to control it.
To change this, perform the following steps:
1. Open the Internet Options dialog in Internet Explorer and click on the "Security" tab:


2. Select the "Internet" zone and UNCHECK the "Enable Protected Mode" checkbox:

3. Select the "Local intranet" zone and UNCHECK the "Enable Protected Mode" checkbox:

4. Select the "Trusted Sites" zone and UNCHECK the "Enable Protected Mode" checkbox:

Finally, click the OK button and restart Internet Explorer. You will now be able to control both Internet Explorer for automating all kinds of web sites and web applications and the Custom Dialog.


Configure Internet Explorer and Custom Dialog for Servers


If you have your Robot installed on a Server, for automating your web related tasks, note the following.
Windows Servers have the "Internet Explorer Enhanced Security Configuration" (IEESC for short) feature turned "On" by default, which prohibits the Processes from properly launching an Internet Explorer or the Automation Browser via the "Launch New Internet Explorer" Action.
Likewise, as long as the IEESC feature is "On" any and all web-automation Actions will too fail to work as intended. Moreover, the design-time Web Helpers of the aforementioned actions will not work either as long as the IEESC feature is "On".
In order to be able to overcome the above, you would have to change the Security Configuration as per the screenshots below in 3 small steps.
1. Open the "Server Manager" and go to "Local Server"
2. Click on the "IE Enhanced Security Configuration"
3. Both for Administrators and Users select the "Off" radio button and click "OK".


You will now be able to interact with both Internet Explorer and the Customer Dialog through your Robot.