A Custom Dialog is acting like a web page with its own HTML and CSS. In the Properties window of the "Display Custom Dialog" Action, you can define your own CSS rules to your Custom Dialog. 

Follow the instructions below in order to apply your own CSS:

1st method - the easy way

1. Create new custom classes along with their attributes. For example,

margin-top: 6\%;
font-style: italic;

.radio1, .radio2{
color: blue;

2. Apply the above to the custom dialog elements. For example,

2nd method - the advanced way

Create CSS rules based on the HTML of the dialog. In order to do so, we need to access the HTML code of the dialog.

Follow the steps below in order to open the dialog as a normal web page:

1. Create a new Process and add the following Actions:

2. "Wait" Action / Wait for: 2 seconds

3. "Launch New internet Explorer" Action / Operation: Attach to Running IE / Select IE: use Foreground IE

4. "Get Details of Web Page" Action / Operation: Get Web Page Source

5. "Write Text to File" Action / File Path: your_file_path.html / Text to Write: %WebPageProperty%

6. "Launch New internet Explorer" Action / Operation: Launch New IE / Initial URL: your_file_path.html 

7. Leave your Custom Dialog open on the screen and run the Process.

8. Immediately click on the dialog so that it becomes focused.

Your new Process consists of five (5) Actions (steps 2-6) and is able to open a new Internet Explorer and display your Custom Dialog as a normal web page.

Now that the dialog is open in a form of a web page, you can create your own CSS rules, based on the CSS Selectors of the page.

Check this Forum topic that demonstrates the use custom CSS rules.