Windows Automation Reference
Version: 10.0 / Modifications: 0
Introduction
Welcome to a new dimension of Microsoft Windows automation. Using the Tomorrow Software Windows Automation Extension you can now not just script up the flow of a Windows Application – but you can also combine it with data from many other sources and the powerful rule writing capabilities of the Tomorrow Software Multi-Protocol engine.
The extension is based on the popular AutoIt automation product and we have included tools from that product to help your automation efforts. AutoIt is a free product, however, if you find the product and the Windows Automation extension useful, we would encourage you to make a donation to the creators of AutoIt at:
https://www.autoitscript.com/site/donate/
Licensing
The licensing of the Tomorrow Software Windows Automation Extension is the same as most other extensions that we provide. You simply need a valid Tomorrow Software license.
The license for the AutoIt tools described in this reference guide is found in the “data” folder where you also found this document. In a quick summary it is a classic free software license.
Getting started
Before you begin your first automation project, you need to make some updates to your Tomorrow Software installation.
Required Updates
The first step is to update/install the following components via the update server:
Tomorrow Software console (B18020 or later)Base Rules (2018-04-26 or later)Parallel Processing Rules (2018-04-23 or later)Windows Automation Rules (2018-04-23 or later)
If you received this document through some means other than the update server, then you will also need to install the Windows Automation repository.
At this point, stop the Tomorrow Software instance.
Updating the Java Runtime Environment
The JRE that ships with Tomorrow Software is a basic 32 bit JRE. The version may depend on when you received your copy of the product.
To successfully run automation projects, you need to update the JRE to at least version 8 for your platform (32 bit or 64 bit).
You can download the correct JRE from here: http://www.oracle.com/technetwork/java/javase/downloads/jre8-downloads-2133155.html
Once you have installed the JRE, you need to update the JRE folder under the Tomorrow Software installation with the JRE that you installed on your Windows PC. You do this by renaming the original JRE folder and creating a new one by copying the JRE from C:\Program Files\Java\jre1.8.0_(version) and renaming it to jre.
Installing the required tools
The final step is to install the Au3Info tool. You need this tool to inspect running Microsoft Windows programs and identify the names of controls that you can manage. The easiest way to install the tool is to download it from the Windows Automation repository’s data folder and save it to your desktop (or some other convenient location).
There are two versions available:
Au3Info.exe is for 32-bit Windows systemsAu3Info_x64 is for 64-bit Windows systems
Make sure that you download the right version.
Your first automation
In this example we will take you through the automation of creating a document in Windows Notepad and saving it.
Start by restarting the Tomorrow Software Server instance, log in and create a new repository called “Notepad Exercise”.
Then create a new rule set called “NotePadDemo”:
and open it up in the rules editor.
Starting an application
The very first thing we need to do is start notepad itself. To start an application, simply drag the Run Application rule onto the canvas:
And set the properties as shown:
This step alone will cause Notepad to start up. You do not need to provide a directory since notepad will be in the system path.
Since we are going to do something more than just start the application, we need to make sure that it is fully loaded before we start pressing keys. So we add a Wait Active rule:
With the properties set as follows:
Identifying windows
Here it is relevant to pause for a minute and look at those properties.
Firstly, the Windows Label. Many of the rules provided in the Windows Automation framework use the Label and Text combination to identify windows to work with. The logic of this combination is as follows:
The Label match is basically starting from the beginning of the label matching as many characters as provided in the rule.
In this case we match the entire label.
The optional Text matching refers to a text within the window that was opened. This could be any word visible on the page or within a dialog box. This matching is used for more precise pinpointing of a window.
We will perform such a match later in this section.
Entering text
For now, we will simply send some keystrokes to Notepad to create a document we can save:
Testing
Let’s try and run our three new rules and see what happens. In the Notepad Exercise repository create a new configuration as follows:
And set the input source to:
We can now deploy our configuration to any convenient active server (You can use a Multi-Protocol server or even Qwerty). As long as you tick the “Restart immediately”, shortly after the deployment is complete, you will see Notepad start up and the text appear:
Sending formatted text
You may have noticed that the text entered in our example was set as “Raw”:
Your other option would be to use the formatting text feature:
This feature allows you to send specific keystrokes with great ease. For example:
You can combine these keys: ^!a would be Ctrl-Alt-a.
If you need to send any of those characters without sending them as special keys, you must enclose them in curly brackets. For example {!} to send a !
You can also send normal Windows keys by enclosing them in curly brackets. For example:
The name used in the brackets can be most normal windows keyboard designations.
If you need to repeat a few keystrokes, you can do this by entering the key name followed by a count. For example:
Will result in the delete key being hit 5 times.
So we could in theory expand on our example to make notepad try to close once the text was entered. The keystrokes for closing a Window is Alt-F4. We would do this in formatted text as follows:
Doing this results in the following outcome:
You can try this if you wish, just remember to switch it back to “Hello World!!” and “Raw “afterwards to continue this exercise.
Reading window text
It is one thing being able to send keystrokes, but more often than not for automation, you will need to know the content of specific fields or you may need to be able to set the value of specific named fields without just using keyboard navigation.
This is where the tool from AutoIt (that you installed earlier) comes into play. Start up the correct version of Au3Info:
Click on the Finder tool and drag it onto the main Notepad window:
You will see that the tool provides you with the basic Windows information (Title and Class). It also provides us with the Basic Control Info, which is that the field is of the Class “Edit” and it is instance “1”.
What we need at this stage is the ability to identify a specific field in a specific window. The best and safest way to do this is to click on the “Control” tab:
And then double-click on the “Advanced Mode” entry. This copies the identifier [CLASS:Edit; INSTANCE:1] to the clipboard for us so that we can use it easily.
So all we need now is to add a “Get Control Text” rule (and a List Variables so we can see what’s going on):
The properties for the Get Control Text would be as follows:
The control identifier being easily set by entering two double-quotes and pasting the content of the clipboard from the AutoIt tool in between them.
A quick run and peak at our console will confirm that this is working:
Text outside controls
There are certain circumstances where text is not necessarily linked to a specific control. The Windows Calculator is one such example. It actually stores the result window not in the control, but in the window itself. If you need to get to this text, you can use the Get Window Text rule instead of the Get Control Text rule.
Hint: When you extract text from the Window itself, it is often formatted across multiple lines. An easy way to get visibility of control characters in text is to escape them as if they would be going into a URL. You can do this with the Escape rule.
Closing windows
It is now time to close our window. This is simply done with the Close Window rule:
The properties should look familiar now:
The result of adding this rule will inevitably be:
Now, we wish to wait for this dialog to appear and then hit Enter to save the file we just created. Once again, this should now be familiar territory:
With the properties being set as follows:
Note the use of Window text in the “Wait for save box” rule. It is conceivable that Notepad my put out many dialogs that is simple labeled “Notepad”, so the extra check for the word “Save” somewhere on the dialog box helps us confirm we are in the right place.
Advanced controls
But now things are getting a little tricky. Once we hit enter, we need to wait for the “Save As” dialog to appear:
On the surface, this may look quite simple. We wait for the dialog box to appear, we find the controls for the directory and file name, put in some values and hit Save.
The first step is not too hard:
Next, we discover (using the AutoIt tool) that the directory control is named “ToolbarWindow32”:
However, through experimentation it quickly becomes obvious that you can’t just set the control value to “Address: MyDirectory” using the Set Control Text rule. It simply has no effect. So, we need to introduce a workaround. In this case, some experimentation shows that if you click on the far right corner of the control, you can actually enter a directory name:
And the text is preselected, so if we can just do the same mouse clicks in rules, we will be able to override the text in the control and continue. This requires a few steps
Getting a control position
We start by getting the control position so that we can figure out where to click within it:
Adding a List Variables rule and running this results in the following output in the console:
So now we know the position and dimensions of the control. The next step is to figure out the correct position to click. A simple calculation rule will take care of that:
All that remains now is to “click”:
Most of the above should now be clear. We are basically clicking the far left side of the control, using the left mouse button. If you don’t provide an X or Y position, the center of the controls axis respectively will be clicked.
All that remains now is to set the control value by sending the right key strokes:
Notice that we hit the Enter key as part of this exercise. This is because the Save As dialog box changes to the directory entered, once the Enter key is hit.
If you are following this example, make sure that you pick a directory that actually exists. In our example, we have created C:\DemoData purely for this exercise.
The next job is to set the actual file name. Using AutoIt, we discover that the control name for this is “[CLASS:Edit; INSTANCE:1]”. So this looks pretty straight forward. However, setting the control text by itself:
Does not work well. The resulting file name actually becomes “mydemo.txt*.txt”.
So formatted text once again comes to the rescue. We preface the new file name with a Ctrl-a (select all) followed by Delete to clear the field:
Note that there are other ways you could achieve the same goal. This is just an illustrative example.
All that remains is to hit the Save button. Any old Windows Keyboard warrior will know that underlined text character in a Windows dialog box can be invoked using Alt+[underlined key]:
In this case, Alt+S will save the file. So we go ahead and invoke it:
If you run this complete example, you will now have a file in your designated folder called “mydemo.txt”
Handling exceptions
Of course, if you run our scenario twice, you will encounter another message dialog telling you that the file already exists:
It is important to handle the kinds of exceptions as otherwise your automation project may become unreliable. In our case, we wait for the “Already exists” dialog to appear, with a timeout telling us if we need to handle it or not:
In the above example, the file will simply be replaced if it already exists.
Interference
A significant problem with Windows automation is interference. Essentially the automation rules are sending keystrokes and mouse clicks to applications. If someone (a human being mostly) tries to also enter keys or click the mouse at the same time, the automation is likely to fail. For this reason, automations should always run on a dedicated machine with no other activity.
When running a cluster of Tomorrow Software Server instances as a REST service, you need to consider the avoidance of interference traffic impacting automation requests. For example, a load balanced clustered web-based service may have heartbeat health check request pings to confirm service availability, or other unwanted requests; such traffic needs to be filtered (not necessarily blocked) but prevented from reaching the automation rulesets.
Parallel processing
A final issue to be aware of when running automations is that multiple concurrent automations also interfere with each other. For this reason, the best approach is to queue automations if they need to run on the same server. The easiest way to do this is with the “Launch Queued Process” rule:
This rule will ensure that X Engine wide, only one automation rule set will run at any one point in time. However, the rules are not held up whilst these automation requests are queued.
If you need to wait for an automation process to complete before continuing, the best rule to use is “Wait for Queued Process”. This rule will place the automation request on the queue and will not continue until the automation has completed.
Windows automation as a service
Scaling up
Given you can only run one automation process at any one point in time, you may need a load balanced setup to share automation requests over multiple servers.
The best way to do this is by wrapping the automation request into a REST service and deploying it to multiple virtual server instances behind a load balancer in round robin mode.
Using this approach, the load balancer will find the next available server and distribute the load evenly.
A core virtual server instance should be created so that it can be cloned whenever more capacity is needed.
Set-up
The default BaseApp Tomorrow Software Server service instance is suitable for running as a REST service, please refer to the instructions file Read me.txt located in Tomorrow-Software-Server-10.0.0]/BaseApp/ for set up. Also refer to the Product Reference.pdf section entitled “Removing other unnecessary components” to remove the Tomorrow Software Console and other unwanted demo applications and server instances not required.
Please note that Windows Automation instances cannot be run as a service. They must be started using a bat file in the Windows startup group.
Example to run at start up: Windows Server 2012
Modify Local Group Policy Editor > Administrative Templates > System > Logon > Run these programs at user logon
Enable this option, press show and enter the following value, and apply/OK to this configuration.
Where c:\Tomorrow\Tomorrow-Software-Server-10.0.0 is this example directory path.
When using this option you need to edit the default Tomorrow.bat file to add the following three lines before cd server to accommodate the start up directory path as follows, once again where c:\Tomorrow\Tomorrow-Software-Server-10.0.0 is this example directory path.
Active Desktop using RealVNC
A significant limitation with Windows automation (like most GUI automation tools) is that it requires an active desktop to run. So, when you log out of any remote desktop connection or lock the computer, automation is paused/stuck until you reconnect. It is therefore impractical to retain open RDP connection for multiple Tomorrow Software Server instances when running as a REST service with high availability demands. The following is a working example to overcome this limitation.
Example: In Windows Server 2012 set the Turn off the display option to Never in Control Panel Power Options.
You still need a way for the remote server to have it’s head/desktop to be unlocked and active. The best way to do this is to use the VNC protocol rather than RDP. There are numerous VNC software (server and client) available that are also free and/or open source.
For this example, we have tested with RealVNC - https://www.realvnc.com/download/vnc/latest/ VNC for Windows version 5.2.3.
Please ensure you refer to Licensing terms https://www.realvnc.com/products/vnc/ as a License key is required to install and use Real VNC for your environment and organisation.
RealVNC VNC Server uses modes to provide remote access to computers in different circumstances, to meet different needs.
VNC Server needs to install on the Tomorrow Software Server instance, and VNC Viewer needs to be installed on a ‘controller’ server.
Given the Tomorrow Software Console server will have access to the server instances, this server is a good candidate to run VNC Viewer, although a dedicated server with access to the instances can perform this connectivity too.
VNC Server installs and runs on default port 5900, so ensure any security group policies have been amended to permit connection using this port, together with ports that are running the REST service. The BaseApp to use as a REST service runs as default on port 10001 as defined in the rulesengine.properties settings.
RealVNC installation notes
During the standard RealVNC installation process, ensure you select the appropriate components for your REST service instance and Console Server or controller.
There is also an install option to add an exception to the Windows firewall during installation, but if you are still experiencing connection problems you may still be required to inspect your server firewall settings.
Before starting the VNS Server service, it’s useful to know all VNC applications are controlled by VNC parameters, set to suitable default values for most users out-of-the-box.
Please refer to this link for RealVNC parameter names reference information. https://help.realvnc.com/hc/en-us/articles/360002251297-VNC-Server-Parameter-Reference-
The easiest way to set the authentication scheme and credentials for the VNC Viewer controller in order to connect to VNC Server is to start the VNC Server (User Mode) desktop application.
For example, set the simple authentication scheme using VNC password in the VNC Server – Options > Users & Permissions option as follows.
Once the authentication scheme and access credentials have been set, and Licensing updated if required, ensure you stop the running VNC Server (User Mode) by pressing the More button, followed by Stop VNC Server as follows.
The parameter IdleTimeout specifies the number of seconds to wait before disconnecting users who have not interacted with the host computer during that time. The default value for IdleTimeout is 3600 seconds, so you need to set this parameter to 0 in order to never disconnect idle connections. You need to add the IdleTimeout parameter in Windows Registry Editor when running VNC Server as a Windows service as follows.
Using Registry Editor, navigate to
HKEY_LOCAL_MACHINE\Software\RealVNC\vncserver.Select
New > String Valuefrom the shortcut menu and createIdleTimeout.Select Modify from the shortcut menu, and specify appropriate Value data, 0.
With VNC Server successfully installed and parameters set, amend the VNC Server service with Startup Type set to Automatic.
Also, the Allow service to interact with desktop option must be checked as follows.
With the IdleTimeout parameter set to 0 as a minimum, restart the server and START the VNC Server service.
You are now ready to connect to the Tomorrow Software server instance running VNC Server from the controller running VNC Viewer.
Connect to the Tomorrow Software Console server (or controller) using a standard Windows remote desktop connection; install the default VNC Viewer components, and start the VNC Viewer application from the desktop shortcut.
The VNC Viewer application will then prompt to enter the host name or IP address of the REST Service server instance running VNC Server.
With the ‘Let VNC Server choose’ option for encryption selected you will prompted as follows for the password set on VNC Server earlier in the VNC Server – Options > Users & Permissions option.
If connection is successful the VNC Viewer will launch a connected window to the server, at which point you can login using your Windows user credentials, and you can proceed to repeat the process to make a VNC connection to all VNC Server instances if operating in a scaled cluster.
Even with VNC Viewer closed, because it is simply a relay of the host’s screen to your desktop (works differently than RDP), when disconnected, it just stops relaying, and that is all. The relay works like a splitter connection, both the local head/monitor has access, and the VNC Viewer has access.
By this design, VNC will continue to retain an active desktop even though you’re not connected over VNC, as long as the host desktop is logged in and not locked.
The environment – Tomorrow Software Console server, and multiple connected REST service server instances as defined in the Tomorrow Software Console server definitions are now ready for use; the VNC Viewer windows residing on the Tomorrow Software Console server (or controller) can be closed, and the remote desktop connection closed, and desktop will remain unlocked and active.
Last updated
