Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Rule sets can be compared to a visual programming language. What you can do with the rule sets and custom rules is almost unlimited. In the following sections we will cover rule sets and the rules editor in more detail.
As already shown in the introduction, you can add documentation information to each of your rule sets. As rule sets are often interlinked, we strongly recommended doing this.
The file name for your rule set should be a single word (no spaces). It can be mixed case. To rename a rule set, simply change the name and click on rename.
To avoid more than one person working on the same rule set at the same time, the rule editor employs a locking mechanism. Once a rule set is opened in the rule editor, another user can only open it for viewing. You will receive notification messages if you attempt to open a rule set for editing in more than one window at a time.
The “Maintain Rule Set” page (shown above) will show which user currently holds a lock.
The rules editor is designed to be easy to use. There are however a few tricks that will make using the rules editor even easier. The following covers a few of these tricks.
If you need to move multiple rules around in unison or select them so that you can copy them to another rule set, hold down the CTRL key whilst clicking and selecting rules. Alternatively, you can hold down the mouse key and drag a rectangle around the rules you wish to select.
Rules can be cut, copy or pasted within the same rule set or between open rule sets by right-clicking whilst the rules are selected. To paste the rules into a new position, right-click on the canvas where the rules should be placed and select Paste.
Note: Not all browsers support the right-click feature. For this reason, edit options can also be obtained by holding down shift and left-click.
If you mistakenly connect one rule to another, you can remove the connection by right-clicking on the chain point and selecting Disconnect.
Many of the rules in the system take a variable, text or a number as a parameter. Generally speaking, variables may only contain letters, numbers and the characters: ‘_’, ‘:’ and ‘.’, and they must start with a letter.
There will be times where a variable can be confused with a text literal, and for that reason text literals should always be enclosed in double quotes. For example: ABC is a variable whereas “ABC” is the text ABC.
Numbers on the other hand are unambiguous and can just be keyed as numbers.
When entering a CSV list of values, there is no need to enclose the entire block in double quotes. Since CSV text has commas in it, it will automatically be detected as a list of string elements.
There is no need to close a rule set to test it. You can keep multiple rule sets open in multiple windows, deploy your rules, test and then return to the already open windows to continue editing. Click the save button rather than the close button to stay on the page.
Several of the standard rules make use of a string matching feature known as regular expressions. Books have been written about regular expressions and it is beyond the scope of this manual to cover more than the very basics.
In its most basic form, you can use regular expressions to see if a certain text is available within another, to count characters and to look for certain pieces of text at certain positions within words.
An example of a regular expression would be: (ab|cd)
This expression will check if a text contains the character sequence “ab” or the sequence “cd”. So both “baby” and “lcd” would be a match.
The following tables list some of the common uses of regular expressions and how they can be used to validate text.
Lists and data sets provide an efficient way to work with keyed data that needs to be stored either in memory or a database.
Lists are capable of storing normal variables, lists or data sets, allowing effectively for multi-dimensional arrays.
There are two types of lists: Regular lists and fixed sized lists. When you insert a value into a list, the list will automatically be created as a regular list if it hasn’t already been created.
Fixed size lists are extremely useful for memory caching. Elements inserted into a fixed size list will stay there until the maximum size of the list is reached. Once that happens, the oldest element that has not been accessed (read or updated) will be removed from the list.
Elements inserted into a list must have a fixed key. If a new element is inserted with an existing key, the existing element will be replaced.
It is possible to create global lists by creating the list in a startup rule set and then setting it as a global variable. When the global list is read from a normal rule set, any changes made to that list as a local variable will directly affect the global list. This provides a means for caching data at a global level.
Data sets provide a way to create a collection of correlated data. For example, you can define a data set called “Fruit” with the properties Name, Color and Shape. Once you have defined a data set, you can create instances of it in a database or in a list. For example: Apple, Red and Round.
The X Engine will automatically handle the correct storage of the data set in a database and properties of the data set can be added and/or removed at any time in its lifecycle. So, if at a later stage you need to store another property in your Fruit data set, you can simply add it to the definition.
Data sets should be defined in a start-up rule set and can only be defined once within the life cycle of a deployment.
Data sets can optionally contain a number of lists. However, lists stored within a data set may only be single dimensional (you cannot have lists within lists).
Once a data set is stored within a database or a list, you can read it by key, delete it, update it etc. To update a data set within a list, you simply create it again with the same key name.
A quick way to get started building rules for a new web application or for stress testing is to use the rules wizard. The rules wizard uses live test data to extract URIs visited and build a structured collection of rule sets.
The rules wizard creates a large number of files, so we strongly recommend that you create a new repository to write the new files to.
Before using the rules wizard, you must first deploy and start either the RuleWizardConf or the StressTestConf configuration found in the Rules Wizard repository to a test server that is protecting the target application (alternatively you can use the zero installation test method which is covered later in the manual). Once done, simply begin navigating the various components of the target web application.
Once you have visited all of the pages you wish to cover with your initial rule set, return to the console and go to the server status screen:
In this example the Qwerty demo application was chosen and there are 19 test records ready to be processed.
Now click on the "New rules wizard" button. You will be presented with the following page that controls the wizard:
At this point you can have the rules wizard create a filter for the rules that automatically exclude static content. It is a comma separated list and you can easily add new elements.
Make sure that you select either New rule set or Stress test, depending on your requirements.
When you have selected an appropriate repository (in our case, we have use the repository ‘Name’) and reviewed the exclusion list, click on "Create".
After a brief pause, the X Engine will write out a complete configuration and collection of rules. The following pages show how these rule sets are structured for a new rule set:
In keeping with best practice for rules writing, the rules wizard always creates a "Load" rule set. This contains rules that are generic for all URIs. The name of the page will be the name of the repository followed by the word "Load".
This rule set will first check for malformed HTTP requests and, if found, will reject them. Subsequently it filters out static content, adds a tracker rule and then proceeds to the main rule set.
The main rule set page contains a basic structure that determines the URI being visited and then uses a switch to re-direct to a rule set covering that URI.
Each one of those rule sets, in turn, are blank, but are already created and ready to have rules added to them. The new rules wizard creates a quick foundation to get you started with writing rules for your application.
When the rules wizard creates each of the blank templates it includes sample information of the fields gathered and their values in the page description:
You can use this information to identify the fields available to your rules.
As described above, an alternative use of the New Rules Wizard is to create a set of rules for stress testing an application. To do this, you must first deploy the StressTestConf configuration from the Rule Wizard repository to the web application on the browser proxy.
Once deployed, you should work through the application to be stress tested, step by step. Try to complete pages as normally as possible, making sure not to pause unnecessarily during the process, as wait times are recorded.
During the new rules wizard creation, select Stress test instead of New rule set.
The following pages show how the rule sets are structured for a stress test:
In keeping with best practice for rules writing, the rules wizard always creates a "Load" rule set. This contains rules that are generic for all URIs. The name of the page will be the name of the repository followed by the word "Load".
This rule set allows you to set up a multitude of things. First of all, the target server identity, and also the user agent to use. It is very common to modify the Load rule set to pick up randomized values for elements such as users. The following shows a modified Load rule set to include diversified User ID and Password configurations for a Qwerty stress test:
In this scenario, the users and their passwords are picked from a CSV file of valid entries. The settings for the CSV Line Picker rule are as follows:
It is important to notice how the THREAD_NO variable is used to pick a line in the CSV file. The THREAD_NO is incremented for every stress testing agent and should the number exceed the number of entries in the CSV file, the rule will start reading from the beginning again.
The main rule set page contains a basic structure that determines the URI being visited and then uses a Number Sequencer rule to re-direct to a rule set covering that URI.
The wait rule set page is used to define wait times. This delay can be fixed, or it can be a function of the recorded wait time during rule set creation.
The following shows a wait time delay:
The wait time properties can be set as follows:
The default for the percentage random is zero, but it can be randomized to create a more realistic user load.
An alternative is to either remove the wait time completely or insert a fixed delay:
The fail page is used to define any action to be taken if a stress test page invocation fails. The default is to abort the flow and stop the thread:
Each page that is detected will be assigned a unique page name and will be given a unique rule set. The following shows the main page in the Qwerty application:
In this case, the page is a GET and the invocation is relatively simple. For POST requests, the structure looks slightly different:
In this case, the recorded POST variables are set up in a single rule. You can override these POST variables to use pseudo random values (as shown previously in the Load rule set).
The individual pages are where you should include test data, and possibly review the response data from significant pages to ensure that the stress test is progressing well.
This section covers how to simplify the rule sets and keep them readable by dividing them into smaller blocks. You do this by using inclusions and exit points.
To illustrate this, please find below a rule set that has the ability to do some basic fraud prevention analysis on incoming HTTP requests:
If you were to copy all of the rules into your rule set every time you needed to perform these checks, you would end up with very large unwieldy rule sets. To avoid this problem, Composable Architecture Platform gives you the ability to include (or embed) existing rule sets into new rule sets as if they were just another rule. The following example illustrates this:
The BrowserInfoCheck rule set as part of the Qwerty, AccountMain rule set is behaving just like any other rule. This was done by dragging it from the Rule Sets folder in the editor and onto the editor canvas. The key to making this approach work is to use Rule Set Exit rules in the embedded document. The rule set exit rules in any given rule set determine what chain points will be available for that rule set when it is embedded into another page.
If you look back to the BrowserInfoCheck rule set, you can see the OK, Error and Warning exits.
There are no limitations on the number of rules sets that can be embedded within one another.
Embedded rule sets have many advantages:
Complex rule sets can be built for a given function, keeping the logic central
You can share rule sets between functional areas without the need to copy
You can have experts build rule sets for novice users
If you have rule sets that perform a unique discrete function, it can be advantageous to wrap them up so that they look like a new rule that can be included at the repository level. You can specify the group, name and a set of input and output parameters for that rule. This makes the rule set show up in a specific group and parameters subsequently show up as properties of the included rule set.
To define parameters, go to the rule sets “Rule Info” tab:
Enter your group and name and click on the Add Parameter button to create a new parameter:
The Parameter Name defines the name of the variable used internally by the rule set for this parameter.
The Parameter Label fields determine what is displayed as the property description when the rule is included.
The Parameter Type determines how the parameter is used. There are 3 options:
An input parameter is set when the rule set is called. So, a top level rule set could use the variable “Name” within it and you could define a parameter equally named “Name”. Once a rule set has parameters, it can no longer “see” or access variables outside its own boundaries.
So, at the top level the variable “Name” could contain the value “Smith”, yet a value of “Jones” could be passed to the parameterized rule set (also with the variable “Name”) and the embedded rule set will only see the value “Jones”.
Input parameters to a rule set can be constants (in quotes), a variable or a value from a drop-down list. To make the input parameter a drop-down list, supply a CSV list of values in the Set values field.
An output parameter is set when the process flow exits the rule set at the top level. So, when the embedded rule goes down a chain point or returns from its function, the original set of variables will come back into effect and only variables set against the output parameter will be transferred from the embedded rule set to the top level rule set.
Output parameters can only be variables.
I/O parameters are variables that are read on input and set again on return.
I/O parameters can only be variables
To illustrate how parameter can be used, consider the following parameter definitions against a rule set:
And the following rule set:
The above rule set will simply add to or subtract from a value.
When this rule set is included into another rule set, the following parameters become visible:
The user can now pass any give variable name to this rule set and the included rule set will perform the expected math on it (using its own internal variable names) and return the result in the passed variable.
This approach enables the building of generic rules that are guaranteed to not have a variable name overlap with other rule sets.
You can now go to the repository level and select the individual rule sets that should be exported from that repository:
When another repository includes this particular repository, the only rule set available will be names “Quick Add” and will be found in the “Math” group as specified on the Rule Info tab.
There may be times when you wish to test rules against a site without affecting that site in any way and without installing anything. Typically, this applies to an initial first trial of Composable Architecture Platform, tests against websites external to your organization, or for convenience where you are waiting on the installation of a built-in proxy or inline filter.
The zero installation rules testing approach relies on the forwarding proxy and browser proxy built into the console application. The following example will reiterate the sample from the earlier section, Quick product introduction using Google's search page.
The first step in this example is to prepare the browser proxy so that all traffic to and from Google is successfully routed via the Composable Architecture Platform Proxy Server. This will give you visibility of the data and provide all of the information needed to manipulate it.
Many browsers have in-built security features to prevent user access to websites whenever there is an untrusted SSL certificate, and will block the incoming request without exception. In this example because it is not possible to install Google’s SSL certificate to the Proxy Server, to overcome this set the Server Definitions for the Proxy Server located in Administration by clicking on the Proxy Server as follows.
Click on the Forwarding tab and set the Request redirection properties for Google as follows. This example is for a UK IP address request that follows the redirect of Google.com to Google.co.uk based upon the IP geolocation from the originating browser.
The first line entry is for example format use only and has no impact on the Proxy Server:
http://thishost>http://thishost:8001
http://google.com>https://google.com
http://www.google.com>https://www.google.com
http://google.co.uk>https://google.co.uk
http://www.google.co.uk>https://www.google.co.uk
Once the redirection settings have been input, scroll to the bottom of the page and save the modified Proxy Server definition.
The Proxy Server will now successfully route the http to https protocol redirection and allow the browser to access the website even without a correct SSL certificate.
To prepare for the test, start by deploying and starting the BasicWebTrial configuration from the Product Trial repository to the Proxy Server. This will allow you to see the data flowing from the website.
Depending on the type of web browser being used, you should now set the manual proxy configuration to tell the browser to go through the Composable Architecture Platform built in proxy.
The following shows how to configure the browser proxy in Firefox Quantum 60.0 on Windows 2012 Server:
Select Options then click on Network Proxy > Settings:
Set the proxy options as shown below:
Please note that if you have changed the ports for the browser proxy in the server settings for the built-in proxy, then these settings may differ.
Now go to the website you wish to test (in this example: http://google.com). The data will quickly start flowing to the console:
You can now start writing rules, deploy them to the Proxy Server, and test them. The only place this will have any effect is for transactions coming from the browser with the browser proxy set. For example, you could create a simple content rule that replaces the Google logo, adds a link to your intranet, or scans for bad language used in searches. The possibilities are endless.
By default, the built in Proxy Server settings will only accept connections from the local host (127.0.0.1). This restriction is in place to ensure a given Composable Architecture Platform Server installation does not become an open proxy. The various server settings can be controlled via the server definition of the Proxy Server.
Each configuration is divided into 6 tabs with settings. In this section we cover these tabs in detail.
The General tab contains all of the basic information about a configuration.
The file name should be a single word (no spaces). You can rename the configuration by changing the name and clicking "Save".
The description provides an easy way for other people to understand what your configuration does and serves as basic documentation.
Every configuration must have a rule set. This is a mandatory field.
Content rule sets allow you to manage specific content (new pages, images and so on), that you can introduce to the application. If the target server definition has a context path set for serving content files, you can optionally check the Use server context path checkbox to use that as the defined directory path.
If you have rules that should be executed before your main rules, or immediately after they have all completed, you can define them in the configuration.
Please note that if you set a startup or completion rule set, the X Engine will be restricted to running in a single thread to ensure the correct order of events.
Modes are named collections of rule sets and content rule sets that can be used to replace the default rule sets that are running. For example, if you wish to take a website offline for maintenance, you can create a “Maintenance” mode and assign it to rules that display a maintenance page instead of your normal website.
The input source tab provides details to the X Engine about where input is coming from and how to deal with it.
The server can be either Production, Multi-Protocol, or Test. This makes the configuration target a specific server type and determines which input adaptors are available to select from.
A critical part of the configuration is the input adaptor or “source of data”. The options available depend on the type of server selected. As a general rule, the file name or URL being processed will be made available by the input adaptor as the variable URI (Uniform Resource Identifier).
For file names, this includes the full file path in the file system dependent format.
Input adaptors are frequently added via extensions. At the time of writing, the following input adaptors are available by default:
Execute a load test against a server: This input adaptor is only available for production servers. It allows you to take a stress test rule configuration generated by the “New rules wizard” modified to suit the application and use that to generate a load against a website. You can control ramp up times and total threads as well as think times.
Please see the “Using the New Rules Wizard for stress testing” section in this manual for more information.
Process multi-protocol input: This input adaptor is only available for Multi-Protocol servers.
It allows you to take input from any protocol defined within the administration section of the console and control the input, proxy and output of that protocol.
Protocols supported include (but are not limited to): MySQL, DNS, Telnet, FTP, ISO8583 and SMTP.
Transports include: SSL, TCP and UDP.
Process a single CSV file: This input adaptor is only available for test servers. It allows you to define each column in a CSV file that you wish the server to process. The file must be present amongst the test data files uploaded to the console.
Process a single multiline CSV file: This input adaptor is only available for test servers. It allows you to process CSV files that have records spread over more than one line. Typically, this could be a file that looks as follows:
To process the above file, you would need the following definition in the Input Fields tab of your configuration:
The break column (BREAK_COLUMN) defines which column number is used to identify a unique record. The record column (RECORD_COLUMN) defines which column number contains the record type.
Each individual field for an entire record is then defined as a field name, with the label indicating which record and column number that field can be found in.
Process a single identifier delimited file: This input adaptor was designed to rapidly traverse files that contain somewhat structured data, where each piece of data is preceded by a recognizable identifier, and all of the identifiers are in the same order (although missing identifiers are tolerated).
This adaptor relies on the data in the file containing a format where an identifier can be used to spot breaks in the data. The following example illustrates how this adaptor can be used:
Sample data that could be provided to this adaptor could be:
The first label listed in the input fields for the configuration MUST be the break for each new record. A record can be on more than one line in the file.
Processing the above shown data would result in two records, the first with the variables set as follows:
The second record contains:
Strings are always being trimmed of leading and trailing blanks but can contain more than one word. If you have identifiers in the file that you wish to ignore, you must still specify them in the list or they will be considered part of a context of a previous identifier.
Process a single XML file: This input adaptor is only available for test servers. It allows for an XML file to be processed by the X Engine. Each XML tag in the file (and its attributes) will be converted to a unique variable name. For example, the following XML document:
results in the following variables being generated:
It is important to note that all tags below the root tag will have a counter attached to them to ensure uniqueness. This is what results in the “_1“ being added to the “name” tag in the example above.
If more than one “name” tag is present, the conversion will be as follows:
Which results in the following variables being generated:
As this process can result in some rather long variable names (especially when processing XML documents such as SOAP requests), the use of the Alias rule is encouraged to simplify rule writing.
Process all CSV files in a directory: This input adaptor is only available for production servers. This input adaptor will look for files in a folder/directory. When one is present, it will process it and then delete the file. Each field in any supplied CSV file must be defined in the configuration.
Process all identifier delimited files in a directory: This input adaptor is only available for production servers.
This input adaptor will look for files in a folder/directory. When one is present, it will process it and then delete the file. The data within the supplied file is converted into unique variable names as outlined in the “Process a single identifier delimited file” adaptor.
Process all multiline CSV files in a directory: This input adaptor is only available for production servers.
This input adaptor will look for files in a folder/directory. When one is present, it will process it and then delete the file. The data within the supplied file is converted into unique variable names as outlined in the “Process a single multiline CSV file” adaptor.
Process all XML files in a directory: This input adaptor is only available for production servers.
This input adaptor will look for files in a folder/directory. When one is present, it will process and then delete the file.
The tags within the supplied XML document are converted into unique variable names as outlined in the “Process a single XML document” adaptor.
Process free format test data: This input adaptor is only available for test servers.
This adaptor is specifically designed to receive data from a file generated by the “Test Data Creation” rule (TST files). There is no need to define any input fields as the data within the file are composed of a field definition list as well as a data value list for each record.
This adaptor is designed to process data from production servers that process web application inputs with the actual variable names changing for each request.
The test server will be able to emulate what happens on an actual production application server without the requirement to simulate anything in a test environment.
This adaptor is also useful for pre-testing any new rule set to evaluate the impact of installing it into production.
Process on heart beat: This input adaptor is only available for production servers.
This input adaptor is used to process the same rule set at regular intervals. You can specify the delay between each run in ms.
Process once and stop: This input adaptor is only available for production servers.
This input adaptor will run the rule set once upon startup and then stop. This is predominantly used for testing rules.
Receive input via HTTP POST: This input adaptor is only available for production servers.
This input adaptor is designed for high-speed processing of a specific HTTP POST (for example from a known JSP or HTML page). Each field that the X Engine is expected to process must be defined in the configuration, just as if the input came from a CSV file.
The field names listed must be the same name (case sensitive) as they appear in the form post from the HTML that submits the request.
It is important not to confuse this adaptor with the “Receive web application input” adaptor, which is slightly slower but much more flexible.
Receive web application data: This input adaptor is only available for production servers.
This input adaptor is probably the most flexible, but also most complex. It is capable of receiving data from any HTTP request, be it a GET or a POST, and translate it into variables that can be used by the X Engine.
The adaptor understands and translates standard HTML, XmlHttpRequest (AJAX) and SOAP requests as long as the appropriate content type is set in the HTTP request.
For HTML POSTs and GETs, the URL parameters and form fields are translated directly into input variables, with each variable name matching the corresponding parameter or field name. For XmlHttpRequest and SOAP requests, the tags within the supplied XML document are converted into unique variable names as outlined in the “Process a single XML document” adaptor.
This particular input adaptor allows you to enforce some web application security settings:
For HSTS:
At the very minimum you must provide a ‘Max age’ value in seconds. The most recommended value is 31536000 seconds (one year).
Optionally you can check the box to include sub domains.
Note: Any of the above settings that modify cookies require that you are running on a Servlet Specification 3.0 or later web application server. For the standard installation that means Jetty 9 or later.
After the selection of an input adaptor in a configuration, there are a number of fields specific to that adaptor. The largest difference is typically between test and production adaptors, we will show two examples here:
The above scenario is for a production input adaptor that processes all files "dropped" into a given directory. The input adaptor will poll that directory and whenever a new file is added it will be processed and then deleted.
The additional fields are as follows:
Selecting this option causes the X Engine to always collect test data by default. You can also start collecting test data on demand using the server status view.
This is the maximum number of records that the X Engine will keep in memory. For data with a large record size, this value should be set properly to avoid retaining too much memory.
This is the sub-directory from the home folder where the X Engine will look for files. For other input adaptors that do not use files (such as the web application adaptors) this can be a different field name providing different information.
Selecting this option will cause the X Engine to automatically start when the server is started. There is no need to selectively click the start button.
Test servers behave differently to production servers in that they always take a single file as input data, process that file and then stop. The following is an example of the settings for a test input adaptor:
Test data is the name of the file to process. This file must be in the test data section of the console tree in the same repository as the configuration.
The testing flags are used to control how the X Engine interacts with the environment around it.
The remaining input source fields are generic. The following is a list of their meanings:
This flag determines if messages written to the console (via any of the List rules) are also written to the system's standard out log.
Enable Debug Mode turns “List Variable for Debug” rules as well as “Exit” rules with “List Variables on Exit” set to “Debug mode”, on, so that all variables will be listed at selected points throughout the rule sets.
The Fail Open setting determines how the X Engine deals with a fatal error. If selected, the X Engine will automatically stop and let all normal traffic proceed transparently should a fatal error occur. If unselected, the X Engine will attempt to recover from the failure and continue running.
This setting is used to control how the X Engine detects infinite loops. Effectively every connection (chain) between rules has a counter built into it. When the number of chain events reaches the count set here, the X Engine will consider itself looping and will terminate to avoid impacting other services.
This setting determines how much performance data is collected as part of the X Engine execution. Please see the performance data section for more information.
Input fields are used for a variety of purposes. They can be used to identify column settings for input adaptors and also to determine global settings that can be changed at the configuration level without changing any rules. The following shows such an example:
When the X Engine is running, it is possible to set global fields. These are fields that can be accessed by the X Engine at any stage and are not dependent on input from other sources. Global fields can be changed during the execution phase of a rule set, allowing you to potentially alter the flow of rules, set different thresholds or check for different conditions.
The following shows an example of defined global fields:
It is important to know that global fields are persistent. This means that the default value set in the configuration only applies for the very first time a global field is set. After that point, the global field retains its set value, even after the X Engine is restarted.
The field name is the global variable name set when the value is changed. The Label is the label that the user sees.
The field type for each global field is important, as are the allowed values. You can set fields types as follows:
Text: This creates a simple text field that can be changed. The allowed values have no effect.
Number: This creates a simple text field for numeric input that can be changed. The allowed values have no effect.
Switch: This creates an on/off style switch that can be changed. The allowed values represent the values set for the ON condition, followed by the OFF condition.
Slider: This creates a slider that can be changed. The allowed values represent the min value, max value and optionally a third value representing the increment. Only integers can be used.
List: This creates a drop-down list of values that can be picked. The allowed values represent each of those selections.
The following shows an example of how the above defined values are displayed in the server settings change function:
Many of the rules available in Composable Architecture Platform are capable of accessing data in databases, either locally to Composable Architecture Platform or externally stored somewhere within the network.
As a user, you will need to know the name of the database that you want to connect to, and in some cases also the table and schema names.
Configurations need to list all of the databases that the rules within them are capable of accessing. This allows the deployment system to provide
Additional information to the Composable Architecture Platform servers about how to access those databases at a technical level. The databases themselves are normally defined by a system administrator.
The following shows a configuration of a sample database:
You must enter the database name and the type of database (driver). The list will vary depending on the types installed on the network.
If you are writing rules that may be used on different systems where the database names may differ, you can use a database alias name. The database alias name in the rule sets will be mapped back to the database name defined in the configuration.
There may be times where you wish to access to a database with a schema name that varies between test and production systems. To allow this, you can override the schema name in the configuration. If you leave the schema name blank, the default value configured by the system administrator will be used.
The system defines where the actual database is located. You can combine this with the defined servers to allow JDBC drivers to connect to any given location.
You have the ability to set a list of rule sets that execute at a given time interval. These rule sets are independent of any input data and simply run on a repetitive cycle. There are two types of cycles: Delay and Real Time:
Delay timers simply execute the rule set, then sleep for the delay period and then execute again.
Real time rule sets will run at a precise interval, regardless of the time it takes to execute the rules.
An example of a timer setting is as follows:
In the above example, the OnTimer rule set will be executed, then pause for 30 seconds and then repeat.
Please note that the data object used within the timer stays the same. This means that you can set variables within the timer rule set and use those same variables the next time the rule set executes (for example variables for a counter).
Note that for web applications, the timers will not start until the first real transaction has been processed.
For each configuration you create, you have the option of producing a complete documentation set. To create this, click on the Document button:
Then select the target server and click on the Document button.
A pop-up window will appear that lists a complete view of the configuration, the selected server, JDBC drivers, databases, rule sets, data files and so on.
Selectively you can include the actual details of the contents of data files and content files. For each of the respective files, simply tick the check box as shown below and click on Save.
IMPORTANT: The output in the pop-up window is very browser dependent and the quality of the results may vary. We recommend using the print preview option in individual browsers to see the final result.
At the time of writing the following conditions applied:
Firefox 11.0 produced a fairly faithful representation of the intended output but was slow.
Internet Explorer 9 produced a reasonable representation of the intended output, was faster than Firefox, but did not always respect page breaks.
Chrome 18 produced a terrible graphical look as it does not print background images in pages but was otherwise fast and true to page formatting.
Please see the “” section in this manual for more information.
Preload is a method whereby the most common browsers will load a list of sites that MUST use HSTS. Google maintains a list of sites that are preloaded as requiring HSTS and that list is used by the Chrome, Safari and Firefox browsers. To have your site registered as preload, you must apply here: . Google will verify that you have the preload flag set against the HSTS header before adding you. If you do not have this flag, Google will reject your application to be added to their list.
Character(s)
Description
Example
Matches
Does not match
[]
Matches any of the enclosed characters
AB[cC]
ABC, ABc
abC, def
[^]
Does not match any of the enclosed characters
AB[^cC]
ABD, ABz
ABc, ABC
[a-z]
Range
AB[C-F]
ABC, ABD, ABF
ABG, ABc
.
Any character except newline
N.
N1, N2
N
\d
Any digit between 0 and 9
N\d
N1, N2
NA
\D
Any non-digit character
N\d
NA
N1, N2
\w
Any letter, digit or underscore (equal to [a-zA-Z0-9_])
ab\wd
abcd, abCd, abad
Ab/1d
\W
Any character other than a letter, digit or underscore (equal to [^a-zA-Z0-9_])
ab\Wd
Ab/1d
abcd, abCd, abad
\s
Any single white space character (space, new line etc.)
ab\sde
ab de
abde
\S
Any single non-white space character
ab\Sde
abcde
ab de
Character(s)
Description
Example
Matches
Does not match
*
Zero or more occurrences
Tes*
Test, Television, Tess
Two, Trees
?
Zero or one occurrence
Colou?r
Color, Colour
Colouur
+
One or more occurrences
Tes+
Test, Tess
Television
{x}
Exactly x occurrences of the previous character
Hel{2}o
Hello
Helo,Helllo
{x,}
x or more occurrences of the previous character
Hel{2,}o{2,}
Helllloooo
Hello, Heloo, Howdy
{x,y}
Between x and y occurrences of the previous character
Hel{2,3}o
Hello, Helllo
Helo, Hellllo
Character(s)
Description
Example
Matches
Does not match
()
Group together
(abc){2,}
abcabc
abc
x|y
Either x or y
(abc|def){2,}
defabc, abcabc, ghjabc
abc
Character(s)
Description
Example
Matches
Does not match
^
Match at beginning of string
^hello
hello world
say hello
$
Match at end of string
hello$
say hello
hello world
\b
Match at beginning or end of a word
\bhel
say hello, hello world
Chellos
\B
Match inside a word
\Bhel
Chellos
say hello, hello world
(?=x)
Only if followed by x
ab(?=c)
abc
ab, abd
(?!x)
Only if not followed by x
ab(?!c)
ab, abd
abc
Record ID | Record Type | Value Column 3 | Value Column 4 |
12345, | R1, | John | Doe |
12345, | R2, | Melbourne | Australia |
23456, | R1, | Bob | Smith |
23456, | R2, | London | United Kingdom |
34567, | R1, | Jane | Doe |
34567, | R2, | Auckland | New Zealand |
Flag | Usage |
Update Internal Data | If selected, the rules in the X Engine will update data with tables that it can directly access. This includes the internal case manager. |
Update External Data | If selected, the X Engine will write data to external systems that are not database connected. This could, for example, be an external case manager that receives cases via a web services call. |
Send Alerts | If selected, the X Engine will send alerts such as emails, SMS messages and other forms of external messages. |
There are two ways that Composable Architecture Platform can be used in a Web Services environment. It can operate as an inline filter, just as it would for a normal HTTP request intercept.
Alternatively, it can be configured to act as a Web Service itself by installing a SOAP servlet into a new (or existing) web application and placing the filter in front of it. To facilitate this, a default servlet is provided named:
software.tomorrow.engine.server.RulesEngineSOAPServlet
This servlet will always provide a valid SOAP response regardless of the input provided to it. The precise response is as follows:
The Composable Architecture Platform inline filter and built in proxy offers the ability to wrap and control the behaviour of a Web application. Usually this is done in conjunction with the Receive Web Application data input adaptor.
Rules in this environment can see all of the post data from the Web browser. By using rules such as the HTTP Request Tracker, HTTP Header Reader and other various rules for interacting with the application’s session object, it is possible to do a lot of additional pre-checking of the request being forwarded to the application before it is even processed.
Rules such as the HTTP Request Saver and HTTP Request Restorer also allow for requests to the backend server being temporarily placed on hold, pending actions by the X Engine.
When writing rules, it is a good idea to keep in mind that you may wish to test your rules without the application server being present. To do this, it is usually a good idea to create a Load rule first, which reaches out to all of the elements of the Web application, the request, dynamic databases, and so on to collect all of the data required for a decision before embarking on making that same decision. By doing so, you can insert a Test Data Creation rule at the point where all of the data is ready, thus allowing you to properly test the business functions of your rule set separately on a test server with all of the relevant data being available.
If you need to look at the output from a given request, insert an HTTP Server Execute rule. This will forward the current request to the server for processing and bring back the resulting response in a single variable. You can then scan that variable for information (for example, look for a balance value or a name) using the Scanner rule. This rule allows you to define the text surrounding information that you are interested in, and then extract that information into a variable.
Similarly, you can manipulate the response before forwarding it to the Web browser. As the response is simply stored in a variable, you can use the Replacer rule to modify text within given tags or specific locations inside the response page.
Once you have finished working with the output from a request, it is imperative that you forward it to the Web browser using the HTTP Response rule.
Finally, there may be situations where you wish to simply append additional data to the end of a Web page before it is transmitted to the user. Typically, this would be in the form of JavaScript appended to the end of the page. The best way to do this is to store the data in a data file, upload it to the server, read it into a variable using the File Reader rule, and append it to the server response using the HTTP Response Addition rule.
This section showcases a number of common rule set patterns used when working with Web applications.
One of the first problems always facing you when you start working with a new application is the ability to understand which data flows where under each circumstance. The following rule set pattern, which includes comments, provides a good starting point:
First, the HTTP Request Tracker rule takes care of getting the browser information and adding tracking cookies. Second, a fast lookup to the MaxMind geo location database (which requires a subscription) identifies the origin of the request. Third, the request is written to the console so that you can monitor it in real time. Finally, the data is written to the test data queue so that you can download it for analysis.
This rule is best deployed to a test system during the initial rule writing phase to better understand what variables and pages are available. It is prudent to deploy it initially during a live install to retrieve a large portion of live test data and play it through the desired rule set on the Composable Architecture Platform test server.
A common issue in dealing with application servers that not only serve dynamic content, but also static data (in the form of images, style sheets, fixed HTML), is to filter this content before it even hits the core rule set. This is best done in the Load rule using a Name Splitter and Switch rule as shown:
The name splitter conveniently extracts the extension of the object being requested using the following properties:
The Switch rule operates on the EXT variable. By adding new chain points for each type of static content they are eliminated from reaching the rule set.
It is often a good idea to know the time a user has spent on a form. This is the foundation for filtering and/or slowing down “screen scrapers” or “data extraction” bots. This is an example of what a browser timing rule set looks like:
The basic concept is to first check if a session is present. If not, this rule set does not proceed. In some sites, this may be overly simplistic and may require modification, but for most sites it will be valid.
The rule set then goes through a series of checks. It reads the last time any request was made to the application, timestamps the current request and stores it. If it is possible to measure a time delay (via the previous timestamp), the method is a POST and the delay (in this case) is less than a second, an attack is assumed since no human can complete a form in less than a second.
A more sophisticated version of this rule set would include a CSV lookup to a list of known forms and the estimated time required to complete them. Based on that, a very effective defense against scripted readers can be mounted.
For reference, the rule set properties are listed below:
In many instances, it is preferable to collate data for decisions over the course of multiple pages. The best way to do this is to use the HTTP Session Writer rule. The rules allow you to specify a list of variables and a list of corresponding key names so that they can be stored in the application server’s session.
The application server’s session provides a convenient place to store data that should only live for the time of the user’s online experience. As the application itself also has access to the session and can set its own keys, it is a good idea to choose key names that are unlikely to conflict with the application. For example, do not use keys such as “user” or “balance”. Instead, use “tomorrow_user” or “tomorrow_balance” (or some other unique prefix).
When the time comes to obtain all of the data in a single request, use the HTTP Session Reader rule. Specify all of the keys names you wish to read and the corresponding variables to restore them to, and you will have all of the available data required.
At times you may wish to serve up a Web page that is not known to the application. Examples of this include a two-factor request page, a challenge page or an information page for a rejected request. The easiest way to do this is to use a content rule set in the configuration, which will handle the delivery for you.
An alternative to using a content rule set is to create a template HTML document, upload it as a data file and deploy it to the target server.
Once it is deployed, it can be read using the File Reader rule. Next, have dynamic contents inserted using the Tag Replacer rule or the String Replacer rule. Finally, the HTTP Response rule can be used to serve the page back to the user. The following pattern shows this in action:
There will be many times where you may wish to create a specific link to a page that does not form part of the application. You will only need to do this for application servers that do not allow you to control content via the content delivery rules. If you are in that unfortunate situation you can use the following approach:
You will need to “piggy-back” onto an existing page using URL parameters.
For example, the main page of an application could be “main.jsp”. However, by appending URL parameters to a link (for example as http://myapplication/main.jsp?ShowGif=penguin), you can use the following rules pattern to detect not just images and display them on request, but also additional pages that you may need to link to.
This pattern effectively sits ahead of the normal rule set for that page and allows you to serve up anything you need. The Switch rule makes it easy to handle multiple different files.
The properties for the first rule are:
The Switch then operates on the ShowGif variable. The File Reader then reads the correct file and sends it back to the user.
Once your application starts to deliver custom content, you would generally want it to “look and feel” the same way as the application it becomes a part of. The best way to manage this is to use style sheets. Many applications already have style sheets in place, and provided your new page is served up within a frame, it will automatically be applied to the new page.
However, if your page must stand alone or if it contains specific structures that are not covered by the standard style sheet, you may need to add style sheet tags to the template or an import reference to the applications style sheet. The HTML syntax for this is as follows:
The name (href) of the style sheet will depend on the actual application, and some applications have more than one style sheet. The best way to find the ones that apply is to view the source of one of the pages within the application.
Sometimes you may wish to allow the server to execute the request from the user so that you can look at the response it provides. The HTTP Server Execute rule provides the means to do that, as shown in the following example:
Note that this example also inserts extra data into the response. The next section covers this in more detail.
Once the response data from a HTTP Server Execute rule has been obtained, you may wish to alter it before it is forwarded to the user. Examples of this include removing high-risk features if the user is coming into the application from an anonymous proxy or a country known for high levels of risk or add a picture (such as for advertising).
Composable Architecture Platform includes a number of string manipulation rules to make this task easy. The following shows the properties for the example mentioned above. It alters the response by adding an image to the page:
You can’t see it in the above example, but the full replacement text property is:
This maps back to the much earlier example of creating links to pages or content now already known to the application. The above will cause the browser to make a second request on the application server for the page URL shown. You can intercept the request and use it to return the image to the browser.
In this particular case, the image is inserted into the HTML in a spot that looks like this:
When you detect a condition that requires action or further user input (such as a two-factor input request), your best option is to redirect to a “piggy-back” page as described above. You can do this by using an HTTP Redirect rule.
Please note that this cannot be done after an HTTP Server Execute rule. This is because the server has detected content already being written to the response and will no longer allow redirection.
The workaround for this is to send a response to the browser that causes it to redirect instead. This can be done with the HTTP Response rule, sending back a line of text as follows:
This pattern once again illustrates the “piggy-back” pattern in action.
Flight recorders can be used for more than logging of critical events for forensics. The following is an example of a flight recorder used simply to record web stats for a specific page:
The properties for the Flight Recorder Trigger are as follows:
The rule will trigger a single record into a flight recorder, giving you information about the user, browser, country or access, referrer and any other fields that you may wish to store. This can later be used to graph access to your application and give you valuable development and marketing feedback. Please see the “Working with Flight Recorders” section later in this book for more details.
Configuration Settings
Rule Sets
Tips and techniques for working with the rules editor
