Web Application Rule Set Patterns
The Composable Architecture Platform inline filter and built in proxy offers the ability to wrap and control the behaviour of a Web application. Usually this is done in conjunction with the Receive Web Application data input adaptor.
Rules in this environment can see all of the post data from the Web browser. By using rules such as the HTTP Request Tracker, HTTP Header Reader and other various rules for interacting with the application’s session object, it is possible to do a lot of additional pre-checking of the request being forwarded to the application before it is even processed.
Rules such as the HTTP Request Saver and HTTP Request Restorer also allow for requests to the backend server being temporarily placed on hold, pending actions by the X Engine.
When writing rules, it is a good idea to keep in mind that you may wish to test your rules without the application server being present. To do this, it is usually a good idea to create a Load rule first, which reaches out to all of the elements of the Web application, the request, dynamic databases, and so on to collect all of the data required for a decision before embarking on making that same decision. By doing so, you can insert a Test Data Creation rule at the point where all of the data is ready, thus allowing you to properly test the business functions of your rule set separately on a test server with all of the relevant data being available.
If you need to look at the output from a given request, insert an HTTP Server Execute rule. This will forward the current request to the server for processing and bring back the resulting response in a single variable. You can then scan that variable for information (for example, look for a balance value or a name) using the Scanner rule. This rule allows you to define the text surrounding information that you are interested in, and then extract that information into a variable.
Similarly, you can manipulate the response before forwarding it to the Web browser. As the response is simply stored in a variable, you can use the Replacer rule to modify text within given tags or specific locations inside the response page.
Once you have finished working with the output from a request, it is imperative that you forward it to the Web browser using the HTTP Response rule.
Finally, there may be situations where you wish to simply append additional data to the end of a Web page before it is transmitted to the user. Typically, this would be in the form of JavaScript appended to the end of the page. The best way to do this is to store the data in a data file, upload it to the server, read it into a variable using the File Reader rule, and append it to the server response using the HTTP Response Addition rule.
This section showcases a number of common rule set patterns used when working with Web applications.
Starting out (passive listening)
One of the first problems always facing you when you start working with a new application is the ability to understand which data flows where under each circumstance. The following rule set pattern, which includes comments, provides a good starting point:
First, the HTTP Request Tracker rule takes care of getting the browser information and adding tracking cookies. Second, a fast lookup to the MaxMind geo location database (which requires a subscription) identifies the origin of the request. Third, the request is written to the console so that you can monitor it in real time. Finally, the data is written to the test data queue so that you can download it for analysis.
This rule is best deployed to a test system during the initial rule writing phase to better understand what variables and pages are available. It is prudent to deploy it initially during a live install to retrieve a large portion of live test data and play it through the desired rule set on the Composable Architecture Platform test server.
Filtering out static content
A common issue in dealing with application servers that not only serve dynamic content, but also static data (in the form of images, style sheets, fixed HTML), is to filter this content before it even hits the core rule set. This is best done in the Load rule using a Name Splitter and Switch rule as shown:
The name splitter conveniently extracts the extension of the object being requested using the following properties:
The Switch rule operates on the EXT variable. By adding new chain points for each type of static content they are eliminated from reaching the rule set.
Timing a form
It is often a good idea to know the time a user has spent on a form. This is the foundation for filtering and/or slowing down “screen scrapers” or “data extraction” bots. This is an example of what a browser timing rule set looks like:
The basic concept is to first check if a session is present. If not, this rule set does not proceed. In some sites, this may be overly simplistic and may require modification, but for most sites it will be valid.
The rule set then goes through a series of checks. It reads the last time any request was made to the application, timestamps the current request and stores it. If it is possible to measure a time delay (via the previous timestamp), the method is a POST and the delay (in this case) is less than a second, an attack is assumed since no human can complete a form in less than a second.
A more sophisticated version of this rule set would include a CSV lookup to a list of known forms and the estimated time required to complete them. Based on that, a very effective defense against scripted readers can be mounted.
For reference, the rule set properties are listed below:
Collating data over multiple pages
In many instances, it is preferable to collate data for decisions over the course of multiple pages. The best way to do this is to use the HTTP Session Writer rule. The rules allow you to specify a list of variables and a list of corresponding key names so that they can be stored in the application server’s session.
The application server’s session provides a convenient place to store data that should only live for the time of the user’s online experience. As the application itself also has access to the session and can set its own keys, it is a good idea to choose key names that are unlikely to conflict with the application. For example, do not use keys such as “user” or “balance”. Instead, use “tomorrow_user” or “tomorrow_balance” (or some other unique prefix).
When the time comes to obtain all of the data in a single request, use the HTTP Session Reader rule. Specify all of the keys names you wish to read and the corresponding variables to restore them to, and you will have all of the available data required.
Serving up a page not known to the application
At times you may wish to serve up a Web page that is not known to the application. Examples of this include a two-factor request page, a challenge page or an information page for a rejected request. The easiest way to do this is to use a content rule set in the configuration, which will handle the delivery for you.
An alternative to using a content rule set is to create a template HTML document, upload it as a data file and deploy it to the target server.
Once it is deployed, it can be read using the File Reader rule. Next, have dynamic contents inserted using the Tag Replacer rule or the String Replacer rule. Finally, the HTTP Response rule can be used to serve the page back to the user. The following pattern shows this in action:
Creating links to pages and content now known by the application
There will be many times where you may wish to create a specific link to a page that does not form part of the application. You will only need to do this for application servers that do not allow you to control content via the content delivery rules. If you are in that unfortunate situation you can use the following approach:
You will need to “piggy-back” onto an existing page using URL parameters.
For example, the main page of an application could be “main.jsp”. However, by appending URL parameters to a link (for example as http://myapplication/main.jsp?ShowGif=penguin), you can use the following rules pattern to detect not just images and display them on request, but also additional pages that you may need to link to.
This pattern effectively sits ahead of the normal rule set for that page and allows you to serve up anything you need. The Switch rule makes it easy to handle multiple different files.
The properties for the first rule are:
The Switch then operates on the ShowGif variable. The File Reader then reads the correct file and sends it back to the user.
Adapting templates to style sheets
Once your application starts to deliver custom content, you would generally want it to “look and feel” the same way as the application it becomes a part of. The best way to manage this is to use style sheets. Many applications already have style sheets in place, and provided your new page is served up within a frame, it will automatically be applied to the new page.
However, if your page must stand alone or if it contains specific structures that are not covered by the standard style sheet, you may need to add style sheet tags to the template or an import reference to the applications style sheet. The HTML syntax for this is as follows:
The name (href) of the style sheet will depend on the actual application, and some applications have more than one style sheet. The best way to find the ones that apply is to view the source of one of the pages within the application.
Forwarding the request to the server
Sometimes you may wish to allow the server to execute the request from the user so that you can look at the response it provides. The HTTP Server Execute rule provides the means to do that, as shown in the following example:
Note that this example also inserts extra data into the response. The next section covers this in more detail.
Manipulating the response to the user
Once the response data from a HTTP Server Execute rule has been obtained, you may wish to alter it before it is forwarded to the user. Examples of this include removing high-risk features if the user is coming into the application from an anonymous proxy or a country known for high levels of risk or add a picture (such as for advertising).
Composable Architecture Platform includes a number of string manipulation rules to make this task easy. The following shows the properties for the example mentioned above. It alters the response by adding an image to the page:
You can’t see it in the above example, but the full replacement text property is:
This maps back to the much earlier example of creating links to pages or content now already known to the application. The above will cause the browser to make a second request on the application server for the page URL shown. You can intercept the request and use it to return the image to the browser.
In this particular case, the image is inserted into the HTML in a spot that looks like this:
Taking charge of the application flow
When you detect a condition that requires action or further user input (such as a two-factor input request), your best option is to redirect to a “piggy-back” page as described above. You can do this by using an HTTP Redirect rule.
Please note that this cannot be done after an HTTP Server Execute rule. This is because the server has detected content already being written to the response and will no longer allow redirection.
The workaround for this is to send a response to the browser that causes it to redirect instead. This can be done with the HTTP Response rule, sending back a line of text as follows:
This pattern once again illustrates the “piggy-back” pattern in action.
Using Flight Recorders for basic web stats
Flight recorders can be used for more than logging of critical events for forensics. The following is an example of a flight recorder used simply to record web stats for a specific page:
The properties for the Flight Recorder Trigger are as follows:
The rule will trigger a single record into a flight recorder, giving you information about the user, browser, country or access, referrer and any other fields that you may wish to store. This can later be used to graph access to your application and give you valuable development and marketing feedback. Please see the “Working with Flight Recorders” section later in this book for more details.
Last updated
