DNS Multi Protocol
In the following case study, we will explore adding a new protocol (DNS) to the capabilities of Tomorrow Software.
For simplicity, we will restrict this to just a single DNS A record.
We will show how to proxy the protocol, how to modify the data coming back from the DNS server and how to capture a network packet and use it later as a template for requests from non-Multi-Protocol input adaptors.
Defining the protocol
This case study assumes that you intend to work with a brand new protocol, if using a predefined protocol (such as MySQL or Telnet) then you can skip this section.
Before you can begin to work with a new protocol, you need to define it. In this case study we will create a basic DNS A Record protocol interpreter. It is not a complete DNS example, but will serve well as an example on how to use the multi-protocol capabilities of Tomorrow Software.
The DNS protocol explained
The DNS protocol was chosen for this case study due to its simplicity and because it is well documented.
A simple internet search for “DNS Packet Format” will provide the complete details, but the following is a simplified primer.
At its core, it has the following structure in both the request and response:
A header block:
0..7 - 8..15 | 16..23 - 24..31 |
Message ID A unique number that the sender can use to tie a response to a request | Flags 16 bit flags. The most important of those is the first bit, which is 0 for a query and 1 for a response |
Number of questions A simple 16 bit count of questions | Number of answers A simple 16 bit count of answers |
Number of authoritative answers A simple 16 bit count of answers that are authoritative | Number of additional answers A simple 16 bit count of additional answers |
Followed by the actual questions or answers block. Questions contain the domain being queried, followed by two 16 bit fields, the first of which is the question type (1 = A record, 2 = NS record and so on) and the second of which is the question class (always 1).
The domain name being queried will have its dots removed and each section of the name is supplied with a leading byte providing the section length, followed by a zero byte to indicate all sections have been provided. For example:
labs.tomorrow.eu will be turned into: [4]labs[8]tomorrow[2]eu[0]
Breaking down the protocol with protocol rules
Before we can start doing anything with the DNS packets, we need to break them down and make them available to our normal rules. We do this in the administration section under “Protocols”.
Just like normal rules, start by creating a rule set named dns_in (as shown) and open it in the rule editor.
You will notice that the rules catalogue for protocols is much smaller than the regular rules catalogue:
You can explore these rules to get a feel for what is available.
Before starting to write the rules, it is important to understand streams, protocol variables, VAO variables, VAO stream variables and stream windows.
Multi-Protocol Streams
Whenever a packet is read using the Multi-Protocol server version of the X Engine, it will be read in the form of a stream. For almost all protocols there are two streams: request and response. It is the job of the Multi-Protocol server to break down the binary content of the stream into variables that can be used and manipulated by the regular X Engine.
The regular X Engine is then capable of modifying the content of the stream before proxying it to the real target server. Upon a reply from the real server, the reply will also be treated as a stream and can equally be broken down and manipulated or simply returned to the original requester.
VAO Variables
Setting a VAO variable directly refers to setting variables in the input for the regular X Engine, when the Multi-Protocol server hands over control to the regular X Engine.
Protocol Variables
To help the protocol rule writer control the workflow around breaking down a protocol, a set of variables known as protocol variables are used. These are basically String objects, and unlike the regular rules, can be treated as such. This means that assignments to a protocol variable via the Set Variable rule can use all of the regular Java language conventions such as:
Notice the use of “”+ in the last example. This is a convenient way to convert a Java integer into a String object.
VAO Stream Variables
VAO Stream variables on the other hand are directly tied to the request or response stream. If you modify a VAO Stream variable within the regular X Engine, then the underlying stream will also be modified. VAO Stream variables use format converters so that the underlying stream can be a binary field, but it will be presented as a regular integer (or some other valid representation) in the regular X Engine.
VAO Stream windows
VAO Stream windows are used to handle the very common occurrence where part of a protocol stream may contain information such as the length of another part.
A classic example of this would be the “Content-length” header in a HTTP response stream.
If designating that a VAO stream variable is also a stream window, any modifications that you make to the content of the stream window will automatically be reflected in the value of the variable.
Breaking down the request stream
We are now ready to create our first protocol rules. Return to the dns-in rule set we opened earlier.
According to the protocol definition, the first field we need to read from the stream is the message ID. We do this by adding a “Read Fixed Data Type” rule:
And setting the properties as follows:
Let’s examine what is going on here:
We are using a “Fixed” data type. This refers to data types that have a fixed unchangeable length within the stream. In our case we pick an unsigned integer, MSB first (MSB referring to Most Significant Byte).
We set the length to 2 bytes.
We picked a variable name of messageid. This is the protocol variable name.
We specified that the stream we are going to work with is the request stream. This is optional as the Multi-Protocol server version of the X Engine is smart enough to know the main stream being worked with, however for clarity, it is recommended to specify it.
We specified a Stream Variable name of MessageID. This means that when the regular X Engine is invoked, it can access and modify the MessageID variable, which will have a direct impact on the stream.
Next, we wire the rule up to the rule set entry point, and also add the “Abort Connection” rule so that we can handle protocol failures gracefully.
The next couple of bytes contain the DNS flags. Since they are bit level, we will read the two bytes as a binary string of 0s and 1s. We once again use the Read Fixed Data Type rule, but this time set the following properties:
This will ensure that the value contained in these 16 bits is represented as a string, looking something like this: “1000010110000000”, where each bit signals a particular meaning as per the DNS protocol specification.
We follow this with 4 simple rules to read the question and answer count:
Each of these new rules reads a 2 byte unsigned integer and are wired to the “Abort Connection” rule on failure.
Next, we need to deal with the actual query payload. For a simple query, this means handling the variable number of elements in the domain name being queried. Theoretically, more than one query could be included in a single DNS request, however for the sake of simplicity, we are ignoring that for now.
The full construct of breaking down the domain name looks like this:
What is happening here is as follows:
The Count and DomainElement variables are each being set to the value “1”.
The while loop is then created using the following properties:
This is followed by a Read Data Type, which is capable of reading a set of bytes with a variable length. In this case, it is the length prefixed String. The properties to perform this read are as follows:
Finally, the Count variable is incremented, using the technique described earlier:
All that remains in breaking down the protocol request is to read the query type and class. As both are simple 2 byte unsigned integers, we can do this with ease:
Finally, we tell the Multi-Protocol server to hand over to the X Engine:
The complete rule set looks like this:
Making the protocol rule set available in rules
Before we can use the protocol rule set in rules, we need to give it a short description and check the box that allows rules access:
Setting up Tomorrow Software rule sets
We now have everything we need to perform a test of our protocol breakdown.
The next step is to create the regular rule sets that are going to set up a port to listen on and receive the packet.
Start by creating a new repository and create a rule set called DNSStart. It will only have one rule:
Save the rule and then create another rule set called DNSMain. It will also only have one rule:
The configuration for these two rule sets is also very simple:
It is now time to start up the stand-alone Multi-Protocol server instance. It is found in the Multi-Protocol folder. The easiest way to do this is to execute either the tomorrow.bat file or the tomorrow.sh file.
Once the instance is running, you can deploy your new configuration to it and start it.
Note: If you are not seeing any Multi-Protocol server instances when you try to deploy, please check that you have a server defined with the server type Multi-Protocol, and that it is configured to the correct management port of your Multi-Protocol server instance.
If you check the log for the Multi-Protocol server instance, you should see the following message:
First test
We now need a tool to send DNS packets to any given DNS server easily. There are many options available on the net, and we selected DNSDataView from http://www.nirsoft.net was chosen for this example.
The first thing we will do is trigger a simple DNS A Record retrieval packet against our Multi-Protocol server instance:
This will obviously not generate a reply yet, but you will see the packets generated in the console output. There will be several, because DNSDataView retries 5 times:
As you can see, the query is broken down into stream variables that are all on the request stream. Also notice how the protocol rules have sliced the query neatly into its three parts: www, testing and com.
Breaking down the response stream
Now that we know the request stream is working, we can proceed to create the protocol rules for the response stream as well. Fortunately, for DNS this is very simple, at least if only dealing with a single DNS A record as in this case study. The first part of the response stream is essentially an echo of the request.
So, start by copying dns_in to a new protocol ruleset named dns_out.
Then we modify each rule to point to the “response” stream and provide a new name for each stream variable by adding the letter R in front of each name:
Once done, we can proceed to read the actual response data.
Things get a little funny here, because the designers of the DNS protocol lived in a time where bandwidth was a scarce resource, so they built “compression” into the protocol.
The way they did it was by manipulating the first bit of the length field of the reply. If this bit is set, then the actual site name (www.testing.com) being replied to can be found by using the rest of the bits + the following byte to create an offset to where the name can also be found in the packet. However, given that in this example we only have one query, we will ignore that and just read the bytes:
With that in mind, the rest of the dns_out protocol rule set becomes fairly simple:
All of the above rules simply read unsigned integers MSB first. Not all have the same length though. RPointer, RType, RClass and RLength are all 2 bytes long. RTTL is 4 bytes long and RIP1 – RIP4 are all 1 byte (each part of the IP address).
The final step to complete the rule set is to name it and allow it to be used in rules:
Proxying a multi-protocol packet
We are now ready to proxy our protocol packet and do something useful with it. We need to return to the regular DNSMain rule set and make some changes:
The first rule you see above is actually the “Proxy Input Request” rule. However, once you change the selected protocol, it automatically changes its name to the protocol it is using.
The complete properties used are:
The Host name/IP shown above is Google’s DNS server. You could choose to use your own to complete this case study.
Testing the proxy
Deploy the dns_example configuration to the Multi-Protocol server instance and restart the rule set. Then go back to DNSDataView and get ready to launch another query. Since we are using Google’s DNS server, we are going to query “www.google.com”.
This time, we get a reply:
And the console shows that the proxy worked:
Looking through the various stream variables, the significant ones are RIP1-4, which tells us that “www.google.com” can be found at 216.58.199.36.
Manipulating a stream
We will now use the regular rule set to manipulate the response stream.
You may notice that the “Set Variable” rule is used to change the RIP4 value to 100. As the Multi-Protocol server version of the X Engine is a two-way mapping of the variables to the stream, changing one of the variables also changes the stream.
We will demonstrate this by deploying the rule sets and re-launching the DNS request:
As the tool shows, we have just changed the output of a DNS request in real time.
The usefulness of this is probably limited (given the recursive nature of DNS), but one example could be making the DNS server respond with a different IP address based on the requesters physical location or setting up internal honeypots.
Crafting protocol packets within rule sets
Proxying packets using the Multi-Protocol server instance is one way to use the protocol packets. There may be times when you wish to use a protocol to access an external service. However, crafting network packets by hand is incredibly time consuming and fraught with error risks.
To get around this, Tomorrow Software includes a feature to capture a packet and use it as a template. Capturing a packet is incredibly easy. Simply modify the rule set to write the stream to a file:
Once you have a captured packet, you can easily modify it using simple stream variables. The following shows how the captured packet is read before being sent to the test DNS server using the “Write Stream to Server” rule:
Using this approach, we have added DNS lookup capability to rules using no code whatsoever.
Last updated
