Understanding the magic.properties Configuration Settings

Each Programmable Data Agent install is highly configurable. The configuration file is installed in the Programmable Data Agent class path alongside the magic jar file.

The tables below document the settings in magic.properties and — at the end — a small number of JVM-level container limits that share failure symptoms with some magic.properties settings and are documented here for tuning convenience.

Core settings

Setting	Details
homeDir	The home directory for the Programmable Data Agent. All files and rule sets deployed to the server will be stored in this folder; backup copies of any rule sets that have been replaced are stored in a subfolder called backup. The value can be an absolute or relative file path.
port	The TCP/IP port where the Programmable Data Agent will listen for instructions from a Composable Agentic Platform console. If using vertical scaling (multiple application server instances on the same server sharing a common configuration folder), provide a list of port numbers separated by commas. Create a server instance in the console for each of the ports specified in the port list. Each server clone can then be managed as an individual Programmable Data Agent. For vertical scaling, rules for the "Master" (first port in the list) are stored in the top-level home directory, whereas rules for any of the "slaves" are deployed to a sub-directory of the home directory named CloneX — where X is a sequential number.
validIPs	Comma-separated list (CSV) of IP addresses of Composable Agentic Platform consoles that are allowed to send commands to this server. Optional. For stronger security, specify a list of trusted console servers that can communicate with this server.
failOpen	Determines the behaviour in case of a failure. If set to true, the Programmable Data Agent will terminate itself if an internal problem occurs. If set to false, the Programmable Data Agent will attempt to recover from any failure. Use the Fail Safe Point rule to manage this at a finer-grained level.
bannedVariableNames	Nominates variables that are never allowed to be set within the Programmable Data Agent when the variables are accessed inline in a proxy or servlet filter. For example, to make sure a user is unable to see an end user's password, nominate the field name for the password here. The rule set will not see the variable. The list is a CSV of disallowed variable names and is case sensitive.
bannedVariableMasks	Determines what happens when an attempt to set a given banned variable is made within the Programmable Data Agent. CSV of tokens that must match the list in bannedVariableNames. Valid tokens: REMOVE (the variable is not set at all), MD5HASH (set but encoded with an MD5 hash), SHA1HASH (set but encoded with an SHA-1 hash), PCIMASK (treated as a credit card number where only the first 6 and last 4 digits are visible). Masks and hashes can be combined using the & operator. For example SHA1HASH&PCIMASK will result in two new variables: [fieldname]_SHA1HASH and [fieldname]_PCIMASK.
EncryptionKey	If the connection between the console and the Programmable Data Agent is to be encrypted, specify a secret password here. The same password must be provided when the host is defined in the console. If an incorrect match is made between the console and the host, the server is seen as offline in the console. Case sensitive. Optional.
EncryptionAlgorithm	If using an encrypted connection between the console and the Programmable Data Agent, specify the encryption algorithm to be used here. If left blank, PBEWithMD5AndDES is used when an EncryptionKey is specified. The same algorithm must be configured on both ends — mismatch causes the agent to appear offline in the console. The supported values (provided by the SunJCE provider in the JDK) are: PBEWithMD5AndDES (default — kept for backward compatibility; not recommended for new deployments) PBEWithMD5AndTripleDES PBEWithSHA1AndDESede (recommended) PBEWithSHA1AndRC2_40 PBEWithSHA1AndRC2_128 PBEWithSHA1AndRC4_40 PBEWithSHA1AndRC4_128 Recommended setting: PBEWithSHA1AndDESede (3DES + SHA-1 PBKDF). It offers ~112-bit effective strength — the strongest of the seven SunJCE PBE algorithms supported by the agent's encrypted socket — and pairs with the shared EncryptionKey to provide mutual proof of possession on the channel. The control story is layered, not algorithm-only. A console↔agent channel is a management channel and is not normally exposed on an open, public network — it is expected to live inside a private subnet, management VLAN, VPN, or VPC, with validIPs restricting which console hosts may connect and a firewall rule enforcing the same at the network layer. PCI-DSS 4.0 §4.2.1 ("strong cryptography during transmission over open, public networks") therefore does not apply to this channel in a typical deployment; the network boundary plus the shared EncryptionKey plus 3DES collectively provide defence in depth. If the channel does cross an untrusted network (multi-region without a VPN, internet-exposed agent, etc.), front the agent port with an mTLS terminator (stunnel, Envoy, service mesh) so the on-the-wire bytes are TLS 1.3 + AES-256-GCM, and treat EncryptionAlgorithm as an inner obfuscation layer. Forward-looking: the SunJCE PBEWithHmacSHA*AndAES_* family (AES-128/AES-256 with HMAC-SHA-256) is not currently usable with the agent's encrypted socket implementation, and 3DES is officially deprecated by NIST SP 800-131A Rev 2. A core change is planned to add AES-PBE support; once delivered, that will become the recommended setting and PBEWithSHA1AndDESede will move to "legacy".

Request body handling

preserveStream

Controls whether form-encoded POST bodies are buffered into a resettable stream so rules and downstream components can read the body more than once.

Default: false

Set true when either of these is true for your PDA:

1. The PDA is a Forwarder (serverType=4) using the built-in proxy — the proxy streams the request body to the upstream server after the rules chain runs. Without a preserved body, BIP forwards zero bytes upstream. The stock root PDA ships with preserveStream=true for this reason.

2. The ruleset contains rules that re-read the raw request body after the input adapter has parsed it, and the typical inbound Content-Type is application/x-www-form-urlencoded or multipart/form-data. Rules in this category include HttpServletExecution, HttpInvokation, HttpRequestDataReader, HttpRequestDerandomizer, and HttpResponseAddition.

Leave false for normal filter deployments that only read request parameters via VAO attributes (REQUEST_* variables) and do not re-stream the body. This is the correct setting for most application-facing PDAs that terminate the request.

Non-form POST bodies are unaffected by this flag. JSON, XML, SOAP, AMF, multipart, and binary POSTs are buffered automatically regardless of the preserveStream value — this setting only controls form-encoded bodies. Chunked POSTs (Transfer-Encoding: chunked, any Content-Type) are also buffered automatically.

Performance impact. When true, every form POST allocates a ByteArrayOutputStream sized to the request's Content-Length and copies the full body into the JVM heap. Negligible at low volume, measurable under sustained high-throughput form traffic — do not enable needlessly.

Stock install defaults by agent:

Agent	preserveStream	Reason
root (Forwarder)	true	BIP proxy requires body re-streaming
TestServer1	false	Test agent, no body re-read
StressTest1	false	Load generator, no body re-read
MPServer1	false	Multi-Protocol server, not HTTP-form-based
Console / consolex	false	Admin UI, terminates all requests

Request size limits

The following three settings cap different dimensions of an inbound request. They are evaluated independently — a request must stay under all three or it is rejected. The first is set in magic.properties; the other two are JVM system properties owned by the Jetty container and documented here because they share failure symptoms with maxRequestSize.

maxRequestSize

Caps the total request body size in bytes that the PDA will accept. Checked once, at filter entry, before the request reaches any rules. If exceeded, the client receives HTTP 413 and the request is dropped.

Applies to all POST types — form, multipart, JSON, XML, binary.

Only effective when the client sent a Content-Length header. Chunked requests without a Content-Length bypass this check and are only bounded by the two Jetty limits below and by JVM heap.

Default: 10 MB (10000000). Typical production values range from 10 MB to 250 MB (Console and consolex stock) to accommodate extension ZIP uploads and multipart file transfers.

Jetty: maxFormContentSize (JVM system property)

Caps the total bytes the Jetty container will parse from an application/x-www-form-urlencoded request body into request.getParameterMap(). When exceeded, Jetty raises HTTP 400 "Unable to parse form content" before the rules chain sees the request.

Default: 200 KB (200000).

Applies only to application/x-www-form-urlencoded requests. Has no effect on multipart, JSON, XML, or binary POSTs.

Not set via magic.properties — override at JVM startup:

-Dorg.eclipse.jetty.server.Request.maxFormContentSize=1048576

(example: 1 MB)

When to raise it: the ruleset legitimately accepts a large form POST — typically many hidden fields or base64-encoded blobs posted as form fields. The symptom is always the same HTTP 400 "Unable to parse form content"; if that appears and the body is under ~1000 keys but over 200 KB, this is the setting to raise.

Jetty: maxFormKeys (JVM system property)

Caps the number of distinct keys Jetty will parse out of a form POST, independent of byte size. Same HTTP 400 "Unable to parse form content" symptom as maxFormContentSize, which makes them easy to confuse.

Default: 1000.

Override at JVM startup:

-Dorg.eclipse.jetty.server.Request.maxFormKeys=20000

TomorrowX stock start scripts already set this to 20000 for PDAs (Tomorrow.sh) and 8000 for the Console (start.sh).

When to raise it: the inbound form has more than ~1000 distinct fields. Rare. Always check this first when the 400 error shows up and the body is well under 200 KB.

Choosing between the three when a request is rejected

Rejection	Status	Likely cause
HTTP 413 "Maximum submit size is N"	413	maxRequestSize exceeded (check Content-Length)
HTTP 400 "Unable to parse form content" (body > 200 KB)	400	maxFormContentSize exceeded
HTTP 400 "Unable to parse form content" (many keys, small body)	400	maxFormKeys exceeded
Empty body forwarded upstream (Forwarder PDA only)	200	preserveStream=false — flip to true