# Data Manipulation

## Separation of code and data.

Avoid hard-coded html in the rules, instead use tag replacement techniques or templating techniques *(custom templating, template engines, frameworks or CMS’s)* that help separate code *(business logic)* from data *(html output).*

This helps web designers to focus only on the code relevant to the user interface and not any business logic.

Custom templating should be designed in such a way that it is directly related to the data output

with the use of “if”, “while”, “for”, “foreach”, “switch” control structures

and it should be readable.

The selected templating technique should be consistent throughout a solution, avoiding confusion.

Two tag replacement techniques are detailed in the **Appendix A**. For more advanced options, like the use of control structures, a templating engine should be used.

## Content Files

Include static files, such as `js/css` libraries in the “Content Files” node.

File and folder organization should be agreed upon by the team, and remain easy to understand and structured.

## Data Files

Include data files, such as `html`, `CSV` files in the “Data Files” node. A benefit of reading from “Data Files” is that it will be memory cached, and you can update it without redeploying all rule sets, but simply by deploying the data file on its own.

<figure><img src="/files/ysLaP1XqCWi4HCAZFZ2X" alt=""><figcaption><p>maintain data file example</p></figcaption></figure>

## Data Sets

For smaller applications Data Sets are a preferred way to store data, instead of using SQL statements. Data Sets are incredibly flexible and do not require modifications to the schema for every new field or feature.

See the Data Set rules to study an example.

Data Sets are more than adequate for anything up to about 100,000 elements and they offer flexible “key-on-anyfield” and “add a new field any time you like without stopping” capabilities.

## Data Input

Custom functions in the Tomorrow Software console can be used as an interface for corporate end-users who have been assigned the role of application/user data maintenance, such as localization data, selection list data, user data, csv files or databases.

User access to desired custom functions can be granted to users through the Tomorrow Software console, by a Tomorrow Software administrator.

## Memory Considerations

Consider increasing maximum memory usage when reading from large files that their content needs to be saved to a variable for further processing *(i.e., “File Reader” rule)*, by changing java startup properties in your `Tomorrow.bat` file. If the file just needs to be forwarded to a http client, then use the “HTTP File Response” rule.


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.tomorrowx.com/cap/best-practices/data-manipulation.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.