This blog dives into a few powerful and super useful server memory management features. Most are undocumented. Please do check with your Tableau Technical Account Manager before you use them. In my opinion, they are must-have  controls for last enterprise Tableau server.

Problem Statement:  If your Tableau has memory governance or control issue or sometimes most user render slow (spinning) due to memory, read on…

TSM Commands:

1. tsm configuration set -k native_api.memory_limit_enabled -v true -frc

What it does is to enable the special memory limit feature.  You need -force-keys option (-frc) .

Now you can set specific memory limit for the Tableau server processes.

2. tsm configuration set -k native_api.memory_limit_per_process_gb -v 50 -frc

What it does is to set each process memory limit as 50G. Let’s say that you have 4 VizQL server on one node, what this command does is to make sure each of the 4 VizQL process can’t use more than 50G memory – What a great safe net enterprise Tableau server must have.

 What happens if the memory over 50G?

• When memory reaches 80% of the limit (50G x 80% = 40G), the Server Resource Manager will have reclamation process.
• When memory reaches 100% of the limit, the process will get re-started by itself so it will not impact the whole server.

 

 

 

 

3. Hyper on  a Separate Server (HoSS) special setting needed

Starting V2018.3, Tableau officially supports new architecture configuration – a node has nothing but File Store (Hyper) and Data Engine (Data Engine has to be on every node). The
intent is to give as much server resources as possible to Hyper. You can have multiple HoSS nodes in one cluster.

How to set memory list of the Hyper nodes?

The above  tsm configuration set -k native_api.memory_limit_per_process_gb -v 50  will also limit ‘hyperd’ process (‘hyperd’ is Hyper process name) memory  to be 50G as well, which is good and bad. Unlike VizQL node where multiple VizQL processes can be configured on the same node), only ONE ‘hyperd’ process can be configured on one HoSS node. Can you Have higher memory limit for HoSS node while still keep lower memory for VizQL nodes? The answer is YES.

4. tsm configuration set -k hyper.srm_memory_limit_per_process_gb -v 180 -frc 

What it does is to limit hyperd memory total for the node to 180G (this tsm may not be available on V2018.x) and still keep max 50G memory for each VizQL process. This setting is a life-saving feature for me!!!!

The 180G is one example only, you may want to set it to 300G, 400G or more demands on your hardware and workload. My rule of thumb is to set it to 70% of your hardware total memory. For example, if server has 256G, set it as 180G.

 How HoSS memory works?

• When memory reaches 80% of the limit (180G x 80% = 144G), the Server Resource Manager will have reclamation process.
• When memory reaches 100% of the limit, the process will get re-started by itself – that is why you need 2 or more HoSS nodes. When one HoSS hyperd restarted, the other HoSS can still server user queries.

Now you know how to control hyperd total memory.  You should be asking how to control each query’s memory limit. Is there such setting? Good news is Yes from V2019.3 (although I did not see this feature documented till V2020.1 beta)

5. tsm configuration set -k hyper.session_memory_limit -v 5g -frc

What it does is to limit each hyper query’s memory 5G  – that is to make sure one large query will not bring the whole hyperd down (it happens and it does happen). This is must-have feature for large Tableau deployment.

It controls the maximum memory consumption that an individual query can have. Specify the number of bytes. Append the letter ‘k’ to the value to indicate kilobytes, ‘m’ to indicate megabytes, ‘g’ to indicate gigabytes, or ‘t’ to indicate terabytes. For example, hyper.session_memory_limit='900m'. Alternatively, specify the session memory limit as a percentage of the overall available system memory.

This feature has been available since V2019.3 but Tableau did not document it. Likely Tableau’s official documentation will start to list this feature since V2020.1.

The following error msg will be what shows on user’s browser when the individual query reached the limit.

This feature will work when Desktop user connects to published data source. The  following will be the error msg for Desktop user connects to published data source on server when individual query limit reaches:

 

Conclusions: 

Enterprise server admins always need to have more controls on server resources in order to scale Tableau.   Those commands are must-have for large server deployments:

• tsm configuration set -k native_api.memory_limit_enabled -v true -frc
• tsm configuration set -k native_api.memory_limit_per_process_gb -v 50 -frc
• tsm configuration set -k hyper.srm_memory_limit_per_process_gb -v 180 -frc
• tsm configuration set -k session_memory_limit -v 5g -frc

Since those are not documented, please do talk with your Technical Account Manager and test it out before use. I have been using them.