Multithreaded Publishing

The Feature multithreaded publishing allows using multiple cores for the publish process to reduce overall publishing times.

1 Configuration

The feature can be activated like this

node.conf

$FEATURE["multithreaded_publishing"] = true;
$PUBLISH_LOAD_BALANCING["threadlimit"] = 2;

Currently, the publish process will always use the configured number of threads for rendering pages, independent from the overall system load.

Currently, only one phase of the publish process – rendering of pages – is using multiple threads, when multithreaded publishing is activated.

Find the article about the publish process for further details.

2 Performance enhancements when rendering pages

While rendering pages, loading the dependencies and preparing the data of the pages to be published will be done in batches, to reduce the number of necessary SQL statements (which will increase the overall publishing performance). The size of these batches can be configured with the following swetting (the default value is 100):

node.conf

// setting higher numbers will increase the memory consumption, lower numbers will increase the number of SQL statements
$MULTITHREADED_PUBLISHING["preparedeps"] = 100;

3 Publishing pages into Content Repositories

Writing pages into the Content Repository is done in a separate thread for each Content Repository. There is a job queue for each Content Repository, that is filled with jobs (one job for each page to be written) and processed by the separate thread. In order to minimize the number of SQL statements necessary to write the objects into the Content Repository, the jobs are grouped and the data from the Content Repository will be read for each group in combined statements. The default group size is 100, but can be changed (see below).

Additionally, the queue has a limit for the total number of jobs, so that it will not grow too big and occupy too much memory, when writing pages into the Content Repository is slower than the threads that render the pages. The default queue limit is 1000. but also can be changed (see below).

node.conf

$MULTITHREADED_PUBLISHING["groupsize"] = 100;
$MULTITHREADED_PUBLISHING["queuelimit"] = 1000;

A higher value for the group size means viewer SQL statements, but higher memory consumption.