Techila Distributed Computing Engine Reference Guide

By default, all Jobs in a Techila Distributed Computing Engine environment will be executed by using a dedicated Techila Worker user account. The techila_ad_impersonate parameter can be used to enable Active Directory impersonation, which allows the system to execute the Job using the End-User’s AD account. Please note that before AD impersonation can be used, it must be configured by the Techila Administrator.

General syntax for this parameter is shown below:

The following syntax could be used to enable AD impersonation.

When AD impersonation is enabled, the entire computational process will be executed under the user’s own AD account.

By default, all Techila Workers will be allowed to participate in a Project. The techila_worker_features parameter is a filter for Techila Worker features. A Techila Worker must have the specified feature (or feature value) to receive Jobs from the Project. Filters are defined according to RFC 1960 (LDAP Search Filter). General syntax shown below:

The <feature> and <value> notations should be replaced with applicable values. The example below defines that only Workers with feature department with value research can participate in the Project.

Worker features can be configured by a Techila Administrator according to the instructions in Techila Distributed Computing Engine Administration Guide. An illustration of a configured Techila Worker feature with matching key-value pairs as described above is shown below.

By default, all Techila Workers will be allowed to participate in a Project. The techila_worker_filter parameter can be used to define a filter to limit which Techila Workers can participate in the Project. The Techila Worker is removed from the Project if the filter doesn’t match. Filters are defined according to RFC 1960 (LDAP Search Filter). General syntax shown below:

techila_worker_filter (<property>=<value>)

The property field in the filter may contain limits for one or more of these properties:

Using these properties, you can perform equality (=), greater or equal than (>=) and smaller or equal than (⇐) checks. Please note that these LDAP filters do not support greater than (>) and smaller than (<) operations. For more information about supported operations, please refer to RFC 1960.

Example 1: Define that only Techila Workers with more than 2 GB memory are allowed to receive Jobs from the Project.

The image below illustrates how using this filter could be used to limit which Techila Workers can participate in the Project.

Example 2: Define that only Techila Workers which have an average round-trip-time of 100 milliseconds or less can participate in the Project.

The image below illustrates how using this filter could be used to limit which Techila Workers can participate in the Project.

Example 3: Combine the filters used in examples 1 and 2 so that both conditions must be met in order for a Techila Worker to participate in the Project.

The image below illustrates how using this filter could be used to limit which Techila Workers can participate in the Project.

Example 4: Combine the filters used in examples 1 and 2 so that atleast one of the conditions must be met in order for a Techila Worker to participate in the Project.

The image below illustrates how using this filter could be used to limit which Techila Workers can participate in the Project.

This parameter can be used in conjunction with the parameter techila_worker_filter. When techila_worker_filter_postpone parameter is set to true, the techila_worker_filter will not remove Techila Workers from the Project, but instead will retry after 30 seconds. Useful for example when used in conjunction with time limits for the computations.

For example, the following syntax could be used to only allow Techila Workers with more than 2 GB of free memory to participate in the Project. If the Techila Worker does not have sufficient amount of free memory available, the Job will be retried on the Techila Worker after 30 seconds.

By default, all Techila Worker Groups assigned to you will partipate your computational Projects. The techila_worker_group parameter can be used to define a subset of these Worker Groups to limit which groups are allowed to participate in the Project. When defined, Techila Workers are only allowed to participate in the Project if they are assigned to a Techila Worker Group in the list.

General syntax:

Example: The following syntax could be used to only use Workers in the "Math" and "Physics" Techila Worker Groups.

By default, Techila Workers will participate in your Project regardless of which operating system the Techila Worker has. The techila_worker_os parameter can be used limit which operating systems are eligible to process Jobs belonging to the Project. Techila Workers are only allowed to accept Jobs from the Project if their operating system is included in the list. Multiple operating systems can be defined as a semicolon (;) separated list.

General syntax:

The values that can be used are:

The string windows will automatically include all Techila Workers with a Microsoft Windows operating system (Windows 10, Windows 7, Windows Server 2008, etc.) The operating system and architecture of Techila Workers can be viewed from the Techila Web Interface.

Example: The following syntax could be used to allow all Techila Workers with a 64-bit Microsoft Windows operating system to participate in the Project.

By default, Jobs are assigned to Techila Workers in a random order. The techila_compute_jobs_in_order parameter can be used to assign Jobs to the Techila Workers in the order they were created, i.e. assign Job 1 first, and then Job 2 and so on.

Note! If used with the Streaming feature, setting techila_compute_jobs_in_order to true does not guarantee the results are transferred to the End-User`s computer in the same order the Jobs are created. For example, if the execution time of Job 2 is significantly shorter than Job 1, Job 2 will be completed first.

In case any of the Jobs expire, they are moved to the end of the Job queue.

General syntax:

Default value for this parameter is false.

Example: The following syntax could be used to enable this feature and assign Jobs to Techila Workers in the order they are created.

By default, Projects will be cancelled if the number of errors in the Project exceeds the error limit. The Project error limit is reached if over 90% of Jobs have generated at least one error and if 10% Jobs have more than one error. This behaviour is illustrated in the figure below.

The techila_do_not_cancel_on_errors parameter can be used to disable Project cancellation due to large error counts.

General syntax:

Default value for this parameter is false.

Example: The following syntax could be used to prevent Projects from being cancelled due to large error counts.

By default, a Job can be marked as expired in situations where it is taking longer than expected to complete the Job. When a Job expires on a Techila Worker, the Job will be placed back in the Job queue. If there is available computing capacity, the expired Job will be assigned and started on one of the Techila Workers that is available. This means that there can be several "copies" of the same Job running simultaneously on different Techila Workers. If no computing capacity is available, the job will remain in the Job queue until capacity becomes available or until the Project is cancelled or the original Job is completed.

Job expirations are typically most useful when the Snapshot feature is enabled. This will enable expired Jobs to be started from the most recent snapshot file.

A Job will be expired if any of the following conditions are met:

The figure below illustrates how Jobs are processed when Job expiration is enabled.

The techila_do_not_expire parameter can be used to prevent Job expirations.

Job expirations can also be disabled by the Techila Administrator using the Techila Web Interface.

General syntax for this parameter:

The default value for this parameter is false.

The example syntax below could be used to disable Job expiration in a Project:

The figure below illustrates how Jobs are processed when Job expiration is disabled.

By default, Jobs that do not complete successfully will be automatically restarted on an another Techila Worker when capacity becomes available. The techila_interrupt_on_retry parameter can be used to disable these automatic retries. If automatic retries are disabled and a Job does not complete successfully, the entire Project will be interrupted and execution of all other Jobs in the Project will be terminated.

General syntax for this parameter:

The default value for this parameter is false.

The example syntax below could be used to disable automatic retries in a Project:

By default, the contents of the temporary execution directory will not be included in the log files. The techila_list_files_in_log parameter can be used to change this behavior so that the directory contents will be included in the log file. Must be used with the techila_stream_logs and techila_stream_stderr parameters.

The following syntax could be used to enable the techila_list_files_in_log parameter:

Project logs will be stored in the location defined in the techila_settings.ini file with parameter stderrdir.

The screenshot below illustrates how the directory content will be visible in the log file.

By default, when a Job generates an error, the error message will not include information about what files were located in the temporary working directory on the Techila Worker. The techila_list_files_on_error parameter can be used to make the error message include a list of files that were found in the temporary working directory.

The following syntax could be used to enable the parameter:

If a Job generates errors, the files in the temporary working directory will be displayed in the error message as illustrated in the screen capture below.

The error message will also be stored in the Project error logs on your computer. The location and name of the error logs is defined in the techila_settings.ini with the errordir parameter.

This parameter can be used to control how much information the techila_list_files_in_log and techila_list_files_on_error parameters display. When the techila_list_files_recursively parameter is set to false, only the files in the temporary working directory will be returned. When this parameter is set to true, the contents of all sub-directories in the temporary working directory will also be returned in a recursive listing.

The screenshot below illustrates how the directory content and subdirectory contents will be visible in the log file.

This parameter can be used to enable Job messages. When enabled, Job messages will be transferred from the Job to your computer. These messages contain debug information about the internal operations performed on the Techila Worker. These messages will be stored in the Techila SDK log files. Must be used with techila_stream_logs and techila_stream_stderr parameters.

The screenshot below illustrates how the Job messages are displayed in the log file.

Allows streaming log files from the Jobs. This parameter is used to enable other parameters, such as techila_enable_jobmessages and techila_list_files_in_log.

The following syntax could be used to enable this parameter:

The standard output (stdout) stream is an output channel between a computer program and its environment and is used to display messages. Storing the stdout stream generated by Techila Workers can be useful when more information regarding the current state of the computations is required.

To direct the stdout stream generated on Techila Workers during a Job, the following Project parameter needs to be defined:

Note! In addition to defining the Project parameter above, at least one of the following parameters needs to enabled in the techila_settings.ini file in order to store the stdout stream.

The stdout stream will be appended to the files every time the End-Users program polls the Techila Server for the Project status.

The stdoutfile parameter defines the name and location of the file, which will be used to store the stdout streams. The stdout stream from all Techila Workers will be appended to the file, meaning the size of the file can grow quickly in cases where large amounts of information is directed to the stdout stream on the Techila Workers.

The standard error (stderr) stream is an output channel between a computer program and its environment and is used to display error messages. Storing the stderr stream generated by Techila Workers is useful for when the error messages need to be analysed in more detail.

To direct the stderr stream generated on Techila Workers during a Project, the following Project parameter needs to be defined in the Local Control Code. More information about the Local Control Code concept can be found in Introduction to Techila Distributed Computing Engine.

Note! In addition to defining the Project parameter above, at least one of the following parameters needs to enabled in the techila_settings.ini file in order to store the stderr stream.

The stderr stream will be appended to the error files every time the End-Users program polls the Techila Server for the Project status.

The stderrfile parameter defines the name and location of the file, which will be used to store the STDERR streams generated on the Workers. The STDERR stream from all Workers will be appended to the file.

The techila_stream_stats parameter can be used to gather performance statistics of Jobs when they are executed on the Techila Workers. These statistics include information about the used CPU time and real time, amount of RAM memory used and the amount of Input / Output (IO) operations. When transferring statistics, you can also define an interval which defines how often the performance statistics are recorded.

In order to transfer performance statistics from the Job back to your computer you will need to enable the following Project parameters.

Additionally, you will need to define an output file which will contain the statistics and define the logging interval for the statistics. General syntax shown below:

The <filename> notation will need to replaced with the name of the file that will be used store the statistics. This file will be generated automatically (you do not need to create it). The <time_in_seconds> notation needs to be replaced with the logging interval. For example, the following syntax would store the log statistics to a file named stats.txt at 2 second intervals.

After the Job has been completed, the statistics can be read from the location defined in the techila_stream_stderr parameter. The example screenshot below shows what the statistics look like in the log file.

This parameter can be used to enable email notifications when the Project is completed. These email notifications will be sent to the email address specified in the End-User Settings in the Techila Web Interface. The general syntax for this feature is:

The syntax shown below illustrates how you can enable this feature.

The default value for this parameter is false.

Please note that in order for the email notification feature to function correctly, the following configurations must be performed:

The recipient`s email address can be configured by End-Users in the Techila Web Interface. This configuration does not require administrative permissions. The email address can be configured in the End-User Settings page, which can be accessed by clicking on the login name next to the Logout button.

The SMTP email configuration requires administrative access to the Techila Web Interface. The configuration can be done in Admin -→ Advanced -→ Config -→ Techila Server Mail Handler in the Techila Web Interface.

This parameter can be used to enable email notifications in situations where the Project fails. Please note that in order to use this feature, the following configurations must be done:

More information about the configurations can be found in techila_mail_on_completion.

This parameter specifies that only one Job of the Project is allowed to run on a Techila Worker at any given time. Without this parameter, the number Jobs that can be run on a Techila Worker is typically equal to the number of CPU cores on the Techila Worker.

Using this parameter will also prevent Jobs from other Projects to be assigned to any Techila Worker that is currently processing a Job from this Project. Please note that if a Techila Worker is already computing a Job from another Project, a Job with the multithreading flag can still be assigned to the Techila Worker.

General syntax:

The default value of this parameter is false.

This parameter can be enabled with the following syntax:

This parameter is typically used in following situations:

This parameter can be used to specify that all free CPU cores of a Techila Worker should be assigned a Job before assigning Jobs to the next available Techila Worker.

General syntax:

This parameter can be enabled with the following syntax:

Example: Your Techila Distributed Computing Engine environment has two Techila Workers (Worker 1 and Worker 2) which each have four CPU cores. If you then create a Project (with techila_optimize_all_cores_first set to true), four Jobs will be assigned to Worker 1 (because it has 4 CPU cores). After all cores of Worker 1 have been filled, Jobs will be assigned to Worker 2.

This parameter can be used to specify that only the fastest (as determined by the benchmark values) Techila Workers should be used in the computations.

General syntax:

This parameter can be enabled with the following syntax:

When the techila_optimize_fastest_one_core_first is set to true, the fastest 10 % (by default) of the Techila Workers will be used in the computations. The percentage value can adjusted by setting a different value for the techila_optimize_maxpowerdifference parameter. For example, the following Project parameters could be used to include the fastest 25 % of Techila Workers in the computations.

General syntax:

This parameter can be used to specify the maximum allowed difference in benchmark values for Techila Workers in order to participate in a Project. This differences is defined as a percentage value. This parameter will need to be used in combination with the techila_optimize_fastest_one_core_first parameter.

General syntax:

Example: The following parameter pair could be used to allow Techila Workers that have a benchmark value belonging to the top 25% to participate in the Project.

The image below illustrates how these Project parameters determine which Techila Workers are allowed to participate in the Project.

By default, one Job will be assigned to one CPU core on a Techila Worker. This means that if a Techila Worker has 4 CPU cores, the maximum number of Jobs that will be assigned to it will be 4. The techila_optimize_max_jobs_per_worker parameter can be used to limit the maximum number of Jobs that will be assigned to a Techila Worker. This value cannot exceed the number of CPU cores on the Techila Worker.

General syntax:

Example: The following syntax could be used to limit the number of Jobs per Techila Worker to 2.

This parameter can be used to control how Jobs are distributed between the Techila Workers. When the techila_optimize_one_core_first parameter is set to true, one Job will be assigned to each available Techila Worker before assigning a second Job to any Techila Worker. This means that the Jobs will be divided evenly between available Techila Workers.

General syntax:

The following syntax could be used to enable this parameter:

This parameter can be used to control how Jobs are distributed between the Techila Workers. When the techila_optimize_one_core_only parameter is set to true, one Job will be assigned to each available Techila Worker. When a Techila Worker completes a Job, a new Job will be automatically assigned to it. This means that only one Job will be running on any given Techila Worker at any given time.

General syntax:

Assigns Jobs to one core of the Worker only. This is similar to multithreading_project, but doesn’t prevent Jobs from other Projects to be assigned to the same Worker.

By default, the number of Jobs running simultaneously is only limited by your Techila License and the amount of Techila Worker CPU cores available. The techila_project_joblimit parameter can be used to define a limit on how many Jobs of the Project will be executed simultaneously.

General syntax:

Example: The following syntax could be used to limit the number of Jobs running simultaneously to 20.

By default, Jobs from a Project will be assigned to Techila Workers as soon as you create the Project. The techila_project_schedule can be used to define a schedule for your Project, which defines the time windows when Jobs from the Project will be assigned to Techila Workers. Please note that defining a schedule does not prevent the Job execution from continuing on the Techila Workers after the given time window, it only affects the time when the Jobs are assigned for computation.

General syntax:

Time ranges are strings, which may contain several time ranges separated with a semicolon (;). Each of the time ranges may contain days of week (0-7 where both 0 and 7 are Sunday) and starttime-endtime values separated with a comma. The examples below show syntaxes for defining time windows:

Example 1: From Monday to Friday 00:00 - 07:00 and 17:00 - 24:00, and a full day from Saturday to Sunday.

Example 2: Every day from 00:00 - 07:00, 11:00 - 12:00 and 17:00 - 24:00.

Example 3: Only on weekends.

By default, temporary result files stored on your computer will have the .data suffix. The techila_results_filesuffix parameter can be used to define an alternative file suffix for these temporary result files.

General syntax:

The following example syntax could be used to set the suffix of the temporary file names to .myfiletype.

These temporary result files will be automatically removed from your computer after they have been processed.

The techila_semaphore_<x> parameter can be used to create a Project specific semaphore. The name of the semaphore will need to be defined in <x>. The amount of semaphore tokens will need to be given as the parameter value. More general information about semaphores can be found in Introduction to Techila Distributed Computing Engine.

General syntax:

For example, the following syntax could be used to create a Project specific semaphore named mysema which has 5 tokens.

By default, Projects will be started as soon as suitable computational capacity is available. The techila_prereq_projects parameter can be used to define a comma separated list of Project IDs, which must be completed before starting this Project.

General syntax:

For example, the following syntax could be used to specify that Projects 1234 and 1235 need to be completed before starting this Project.

By default, execution of a Job will be aborted on a Techila Worker if it generates an error. This behaviour can be changed by defining a whitelist of error codes that should not result in Job termination. The list is defined as a Bundle parameter:

For example, the following syntax could be used to whitelist error code 1.

After white listing error codes, the techila_max_retries parameter can be used to define how many times the Job is retried. If you do not define a value for techila_max_retries the Job will be retried the default amount (12 times).

The following example syntax could be used to retry a Job 20 times if the Job returns error code 1.

Alternatively, you can also ignore specific error codes by using the ignore parameter as shown below.

With the example above, a Job is marked successfully completed even if exits with code 11. This can be useful in situations in where the computations are completed successfully, but the application exits with a non-zero code.

This parameter only takes effect if you have defined a whitelist of error codes that should be retried using Bundle parameters as shown below.

When a whitelist of retryable error codes has been defined, the techila_retry_delay parameter can be used to override the default delay (10 seconds) between retries.

The following example syntax could be used to set the retry delay to 20 seconds if the Job returns error code 1.

This parameter can be used to define an optional regular expression to be checked against stderr and stdout before allowing the retrying of Job execution. When defining the regular expression statement, dotall mode can be enabled by adding a (?s) notation at beginning of the regex statement. Enabling dotall mode will mean that the entire stderr and stdout streams will be checked for a match. More information about the dotall mode can be found here.

Example 1: The following parameter pair could be used to retry Job execution if the Job exits with error code 1 and the string myErrorString is detected in the output.

Example 2: The following parameter pair could be used to retry Job execution if the Job exits with any error code and the string myErrorString2 is detected in the output.

This parameter can be used to define optional comma separated error log file(s) to be included into the error message. By default a maximum of 5kB of the beginning and 5kB of the end of the file are included to the error message. If you want to include the contents of the entire file, add ;includeall=true in the file definition.

Example: The following parameter could be used to include the content of file myfile.log in the error message.

The screencapture below illustras how the file contents are displayed in the error message in the Techila Web Interface.

By default, a Techila Worker is removed from a Project if a Job belonging to the Project generates an error on the Techila Worker. After an error is generated on a Techila Worker, no additional Jobs from the Project will be assigned to the Techila Worker.

The IgnoreWorkerErrors parameter can be used change this behavior and keep any Techila Workers that generate errors in the Project.

General syntax:

When a Job is processed on a Techila Worker, a temporary working directory is automatically created and used to store files used in the computations. By default, this temporary working directory will be automatically removed after the Job is terminated.

The DebugDoNotRemoveDir parameter can be used to prevent the directory from being removed. This can be useful in debugging situations where you want to examine the files that were generated in the temporary working directory.

The syntax for enabling the parameter is shown below:

The value of 1 must be given as a string.

The DebugDoNotRemoveDirOnError parameter works similarly as DebugDoNotRemoveDir, but is only triggered if the Job generates an error while it is being processed.

The syntax for enabling the parameter is shown below:

The value of 1 must be given as a string.