Techila Distributed Computing Engine with R

This package can be installed using the following R command:

After downloading and installing the package, the functions in the rJava package should become accessible from R. You can verify that the installation procedure was successful by loading the package with the following R command:

If the installation has failed, please ensure that the following environment variables are set correctly:

This package can be installed using the following R command:

After downloading and installing the package, the functions in the R.utils package should become accessible from R. You can verify that the installation procedure was successful by loading the package with the following R command:

The techila package is included in the Techila SDK and contains TDCE R commands.

Please follow the steps below to install the techila package. The appearance of screens may vary, depending on your R version, operating system and display settings.

The list below contains some of the R specific activities that are performed automatically when the peach function is used to create a computational Project.

The list below contains some of the R specific activities that are performed automatically when using the cloudfor function to create a computational Project.

The foreach example included in the Techila SDK is shown below:

This example will create a Project consisting of five Jobs. The code starts by loading the required packages: techila and foreach.

After this, the TDCE foreach backend is registered and the Techila SDK’s techila directory location is defined.

The foreach syntax used to perform the computations in TDCE starts by defining that the computational result should be stored in variable result and that the number of iterations should range from 1 to 10.

Each Job will perform two iterations. Because the total number of iterations was set to 10, this means the Project will consist of five Jobs.

The results will be combined with the c operator. This means that the results will be returned as a numerical vector, instead of a list.

The %dopar% notation will push the computations to the TDCE environment. (If you would change this to %do%, the operations would be executed sequentially on your computer.)

The code that will be executed in each iterations is quite trivial, consisting of simply calculating the square root of the loop counter i.

After the Project has been completed, the results will be returned from the TDCE environment and printed to the R console on your computer.

To create the computational Project, change your current working directory in your R environment to the directory that contains the example material relevant to this example.

After having browsed to the correct directory, you can source the Local Control Code using command:

After having sourced the file, create the computational Project using command:

This will create a Project consisting of five Jobs, each Job performing two iterations of the foreach loop structure. The example screenshot below illustrates what the expected output looks like.

In order to execute function from the plyr package in parallel using TDCE, the following packages need to be loaded: techila and plyr. After loading the packages, the TDCE backend can be registered using the syntax illustrated below

The notation in <> needs to be replaced with the location of your techila directory. For example, if your Techila SDK is located in C:/techila, then you could register the backend with the following syntax.

After registering the TDCE backend, functions from the plyr package can be executed in a TDCE environment by setting .parallel=TRUE as shown in the example snippet below.

The array ozone is included in the plyr package and is a 24 x 24 x 72 numeric array. The code snippet above would calculate the average value for each row in the ozone array in a separate Job, meaning the Project would consist of 24 Jobs. The plyr package has been transferred to the Techila Workers by using the .packages parameter.

This example illustrates how the computations performed with ddply can be executed in parallel in a TDCE environment. The example uses the iris data frame, which is included in the plyr package. The code for the example in the Techila SDK is shown below:

The code starts by loading the required packages: techila and plyr. After loading the packages, the TDCE foreach backend is registered and the Techila SDK’s techila directory location is defined.

The operation will be performed on data frame iris, which will be split into parts according values in Species . The operation numcolwise(mean) will be executed for each data frame part. The computations are marked for parallel execution and the required plyr package will be transferred to all participating Techila Workers.

The iris data frame contains three unique values in the Species column and the data for each Species will be processed in a single Job. This means that the Project will consist of three Jobs.

To create the computational Project, change your current working directory in your R environment to the directory that contains the example material relevant to this example.

After having browsed to the correct directory, you can source the Local Control Code using command:

After having sourced the file, create the computational Project using command:

This will create a Project consisting of three Jobs, each Job processing the data for one Species. The example screenshot below illustrates what the expected output looks like.

The locally executable program used in this example is shown below.

The code contains a single for-loop, which contains a single multiplication operation where the value of the i variable is squared. The value of the i variable will be replaced with the iteration number, which will be different each iteration. The result of the multiplication will be stored in the result vector at the index determined by the value of the i variable.

The locally executable program can be executed by changing your current working directory in R to the directory containing the material for this example and executing the command shown below:

Executing the command shown above will calculate 10 iterations. The values stored in the result-array are shown in the image below.

The cloudfor version of the locally executable program is shown below.

The command library(techila) will be executed when the file is sourced. After executing the command, the functions in the techila-package will be available.

The for-loop in the locally executable version has been replaced with a cloudfor-loop. The %t% notation after the cloudfor-loop defines that the code inside the following curly brackets should be executed in the Techila Distributed Computing Engine (TDCE) environment. In this example, the executable code block only contains the operation where the value of the i variable is squared.

The .sdkroot control parameter is used to define the location of the techila directory. In this example, a relative path definition has been used. This definition will be used in all of the R example material in the Techila SDK.

The .steps control parameter is used to define that two iterations should be calculated in in each Job. This means that for example if the number of loops is set to 10, the number of Jobs will be 5 (number of loops divided by the value of the steps parameter).

The computational Project can be created by executing the cloudfor version of the program. The cloudfor version can be executed by changing your current working directory in R to the directory containing the material for this example and executing the command shown below:

After you have executed the command, the Project will be automatically created and will consist of five (5) Jobs. These Jobs will be assigned and computed on Techila Workers in the TDCE environment. Each Job will compute two iterations of the loop structure and will return a list containing two values returned from the loop evaluations. This list will be stored in an output file, which will be automatically transferred to the Techila Server.

After all computational Jobs have been completed the result files will be transferred to your computer from the Techila Server. The values stored in the output files will be read and stored in the result-array. This array will contain all the values from the iterations and will correspond to the output generated by the locally executable program discussed earlier in this Chapter.

The locally executable program used in this example is shown below.

During the initial steps of the program, the tables stored in files datafile.txt and datafile2.txt will be read and stored the variables values and targetvalue respectively. The targetvalue variable will contain one row of data and the values variable will contain four rows of data with a similar structure.

The computational part consists of comparing the values of the rows stored in the values variable with the row stored in the targetvalue variable. Each line is compared during a separate iteration of the outermost for-loop. A graphical illustration of the data is shown in the image below.

The result of the comparison will be stored in the result-matrix, which will contain a row of FALSE values for rows that did not match. The matching row will be marked with TRUE values.

The locally executable program can be executed by changing your current working directory in R to the directory containing the material for this example and executing the command shown below:

The cloudfor version of the locally executable program is shown below. Line numbers have been added.

The code starts by loading the techila package, making the functions in the package available.

After loading the package, the code will load the table in file datafile1.txt, store the values to the values variable and determine the number of columns and rows in the table. An empty result array will also be created, which will be used to store row comparison results on the Techila Worker.

Before creating a Project, the code will execute an additional local for-loop. This loop will be used to create four (4) new files. Each file will contain one row of data extracted from the file datafile1.txt.The first row will be stored in a file called input_1, the second row in a file called input_2 and so on. These files will be used as Job-specific input files and will be transferred to the TDCE environment later in the program.

After generating the files, a list containing all file names starting with input that are located in the current working directory will be created. This list will be used later in the program to define a list of files that should be used as Job-specific input files.

The cloudfor-loop used in this example will range from one (1) to number of rows in the entire data table. The results of the computational Project will be stored in the result variable.

The number of iterations performed in each Job will be set to one (1) by using the .steps parameter. This means that the Project will contain four (4) Jobs, one Job for each row of data.

The location of the techila directory is set with the .sdkroot parameter.

The .datafiles parameter defines that datafile2.txt should be transferred to all participating Techila Workers.

Respectively, the .jobinputfiles parameter is used to transfer Job specific input files. In this example, the filenames parameter contains one list item, meaning one file will be given for each Job. File input1 will be assigned for Job 1, file input2 for Job 2 and so on. Each file will be renamed to input after it has been transferred to the Techila Worker.

After transferring these files to the Worker(s), they will be loaded using the following two lines:

These lines will be read the contents of datafile2.txt (will be same in each Job) and input (will be different in each Job).

After reading the files, a similar element-wise comparison will be performed as in the locally executable program. The result of the comparison will be stored in variable result and returned from the Job.

After the Project has been completed, the cloudfor function will return and the results will be stored in variable result. Creating the computational project

The computational Project can be created by executing the cloudfor version of the program. To execute the program change your current working directory in R to the directory containing the material for this example and execute the command shown below:

The Project will contain four (4) Jobs. Each Job will compare the row stored in the Job-specific input file with the row in the file datafile2.txt. The result of the comparison will be stored in the result-array, which will be returned from the Techila Worker.

After all Jobs have been completed, the results will be transferred to the End-Users computer. The results returned by the cloudfor-loop will be in a list form. The values in the list will the stored in a matrix, which will be identical as in the locally executable version.

The source code of the locally executable program (located in the file local_function.r) used in this example is shown below.

The function called local_function contains two perfectly nested for for-loops. The innermost for-loop will call the multiply function, which will perform some simple arithmetic operations using the loop counter values i and j as input arguments. The result of the operation will be stored in the result-matrix at the indices corresponding to the values of the loop counters i and j. The value of the operation will also be printed each iteration. The content and values stored in the result-matrix are illustrated in the image below.

The cloudfor version of the locally executable program is shown below.

The cloudfor version contains three functions:

The run_streaming function contains the cloudfor-loops, which have been used to replace the normal for-loops in the locally executable program. In addition, control parameters have been used to enable the result streaming and for defining the name of the callback function.

The parameter shown above will enable individual Job results to be streamed from the Techila Server as soon as they are available. When streaming has been enabled, individual Job results will be returned from the Techila Server in the order the Jobs are completed. This means that the results will be returned in no specific order. The effect of this is illustrated by the callback function, which will display the results in the order in which they will be received from the Techila Server.

The parameter above defines the function cbfun as the callback function, meaning this function will be used to process each of streamed Job results. In this example, the function will only print the content of the job_result variable, which will contain the value returned from each Job. The values printed during the callback function will most likely be in a different order than in the locally executable version. These results will be automatically reshaped to a 2x3 matrix (same as in the locally executable version) after all results have been received from the Techila Server.

The multiply function is identical as in the locally executable version and contains simple arithmetic operations that use the values of the loop counters as input arguments. This function call is inside the innermost cloudfor-loop, meaning the function will be executed on the Techila Workers.

The computational Project can be created by executing the cloudfor version of the program. To execute the program, change your current working directory in R to the directory containing the material for this example and execute the command shown below:

The Project will contain six (6) Jobs. In each of the computational Jobs, the multiply function will be called with different input arguments. The combinations of the input arguments will be identical as in the locally executable program, meaning the operations performed in the computational Jobs will correspond to the operations performed in the locally executable program.

Individual Job results will be streamed in the order they are completed and will be automatically processed by the callback function. After all results have been completed, the results will be reshaped and the matrix containing the results will be printed.

The source code of the example discussed here can be found in the following file in the Techila SDK:

The code used in this example is also illustrated below for convenience.

The code starts by executing the operating system command whoami, which displays the current domain and user name. This command will be executed on the End-User’s computer, meaning the command will return the End-User’s own AD user name. The user name will be stored in the local_username variable.

The cloudfor-loop used in this example will create a computational Project, which will consist of one Job.

AD impersonation is enabled by using the Project parameter techila_ad_impersonate. With this parameter enabled, the entire computational process will be executed using the End-User’s own AD user account.

The whoami command is then used to get the identity of the Job’s owner on the Techila Worker. Because AD impersonation has been enabled, this command should return the End-User’s AD user name and domain. If AD impersonation would be disabled (e.g. by removing the techila_ad_impersonate Project parameter), this command would return the Techila Worker’s AD user name.

After the Project has been completed, information about which AD user account was used locally and during the computational Job will be displayed.

To create the computational Project, change your current working directory in your R environment to the directory that contains the example material relevant to this example.

After having browsed to the correct directory, you can source the Local Control Code using command:

After having sourced the file, create the computational Project using command:

After the Project has been completed, information about the AD user accounts will be displayed. Please note that the output generated by the program will change based your domain and AD account user names. The example screenshot below illustrates the program output when the End-User’s own AD account name is techila and the domain name is testdomain.

The source code of the example discussed here can be found in the following file in the Techila SDK:

The code used in this example is also illustrated below for convenience.

The code will create a Project consisting of four Jobs. Simultaneous processing in Jobs is limited by using Project-specific and global semaphores. After completing the Project, the information about the semaphore usages will be displayed.

The Project parameter .ProjectParameters = list("techila_semaphore_examplesema" = "2") will create a Project-specific semaphore named examplesema. The number of tokens in the semaphore will be set to two. This means that a maximum of two tokens can be reserved at any given time.

When a Job is started on a Worker, the current time stamp will be retrieved and stored in jobStart variable. This will be used to mark the start of the Job.

After getting the time stamp, each Job reserves one token from a Project-specific semaphore examplesema. This command will wait indefinitely, until a semaphore token has been reserved. This means that the first two Jobs that will execute this function, will reserve the tokens. The remaining Jobs will wait until semaphore tokens are released by the Jobs that reserved them.

After getting a semaphore token, the Job gets the current time stamp, which is used to mark the start of the semaphore reservation time. The genload function is then called which will generate CPU load for 30 seconds by generating random numbers. The code for the genload function can be found at the end of the file.

After exiting the genload function, the code calculates how many seconds elapsed between the start of the Job (jobStart variable) and the genload function call (tstart variable). If a Job was able to reserve a token right away, this value should be close to 0. If the Job had to wait for a semaphore token to become available, this value will be close to 30. Please note that if the Jobs were not started at the same time, you will get different values.

The Job will then calculate the time window when the Project-specific semaphore token was reserved relative to the start of the Job and store the information to variable result.

The Job will then release a token belonging to the Project-specific semaphore examplesema, making it available for any other Job that is waiting for a token to become available.

After this, the Job will attempt to reserve a token from a global semaphore called globalsema by using the techila_smph_reserve function (syntax shown below for convenience). The syntax also sets the value of the ignoreerror argument to TRUE, meaning code execution is allowed to continue, even if there was a problem with the semaphore reservation process.

If your TDCE environment has a global semaphore called globalsema, then the function will return the value TRUE. If your TDCE environment does not have a global semaphore called globalsema, then the function will return the value FALSE. Please note that global semaphores will need to be created by your local Techila Administrator. This means that unless your local Techila Administrator has created a semaphore named globalsema, the value returned by the will be FALSE.

The return value is stored in the reservedok variable, which will be used to define which of the following if-statements should be executed.

If the reservedok variable contains value TRUE, the first if-clause will be executed. During these lines, five seconds of CPU load will be generated and the token reservation time window will be calculated. After this, the token belonging to the global semaphore named globalsema will be released. After this, information about the time window when the global semaphore token was reserved by the Job will be stored in the result string.

If the reservedok variable contains value FALSE, the second if-clause on lines will be executed. In this case, a simple string containing an error message will be stored in the result variable.

After the Project has been completed, the last for-loop in the code will be executed. During this for-loop, information about the semaphore reservation times will be displayed on the screen by printing the strings stored during Jobs.

The example figure below illustrates how Jobs in this example are processed in an environment where all Jobs can be started at the same time. In this example figure, the global semaphore globalsema is assumed to exist and that it only contains one token.

The activities taking place during the example Project are explained below.

After the Jobs have been assigned to Techila Workers, two of the Jobs are able to reserve a Project-specific semaphore token and begin generating CPU load. This processing is illustrated by the Computing, Project-specific semaphore reserved bars. During this time, the remaining two Jobs will wait until semaphore tokens become available. After the first Jobs have released the Project-specific semaphores (i.e. after generating CPU load for 30 seconds), Jobs 3 and 4 can reserve semaphore tokens and start processing the generating CPU load.

The global semaphore only contains one token, meaning only one Job can reserve a token at any given time. In the example figure below, Job 1 reserves the token and starts generating CPU load. This processing is represented by the Computing, global semaphore reserved bars. After Job 1 has completed generating CPU load (i.e. after 5 seconds), the global semaphore is released and Job 2 can start processing.

After Jobs 3 and 4 finish generating CPU load, the Jobs will start reserving tokens from the global semaphore.

To create the computational Project, change your current working directory in your R environment to the directory that contains the example material relevant to this example.

After having browsed to the correct directory, you can source the Local Control Code using command:

After having sourced the file, create the computational Project using command:

Please note that the output generated by the program will change based on whether or not the global semaphore named globalsema is available. The two example screenshots below illustrate the output in both scenarios.

Please also note that there might be overlap in the reported time windows. This is because the time windows are measured from the timestamp generated at the start of the code, which means that e.g. initialization delays can cause the reported times to overlap.

The example screenshot below illustrates the generated output when the global semaphore exists.

The example screenshot below illustrates the generated output when the global semaphore does not exist.

The locally executable R script called local_function.r contains one function called local_function, which consists of one for loop. The algorithm of the locally executable function used in this example is shown below.

The program requires one input parameter, which defines the number of iterations that will be performed in the for loop. Every iteration performs the same arithmetic operation: 1+1. The result of the latest iteration will get appended to the result vector. The result vector for three iterations is shown below.

To execute the function, please source the R code using command:

After the R script has been sourced, the function can be executed using command:

After executing the function, the numerical values stored in the result variable will be displayed.

All arithmetic operations in the locally executable function are performed in the for loop. There are no recursive data dependencies between iterations, meaning that the all the iterations can be performed simultaneously. This is done by placing the computational instructions in the Techila Worker Code (distribution_dist.r).

Local Control Code in the R script run_distribution.r is used to create the computational Project. The Techila Worker Code in the distribution_dist.r file is transferred to the Techila Workers where script will automatically be sourced at the preliminary stages of the Job. After the R script has been sourced, the function distribution_dist will be executed.

The Local Control Code used to control the distribution process is shown in below.

The script defines one function called run_distribution, which requires one input parameter. This input parameter will be used to specify the number of Jobs into which the Project should be split. This will be further done by using the jobs input parameter, which defines the length of the peachvector.

In this example, no input arguments are required by the function that will be executed on the Techila Workers. This means that the params parameter does not need to be defined.

At the final stages of the code, as soon as the results have been transferred back to the End-User’s local computer, the results will be converted to numeric format.

The Techila Worker Code that performs the computations is shown below.

Operations performed in the Techila Worker Code are equivalent to one iteration of the locally executable loop structure. As no input parameters will be transferred to the Techila Worker Code, identical arithmetic operations are performed during all Jobs. The interaction between the Local Control Code and the Techila Worker Code is illustrated in the image below.

To create the computational Project, please change your current working directory in your R environment to the directory containing the example material for this example.

After having browsed to the correct directory, you can source the Local Control Code using command:

After having sourced the R script, execute the function using command:

This will create a computational Project consisting of three Jobs. Each of the Jobs will be extremely short, as each Job consists of simply summing up two integers; 1+1. The computations occurring during the computational Project are illustrated in the image below:

The algorithm for the locally executable function used in this example is shown below.

This function requires two input parameters; multip and loops. The parameter loops determines the number of iterations in the for loop. The parameter multip is a number, which will be multiplied with the iteration number represented by x. The result of this arithmetic operation will then be appended to a vector called result, which will be returned as the output value of the function. The result vector in a case of five iterations is shown below.

To execute the function, please source the R code using command:

As soon as the R script has been sourced, the function can be executed using the command shown below:

After executing the function, numerical values stored in the result variable will be displayed. If you executed the function using the input parameters shown above, the following values will be printed:

All the computations in locally executable R script are performed in the for loop and there are no dependencies between the iterations. Because of this, the locally executable program can be converted to a distributed version by extracting the arithmetic operation into a separate piece of code. Input parameters for the executable R function will be transferred with the params array of the peach function.

The Local Control Code used to control the distribution process is shown in below.

The function run_parameters requires two input parameters multip and jobs. The multip parameter is listed in the params array, meaning it will be transferred to Techila Workers and given as the first input argument to the executable function.

The jobs parameter is used to define the length of the peachvector, meaning the value of the jobs parameter will define the number of Jobs in the Project. Elements of the peachvector will also be given as the second input argument to the executable function. This is because the second entry in the params parameter is the "<param>" notation, which will automatically be replaced with a different peachvector element on the Techila Workers.

The algorithm for the Techila Worker Code is shown below.

The Local Control Code discussed earlier defined two parameters in the params parameter. One of these parameters was a static parameter (multip) and the other was a dynamic parameter ("<param>"). In the Techila Worker Code, the static parameter is being represented by a parameter called multip, which will be constant across all Jobs. The dynamic parameter in the Local Control Code is represented by jobidx parameter, which will get replaced by a different element of the peachvector in each Job. As a result we can say that jobidx parameter simulates the iteration number of the locally executable function.

The interaction between the Local Control Code and the Techila Worker Code is illustrated in the image below.

To create the computational Project, change your current working directory in your R environment to the directory that contains the example material for this example.

After having browsed to the correct directory, you can source the Local Control Code using command:

After having sourced the R script, the computational Project can be created using following command:

This will create a computational Project that will consist of five Jobs. The parameters in the params parameter in the peach function call will be given values based on the input arguments of the run_parameters function. The Techila Workers will execute the parameter_dist function using one static and one dynamic input parameter.

The static parameter multip to will be set to two (2). The peachvector will contain the integers from one to five. These integers are used to define the value of the jobidx parameter in the Techila Worker Code. The computational operations occurring during the computational Project are illustrated in the image below.

The locally executable R script used in this example is shown in below.

During the initial steps of the function, the table in the file datafile.txt will be stored in the contents variable by using the read.table command. The computational part consists of calculating the sum of each column in the table that is stored in the contents variable. The sum of one column is calculated during each iteration.

To execute the function, please source the R code using command:

As soon as the R script has been sourced, the function can be executed using the command shown below:

After executing the function, a line will be printed that will display the sums of each column in the table. The printed values should correspond to the values shown below:

In the distributed version, the file datafile.txt will be transferred to Techila Workers by using the files parameter of the peach function. The values of one column in the table will be summed during one Job.

The Local Control Code that is used to create the computational Project is shown below.

The function run_data_files requires no input parameters. The number of Jobs in the computational Project will be determined by value of the jobs parameter. This is done by using the jobs parameter to define the length of the peachvector. Elements of the peachvector are also used as a dynamic input parameter in the parameter params as indicated by the "<param>" notation.

The files parameter contains the name of the file (datafile.txt) that will be transferred to all Techila Workers.

The algorithm of the Techila Worker Code used in this example is shown below.

The Local Control Code introduced earlier defines one dynamic input parameter. This is represented in the Techila Worker Code by the jobidx parameter, which will get replaced by a different element of the peachvector in each Job. In Job 1, the value will be one (1), in Job 2, the value will be 2 and so on. This means that the jobidx parameter can be used to point to the correct column during each Job.

The Local Control Code also introduced one (1) filename in the files array. This file will be copied to the same temporary working directory at the Techila Worker with the executable code, which means that the file datafile.txt can be loaded into memory using the same syntax as in a locally executable function.

To create the computational Project, change your current working directory in your R environment to the directory that contains the example material for this example.

After having browsed to the correct directory, you can source the Local Control Code using command:

After having sourced the R script, execute the function using command:

The number of Jobs in the Project will be automatically fixed to four as the value of the jobs parameter is defined in the Local Control Code. Parameters in the params parameter and the file specified in the files parameter will be transferred to the Techila Workers. The function will be executed using the dynamic input parameter and the Techila Workers will access the file specified in the files parameter from the temporary working directory.

The R script containing the locally executable functions is shown below.

To execute the functions on your local computer, please source the R code using command:

As soon as the R script has been sourced, function1 function can be executed with command:

When called, function1 will perform the summation 1+1 and return 2 as the result.

Respectively, the function called function2 can be executed with command:

When called, function2 will perform the multiplication 10*10 and return 100 as the result.

In this example, the functions in the locally executable R script will be placed directly into the R script containing the Techila Worker Code (multi_function_dist.r).

Local Control Code in the R script (run_multi_function.r) is used to create the computational Project. The funcname parameter in the peach function call determines which function will be called from the functions defined in the Techila Worker Code.

The Local Control Code used to create the computational Project is shown below.

The function run_multi_function requires one input parameter. This input parameter is used to determine, which function will be called during a Job. This is performed by setting the value of the funcname parameter to the value of the input parameter.

The Techila Worker Code ("multi_function_dist.r") used in this example is shown below.

As can be seen, the Techila Worker Code contains the same function definitions as the locally executable R script. The Techila Worker Code will be sourced at the preliminary stages of a computational Job, meaning both functions will be defined during a computational Job. This means either function can be called by with the funcname parameter in the Local Control Code.

To create the computational Project, change your current working directory (in R) to the directory that contains the example material relevant to this example.

After having browsed to the correct directory, you can source the Local Control Code using command:

After having sourced the Local Control Code, a computational Project that executes function1 on Techila Workers can be created using command shown below:

This will create a computational Project that consists of one (1) Job. The computational operations occurring during the Project are illustrated in the image below.

Respectively, function2 can be executed on the Techila Worker by using the command:

The computational operations occurring during the Projects are illustrated in the image below.

The locally executable function for approximating the value of Pi used in this example is shown below.

The local_function-function requires one input argument called loops, which determines the number of iterations in the for loop. During each iteration, two random numbers will be generated, which will be used as the coordinates of the random point. The coordinates of the point are then used to calculate the distance of the point from the centre of the unit circle. If the distance is less than one, the point is located within the unit circle and the counter is incremented by one. As soon as all iterations have been completed, the value of Pi will be calculated.

To execute the locally executable function that approximates the value of Pi, source the R code using command:

As soon as the R code has been sourced, the function can be executed using command:

This will approximate the value of Pi using 10,000,000 randomly generated points. The operation will take approximately five minutes, depending on the speed of your CPU. If you wish to perform a shorter approximation, reduce the number of random points generated to e.g. 1000000.

After the approximation is completed, the approximated value of Pi will be displayed in the R Console as shown below.

Note that due to randomness of the Monte Carlo method, the last decimals in your result will likely differ from the one shown above.

The computationally intensive part in Monte Carlo methods is the random number sampling, which is performed in the for loop in the locally executable function. There are no dependencies between the iterations. This means that the sampling process can be divided into a separate function and executed simultaneously on several Techila Workers.

Note that the seed of the random number generator is initialized automatically on the Techila Workers by the peachclient as explained in R Peach Function. If you wish to use a different seeding method, please seed the random number generator directly in the Techila Worker Code.

The Local Control Code used in this example to create the computational Project is shown below.

The first line of the Local Control code consists of loading the techila library by using the library(techila) command. The line containing the peach function call is responsible for distributing the computations to the distributed computing environment. After the peach function returns, individual results from Jobs will be combined and used to calculate the approximate value of Pi.

The code that is executed on Techila Workers in this example is shown below.

The algorithm is very similar to the algorithm of the locally executable function. The function requires one input argument called loops which is being used to determine the number of iterations. Every iteration calculates the distance of a randomly generated point from the centre. If the distance is less than one, the point is within the unit circle and the count is incremented by one. No post-processing activities are performed in the Techila Worker Code, as the results from individual Jobs are post-processed in the Local Control Code.

To create the computational Project, change your current working directory to the directory containing the example material. Source the Local Control Code using command:

As soon as the R script has been sourced, please execute the function using command:

This will create a Project consisting of ten Jobs, each containing 1,000,000 iterations. The Jobs will be distributed to Techila Workers, where the Monte Carlo routine in the Techila Worker Code is executed. When a Techila Worker finishes the Monte Carlo routine, results are transferred to the Techila Server. After all the Techila Workers have transferred the results to the Techila Server, the results are transferred to the End-Users computer. After the results have been downloaded, the last line in the control code is executed which contains the post-processing operations, which in this case consist of scaling the results according to the number of performed iterations.

The Local Control Code of the Monte Carlo Pi where Callback function and Streaming are used is shown below.

The Local Control Code here consists of two functions, run_streaming and callbackFun. The run_streaming function will distribute the computations using peach. The function called callbackFun is the callback function that will be executed every time a new result will be streamed from the Techila Server to the End-User’s computer. The variables used in the callback function are declared as global to preserve the values of the parameters between function calls. The input argument of the callback function (result) will be replaced by the result returned from the Techila Worker Code.

The Callback function callbackFun contains the arithmetic operations to continuously update the approximated value of Pi. The value will be printed in a continuous manner as results are received from the Techila Server. The Callback function will return the approximated value of Pi, based on the results received so far. This approximated value will be stored in the result vector returned by peach.

The Techila Worker Code used in this example is shown below.

The code is similar to the basic implementation introduced in Monte Carlo Pi with Peach, the differentiating factor is that both the count and loops variables are returned in a list. This means that the callback function callbackFun in the Local Control Code will receive the list as the input argument.

To create the computational Project, change your current working directory (in R) to the directory that contains the example material relevant to this example.

After having browsed to the correct directory, you can source the Local Control Code using command:

After having sourced the R script, execute the function using command:

This will create a computational Project consisting of 20 Jobs, each Job performing a Monte Carlo routine that consists of 100,000 iterations. Results will be streamed from the Techila Server to End-User as they are completed and the approximated value continuously as more results are streamed.

The Local Control Code for creating a project that uses Job-Specific Input Files is shown below.

The datafiles parameter in the jobinputfiles parameter specifies which files should be used in each Job. The syntax used in this example is shown below:

The filenames parameter in the jobinputfiles parameter specifies the names of the files after they have been transferred to the Techila Worker. The syntax used in this example is shown below:

This syntax assigns the name input.txt to all Job-Specific Input Files.

Techila Worker Code used to perform operations on the Job-specific input files is shown below.

In this example, all the Jobs access their input files by using the file name input.txt. Each Techila Worker then sums the numbers in the Job-specific input file. The value of the summation will be returned as the result.

To create the computational Project, change your current working directory in your R environment to the directory that contains the example material for this example.

After having browsed to the correct directory, you can source the Local Control Code using command:

After having sourced the R script, execute the function using command:

This will create a Project consisting of four Jobs. The system will automatically assign a Job-Specific Input File to each Job, according to the jobinputfiles parameter in the Local Control Code. This is illustrated in the image below.

The Local Control Code in the run_detached.r script contains two functions, which can be used for creating the detached Project and for downloading the results.

The first peach function call creates a Project and detaches it by using the parameter pair:

The parameter pair causes the peach function to return after the Project has been created. The Project ID number of the Project will be returned by the peach function and stored in the variable pid. This Project ID number will be used to download results after the Project has been completed.

After the Project has been completed, the results can be downloaded by executing the download_result function.

The peach function call in the download_result function will be used to connect to the Techila Server and request the results. The download request is linked to the previously created project with the following parameter pair:

The pid parameter will specify the Project ID number of the Project that the results will be downloaded for. The value of the parameter is defined as the input argument of the download_result function. The downloaded results will be post-processed to calculate the approximate value of Pi.

The code that is executed on the Techila Workers is shown below.

The Techila Worker Code performs the same Monte Carlo routine as was performed in the basic Monte Carlo Pi implementation presented in Monte Carlo Pi with Peach.The only difference is that the function returns a list containing the results (variable count) and the number of iterations performed (variable loops).

The number of iterations is stored in order to preserve information that is required in the post-processing. Embedding the variables required in post-processing in the result files means, that the post-processing activities can be performed correctly regardless of when the results are downloaded.

To create the computational Project, change your current working directory in your R environment to the directory that contains the example material relevant to this example.

After having browsed to the correct directory, you can source the Local Control Code using command:

After having sourced the file, create the computational Project using command:

This creates a Project consisting of ten Jobs. After all of the computational data has been transferred to the Techila Server, the Project ID number will be returned to the pid variable. The Project ID number can be used to download the results of the Project after all the Project has been completed.

After the Project has been completed, the results can be downloaded from the Techila Server with the download_result function using the syntax shown below:

The Local Control Code used to create several, consecutively created projects is shown below.

The peach function call is placed inside a loop structure, which is implemented with a while statement. New computational Projects will be created until the error in the approximation is below the predefined threshold value. The amount of error in the approximation will be printed every time a new Project has been completed. Note that messages have been disabled in order to provide a more clear illustration of the results received from Projects.

If the peach function returns NULL, the example will be stopped and all temporary files will be removed by using the unload command. The unload command will also be automatically executed after all Projects have been completed. The run_iterative-function will return the amount of error in the approximation after all Projects have been completed.

The algorithm for the Techila Worker Code is shown below.

The seed of the random number generator is defined by the values of jobidx and iteration variables. This is simply to ensure that the number of consecutive Projects required stays within a reasonable limit. The computational operations performed in the Techila Worker Code are similar as in the basic implementation presented in Monte Carlo Pi with Peach, returning the number of random points that are located within the unitary circle.

To create the computational Project, change your current working directory (in R) to the directory that contains the example material relevant to this example.

After having browsed to the correct directory, you can source the Local Control Code using command:

After having sourced the R script, execute the function using command:

The command shown above will create projects consisting of 20 Jobs. Each Job will consist of 100,000 iterations. Projects will be created until the error of the approximated value is smaller than the threshold value. The error of the approximation will be printed every time a Project has been completed.

The Local Control Code used in this example is shown below.

The Local Control Code creates two Data Bundles. Files file1_bundle1 and file2_bundle1 for the first Data Bundle will be read from a folder called storage, which is located in the current working directory. Files file1_bundle2 and file2_bundle2 will be read from the current working directory. Expiration periods of the Data Bundles will be set to 60 minutes for the first Bundle and 30 minutes for the second Bundle.

The Techila Worker Code used in this example is shown below.

The Techila Worker Code contains instruction for reading each of the files included in the Data Bundles. Contents of the Data Bundles will be copied to the same temporary working directory as the executable R code, meaning the files can be accessed without any additional path definitions.

To create the computational Project, change your current working directory in your R environment to the directory that contains the example material relevant to this example.

After having browsed to the correct directory, you can source the Local Control Code using command:

After having sourced the Local Control Code, create the computational Project using command:

This creates a Project consisting of one (1) Job. Two Data Bundles will be created and transferred to the Techila Server, from where they will be transferred to the Techila Worker. If you execute the Local Control Code several times, the Data Bundles will only be created in the first Project, subsequent Projects will use the Data Bundles stored on the Techila Server.

The Local Control Code used in this example is shown below.

The Local Control Code shown above defines two functions; mcpi_dist and run_funchandle. The run_funchandle function contains the necessary commands for creating the computational Project by using the peach`function. The `funcname parameter of the peach function call refers to the mcpi_dist function and is entered without quotation marks.

The mcpi_dist-function contains the executable code that will be executed on Techila Workers during the computational Jobs. This function will be defined when the Local Control Code is sourced in the preliminary steps when creating the computational Project.

Please note that the files parameter is not be used, meaning that no R scripts will be sourced during the preliminary stages of a computational Job.

To create the computational Project, change your current working directory in your R environment to the directory that contains the example material relevant to this example.

After having browsed to the correct directory, you can source the Local Control Code using command:

After having sourced the Local Control Code, create the computational Project using command:

This will create a Project consisting of ten Jobs, each Job performing 1,000,000 iterations of the Monte Carlo routine defined in the mcpi_dist function. The values returned from the Jobs will be used to calculate an approximate value for Pi.

The Local Control Code used in this example is shown below.

In the example shown above, two files (file1 and file2) are defined as output files. These output files will be transferred from the Techila Worker to the End-Users computer at the final stages of the computational Project. After the files have been transferred, the filehandler_func function will be called to process each of the result files.

The filehandler_func will be called once for each additional result file. The function will print the path and name of the file. The variable stored in the result file will be loaded to memory and printed using the print command.

The Techila Worker Code used in this example is shown below.

Each Job in the computational Project generates two files that are called file1.txt and file2.txt.These filenames were defined as output files in the Local Control Code and will be transferred to the End-Users computer.

To create the computational Project, change your current working directory in your R environment to the directory that contains the example material relevant to this example.

After having browsed to the correct directory, you can source the Local Control Code using command:

After having sourced the Local Control Code, create the computational Project using command:

This creates a Project consisting of two (2) Jobs. Two additional result files will be transferred from each Job. Each of the result files will be processed by the file handler function, which will display information on each of the additional result files.

The Local Control Code used in this example is shown below.

Snapshots are enabled with the following parameter pair in the Local Control Code:

No other modifications are required to enable snapshotting with the default snapshot transfer interval. Apart from the parameter pair used to enable snapshotting, the structure of the Local Control Code is similar to the basic implementation as shown in Monte Carlo Pi with Peach.

The Techila Worker Code used in this example is shown below.

During the initial steps in the Techila Worker Code the count and iter values are initialized. These initialization values will be used in situations where a Snapshot cannot be found. If a Snapshot file exists, it will indicate that the Job is being resumed after an interruption. In this case, the content of the Snapshot file will be used to override the initialized values. This will be performed using the loadSnapshot function, which automatically loads the contents of the Snapshot file to the workspace. Iterations will be resumed from the last value stored in the Snapshot file.

Intermediate results will be stored in the Snapshot by calling the saveSnapshot function every1e7th iteration. The variables stored in the snapshot file are iter and count. The parameter iter will contain the number of iterations performed until the snapshot generation occurred. The parameter count will contain the intermediate results.

To create the computational Project, change your current working directory in your R environment to the directory that contains the example material relevant to this example.

After having browsed to the correct directory, you can source the Local Control Code using command:

After having sourced the Local Control Code, create the computational Project using command:

This creates a Project consisting of 10 Jobs, each Job performing 1e8 iterations. Intermediate results will be saved at every 1e7th iteration. Snapshot files will be transferred every 15 minutes from the Techila Worker to Techila Server. If a Job is migrated to a new Techila Worker while the Job is being computed, the latest available Snapshot file will be automatically transferred from the Techila Server to the new Techila Worker.

Snapshot data can be viewed and downloaded by using the Techila Web Interface. Instructions for this can be found in the document Techila Web Interface End-User Guide.

Note that when using the syntax shown above to run the example, the execution time of single Job is relatively short. This might result in the Job being completed before a Snapshot file will be transferred to the Techila Server. If Snapshot data is not visible in the Techila Web Interface, consider increasing the amount of iterations to increase the execution time of a Job.

An installed R package can be stored in an R Package Bundle with the packages parameter as shown below:

Where <package> is the name of the installed package that should be placed in the R Package Bundle.

Multiple packages can be transferred by listing the packages as a comma separated list:

For example, the following syntax could be used to transfer packages pracma and MASS packages.

The material used in this example is located in the following folder in the Techila SDK:

An installed R package can be stored in an R Package Bundle with the bundleit function (included in the techila package) as shown below:

Where <package> is the name of the installed package that should be placed in the R Package Bundle and <path_to_sdk_root> is the location of your techila_settings.ini file.

For example, the command shown below could be used to place the stats package in an R Package Bundle.

The bundleit function will return the name of the Bundle that was created. The general naming convention of the Bundle is shown below:

Where the values enclosed in "<>" would be replaced with system specific values. These values are explained below:

For example, when creating an R Package Bundle of the stats package using R 2.12.1, the name of the R Package Bundle would resemble the one shown below:

The bundle name will be used to include package in a computational Project, so it is advisable to store the Bundle name in a variable. The name of the bundle will be different for each user, differentiated by the first entry in the bundle name (demouser in the example), the package name (stats in the example) and the version number (v1 in the example).

Note that all Bundles must have a unique name, meaning that executing the bundleit command again with identical parameters will not re-create the Bundle. A new version of an R Package Bundle can be created by modifying the version number:

The command shown above would return and print the following value:

By default, the R Package Bundle will only be available for Techila Workers with the same operating system platform as the computer that was used to create the Package Bundle. A Package Bundle can be made available for all Techila Workers with the following parameter pair:

For example, an R Package Bundle of the stats package for all operating system platforms could be created with the following command:

The following example illustrates how to store an R package in an R Package Bundle and use the Bundle in a computational Project.

The material used in this example is located in the following folder in the Techila SDK:

The source code of the example discussed in this Chapter shown below. The commented version of the code can be found in the following file in the Techila SDK:

The above code will create a Project consisting of two Jobs, where each Job will consist of one iteration. Each Job will transfer a short string to the other Job in the Project. After both Jobs have sent (and received) the data, the Project will be completed.

Below is an illustration of the operations that will be performed in this Project when the Jobs are assigned to Techila Workers.

Below is a more detailed explanation on the code sample.

The number of iterations in each Job should be set to one:

The .steps parameter is required for two reasons:

If the .steps parameter would be removed, the code would be executed locally on the End-User’s computer when estimating the execution time of an iteration. This would generate an error, because the interconnect functions are not defined and cannot be executed on the End-User’s computer.

In this example, the code is also structured so that both iterations must be running simultaneously in different Jobs. For this reason, the value of the .steps parameter has been set to one.

At the start of the code, the interconnect network will be initialized with techila.ic.init(). If no interconnect network can be established, the code will generate an error on this line.

The computational code that is executed on the Techila Workers contain two if-statements, which determine the operations that will be executed in each Job. Job 1 will execute the code inside the first if-statement (i==1) and Job 2 will execute the code inside the other code branch (i==2).

Job 1 will start by executing the send_data_to_job, which is used to transfer data to Job 2. Job 2 respectively starts by executing the ‘recv_data_from_job` function, which is used to read the data, which is being transferred by Job 1.

After Job 2 has received the data, the roles are reversed, meaning Job 2 will transfer data to Job 1.

After Job 1 has received the data from Job 2, both Jobs exit their respective if-statements and execute the wait_for_others() function (line 18), which will act as a synchronization point in the Jobs.

After both Jobs have executed the techila.ic.wait_for_others() function, the techila.ic.disconnect() function will be executed, which will close the connection to the interconnect network.

After the Jobs have been completed, the results will be downloaded and stored to the result variable on the End-User’s computer where the results will be printed on the screen.

To create the computational Project, change your current working directory in your R environment to the directory that contains the example material relevant to this example.

After having browsed to the correct directory, you can source the Local Control Code using command:

After having sourced the file, create the computational Project using command:

The example screenshot below illustrates the program output, which will display the message strings that were transferred between Jobs.

The source code of the example discussed in this Chapter shown below. The commented version of the code can be found in the following file in the Techila SDK:

This example will create a Project with three (3) Jobs. Job 2 will transfer the string Hi from Job 1 to Jobs 1 and 3. The transferred string will be displayed on the End-User’s computer after the Project has been completed.

The value of the sourcejob parameter to two (2), which will be used to define which Job will broadcast the data. If you want to transfer data from another Job, simply change the value to either 1 or 3, depending on which Job you want to use to broadcast the data.

The message that will be broadcasted is a string containing the the Job’s index number. The example table below illustrates values of the datatotransfer variable in each Job, in a Project consisting of 3 Jobs.

The cloudbc-function that will be used to broadcast the data from one Job to all other Jobs in the Project is shown below.

The value of the variable sourcejob will determine which Job will broadcast data. The data that will be transferred is defined by the value of the datatotransfer variable. With the values used in this example, Job 2 will broadcast the string Hi from Job 2 to all other Jobs in the Project.

In each Job, the techila.ic.cloudbc function will return the string that was broadcasted and store it in the jobres variable.

The line containing techila.ic.wait_for_others() will act as a synchronization point, meaning Jobs will wait until all other Jobs have also executed the function before continuing.

After all Jobs have reached the syncronization point, they will disconnect from the interconnect network.

To create the computational Project, change your current working directory in your R environment to the directory that contains the example material relevant to this example.

After having browsed to the correct directory, you can source the Local Control Code using command:

After having sourced the file, create the computational Project using command:

When executed, the code will create a Project consisting of three (3) Jobs. Job 2 will broadcast data to other Jobs in the Project. Below figure illustrates the operations that take place when the code is executed with the syntax shown above.

The example screenshot below illustrates the program output, which will display the message string that was broadcasted during the Project.

The source code of the example discussed in this Chapter shown below. The commented version of the code can be found in the following file in the Techila SDK:

As can be seend from the example code, data can be transferred to all Jobs from all other Jobs by using the send_data_to_job and recv_data_from_job functions combined with regular for-loops and if-statements. These for-loops and if-statements will need to be implemented so that each Job that is sending data has a matching Job that is receiving data.

The above example code will create a Project with 4 Jobs where simple strings will be transferred from each Job to all other Jobs in the Project.

The number of Jobs in the Project is determined by calling the techila.get_jobcount function, which will return the number of Jobs (4) in the Project.

The message string that will be transferred between Jobs will contain the Job’s index number to indicate which Job sent the message. The table below shows the messages transferred from each Job.

The two for-loops inside the cloudfor-loop contain the code that will decide the order in which Jobs will transfer messages to other Jobs. The transferred messages will be stored to the dataall list, which will be returned to the End-User’s computer.

The interconnect data transfers that take place during the Project are illustrated in the figure below. The arrows indicate that interconnect data is being transferred. The values in parentheses correspond to the values of the src and dst loop counters. For example, arrow with value (1,3) means that Job 1 is sending the msg string to Job 3. If src is equal to dst (e.g. (2,2)), no data is transferred because the source and target are the same.

To create the computational Project, change your current working directory in your R environment to the directory that contains the example material relevant to this example.

After having browsed to the correct directory, you can source the Local Control Code using command:

After having sourced the file, create the computational Project using command:

When the command is executed, the code will create a Project consisting of four (4) Jobs. Each Job will transfer a simple string to all other Jobs in the Project. These transferred strings will then be returned and displayed on the End-User’s computer as illustrated in the screenshot below.

The source code of the example discussed in this Chapter shown below. The commented version of the code can be found in the following file in the Techila SDK:

This example will create a Project with three (3) Jobs. Each Job will generate a random number, which will be multiplied by using the cloudop function. The multiplication result will be displayed on the End-User’s computer after the Project has been completed.

Each Job will set the random number generator seed at the start of the Job, which ensures that random numbers can be generated repeatedly.

After setting the seed, each Job generates one random number and stores the value in the data variable.

The multiply function is then executed across all Jobs with the input values stored in the data variable. The syntax used in this example defines two input arguments to the cloudop function (<op>, <data>), meaning the third input argument (<target>) has not been defined. This means that the final result of the cloudop function will be stored in variable mulval in all Jobs.

The definition for the multiply function is located at the end of the code. This function accepts two input arguments, multiplies them and returns the multiplication value as the result. This means that the multiply function meets the requirements for functions that can be executed with cloudop.

To create the computational Project, change your current working directory in your R environment to the directory that contains the example material relevant to this example.

After having browsed to the correct directory, you can source the Local Control Code using command:

After having sourced the file, create the computational Project using command:

The example screenshot below illustrates the program output, which will display the value of the multiplication result calculated during the Project. Each Job will return the same value, because the syntax of the cloudop-function used in this example did not define the <target> argument.