Techila Distributed Computing Engine with Python

The typical naming convention of Python scripts presented in this document is explained below:

Please note that some Python scripts and functions might be named differently, depending on their role in the computational Project.

The list below contains some of the Python specific activities that are performed automatically when the peach-function is used to create a computational Project.

The list below contains some of the Python specific activities that are performed when the @techila.distributable() is used to create a computational Project.

The code snippet below shows how you could decorate a function named fun using @techila.distributable() and execute the fun function in TDCE.

If you want, you can copy-paste the code snippet below to your Python environment and run it in your own TDCE environment.

Each call to a decorated function will append the function with its arguments to a list that will be executed in TDCE during a Project when techila.run() is called. This means that by calling the decorated function multiple times you can create a list of functions that will be executed in TDCE simultaneously.

For example, the following code snippet could be used to define a list of 4 function calls and execute each function simultaneously with different input arguments in the TDCE environment.

If you want, you can copy-paste the code snippet below to your Python environment and run it in your own TDCE environment.

You can build the list of function calls in a more efficient manner by using, for example, a for loop. The following example syntax could be used to create a list of 100 function calls and execute them in the TDCE environment.

If you want, you can copy-paste the code snippet below to your Python environment and run it in your own TDCE environment.

By default, each Job executes the function once. However, by using the steps parameter, you can specify that each Job should execute the function multiple times. Grouping function calls can improve performance in situations where one function call takes a relatively short time, but you have a very large range of input arguments to process.

For example, the following syntax could be used to specify that each Job should perform 50 function executions.

The code snippet below illustrates the use of this parameter.

If you want, you can copy-paste the code snippet below to your Python environment and run it in your own TDCE environment.

In situations where your computationally intensive function is located inside multiple, perfectly nested for loops, you can follow the same approach to decorate the function.

The following example syntax could be used to create a list of 10000 function calls and execute them in the TDCE environment.

If you want, you can copy-paste the code snippet below to your Python environment and run it in your own TDCE environment.

By decorating different functions, you can execute different functions in different Jobs inside one Project that is executed in TDCE.

For example, the following syntax could be used to create a list of 4 function calls and execute them in the TDCE environment. Two Jobs would execute fun and the remaining two Jobs would execute fun2

If you want, you can copy-paste the code snippet below to your Python environment and run it in your own TDCE environment.

Result data returned from the TDCE environment can be stored in variables defined in the function call. The following code snippet shows how you could return numpy data types from TDCE to your computer.

If you want, you can copy-paste the code snippet below to your Python environment and run it in your own TDCE environment.

The example code snippet below shows how you can execute the same function multiple times in TDCE and return a numpy data type from each Job.

If you want, you can copy-paste the code snippet below to your Python environment and run it in your own TDCE environment.

The Python script local_function.py containing the locally executable version is shown below.

The file contains two function definitions:

All operations in this example will be executed locally. In the Distributed Version, the fun function will be decorated, which means it will be executed in TDCE.

The Python script run_distribution.py containing the distributed version is shown below.

The file contains two function definitions:

When the run_distribution function is executed, the for loop will execute the decorated fun function. Each time the decorated function is executed, a TechilaItem object will be returned and stored in a list. This means that after the for loop has been completed, the result list will contain multiple TechilaItem objects.

For example, if x=5, the result list will contain 5 TechilaItem objects.

When the code reaches the techila.run() command, a computational Project will be created. The number of Jobs in the Project will be equal to the amount of TechilaItem objects. For example, if there are 5 TechilaItem objects, the Project will consist of 5 Jobs.

Each Job will execute the fun function and return the result of the summation operation (1+1) as the result. These results will be automatically streamed to the End-User’s computer and stored in the result list.

After the Project has been completed, the result list will contain 5 TechilaItem objects. Each TechilaItem object contains a result member variable, which contains the result of the summation operation. The example syntaxes shown below illustrate how you could retrieve these values.

The files needed to run this example are located in the following Techila SDK directory:

To run the local version, change your current working directory to the directory containing the example material and execute the following commands:

To run the distributed version, change your current working directory to the directory containing the example material and execute the following commands:

The Python script local_function.py containing the locally executable version is shown below.

The file contains two function definitions:

All operations in this example will be executed locally, each iteration multiplying different values. In the Distributed Version, the fun function will be decorated, which means it will be executed in TDCE using a set of different input arguments for each Job.

The Python script run_parameters.py containing the distributed version is shown below.

The file contains two function definitions:

When the run_parameters function is executed, the for loop will execute the decorated fun function. Each time the decorated function is executed, a TechilaItem object will be returned and stored in a list. This TechilaItem will also include the values of the input arguments multip and x. This means that after the for loop has been completed, the result list will contain multiple TechilaItem objects, with different sets of input arguments

For example, if loops=5 and multip=2, the result list will contain 5 TechilaItem objects with the following input arguments.

When the code reaches the techila.run() command, a computational Project will be created. The number of Jobs in the Project will be equal to the amount of TechilaItem objects. For example, if there are 5 TechilaItem objects, the Project will consist of 5 Jobs.

Each Job will execute the fun function and return the result of the multiplication operation as the result. These results will be automatically streamed to the End-User’s computer and stored in the result list.

Assuming values loops=5 and multip=2 are used to run the example, the Jobs will perform the following operations.

After the Project has been completed, the result list will contain 5 TechilaItem objects. Each TechilaItem object contains a result member variable, which contains the result of the multiplication operation. The example syntaxes shown below illustrate how you could retrieve these values.

The files needed to run this example are located in the following Techila SDK directory:

To run the local version, change your current working directory to the directory containing the example material and execute the following commands:

To run the distributed version, change your current working directory to the directory containing the example material and execute the following commands:

The Python script local_function.py containing the locally executable version is shown below.

The file contains two function definitions:

All operations in this example will be executed locally. In the Distributed Version, the fun function will be decorated, which means the file datafile.txt will be transferred to TDCE and accessed on Techila Workers.

The Python script run_datafiles.py containing the distributed version is shown below.

The file contains two function definitions:

When the run_datafiles function is executed, the for loop will execute the decorated fun function. Each time the decorated function is executed, a TechilaItem object will be returned and stored in a list. This TechilaItem object will contain the function call to the original function fun, meaning the file datafile.txt will need to be available on the Techila Workers when the function is executed.

When the code reaches the techila.run(datafiles=['datafile.txt']) command, a computational Project will be created. The datafiles parameter is used to transfer the file datafile.txt from the current working directory on your computer to the Techila Workers. After the file has been transferred to the Techila Workers, it will be stored in the same temporary working directory that is used to execute the Job.

Each Job will then run fun function and will read the file from the current working directory on the Techila Worker. This means that the same syntax can be used to read the file on the Techila Worker as was used in the local version.

After the Project has been completed, the result list will contain 5 TechilaItem objects. Each TechilaItem object contains a result member variable, which contains the result of the summation operation. The example syntaxes shown below illustrate how you could retrieve these values.

The files needed to run this example are located in the following Techila SDK directory:

To run the local version, change your current working directory to the directory containing the example material and execute the following commands:

To run the distributed version, change your current working directory to the directory containing the example material and execute the following commands:

The Python script local_multiple_functions.py containing the locally executable version is shown below.

The file contains 3 function definitions:

All operations in this example will be executed locally. In the Distributed Version, functions function1 and function2 will be decorated, and will be executed in TDCE.

The Python script run_multi_function.py containing the distributed version is shown below.

The file contains 3 function definitions:

When the run_multi_function function is executed, the for loop will execute function1 during the first iteration of the for loop, meaning the first TechilaItem object will contain a call to function1. As function2 will be executed during all other iterations, all other TechilaItem objects will contain a call to function2.

When techila.run() is executed, a computational Project will be created. Function function1 will be executed during Job #1 and function2 in all other Jobs (Job #2 - Job #N).

The files needed to run this example are located in the following Techila SDK directory:

To run the local version, change your current working directory to the directory containing the example material and execute the following commands:

To run the distributed version, change your current working directory to the directory containing the example material and execute the following commands:

The Python script run_callback.py containing the distributed version is shown below.

The file contains 3 function definitions:

The function run_callback takes two input arguments: jobs and loops. The jobs argument will determine the number of Jobs in the Project. The loops argument determines the number of loops calculated in each Job.

When the techila.run() function is executed, a computational Project will be created. When Jobs are completed, the results will be automatically streamed to the End-User’s computer. Each time a new result has been received, the decorated function callbackfun will be executed and will be used to process the results returned from the Job. In this example, the callback function will be used to calculate an approximated valuf for Pi.

The files needed to run this example are located in the following Techila SDK directory:

To run the distributed version, change your current working directory to the directory containing the example material and execute the following commands:

The Python script run_packagetest.py containing the distributed version is shown below.

This example also uses functionality from the techilatest/functions.py file shown below for reference.

In order to use functionality from the custom package on the Techila Workers, the package will need to be transferred to the TDCE environment. This can be done by passing the following parameter to the decorator:

This parameter defines that a package called techilatest should be transferred from your computer to the TDCE environment.

Additional packages can be transferred by listing them as comma separated list.

In order to run this example, you will first need to install the techilatest package using the following command:

After installing the package, you can run the example by executing the following commands in python:

The Python script local_function.py contains one function called local_function, which consists of one for-loop. The algorithm of the locally executable function used in this example is shown below.

The function takes one input argument, which defines the number of iterations that will be performed in the for-loop. Every iteration performs the same arithmetic operation: 1+1. The result of the latest iteration will be appended to the result list.

For example, when performing five iterations the result list would contain the following values.

To execute the locally executable function in Python, use the commands shown below:

All arithmetic operations in the locally executable function are performed in the for-loop. There are no recursive data dependencies between iterations, meaning that the all the iterations can be performed simultaneously. The iterations can be performed simultaneously by placing the arithmetic operations to a separate file (distribution_dist.py), which can be then transferred and executed on Workers.

The file containing the Local Control Code (run_distribution.py) will be used to create the computational Project. The Worker Code (distribution_dist.py) will be transferred to the Workers, where it will be automatically imported using command:

This import process will take place at the preliminary stage of Job. After the functions have been imported, the function distribution_dist will be executed. The result returned by the function will be returned from the Job.

The Local Control Code used to create the computational Project is shown below.

The code consists of one function called run_distribution. This function takes one input argument called jobcount (integer). This variable will be used during the peach-function call to define the value of the jobs parameter, which defines the number of Jobs in the Project.

Techila helper functions (including peach) are made available by importing the techila package.

The computational Project is created with a peach-function call. After the computational Project has been completed, the results will be stored in the result list. The number of list elements will be the same as the number of Jobs in the Project. Each list element will contain the result returned from one Job.

The parameters of the peach-function call are explained below.

The funcname parameter shown above defines that the function distributed_dist will be executed in each Job. This function will be defined when the file distribution_dist.py is evaluated as explained below.

The parameter shown above defines that code in file named distributed_dist.py should be imported using command from distribution_dist import *. This import process will make all variables and functions defined in the file accessible during the computational Job. In this example, the file contains the definition for the distribution_dist function.

The jobs parameter shown above defines the number of Jobs in the Project. In this example, the number of Jobs will be based on the value of the jobcount variable. The number of Jobs in the Project could also be defined by using the peachvector parameter. For more information on how to define the peachvector, please see Using Input Parameters.

After the Project has been completed, the results stored in result will be printed. Each element in the array will contain the result for one Job in the Project.

The function will return the result variable as the output.

The Worker Code that will be executed on the Workers is in the file called distribution_dist.py. The content of the file is shown below.

Operations performed in the Worker Code correspond to the operations performed during one iteration of the locally executable for-loop structure. Each Job will simply sum two integers (1+1) and store the value of the summation to the result variable. The result variable will returned from the Job will be eventually received by the End-User as a one of the list elements returned by the peach-function in the Local Control Code.

The interaction between the Local Control Code and the Worker Code is illustrated below.

To create the computational Project, change your current working directory (in Python) to the directory containing the example material for this example.

After having navigated to the correct directory, create the computational Project using commands shown below:

The computational Jobs will be extremely short as each Job consists of simply summing up two integers; 1+1. The computations occurring during the Project are illustrated below.

The algorithm for the locally executable function used in this example is shown below.

This function takes two input arguments; multip and loops. The parameter loops determines the number of iterations in the for-loop. The parameter multip is a number, which will be multiplied with the iteration counter represented by x. The result of this arithmetic operation will be appended to a list called result, which will be returned as the output value of the function.

An example result list (five iterations) is shown below.

The locally executable function can be executed in python using the commands shown below:

After executing the commands, numerical values stored in the result list will be displayed.

All the computations in locally executable Python function are performed in the for-loop and there are no dependencies between the iterations. As a result of this, the locally executable program can be converted to a distributed version by extracting the arithmetic operation into a separate piece of code (parameters_dist.py), which will be executed on Workers.

The Local Control Code (run_parameters.py) contains the peach-function call that will be used to create the computational Project. In order to perform similar operations in the Project as in the local version, input arguments need to be given to the executable function to simulate different iterations of the for-loop. These input arguments will be transferred by the using the params parameter of the peach-function.

The Local Control Code used to create the computational Project is shown below.

Input arguments for the function that will be executed on the Worker are defined with the params parameter. In this example, two input arguments will be passed to the executable function. The first input argument (multip) will be identical for all Jobs and will correspond to the value passed as an input argument to the run_parameters function. The second input argument ('<param>') will be replaced with a different peachvector element for each Job. These elements will be used to simulate the value of for-loop counter used in the locally executable function.

The peachvector will contain elements from one (1) to the value of the jobs parameter. As the '<param>' notation was used in the params parameter, these elements will be passed as input arguments to the executable function. Job #1 will receive the first element (value 1), Job #2 will receive the second element (value 2) and so on.

The number of elements in the peachvector will also define the number of Jobs in the Project. In this example, the number elements will be the same as the value of the jobs variable.

The value returned by the peach-function will be stored in the result list. Each list element will contain the result returned from one of the Jobs.

The function that will be executed on Workers (in file parameters_dist.py) is shown below.

The Local Control Code discussed earlier defined two parameters in the params parameter, which will be passed to the executable function (parameters_dist) as input arguments. The first input argument (multip) will have the same value in all Jobs. The second input argument (jobidx) will be replaced with different peachvector elements. The value of the jobidx parameter will fundamentally be used to simulate the value of the loop counter that was used in the locally executable function.

The interaction between the Local Control Code and the Worker Code is illustrated below.

To create the computational Project, change your current working directory (in Python) to the directory that contains the example material for this example.

After having navigated to the correct directory, create the computational Project using commands shown below:

If prompted, enter the password to your keystore file. The Project will be created and information on the Project progress will be displayed in the console. After all Jobs have been completed, the result list will be printed. Each element in the result list will contain the value returned from one Job. The Jobs will be extremely short as each Job consists of simply multiplying two integers.

The computational operations occurring during the Project are illustrated below.

The locally executable Python script used in this example is shown in below.

During the initial steps of the function, the file datafile.txt will be opened and the rows will be read and stored in the rows variable. This file contains four rows of numbers, where each number is separated by white space.

The function also contains a for-loop, which will be used to sum the values on each row and append the summation result in the contents list. After all rows have been processed the results will be displayed.

To execute the locally executable function in python, use the commands shown below:

Each row in the datafile.txt can be processed independently, meaning Jobs in the computational Project can be configured to process different rows simultaneously. This can be achieved by transferring the file datafile.txt to Workers by using the datafiles parameter of the peach-function.

Each Job can then be configured to process one of the rows in the data file. This can be achieved by using the indexes of the peahcvector (notation (<vecidx>)) to index the file and process a different row in each Job.

The Local Control Code that is used to create the computational Project is shown below.

Input arguments for the executable function are defined using the params parameter. In this example, the notation '<vecidx>' has been defined which means index values of the peachvector will be passed as an input arguments to the function. The value of the notation will be zero (0) for Job #1, one (1) for Job #2 and so on. These index values will be used to choose which row should be processed from the data file.

Data files are transferred to Workers using the datafiles parameter. In this example, the file datafile.txt has been defined and will be transferred to Workers. After the file has been transferred to the Worker, it will be copied in the same temporary working directory with the executable code.

The algorithm of the Worker Code used in this example is shown below.

The code starts by importing the csv package that will be used when reading the contents of the file. All packages included in the standard Python distribution will be automatically available for use on the Workers. Additional packages can be made available by using the bundleit command as explained in Using Custom Python Packages in Computational Projects

The function that will be executed on Workers takes one input argument (jobidx), which will be replaced by the index values of the peachvector as explained earlier (<vecidx> notation in the Local Control Code). This input argument will be used to select which row should be processed.

The first operations in the function will select the row that matches the value of the jobidx variable and sum the values on that row. This corresponds to the operations that are performed in one iteration of the for-loop structure in the locally executable function.

After processing the row, the summation result will be returned as the result from the computational Job.

The interaction between the Local Control Code and the Worker Code are shown below.

To create the computational Project, change your current working directory (in Python) to the directory that contains the example material for this example. After having navigated to the correct directory, create the computational Project using commands shown below:

If prompted, enter the password to your keystore file. The Project will be created and information on the Project progress will be displayed in the console. The number of Jobs in the Project will be automatically fixed to four according to the value of the jobs parameter defined in the Local Control Code.

The Python script containing the locally executable functions is shown below.

To execute the functions on your local computer, please define the functions using the commands shown below:

After the functions have been defined, function1 can be executed using command:

When called, function1 will perform the summation 1+1 and return 2 as the result.

Respectively, the function2 can be executed with command:

When called, function2 will perform the multiplication 10*10 and return 100 as the result.

In the locally executable version of the example, the Python script contains two function definitions.

In the distributed version, the functions can be made available on Workers by including both function definitions in the Python script that will be evaluated at the preliminary stage of the Job. This can be achieved by using the files parameter. As the file containing the functions will be evaluated before any functions are called, this also means that either one of these functions can be used as an entry point in the computational Job (values for the funcname parameter).

The Local Control Code used to create the computational Project is shown below.

The function run_multi_function takes one input argument, which will be used to define the value of the funcname parameter. This means that the input argument will be used to determine, which function will be executed during a Job.

The files parameter lists one file called multifunction_dist.py, which contains the same function definitions that were defined in the locally executable version of this example. This either one of the functions (function1 and function2) can be used as an entry point.

The file containing the Worker Code ('multi_function_dist.py') used in this example is shown below.

As can be seen, the Worker Code contains the same function definitions as the locally executable version of the program. The Python script containing the function definitions will be evaluated at the preliminary stages of a computational Job, meaning both functions will be defined and can be called during a computational Job.

This also means either function can also be used as the entry point when defining the funcname parameter in the Local Control Code.

To create the computational Project, change your current working directory (in Python) to the directory that contains the example material for this example.

After having navigated to the correct directory, create the computational Project using commands shown below:

After having evaluated the Local Control Code, a computational Project that executes function1 on Workers can be created using command shown below:

This will create a computational Project that consists of one (1) Job. The computational operations occurring during the Project are illustrated below.

Respectively, function2 can be executed on the Worker by using the command:

The computational operations occurring during the Project are illustrated below.

The locally executable function for approximating the value of Pi used in this example is shown below.

The function takes one input argument called loops, which determines the number of iterations in the for-loop. During each iteration, two random numbers will be generated, which will be used as the coordinates of the random point. The coordinates of the point are then used to calculate the distance of the point from the centre of the unit circle. If the distance is less than one, the point is located within the unit circle and the counter is incremented by one. After all iterations have been completed, the value of Pi will be calculated.

To execute the function locally on your computer, use the commands shown below:

This will approximate the value of Pi using 10,000,000 randomly generated points. The operation will take approximately one minute, depending on your CPU. If you wish to perform a shorter approximation, reduce the number of random points generated to e.g. 1,000,000.

After the approximation is completed, the approximated value of Pi will be displayed.

The computationally intensive part in the Monte Carlo method is the random number sampling, which is performed inside the for-loop in the locally executable function. There are no dependencies between the iterations, meaning that the sampling process can be divided into a separate function and executed simultaneously on Workers.

Note that the seed of the random number generator is initialized automatically on the Workers by the peachclient as explained in Process Flow in Computational Projects. If you wish to use a different seeding method, please seed the random number generator directly in the Worker Code.

The Local Control Code used in this example to create the computational Project is shown below.

The file contains one function called run_mcpi, which takes two input arguments: jobcount and loops. The jobcount parameter will be used to define the value for the jobs parameter (in the peach-function call) and will define the number of Jobs in the Project. The loops parameter will be passed as an input argument (by using params) to the mcpi_dist function that will be executed on Workers.

After all Jobs have been completed, the peach-function will return a list called result. Each list element will correspond to the result generated in one Job and will contain the number of points within the unitary circle. These values will be combined and used to calculate the approximate value of Pi.

The Worker Code used in this example is shown below.

The function mcpi_dist is very similar to the algorithm of the locally executable function. The function takes one input argument called loops which will be used to determine the number of iterations performed during the Job. During each iteration, the distance of a randomly generated point from the centre will be calculated. If the distance is less than one, the point is within the unit circle and the count is incremented by one.

No post-processing activities are performed in the Worker Code, as the results from individual Jobs will be post-processed in the Local Control Code after all Jobs have been completed.

To create the computational Project, change your current working directory in your Python environment to the directory that contains the example material for this example.

After having navigated to the correct directory, create the computational Project using commands shown below:

This will create a Project consisting of ten Jobs, each containing 1,000,000 iterations. The Jobs will be distributed to Workers, where the Monte Carlo routine in the Worker Code is executed. When a Worker finishes the Monte Carlo routine, results are transferred to the Techila Server. After all the Workers have transferred the results to the Techila Server, the results are transferred to the End-Users computer. After the results have been downloaded, the post-processing operations will be performed and the approximated value of Pi will be displayed.

The Python script containing the Local Control Code for this example is shown below.

The Local Control Code used in this example contains of two functions, run_streaming and callbackfun.

The run_streaming function contains the peach-function call, which will be used to distribute the computations using the peach-function. The syntax of the peach-function defines that individual Job results should be streamed (stream = True) and that the results should be processed by using a callback function ('callback=callbackfun').

The function callbackfun is the callback function that will be executed every time a new Job result is streamed from the Techila Server to your computer. This function is used to combine the individual results and to display the approximation result each time a new Job result has been received.

The input argument of the callback function (jobresult) will be a dict object, which is returned by the code that is executed on the Workers. In this example, the object will contain the number of points inside the unitary circle (count) and the number of iterations performed in the Job (loops). The values returned from Jobs will be added together and stored in the global variables (variables starting with total_) in order to preserve the values between function calls.

Each time a new Job result has been processed, the current value of the approximation will be displayed. The callback function will return the individual Job results in the same format as they were received. These returned values will be stored as a list element in the list returned by the peach-function.

The Worker Code used in this example is shown below.

The code is similar to the basic implementation introduced in Monte Carlo Pi with Peach. The differentiating factor is that the function returns a dict object, which will contain the number of iterations performed in the Job (loops) and the number of points within the unitary circle (count). This object will be automatically passed as an input argument to the callback function callbackfun defined in the Local Control Code.

To create the computational Project, change your current working directory (in Python) to the directory that contains the example material for this example.

After having navigated to the correct directory, create the computational Project using commands shown below:

This will create a computational Project consisting of 20 Jobs, each Job performing a Monte Carlo routine that consists of 1,000,000 iterations. Results will be streamed from the Techila Server to End-User as they are completed and the approximated value continuously as more results are streamed.

The Local Control Code for creating a project that uses Job-Specific Input Files is shown below.

The peach-function syntax used in the example will create a Project consisting of four Jobs. Each of the Jobs will receive one of input files specified in the jobinputfiles parameter. File input1.txt will be transferred to with Job #1, file input2.txt is transferred with Job #2 and so on. Note that the number of entries in the list is equal to the number Jobs in the Project.

After the files have been transferred to Workers, they will be copied to the same temporary working directory with the executable code. Each file will also be renamed to input.txt during this process.

Worker Code used to perform operations on the Job-specific input files is shown below.

In this example, all Jobs will access the input files by using the file name input.txt which was defined in the Local Control Code. These files will be accessed from the same temporary working directory that contains the executable code, meaning no path definitions will be required. Each Job will sum the numbers in the Job-specific input file and return the value of the summation as the result.

To create the computational Project, change your current working directory (in Python) to the directory that contains the example material for this example.

After having navigated to the correct directory, create the computational Project using commands shown below:

This will create a Project consisting of four Jobs. The system will automatically assign a Job-specific input file to each Job, according to definition of the jobinputfiles parameter in the Local Control Code. This is illustrated below.

The Local Control Code (run_detached.py) used in this example is shown below.

The code contains two functions, run_detached and download_result.

The run_detached function contains the peach-function call, which will be used to create the computational Project. The peach-function syntax contains the following parameter that specifies that the Project should be detached after all necessary data has been transferred to the Techila Server.

This parameter will also cause the peach-function to return the Project ID number of the Project that was created. This Project ID number will be stored in the variable pid and will be used later when downloading the results for the Project.

After creating the Project, the download_result function can be used to download the results. This function takes one input argument (pid), which will be used to link the peach-function call to a previously created Project.

After results have been downloaded, the results will be stored in the results list. Each list element will contain the result returned from one Job. The list elements will then be combined and used to calculate an approximated value for Pi.

The code that is executed on the Workers is shown below.

The code used in this example performs the same Monte Carlo routine as in the basic implementation introduced in Monte Carlo Pi with Peach. The differentiating factor is that the function returns a dict object, which will contain the number of iterations performed in the Job (loops) and the number of points within the unitary circle (count).

The number of iterations is stored in order to preserve information that is required in the post-processing. Storing all variables required in post-processing in the result files means, that the post-processing activities can be performed correctly regardless of when the results are downloaded.

To create the computational Project, change your current working directory (in Python) to the directory that contains the example material for this example.

After having navigated to the correct directory, create the computational Project using commands shown below:

This creates a Project consisting of ten Jobs. After all of the computational data has been transferred to the Techila Server, the Project ID number will be returned and stored to the 'pid' variable. This Project ID number can be used to download the results of the Project after all the Jobs have been completed.

After the Project has been completed, the results can be downloaded from the Techila Server with the download_result function using the syntax shown below. Please note that if you execute the function before the Project has been completed, the peach-function will wait for the Project to be completed.

The Local Control Code used to create several, consecutively created projects is shown below.

The Local Control Code used in this example contains three logical steps:

These steps are explained below:

Step 1: The connection to the Techila Server will be initialized by calling the init function before performing the peach-function call. When init is called before the peach-function, the peach-function will not uninitialize the connection after downloading the results. This means that the connection will remain available for subsequent peach-function calls. This also means that when creating several consecutive Projects, performing a separate connection initialization is typically more efficient than performing a new initialization with each peach-function call.

Step 2: The peach-function call used to create the computational Projects is placed inside a loop structure, which is implemented with a while statement. Each computational Project will be used to compute an approximation for the value of Pi. The results generated in the Projects will be combined and compared to the value of Pi (math.pi) to compute the current error in the approximation.

New computational Projects will be created until the error in the approximation is below the predefined threshold value. The amount of error in the approximation will be printed every time a new Project has been completed. Note that messages have been disabled (messages = False) in order to provide a more clear illustration of the results received from Projects.

Step 3: After the error in the approximation drops below the threshold, the loop structure will be exited. The connection to the Techila Server will be uninitialized by using the uninit function, which will also remove all temporary files created by the peach-function.

The run_iterative-function will return the amount of error in the approximation after all Projects have been completed.

The algorithm for the Worker Code is shown below.

The executable function takes three input arguments: loops, jobidx and iteration. The loops variable is used to define the number of iterations performed in the while-loop. The jobidx and iteration variables are used to specify the seed of the random number generator. This is done in order to ensure that the number of required Projects required stays within a reasonable limit.

Apart from the modifications described above, the computational operations performed in the Worker Code are similar as in the basic implementation presented in Monte Carlo Pi with Peach, returning the number of random points that are located within the unitary circle.

To create the computational Projects, change your current working directory (in Python) to the directory that contains the example material relevant to this example.

After having browsed to the correct directory, create the computational Projects using commands:

Executing the commands shown above will create several, consecutively created Projects each consisting of 20 Jobs. New Projects will be created until the error in approximation is smaller than the threshold value. The error of the approximation will be printed every time a Project has been completed.

The Local Control Code used in this example is shown below.

The Local Control Code used in this example will create two Data Bundles. Files file1_bundle1 and file2_bundle1 will be read from a folder called storage and stored in the first Data Bundle. Files file1_bundle2 and file2_bundle2 will be read from the current working directory and stored in the second Data Bundle. Expiration periods of the Data Bundles will be set to 60 minutes for the first Bundle and 30 minutes for the second Bundle.

The Worker Code used in this example is shown below.

The Worker Code contains instruction for reading each of the files included in the Data Bundles. The files in the Data Bundles will be copied to the same temporary working directory as the executable Python code, meaning the files can be accessed without any additional path definitions.

After reading the files, the executable function will return a list containing the values read from the files. The value returned from the Job will be eventually returned by the peach-function defined in the Local Control Code.

To create the computational Project, change your current working directory (in Python) to the directory that contains the example material for this example.

After having navigated to the correct directory, create the computational Project using commands shown below:

This creates a Project consisting of one (1) Job. Two Data Bundles will be created and transferred to the Techila Server, from where they will be transferred to the Worker. Please note that if you execute the Local Control Code several times, the Data Bundles will only be created with the first Project. Any subsequent Projects will use the Data Bundles stored on the Techila Server.

The Local Control Code used in this example is shown below.

The Local Control Code shown above defines two functions; mcpi_dist and run_funchandle.

The run_funchandle-function contains the peach-function call that will be used to create the computational Project. The funcname parameter of the peach-function call refers to the mcpi_dist-function and is entered without quotation marks.

An additional function dependency has beed defined by using the funclist parameter. With the syntax used in this example, the point_dist function will also be transferred to the Techila Distributed Computing Engine environment and can be used during the computational Jobs.

The mcpi_dist-function contains the executable code that will be executed on Workers to start the computational Jobs. During the computational Job, the point_dist function is used to generate the random points.

Please note that the files parameter is not used, meaning that no Python scripts will be sourced during the preliminary stages of a computational Job.

To create the computational Project, change your current working directory in your Python environment to the directory that contains the example material for this example.

After having navigated to the correct directory, create the computational Project using commands shown below:

This will create a Project consisting of ten Jobs, each Job performing 1,000,000 iterations of the Monte Carlo routine defined in the mcpi_dist function. The values returned from the Jobs will be used to calculate an approximate value for Pi.

The Local Control Code used in this example is shown below.

In the peach-function syntax shown above, regexp notations have been used to specify all files starting with the string output_file as output files. These files will be transferred from the Workers to your computer. The regexp notation is required, because each of the output files will have a different name.

After the files have been transferred to your computer, the filehandler_func function will be used to process each of the output files. The filehandler_func will be called once for each output file. Each time the function is called, the value of the input argument (variable file) will be automatically replaced with the name and path of output file. In this example, the file handler function will be used to print the path and name of the file and to copy the file to the current working directory.

The Worker Code used in this example is shown below.

Each Job in the computational Project will generate one output file. The name of the output file will include the value of the input argument jobidx, meaning the name of the file will be different for each Job. Job #1 will generate a file called output_file1, Job #2 will generate a file called output_file2 and so on. The names of all generated output files will match the regexp filter defined in the Local Control Code, meaning all files will be transferred to the End-Users computer.

To create the computational Project, change your current working directory (in Python) to the directory that contains the example material for this example.

After having navigated to the correct directory, create the computational Project using commands shown below:

This creates a Project consisting of five Jobs. Each Job will generate one additional output file, which will be processed by the file handler function. The file handler function will display the temporary locations and content of each output file. The files will be copied to the current working directory as can be seen from the output generated by the listdir function.

The Local Control Code used in this example is shown below.

Snapshots are enabled with the following parameter pair in the Local Control Code:

No other modifications are required the Local Control Code to enable snapshotting with the default snapshot parameters. Apart from the parameter pair used to enable snapshotting, the structure of the Local Control Code is similar to the basic implementation as shown in Monte Carlo Pi with Peach.

The Worker Code used in this example is shown below.

The snapshot helper functions are imported to the namespace using the following command:

After the import command has been executed, the helper functions can be called in the Worker Code.

During the initial steps in the executable function the count and i variables are declared global and initialized. The initialization values will be used in situations where a snapshot file cannot be found (e.g. when starting the Job). If a snapshot file exists, it will indicate that the Job is being resumed after an interruption. In this case, the content of the snapshot file will be used to override the initialized values. This will be performed using the load_snapshot function, which automatically loads variables stored in the snapshot file to the Python environment. After loading the snapshot file, computations will be resumed from the last iteration value (i) stored in the snapshot file.

Intermediate results will be stored in the snapshot file by calling the save_snapshot function every1e7th iteration. The variables stored in the snapshot file are i and count. The parameter i will contain the number of iterations performed when the snapshot generation occurred. The parameter count will contain the intermediate results.

To create the computational Project, change your current working directory (in Python) to the directory that contains the example material for this example.

After having navigated to the correct directory, create the computational Project using commands shown below:

This creates a Project consisting of 10 Jobs, each Job performing 1e8 iterations. Intermediate results will be saved at every 1e7th iteration. Snapshot files will be transferred approximately every 30 minutes from the Worker to Techila Server. If a Job is migrated to a new Worker while the Job is being computed, the latest available snapshot file will be automatically transferred from the Techila Server to the new Worker.

Snapshot data can also be viewed and downloaded by using the Techila Web Interface. Instructions for this can be found in the Techila Web Interface End-User Guide.

Note that when using the syntax shown above to run the example, the execution time of single Job is relatively short. This might result in the Job being completed before a snapshot file will be transferred to the Techila Server. If snapshot data is not visible in the Techila Web Interface, consider increasing the amount of iterations to increase the execution time of a Job. You can also experiment by specifying a shorter snapshot interval with the snapshotinterval parameter as explained earlier in this Chapter.

In this example, a package called techilatest will be installed on the End-Users computer. This package will then be transferred to the Techila Server by using the bundleit function. The package is located in the same folder that contains the example material for this example and contains one module called functions.py.

The content of the function.py file is shown below. Instructions for installing the package can be found at the end of this Chapter.

The functions shown above can be used to perform basic arithmetic operations. Each function takes two input arguments. These functions will be called during the computational Jobs. The input arguments will be defined in the Local Control Code shown below.

The Local Control Code (in file run_packagetest.py) used in this example is shown below.

The script contains two functions, which will be used to perform the following operations:

The function create_package contains the bundleit-function call, which will be used to create a Bundle from the techilatest package. The function will return the name of the Bundle (bundlename) that was created, which can be used to import the Bundle to the computational Project.

The function run_packagetest contains the peach-function call, which will be used to create the computational Project. The function takes one input argument (bundlename) which will be used to define which Bundle should be imported to the Project (imports = [bundlename]).

The peach-function syntax also defines two input arguments (params=['<vecidx>','<param>']), which will be passed to the executable function. These input arguments will eventually be passed to the functions in the techilatest package.

The Worker Code used in this example is shown below.

The first line in the Worker Code will load the functions module from the techilatest package. The import can be performed because the Bundle containing the package has been transferred to all participating Workers. After importing the module, the functions can be accessed during the computational Job.

To install the techilatest package, launch a command prompt / terminal and change your current working directory to the directory that contains the example material for this example. Install the package using command:

Executing the command should generate a similar output as shown below (paths may differ).

After installing the package, launch Python and follow the instructions below to run the example.

Evaluate the Python script containing the Local Control Code using command shown below:

After sourcing the file, create a Bundle from the package techilatest using command:

Create a computational Project using command:

This will create a Project containing five Jobs. The Bundle containing the techilatest package will be transferred to each participating Worker and the package will be available on these Workers. After the Project has been completed, the values generated in the Jobs will be displayed. Each displayed row contains the values returned from one Job and contains the values returned from the functions in the techilatest package.

The Local Control Code used to create the Project is shown below.

The script shown above defines two functions; run_binaries and fh.

The run_binaries function contains the peach-function call, which will be used to create the computational Project. The parameters of the peach-function are explained below:

The executable parameter has been set to True, indicating that the executable code will be a precompiled binary.

The python_required parameter has been set to False, indicating that Python Runtime components will not be required on the Worker.

As the executable program will be a precompiled binary, the value of the funcname parameter can be defined quite freely. The value of the funcname parameter will only be used to specify the name of the Project (visible in the Techila Web Interface) and for naming the .state file that will be generated in to the current working directory when creating the Project.

The binaries parameter defines that the executable mcpi.exe will be executed on Workers with a Windows operating system and the executable mcpi on Workers with a Linux operating system. These executables will be transferred from the current working directory to the Techila Server when the Project is created.

The project_parameters parameter sets a value for the loops key according to the value of the loops variable. This definition will be required in order to refer to the value using the %P(loops) notation when defining the params parameter.

The outputfiles parameter defines that the file called output.data should be returned from the Workers to the End-Users computer. The syntax also defines the resource output, meaning the notation %O(output) can be used to refer to the name of the output file.

The params parameter defines three input arguments for the executable binary.

The first input argument (%P(jobidx)) will be replaced by different values for each Job. The value of this notation will be 1 for Job 1, 2 for Job 2 and so on. This value will be used to specify the seed for the random number generator.

The second input argument (%P(loops)) will be replaced by the value defined for the loops parameter in project_parameters. The value of this notation will therefore be 100000000 for all Jobs. This value will be used to specify the number of iterations that will be performed in each Job (will define the length of a computational Job).

The third input argument (%O(output)) will be replaced with the value of the resource output defined in the outputfiles parameter. The value of this notation will therefore be output.data for all Jobs. This value will define the name of the file that will be generated by the executable binary.

The figure below illustrates how the parameters defined in the Local Control Code will be passed to the executable binary as input arguments.

The fh is the file handler function which will be used to process each of the output files generated during the computational Project. In this example, the function will be used to:

The values returned by the fh-function will be stored as list elements in the results list that will be returned from the peach-function. After all output files have been processed, the values will be combined and used to calculate an approximated value for pi.

Each Worker performs the Monte Carlo routine by executing the mcpi.exe or mcpi binary depending on the operating system of the Worker. The binary is executed according to the input arguments that were defined in the peach-function params parameter.

The project can be created with the following commands:

This will create a project consisting of 20 Jobs. During each Job, the mcpi binary will be executed and will perform a Monte Carlo routine that consists of 100,000,000 iterations. After the Monte Carlo routine has been completed, the result of the approximation will be stored a file called output.data. This file will be returned from the Workers to the Techila Server.

As the peach-function syntax enabled result streaming, the results will be streamed from the Techila Server to your computer as soon as they are available. Each time a new result has been transferred, the file handler function will be executed and used to process the output file.

The Local Control Code used in this example is shown below. The commented version of the code can be found in the following file in the Techila SDK:

The Popen method from the subprocess package is used to get the current user name. This Popen method will be executed on the End-User’s computer, meaning the command will return the user name of the End-User. The user name is stored in variable o.

After getting the user name, the string is trimmed and stored in the local_username variable.

The peach-function call creates a Project consisting of one Job. During this Job, the impersonate_dist function will be executed on the Worker. The return value of the peach-function will be stored in variable worker_username, which will contain the user name of the account that was used when the code was executed on the Worker.

AD impersonation is enabled with the Project parameter. This means that the entire computational process (i.e. Job) will be executed using the End-User’s own AD user account.

After the Project has been completed, the user accounts that were used during the example will be printed.

The Worker Code that will be executed in each Job is shown below. The commented version of the code can be found in the following file in the Techila SDK:

The code starts by importing the subprocess package, which contains methods needed to get the current user name.

After importing the package, the Popen method from the subprocess package is used to get the current user name. This command will be executed on the Worker and will return the user account that is used to run the code. If AD impersonation is enabled and is working correctly, this command will return the End-User’s own AD user account details. If AD is not working correctly, this command will return the Techila Worker’s own user account details.

After retrieving the user account information, the Job will be completed and the information will be returned to the Ende-User’s computer where it will be returned by the peach-function.

To create the computational Project, change your current working directory (in Python) to the directory that contains the example material for this example.

After having navigated to the correct directory, create the computational Project using commands shown below:

After the Project has been completed, information about the AD user accounts will be displayed. Please note that the output generated by the program will change based your domain and AD account user names.

The Local Control Code used in this example is shown below. The commented version of the code can be found in the following file in the Techila SDK:

The Project parameter definition is used to create the Project-specific semaphore. The semaphore will be named examplesema and will contain two semaphore tokens. This means that a maximum of two tokens can be reserved at any given time. No other modification are required in the code that is used to create the Project.

Project results will be displayed by using for-loops to iterate over the elements in the results list.

The Worker Code that will be executed in each Job is shown below. The commented version of the code can be found in the following file in the Techila SDK:

The code starts by importing the TechilaSemaphore from the peachlient.py file. The peachlient.py file is automatically copied to the same temporary working directory on the Worker. This temporary working directory will also contain the semaphore_dist.py file.

The current timestamp is then retrieved, which will be used to mark the start time of the Job.

The Job’s index number is retrieved from an environment variable and stored as the first element in the result list.

The with-statement will create a TechilaSemaphore object. This object contains enter and exit methods, which will be automatically called when the with-block is entered and exited. The execution of this code block will start when the Job has reserved a semaphore token from the semaphore called examplesema. Because the Project contains four Jobs and there are two tokens, only two Jobs will start processing the code right away. The remaining two Jobs will wait until the earlier Jobs exited the with-block and released the tokens.

After getting the token, the current timestamp is retrieved. This will be used to mark the start time of the with block.

The genLoad function is then executed, which will generate CPU load by generating random numbers for 30 seconds.

The elapsed time between the start of the Job (jobStart variable) and the genLoad method call (start variable) is calculated. If a Job was able to reserve a token right away, this value should be 0. If the Job had to wait for a semaphore token to become available, this value will be roughly 30. Please note that if all Jobs were not started at the same time, you will get different values.

After this, the code calculates how many seconds the Project-specific semaphore token was reserved.

The result string is then built and the string is addedto the result list, which will be returned from the Job. The string will contain information about the time window when the Project-specific semaphore token was reserved relative to the start of the Job.

After processing the Project-specific semaphore, the second with-block is executed, which will attempt to reserve one token from the global semaphore globalsema. The syntax used sets the value of the ignoreerror argument to True, meaning code inside the with-block will be executed regardless of the status of the semaphore reservation process. If the semaphore token was reserved successfully, the value True will be stored in ts. If the semaphore could not be reserved, the value False will be stored in ts.

If the global semaphore token was reserved successfully, code inside the if-clause will be executed. In this case, the genLoad function will be used to generate 5 seconds of CPU load. After the generating the CPU load, information about when the global semaphore token was reserved will be stored in the result list.

If the global semaphore token was not reserved successfully, the code inside the second if-clause will be executed. In this case, a simple status message will be stored in the result list.

Please note that global semaphores will need to be created by your local Techila administrator. This means that unless your local Techila administrator has created a semaphore named globalsema, the second if-statement (ts == False) will be executed in the Job.

The example figure below illustrates how Jobs in this example are processes in an environment where all Jobs can be started at the same time. In this example figure, the global semaphore globalsema is assumed to exist and that it only contains one token.

The activities taking place during the example Project are explained below.

After the Jobs have been assigned to Workers, two of the Jobs start processing the first with-statement. This processing is illustrated by the Computing, Project-specific semaphore reserved bars. During this time, the remaining two Jobs will wait until semaphore tokens become available. After the first Jobs have released the Project-specific semaphores, Jobs 3 and 4 can reserve semaphore tokens and start processing the first with-block.

The global semaphore only contains one token, meaning only one Job can process the second with-block at any given time. In the example figure below, Job 1 reserves the token and starts processing the second with-statement first. This processing is represented by the Computing, global semaphore reserved bars. After Job 1 has completed processing the second with-block, the global semaphore is released and Job 2 can start processing.

After Jobs 3 and 4 complete processing the first with-statement, the Jobs will start reserving tokens from the global semaphore.

To create the computational Project, change your current working directory (in Python) to the directory that contains the example material for this example.

After having navigated to the correct directory, create the computational Project using commands shown below:

Please note that the output generated by the program will change based on whether or not the global semaphore named globalsema is available. The two example screenshots below illustrate the output in both scenarios.

The example screenshot below illustrates the generated output when the global semaphore exists.

Please note that there might be overlap in the reported time windows. This is because the time windows are measured from the timestamp generated at the start of the 'Execute' method, which means that e.g. initialization delays can cause the reported times to overlap.

The source code of the example discussed here can be found in the following file in the Techila SDK:

The code used in this example is also illustrated below for convenience.

In this example, the peach function call will create a Project consisting of two Jobs. Each Job will generate a variable a and transfer the value of variable a as intermediate data back to the End-User’s computer. After the intermediate data has been received on the End-User’s computer, the data will be processed using an intermediate callback function named myfunction.

Function myfunction will display information about the data that was received from a Job, increase the value of variable a by 2 and send the updated value back to the same Job that sent the data by using function send_im_data. This updated value will be loaded by the load_im_data function call.

Each Job will return the updated value of the variable as the result. These results will be printed after the Project has been completed.

To create the computational Project, change your current working directory (in Python) to the directory that contains the example material for this example.

After having navigated to the correct directory, create the computational Project using commands shown below:

The source code of the example discussed in this Chapter is shown below.

The run_jobtojob function is used to create the Project and display the results. The peach-syntax on will create a Project with two Jobs. During each Job, the function jobtojob_dist will be executed. After the Project has been completed, the results will be displayed with the for-loop.

The jobtojob_dist function will be executed in each Job. This function will be used to transfer simple message strings between the two Jobs in the Project.

Below is an illustration of the interconnect data transfer operations that will be performed in this Project when the Jobs are assigned to Workers.

Below is a more detailed explanation on the effect of each line in the code sample.

The TechilaInterconnect class is imported from the peachclient.py file. After this, a new TechilaInterconnect object, which will provide the necessary methods needed to transfer interconnect data.

The remaining code contains two if-statements. These determine the operations that will be executed in each Job. Job 1 will execute the code inside the first if-statement (ti.jobid==1) and Job 2 will execute the code inside the other code branch (ti.jobid==2).

Job 1 will start by executing the send_data_to_job command, which is used to transfer data to Job 2. Job 2 respectively starts by executing the recv_data_from_job command, which is used to receive the data, which is being transferred by Job 1.

After Job 2 has received the data, the roles are reversed, meaning Job 2 will transfer data to Job 1.

After data has been transferred between Jobs, each Job will execute the ti.wait_for_others() command. This line will act as a synchronization point, meaning Jobs will continue execution only after all Jobs have reached this line.

To create the computational Project, change your current working directory (in Python) to the directory that contains the example material for this example.

After having navigated to the correct directory, create the computational Project using commands shown below:

The commented version of the code can be found in the following file in the Techila SDK:

The source code of the example is shown below.

The peach-function used in this example will create a Project with three Jobs. Each Job will execute the cloudbc_dist function with one input argument sourcejob. The sourcejob argument will be used to define which Job will broadcast the interconnect data. In this example, the value of the sourcejob parameter has been set to 2, meaning Job 2 will broadcast interconnect data.

Each Job will execute the cloudbc_dist function, which will be used to broadcast the data.

At the start of each Job, a message string will be built and stored in variable datatotransfer. This message string will include the value of ti.myjobid, which contains the Job’s index number. This information will be used to distinguish which Job broadcasted the string. The example table below illustrates values of the datatotransfer variable in each Job, in a Project consisting of 3 Jobs.

The datatotransfer variable will be broadcasted from the sourcejob to all other Jobs in the Project using ti.cloudbc. In this example, the value of the sourcejob argument has been set to 2, meaning Job 2 will broadcast data to all other Jobs in the Project. The broadcasted data will be returned from the ti.cloudbc method and stored in the result variable.

After the data has been broadcasted (and received), each Job in the Project will wait until all other Jobs in the Project have reached and executed the ti.wait_for_others() command.

To create the computational Project, change your current working directory (in Python) to the directory that contains the example material for this example.

After having navigated to the correct directory, create the computational Project using commands shown below:

When executed, the code will create a Project consisting of three (3) Jobs. Job 2 will broadcast data to other Jobs in the Project. Below figure illustrates the operations that take place when the code is executed with the syntax shown above.

The source code of the example discussed in this Chapter is shown below.

The peach-function used in this example defines that the Project will contain four Jobs and that during each Job, the alltoall_dist function should be executed.

The alltoall_dist function is executed on the Worker and starts by getting the Job’s index number and stores it in variable jobidx.

The number of Jobs in the Project is retrieved with ti.jobcount and stored in variable jobcount. With the values used in the example, this value will be four (4).

Each Job will then build a simple, unique message string, which will be transferred from the Job. The table below shows the messages transferred from each Job.

The two nested for-loops contain the code that will decide the order in which Jobs will transfer messages to other Jobs. The transferred messages will be stored to the dataall list, which will be returned to the End-User’s computer.

The interconnect data transfers that take place during the Project are illustrated in the figure below. The arrows indicate that interconnect data is being transferred. The values in parentheses correspond to the values of the src and dst loop counters. For example, arrow with value (1,3) means that Job 1 is sending the msg string to Job 3. If src is equal to dst (e.g. (2,2)), no data is transferred because the source and target are the same.

To create the computational Project, change your current working directory (in Python) to the directory that contains the example material for this example.

After having navigated to the correct directory, create the computational Project using commands shown below:

When the command is executed, the code will create a Project consisting of four (4) Jobs. Each Job will transfer a simple string to all other Jobs in the Project.

The source code of the example discussed in this Chapter is shown below.

The run_cloudop function contains the code for creating the Project and displaying the results after the Project has been completed.

The peach-function used in this example will create a Project with three Jobs. Each Job will execute the cloudop_dist function with one input argument '<vecidx>'. The value of the '<vecidx>' notation will be different in each Job: 1 in Job 1, 2 in Job 2 and 3 in Job 3.

Each Job will execute a function called multiply using the cloudop method across all Jobs. This function takes two input arguments, multiplies them and returns the multiplication result. This means that the multiply function meets the requirements for functions that can be executed with cloudop.

In this example, the multiply function will multiply the random numbers generated in each Job. As the <target> argument is not defined in the cloudop call, the final result of the multiplication will be returned in each Job and stored in the mulval variable.

To create the computational Project, change your current working directory (in Python) to the directory that contains the example material for this example.

After having navigated to the correct directory, create the computational Project using commands shown below:

When the command is executed, the code will create a Project consisting of three (3) Jobs. During each Job, a random number is generated. These random numbers will then be multiplied by using the cloudop function.