Techila Distributed Computing Engine with C

1. Introduction

This document is intended for End-Users who wish to access the Techila Distributed Computing Engine (TDCE) environment from applications developed with C. If you are unfamiliar with the terminology or the operating principles of the TDCE system, information on these can be found in Introduction to Techila Distributed Computing Engine.

The structure of this document is as follows:

Prerequisites contains information regarding the preparation steps required to use TDCE with applications developed with C. This chapter also contains a brief introduction on the naming convention of the C source code files and introduces the functions that are used for distributing computations from applications developed with C to the TDCE environment.

Peach Tutorial Examples contains walkthroughs of simplistic example code samples. The example material illustrates how to define input arguments for the executable program and transfer data files to Techila Workers participating in the computations. After examining the material in this Chapter you should be able split a simple locally executable program into two pieces of code (Local Control Code and Techila Worker Code), which in turn can be used to perform the computations in the TDCE environment.

1.1. Prerequisites

To access the TDCE computing environment from applications written with C, you will need the following components:

Valid End-User Key (typically created by your local Techila Administrator)
Techila SDK
techila_settings.ini file that has been configured according to the instructions in Getting Started.
Compiler capable of compiling C code (e.g. the C compiler included in the GNU Compiler Collection (gcc))
Java Development Kit. Note! Java Runtime Environment (JRE) will not work, because the C compilation process will require access to the jni.h file, which is not include in the JRE.

Please note that there are makefiles for each C example included in the Techila SDK. This means that if you want to use these makefiles to compile the examples on your computer, you also need a compatible make utility installed on your computer. The following make utilities are compatible with the makefiles:

GNU make (gmake)
Microsoft Program Maintenance Utility (nmake)

If you want to modify the configuration settings used to compile the examples, e.g. to define a different compiler, please edit file techila\Examples\C\Makefile.inc in the Techila SDK.

1.2. Techila Distributed Computing Engine C API

The Techila Distributed Computing Engine (TDCE) C API enables you to create computational Projects from programs developed with C.

Using the TDCE C API requires linking a library with the TDCE functions to your project and including the Techila.h header file in the C source file from which you want to access the functions.

The syntax for including the header file is shown below.

#include "Techila.h"

During compilation, the code needs to be linked to the following TDCE libraries located in the lib directory in the Techila SDK.

NAME	OPERATING SYSTEM	ARCHITECTURE
Techila32.dll	Windows	32-bit
Techila64.dll	Windows	64-bit
libTechila32.so	Linux	32-bit
libTechila64.so	Linux	64-bit

NAME

OPERATING SYSTEM

ARCHITECTURE

Techila32.dll

Windows

32-bit

Techila64.dll

Windows

64-bit

libTechila32.so

Linux

32-bit

libTechila64.so

Linux

64-bit

For example, consider a program with the source code shown below

#include "Techila.h"

int main(void) {

// Create a test session
int status = techila_initFile("C:/techila/techila_settings.ini");

// Functions to create Project would be implemented here

// Remove session
status = techila_unload(1);
return 0;
}

In a Windows environment, the program shown above could be compiled and linked with the following commands (assuming that the code is stored in a file called run_test.c and the file techila_settings.ini can be accessed from the path C:/techila).

gcc -IC:/techila/lib -I"C:/Program Files/Java/jdk1.7.0_17/include" -I"C:/Program Files/Java/jdk1.7.0_17/include/win32" -Wall -pedantic -std=c99 -c -o run_test.o run_test.c
gcc -LC:/techila/lib -l:C:/techila/lib/Techila32.dll -o run_test run_test.o

Respectively in a Linux environment, the code could be compiled with the commands shown below.

gcc -I/home/user/techila/lib -I/usr/lib/jvm/java-6-sun/include -I/usr/lib/jvm/java-6-sun/include/linux -Wall -pedantic -std=c99 -c -o run_test.o run_test.c
gcc -L/home/user/techila/lib -lTechila32 -ldl -o run_test run_test.o

Please note that the paths in shown in the compilation commands are system specific and will probably be different on your computer. A similar syntax can also be used to compile the example material, if you do not have access to a compatible make utility on your computer.

1.3. The techila_peach Functions in the Techila Distributed Computing Engine C API

Computational projects can be created by using the techila_peach functions included in the TDCE libraries. For more information on available features, please see Introduction to Techila Distributed Computing Engine.

A complete list of available functions can be found in the documentation in the directory techila\doc\doxygen. A list of most commonly used techila_peach functions when creating a computational Project is shown below.

The functions on the left have been grouped in to three categories based on which phase in the computations they are used. The first category contains the function used during initialization. The second group contains functions used for creating the computational Project. The third group contains functions that are typically used when downloading results and removing temporary files after the Project has been completed.

Figure 1. A list of typically used functions when creating a computational Project. Please note the list only contains some of the most commonly used functions when creating a computational Project and does not represent a complete list. Also note that techila_peach_nextFile function is typically placed inside a loop structure in order to ensure that all result files will be processed.

An example of a very simple syntax is shown below.

#include "Techila.h"

void handle_result_file(char* file) {

/* Code for processing each result file after it has been transferred
 from the Techila Server */

}

int main(int argc, char** argv) {

  /* Use techila_settings.ini file located in directory C:\techila\ */
  int status = techila_initFile("C:\\techila\\techila_settings.ini");

  /* Create a peach handle that will be used to link all peach function
   calls to the same project */
  int ph = techila_peach_handle();

  /* Set a name for the Project, which will be visible in the Techila
   Web Interface */
  techila_peach_setName(ph, "Example name");

  /* Specify that the file called `test.exe` should be executed on
   all available 64-bit Windows Workers. */
  techila_peach_addExeFile(ph, "exe", "test.exe", "Windows", "amd64");

  // Set the number of Jobs in the Project to 10
  techila_peach_setJobs(ph, 10);

  /* Specify that a file called `techila_output` should be returned from
   the Workers */
  techila_peach_addOutputFile(ph, "techila_output");

  /* Specify one input argument to the executable (example.exe). The
  value of the `jobidx` parameter will be different for each Job. The
  value will be  1 for Job #1, 2 for Job #2. */

  techila_peach_putExeExtras(ph, "Parameters", "%P(jobidx)");

  / Start the computational Project. */
  status = techila_peach_execute(ph);

  char file[512];
  size_t l = sizeof(file);

  while (l > 0) {
  /* Download result files from the Techila Server */
  status = techila_peach_nextFile(ph, file, &l);

  if (l > 0) {
    file[l] = `\0`;
    /* Function for post-processing result files. */
    handle_result_file(file);
  }
}

  /* Clean up all temporary project data */
  status = techila_peach_done(ph);
  status = techila_unload(1);
}

1.4. Preparation

Before using the TDCE C API, the following environment variables need to be defined as described below.

ENVIRONMENT VARIABLE	CONTENT	OS
JAVA_HOME	Must point to the Java Development Kit installation directory.	Windows, Linux
LD_LIBRARY_PATH	<full path>/techila/lib	Linux
PATH	<full path>\techila\lib	Windows

ENVIRONMENT VARIABLE

CONTENT

JAVA_HOME

Must point to the Java Development Kit installation directory.

Windows, Linux

LD_LIBRARY_PATH

<full path>/techila/lib

Linux

PATH

<full path>\techila\lib

Windows

The following Chapters contain instructions on how to set these environment variables. Please follow the instructions applicable to your operating system:

Windows
Linux

1.4.1. Windows

To add the necessary environment variables in a Windows operating system, please follow the steps below. Please note that the steps are for Windows 7 and some steps might be different if you are using a different Windows version.

Open Windows Control Panel
Click System
Click Advanced system settings
Click Environment Variables
Click New… to create a new user variable.
Enter JAVA_HOME to the Variable name field and the path of your Java installation directory to the Variable value field. Click OK to confirm the changes.

Figure 2. Creating a new, user-specific environment variable in Windows 7.
Choose the environment variable PATH from the User Variables list and click Edit…

Note, if you do not have a PATH variable defined, click New… to create the variable
Add <full path>\techila\lib path to the PATH variable. Click OK to confirm the changes.

Figure 3. When adding a new path entry, use semicolon (;) to separate the entry.

1.4.2. Linux

To add the necessary environment variables in a Linux operating system, please follow the steps below.

Add the following line to .bashrc or other shell startup script.

export JAVA_HOME=<path of Java installation directory>
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:<full path>/techila/lib

If necessary, launch a new shell to make the changes take effect.

1.5. Example Material

The example material discussed in this document can be found under the following folder in the Techila SDK:

techila\examples\C\Tutorial

Examples located in the "Tutorial" folder are examined in Peach Tutorial Examples.

Figure 4. The example material examined in this document is located in the under the "C" folder in the Techila SDK.

1.6. Naming Convention of the C source Code Files

The typical naming convention of C source code files containing the example material presented in this document is explained below:

Files ending with "_dist" contain the Techila Worker Code, which will be compiled into an executable binary when make is executed. These executable binaries will be distributed to the Techila Workers when the Local Control Code is executed.
Files beginning with "run_" contain the Local Control Code, which will be compiled when make is executed. After compilation, these binaries can be executed locally on the End-Users computer and will be used to create the computational Project.
Files beginning with "local_" contain locally executable code, which does not communicate with the TDCE environment.

Please note that some files and/or functions might be named differently, depending on their role in the computations.

2. Peach Tutorial Examples

This Chapter contains walkthroughs of minimalistic examples that illustrate how to implement and control the core features of the Techila Distributed Computing Engine (TDCE) C API. The example material discussed in this Chapter, including C-source code files and data files can be found under the following folder in the Techila SDK:

techila\examples\C\Tutorial

Each of the examples contains three pieces of code:

A locally executable program that can be executed locally and will not communicate with the distributed computing environment in any way. This program is provided as reference material to illustrate how operations in a locally executable program can be performed in the TDCE environment.
A file containing the Local Control Code. This file will be compiled, executed locally and will be used to distribute the computations in the Techila Worker Code to the TDCE environment
A file containing the Techila Worker Code. The source code in this file will be compiled in to a binary and executed on the Techila Workers. This program represents the computationally intensive part of the locally executable program.

Please note that the example material in this Chapter is only intended to illustrate the core mechanics related to distributing computations from applications developed with C.

2.1. Executing a Precompiled Binary on Techila Workers

The walkthrough in this Chapter is intended to provide an introduction on distributed computing using the TDCE C API. The purpose of this example is to demonstrate:

How to modify a simple, locally executable program so that the computational operations will be performed Techila Workers
The difference between Local Control Code and Techila Worker Code
The basic syntax used to create a computational Project when using the C techila_peach functions

The material discussed in this example is located in the following folder in the Techila SDK:

techila\examples\C\Tutorial\1_distribution

2.1.1. Locally executable program

The source code of the locally executable program is located in the file local_program.c. To view the entire source code, please open the file in an editor of your choosing. The process flow of the locally executable program is shown below.

As can be seen from the flowchart and the source code, the program is very simple. The same operation 1+1 is executed during each iteration and the value of the summation is stored in the array myarray.

After all five (5) iterations have been completed; the elements stored in the array will be printed. The computationally intensive part of the program has been highlighted and represents the section that will be executed on the Techila Workers in the distributed version.

Figure 5. Process flow of the locally executable program. The code contains a single for loop where the value of the operation 1+1 is stored in an array in the location determined by the value of the loop counter i.

2.1.2. Distributed version

The process flow of the distributed program is shown below. Sections highlighted with red indicate parts of the program that are executed on the End-Users computer. The section highlighted with blue respectively indicate operations that take place on the Techila Workers.

The process will start by setting initialization values for parameters, which will be used to define the number of Jobs in the computational Project (variable jobs) and for storing the results received from the Techila Workers (array myarray). After setting required parameters, the computational Project will be created by using the techila_peach functions. These functions will be explained later in this Chapter.

After the Project has been created, Jobs will be assigned to Techila Workers. In this case, each Techila Worker will execute the Techila Worker Code without any input arguments, meaning that the operations that occur on the Techila Workers are identical. The operations performed during a Job correspond to the operations that take place during one iteration of the loop structure in the locally executable program. Each Job can be therefore thought to correspond to one iteration of the locally executable loop structure.

After the Project has been completed, meaning all computational Jobs have been completed, results can be downloaded from the Techila Server. The result files will be downloaded by using the techila_peach_nextFile function and post-processed by using a function that will read the result file (which will contain one integer). The value of the integer will be stored to the array myarray.

Figure 6. Process flow of the distributed program. All Jobs will execute the distribution_dist executable without any input arguments.

2.1.3. Local Control Code

The Local Control Code used in this example is in the file run_distribution.c. To view the entire source code, please open the file in an editor of your choosing.

The functionality associated with each line of code is explained below.

int jobs=5;

Sets the value of the jobs variable to 5. This variable will be used later when defining the number of Jobs in the Project.

int status = techila_initFile(NULL);

The techila_initFile function is used to initialize a session with the Techila Server. The function will read all parameters required for initializing a session from the techila_settings.ini file. When the function is given the input argument NULL, the system will automatically search the location of the settings file. You can also define the location of the settings file you wish to use.

When the inifFile command is executed, a password prompt dialog will also be displayed, assuming you have not specified the password in the techila_settings.ini file. Executing this function will also create a session_store file to your home directory, which will contain the user credentials. This file will be removed when the techila_unload command is executed.

The value returned by the techila_initFile function will be stored in the status variable. This value will be passed to the checkStatus function, which will be used to check the status code. If the value of the status code is smaller than zero (an indication of an erroneous situation), the corresponding error code will be displayed.

int ph = techila_peach_handle();

The techila_peach_handle function is used to create a handle for the peach object and needs to be called before calling other techila_peach functions. This peach handle will be used to link all techila_peach function calls to the same Project. In this example, the peach handle is stored in the variable ph. The variable ph will be used in subsequent function calls to link the function calls to the correct project.

techila_peach_setName(ph, "Distribution example");

The techila_peach_setName function used will set the name of the Project to Distribution example. The name specified with this function will also be used to name the state file that will be generated when creating the Project.

if(access("distribution_dist64.exe", F_OK ) != -1 ) {
   techila_peach_addExeFile(ph,"exe", "distribution_dist64.exe", "Windows", "amd64");
  }

The techila_peach_addExeFile function is used to specify the name of the executable binary that will be executed on the Techila Workers and the operating system and platform of the Techila Workers that are allowed to use the binary. In the example above an executable named distribution_dist64.exe would be executed on all Techila Workers with a Windows based 64-bit operating system

The techila_peach_addExeFile function calls in this example have been placed inside if clauses, which will check if a file with the specified name exists in the current working directory by using the access function. The names of the output files have been specified in the makefile and the executable will be named according to the platform specified when running make.

For example, if you specify win64 as the platform when running make, the compilation process will generate a file called distribution_dist64.exe. In this case, the if clause shown above would be true (because the file exists) and the executable would be included in the Project. Respectively, the remaining if clauses would return false because no files with matching names exist.

techila_peach_setMessages(ph,1);

The techila_peach_setMessage function can be used to enable (or disable) message printouts. In this example, messages have been enabled by setting the value to 1.

techila_peach_setJobs(ph, jobs);

The techila_peach_setJobs function is used to define the number of Jobs in the Project. In this example, the number of Jobs is set according to the value of the jobs variable. As the value of the jobs has been set to 5, the number of Jobs in the Project will also be 5.

techila_peach_addOutputFile(ph, "techila_output");

The techila_peach_addOutputFile function can be used to specify names of output files that should be returned from the Techila Workers. In this example, a file named techila_output will be returned from the Techila Workers. This file will be generated when the Techila Worker Code is executed and will contain the result of the computational Job. This file will be generated when the Techila Workers execute Techila Worker Code (source code in the file distribution_dist.c in this example).

status = techila_peach_execute(ph);

The techila_peach_execute function is used to create the computational Project. Executing this function will create all necessary Bundles, which will be linked to the peach handle ph.The Bundles will also be transferred to the Techila Server. After the project has been created with the techila_peach_execute function, the Local Control Code continues execution.

The techila_peach_nextFile function is used to download one result file from the Techila Server. If the Techila Server has not received any result files from the Techila Workers, the function will wait until a result file is available. As the function will only download one result file, the function call is typically placed within a loop structure. This loop structure typically exits only after all result files have been downloaded. The loop structure used in this example is shown below.

The syntax of the techila_peach_nextFile function call is explained below:

status = techila_peach_nextFile(ph, file, &len);

The first input argument will be the peach handle that the download request will be linked to. In this example the ph variable is used, which will link the download request to the same project that was created earlier. The file variable will contain the name (with the full path) of the result file. The len variable will be used to store the length of the filename. When all result files have been downloaded, zero (0) will be returned as the value of the len variable and the while loop structure will exit.

After all result files have been post processed, the program will exit the loop structure mentioned above. At this point, all necessary procedures have been completed, which means temporary files generated during the Project can be removed. This is performed with the function call described below.

status = techila_peach_done(ph);

The techila_peach_done function is used to remove all temporary Project data and for performing clean-up procedures. This function must be called after the Project has been completed. The clean-up procedures are linked to a specific project by defining the peach handle as the input argument.

status = techila_unload(1);

The techila_unload function is used to uninitialize the TDCE management library and will also remove the user credentials from the memory and end the session with the Techila Server. After this command is executed, you will not be able to execute any techila_peach function calls before a new session has been created by using the techila_initfile function.

2.1.4. Techila Worker Code

The Techila Worker Code used in this example is in the file distribution_dist.c. To view the entire source code, please open the file in an editor of your choosing. Code lines responsible for performing the computational operations and generating the output file are explained below.

result = 1+1;

The line above is executed once in each computational Job. The operation performed during one computational Job corresponds to the operations performed in one iteration of the locally executable program.

fp=fopen("techila_output", "w");
fprintf(fp,"%i\n",result);

The two lines shown above, will store the value of the result variable to a file called techila_output. As the Local Control Code defined a file named techila_output as an output file, this file will be transferred from the Techila Worker to the Techila Server after the computational Job has been completed.

Result files that have been transferred to the Techila Server will be automatically transferred to your computer when the techila_peach_nextFile function in the Local Control Code is executed.

2.1.5. Compilation

To compile all programs used in this example, please follow the steps below for the applicable operating system:

Launch a new command prompt/terminal

Change your current working directory to:

<full path to>\techila\examples\C\Tutorial\1_distribution

Compile the example material using the command applicable for your operating system:
- Windows,32-bit: make win32
- Windows,64-bit: make win64
- Linux,32-bit: make linux32
- Linux,64-bit: make linux64

After you have successfully compiled the programs, you can proceed with running the example programs as described below.

2.1.6. Running the example programs

To run the locally executable program, use the applicable command depending on your operating system:

Windows: local_program.exe
Linux: ./local_program

After executing the command the locally executable program will be executed. The program will execute five iterations of the loop structure and print a message after each iteration. After all iterations have been completed, the contents of the myarray array will be displayed on a single line.

The output generated by the locally executable program should resemble the one illustrated below.

Figure 7. The locally executable program will display the result generated each iteration. After all iterations have been completed, the elements stored in the myarray will be displayed.

To run the distributed version of the program, use the applicable command depending on your operating system:

Windows: run_distribution.exe
Linux: ./run_distribution

After executing the command, the computational Project will be created and you will be prompted for the End-User keystore password. Please note that if you have specified the password in the techila_settings.ini file, the password will not be prompted and the system will use the password specified in the techila_settings.ini file.

The computational Project will consist of five Jobs and each Job will perform the operation 1+1. After the computational Jobs have been completed, the results will be downloaded from the Techila Server. When a result file has been downloaded, the program will display the full path of the result file on your computer, read the returned value from the file and store the value in the array myarray. Please note that the files that are displayed are only temporary files and will be automatically removed when the techila_unload command is executed later in the program.

The distributed version should generate an output resembling the one shown below.

Figure 8. When executing the Local Control Code, a computational Project will be created. After all Jobs have been completed, result files will be downloaded. The location of the temporary result files will also be displayed.

2.2. Using Input Parameters

The walkthrough in this Chapter is intended to provide an introduction on distributed computing using the TDCE C API. The purpose of this example is to demonstrate:

How to define input arguments to the executable program

The material discussed in this example is located in the following folder in the Techila SDK:

techila\examples\C\Tutorial\2_parameters

2.2.1. Locally executable program

During each iteration, two integers will be multiplied. The value of the first multiplier multip will be the same during each multiplication. The value of the second multiplier will be different each iteration and will range from one (1) to the total number of iterations in the loop structure. The value of the multiplication is stored in the array myarray. After all five (5) iterations have been completed; the elements stored in the array will be printed. The computationally intensive part of the program has been highlighted and represents the section that will be executed on the Techila Workers in the distributed version.

Figure 9. Process flow of the locally executable program. The code contains a single for loop where the iteration number (i+1) is multiplied with the number two (2).The result of the operation is stored in an array in the location determined by the value of the loop counter i.

2.2.2. Distributed version

The process flow of the distributed program is shown below. Sections highlighted with red indicate parts of the program that are executed on the End-Users computer. The section highlighted with blue respectively indicates operations that take place on the Techila Workers.

The process will start by setting initialization values for parameters, which will be used to:

Define the number of Jobs in the computational Project (variable jobs)
Store the results received from the Techila Workers (array myarray).
Set the value of first input argument for the program that will be executed on Techila Workers: (variable multip)

After setting all required parameters, the computational Project will be created by using the techila_peach functions. Functions used for defining input arguments will be explained later in this Chapter.

After the Project has been created, Jobs will be assigned to Techila Workers. In this case, each Techila Worker will execute the Techila Worker Code (executable parameters_dist) with two input arguments. The value of the first input argument will be the same in all Jobs. The value of the second input argument will be different in each Job, and will range from one (1) with Job #1 to five (5) with Job #5.

The operations performed during a Job correspond to the operations that take place during one iteration of the loop structure in the locally executable program. Each Job can be therefore thought to correspond to one iteration of the locally executable loop structure.

After the Project has been completed, meaning all computational Jobs have been completed, results will be downloaded from the Techila Server. After the results have been downloaded, the values will be read from the result files and stored in the array myarray.

Figure 10. Process flow of the distributed program. All Jobs will execute the parameters_dist executable with two input arguments. The value of the first input argument will be same in all Jobs and whereas the value of the second input argument will be different in each Job.

2.2.3. Local Control Code

The Local Control Code used in this example is in the file run_parameters.c. To view the entire source code, please open the file in an editor of your choosing. Most of the techila_peach function calls used in this example are similar as those described in Executing a Precompiled Binary on Techila Workers and are therefore not explained here. Please see Executing a Precompiled Binary on Techila Workers for more information on the techila_peach function calls not explained in this Chapter.

As the purpose of this example is to illustrate how to pass input arguments to the executable function, the lines responsible for defining input arguments are listed below.

  char multip[]="2";
  techila_peach_putProjectParam(ph, "multip", multip);
  techila_peach_putExeExtras(ph, "Parameters", "%P(multip) %P(jobidx)");

The functionality of each line is explained below.

char multip[]="2";

The line above creates a char array called multip and sets the value to two (2). The value stored in the array will be used to define the value of the Project parameter multip later in the Local Control Code.

techila_peach_putProjectParam(ph, "multip", multip);

The techila_peach_putProjectParam function is used to define Project parameters. In this example, the function is used to define a value for a parameter called multip. After this line is executed, referring to the parameter with the %P(multip) notation will retrieve the value of the multip parameter which was defined earlier.

techila_peach_putExeExtras(ph, "Parameters", "%P(multip) %P(jobidx)");

The techila_peach_putExeExtras function can be used to define input arguments for the executable program. In this example, two entries are defined, which means two input arguments will be given to the executable program when it will be executed on the Techila Workers. Both entries are explained below:

The notation %P(multip) will be replaced with the value of the multip variable and will have the value two (2) for each Job.
The notation %P(jobidx) will be replaced with the value of the jobidx parameter and will be different for each Job. The jobidx parameter is generated automatically on the Techila Server. The value of the parameter will unique for each Job and will range from one (1) to the total number of Jobs in the Project. In other words, the value of the jobidx parameter will be one (1) for Job #1, two (2) for Job #2 and so on.

The way in which the input arguments are transferred to the executable program is illustrated below.

Figure 11. The value of the multip parameter will be stored in char format. This means that it can be passed directly to the techila_peach_putProjectParam function, which only accepts parameters in char format. Because the multip parameter is defined in the techila_peach_putProjectParam function, it can be referenced to with the %P(multip) notation. This means that the value can be used as an input argument when the executable binary is executed on a Techila Worker. The value of the jobidx parameter is received directly from the Techila Server, meaning it does not have to be defined with techila_peach_putProjectParam function.

2.2.4. Techila Worker Code

The Techila Worker Code used in this example is in the file parameters_dist.c. To view the entire source code, please open the file in an editor of your choosing. The operations performed in the Techila Worker Code include three fundamental parts, which are:

Reading input arguments from the command line
Performing the computational operations
Generating the output file

Code lines responsible for these operations have been explained below.

int multip = atoi(argv[1]);
int jobidx = atoi(argv[2]);

The two lines above are the first commands executed during a computational Job when the Techila Worker Code is executed. These lines will read two input arguments from the command line using the atoi function and store the values of the input arguments in the variables multip and jobidx. The value of the multip parameter will be the same in all Jobs and will correspond to the value which was defined in the Local Control Code. The value of the jobidx variable on the other hand will be different for each Job and will range from one (1) to the total number of Jobs (5 in this example).

int result =  multip * jobidx;

fp=fopen("techila_output", "w");
fprintf(fp,"%i\n",result);

The two lines shown above, will store the value of the result variable to a file called techila_output. This file is specified as an output file in the Local Control Code, which means, this file will be transferred from the Techila Worker to the Techila Server after the computational Job has been completed. Result files that have been transferred to the Techila Server will be automatically transferred to your computer when the techila_peach_nextFile function in the Local Control Code is executed.

2.2.5. Compilation

To compile all programs used in this example, please follow the steps below for the applicable operating system:

Launch a new command prompt/terminal

Change your current working directory to:

<full path to>\techila\examples\C\Tutorial\2_parameters

Compile the example material using the command applicable for your operating system:
- Windows,32-bit: make win32
- Windows,64-bit: make win64
- Linux,32-bit: make linux32
- Linux,64-bit: make linux64
  
  After you have successfully compiled the programs, you can proceed with running the example programs as described below.

2.2.6. Running the example programs

To run the locally executable program, use the applicable command depending on your operating system:

Windows: local_program.exe
Linux: ./local_program

After executing the command the locally executable program will be executed. The program will execute five iterations of the loop structure and print a message after each iteration. The message displayed will contain the iteration number and the result of the multiplication. After all iterations have been completed, the contents of the myarray array will be displayed on a single line.

The elements in the array will contain the iteration number multiplied with two (2). The generated output should resemble the one shown below.

Figure 12. Executing the locally executable program in Windows 7.

To run the distributed version of the program, use the applicable command depending on your operating system:

Windows: run_parameters.exe
Linux: ./run_parameters

The computational Project will consist of five Jobs and during each Job, the jobidx parameter will be multiplied with two (2). After the computational Jobs have been completed, the results will be downloaded from the Techila Server.

When a result file has been downloaded, the program will display the full path of the result file on your computer, read the returned value from the file and store the value in the array myarray. Please note that the files that are displayed are only temporary files and will be automatically removed when the techila_unload command is executed later in the program.

The distributed version should generate an output resembling the one shown below.

Figure 13. When executing the Local Control Code, a computational Project will be created. After all Jobs have been completed, result files will be downloaded and the location of the temporary result files will also be displayed.

2.3. Transferring Data Files

The walkthrough in this Chapter is intended to provide an introduction on how you can transfer additional data files to Techila Workers.

Material discussed in this example is located in the following folder in the Techila SDK:

techila\examples\C\Tutorial\3_datafiles

data files can be transferred to the Techila Workers by using the functions listed below:

techila_peach_newDataBundle(int handle);
techila_peach_addDataFile(int handle, LPSTR file);
techila_peach_addDataFileWithDir(int handle, LPSTR dir, LPSTR file);

These functions are explained below.

The techila_peach_newDataBundle function is used state that a new Data Bundle should be created. This function must be called before files can be added to the Data Bundle. The input argument (int handle) will be used to determine which project the Data Bundle will be linked to.

The techila_peach_addDataFile function can be used add files to a Data Bundle, enabling you to transfer files from your current working directory to the Techila Workers. The first input argument (int handle) will be used to determine which Project the file transfers should be linked to. The second input argument (LPSTR file) defines the name of the file that should be transferred. For example, the syntax shown below would transfer a file called datafile.txt to all Techila Workers participating to the Project.

int ph = techila_peach_handle();
techila_peach_newDataBundle(ph);
techila_peach_addDataFile(ph, "datafile.txt");

Multiple files can be transferred by using multiple function calls. For example, the syntax shown below would transfer files datafile1.txt and datafile2.txt to all Techila Workers participating to the Project.

int ph = techila_peach_handle();
techila_peach_newDataBundle(ph);
techila_peach_addDataFile(ph, "datafile1.txt");
techila_peach_addDataFile(ph, "datafile2.txt");

The techila_peach_addDataFileWithDir function can be used to transfer files that are not located in the current Working directory. The first input argument (int handle) will be used to determine which Project the file transfers should be linked to. The second input argument (LPSTR dir) will define the location where the file is located on your computer. The third input argument (LPSTR file) defines the name of the file that should be transferred. For example, the following syntax would transfer a file called datafile3.txt located in the directory C:\temp to all participating workers.

int ph = techila_peach_handle();
techila_peach_newDataBundle(ph);
techila_peach_addDataFileWithDir(ph, "C:\temp", "datafile3.txt");

Please note that all files transferred using these functions will be copied to the same temporary working directory with the executable binary when the computational Job is processed on the Techila Worker. If you wish to preserve the directory structure of the transferred files, this can be achieved by creating a named data bundle by using the Techila Bundle Creator Tool (as described in Bundle Guide) and importing the named Data Bundle to the computational Project.

After a Data Bundle has been created, it can be automatically reused in subsequent computational Projects assuming that nothing has changed regarding the number of Data Bundles created, timestamps of files in the Bundles or the content of the files in the Bundles.

If you wish to create a Bundle that can be reused in several different computational Projects, it is recommended that you use the CLI createBundle command or the Techila Bundle Creator tool for creating the Data Bundle. More information on the use of these tools can be found in Bundle Guide. The Bundle can then be imported in to a computational Project using the techila_peach_addExeImport function.

2.3.1. Locally executable program

The program will open two files datafile0 and datafile1, which both contain a 5x5 matrix of comma separated integer values. The values of the integers will read from the files and will be stored in arrays for further operations. The values read from file datafile0 will be stored in array m1 and values read from file datafile1 will be stored in array m2. After all values have been stored, the elements on each row of the arrays will be summed and summation result will be stored in the array result.

The computationally intensive part of the program has been highlighted and represents the section that will be executed on the Techila Workers in the distributed version.

Figure 14. Process flow of the locally executable program. The code starts with two for loops, where each loop will process one data file and store integers read from the file in arrays. The values stored in the array will be summed, generating an array that contains the sum of elements on each line.

2.3.2. Distributed version

The process flow of the distributed program is shown below. Sections highlighted with red indicate parts of the program that are executed on the End-Users computer. The section highlighted with blue respectively indicates operations that take place on the Techila Workers

In the distributed version of this example, two files called datafile0 and datafile1 will be placed in a Data Bundle and transferred to all participating Techila Workers. The file datafile0 is located in the same directory with Local Control Code and datafile1 is located a sub-directory called storage.

After adding the files to the Data Bundle, the computational Project will be created by using the techila_peach functions. The functions used to add the files to the Data Bundle will be explained later in this Chapter.

After the Project has been created, Jobs will be assigned to Techila Workers. In this case, each Techila Worker will execute the Techila Worker Code with one input argument. The value of the input argument will be replaced with the value of the jobidx parameter and will be different in each Job. The value of the jobidx parameter will range from one (1) with Job #1 to five (5) with Job #5.

The value of the jobidx parameter will be used to determine which line of the files should be processed. In each Job, the integers on the line corresponding to the value of the jobidx parameter will be summed. For example, during Job #1 the integers on the first line will be summed, Job #2 will sum the integers on the second line and so on.

The summation result will be stored in a file named techila_output, which will be transferred to the Techila Server after the computational Job has been completed. After all Jobs in the Project have been completed, results can be downloaded from the Techila Server. The result files will be downloaded by using the techila_peach_nextFile function and post-processed by using a function that will read the result file (which will contain one integer) and store the value of the integer to the array myarray.

Figure 15. Process flow of the distributed program. All Jobs will execute the datafiles_dist executable with one input argument. The value of the input argument will determine which line will be processed. The elements on the line corresponding to the value of the input argument will be summed. The value of the summation will be stored in the output file, which will be transferred back from the Techila Worker to the End-Users computer.

2.3.3. Local Control Code

The Local Control Code used in this example is in the file run_datafiles.c. To view the entire source code, please open the file in an editor of your choosing. Most of the techila_peach function calls used in this example are similar as those described in earlier Chapters and are therefore not explained here. For more information on the function calls not explained in this Chapter, please refer to the earlier Chapters the Tutorial section.

As the purpose of this example is to illustrate how to transfer input arguments to the Techila Workers, the lines responsible for defining a Data Bundle and adding files to the Bundle are explained below.

techila_peach_newDataBundle(ph);

The techila_peach_newDataBundle function is used to define a new Data Bundle. In this example, the variable ph is given as an input argument. This means that the Data Bundle will be linked to the Project that will be created later by using the techila_peach_execute(ph) function call. Linking the Data Bundle to the Project will mean that all participating Techila Workers will automatically download the Bundle from the Techila Server. Files stored in the Bundle will be automatically copied to the same temporary working directory that contains the executable binary.

techila_peach_addDataFile(ph, "datafile0");

The techila_peach_addDataFile function is used to add files from the current working directory to a Data Bundle. In this example, a file called datafile0 will be added to the Data Bundle.

techila_peach_addDataFileWithDir(ph, "./storage/", "datafile1");

The techila_peach_addDataFileWithDir function is used to add files from a specific directory to a Data Bundle. In this example, a file called datafile1 located in a directory called storage will be added to the Data Bundle.

The locations of the files are illustrated below in. The figure on the left indicates the location of the files on the End-Users computer. The figure on the right shows the location of the files on a Techila Worker during a computational Job. All files will be copied to the temporary working directory, which will be created at the preliminary stages of the computational Job.

Figure 16. File locations locally and on the Techila Worker. When files are transferred in Data Bundles to the Techila Workers, all files will be by default copied to the same temporary working directory with the executable binary.

2.3.4. Techila Worker Code

The Techila Worker Code used in this example is in the file datafiles_dist.c. To view the entire source code, please open the file in an editor of your choosing. The operations performed in the Techila Worker Code include the following fundamental parts, which are:

Reading the value of the jobidx parameter from the command line
Accessing the data files datafile0 and datafile1
Summing the integers on a specific line (as determined by the jobidx parameter)
Generating the output file

Code lines responsible for these operations have been explained below.

int jobidx = atoi(argv[1]);

The above line will read one input argument from the command line using the atoi function and store the value of to the variable jobidx. The value of the jobidx variable will be range from one (1) to the total number of Jobs (5 in this example). The value of the variable will be used to determine, which line will be processed from the files.

FILE* f1 = fopen("datafile0", "r");
FILE* f2 = fopen("datafile1", "r");

The lines shown above will open files datafile0 and datafile1 for reading. Both files can be accessed without any path definitions, because the files have been copied to the same temporary working directory with the executable binary datafiles_dist.

while (i < jobidx) {
  r1 = fscanf(f1, "%i,%i,%i,%i,%i", &m1[0], &m1[1], &m1[2], &m1[3], &m1[4]);
  i++;
}

The loop structure shown above will read lines in the file datafile0 until the number of lines read matches jobidx. When the correct line has been read, the integer values on the line will be stored in the array m1. An identical process will also be done for the file datafile1, where the integer values will be to array m2.

for (i = 0; i < SIZE; i++) {
  result=result+m1[i]+m2[i];
}

After the correct line of integers has been read from both files, the values stored in the arrays m1 and m2 will be summed. The SIZE variable has been defined as a macro and has been set to five (5).

fp=fopen("techila_output", "w");
fprintf(fp,"%i\n",result);

2.3.5. Compilation

To compile all programs used in this example, please follow the steps below for the applicable operating system:

Launch a new command prompt/terminal

Change your current working directory to:

<full path to>\techila\examples\C\Tutorial\3_datafiles

Compile the example material using the command applicable for your operating system:
- Windows,32-bit: make win32
- Windows,64-bit: make win64
- Linux,32-bit: make linux32
- Linux,64-bit: make linux64

After you have successfully compiled the programs, you can proceed with running the example programs as described below.

2.3.6. Running the example programs

To run the locally executable program, use the applicable command depending on your operating system:

Windows: local_program.exe
Linux: ./local_program

After executing the command the locally executable program will be executed. The program will read the integers stored in the files datafile0 and datafile0 and store thee values in arrays m1 and m2. After all values have been stored in the 5x5 arrays, the elements of each line will be summed and stored in the result array. After all elements have been summed, the content of the result array will be printed. The generated output should resemble the one shown below.

Figure 17. Executing the locally executable program in Windows 7. The program will sum the elements on each line and display the summation results.

To run the distributed version of the program, use the applicable command depending on your operating system:

Windows: run_datafiles.exe
Linux: ./run_datafiles

The computational Project will consist of five Jobs. During each Job, the integers on one line (as determined by the value of the jobidx parameter) will be summed. After the computational Jobs have been completed, the results will be downloaded from the Techila Server. When a result file has been downloaded, the program will display the locations of the downloaded result files, read the returned value from the file and store the value in the array myarray.

Please note that the files that are displayed are only temporary files and will be automatically removed when the techila_unload command is executed later in the program. After all result files have been downloaded from the Techila Server, the elements stored in the array myarray will be displayed.

The distributed version should generate an output resembling the one shown below.

Figure 18. When executing the Local Control Code, a computational Project will be created. After all Jobs have been completed, result files will be downloaded and the location of the temporary result files will also be displayed. The content of the result file will also be displayed.

2.4. Using Shared Libraries on Techila Workers

Shared libraries such as dynamically linked libraries (.dll`s) and shared objects (.so`s) files can contain additional functions that can be accessed by loading the library during runtime execution. If a program that is executed on a Techila Worker needs access to functions in a dynamically linked library, the file containing the function definitions needs to be transferred to the Techila Workers.

The walkthrough in this Chapter is intended to provide an introduction on how you can transfer a shared library to Techila Workers and access functions from the library.

Material discussed in this example is located in the following folder in the Techila SDK:

techila\examples\C\Tutorial\4_libraries

Dynamically linked libraries and shared object files can be transferred to the Techila Workers using the same functions that are used to transfer additional data files. The functions used for transferring additional data files are explained more thoroughly in Transferring Data Files This chapter focuses on highlighting some of the additional parameters that need to be defined to modify the runtime environment on the Techila Workers.

When a dll-file is transferred to Techila Workers by using a Data Bundle, the dll-file will be copied to the same temporary working directory as the executable program. For example, the syntax shown below would transfer the dll-file named library.dll to the Techila Workers. Because Windows operating systems automatically search the directory where the executable program is located, no additional path definitions need to be specified.

techila_peach_newDataBundle(ph);
techila_peach_addDataFile(ph, "library.dll");

The syntax for transferring the so-files is similar. For example, the syntax shown below would transfer a so-file called liblibrary.so to the Techila Workers. In addition to transferring the file, the LD_LIBRARY_PATH variable must be updated to also include the current working directory. In this example, the value of the variable has been set to contain only the current working directory.

techila_peach_newDataBundle(ph);
techila_peach_addDataFile(ph, "liblibrary.so");
techila_peach_putExeExtras(ph, "Environment", "LD_LIBRARY_PATH;value=.");

2.4.1. Locally executable function

The program starts by including the library.h`h header file, which contains the declaration of a function called `libraryfunction.

int libraryfunction(int x);

This function will be used when the program is executed. The function call is within a loop structure and the value of the input argument will depend on the value of the loop counter. The function will square the value of the given input argument and store the result in the array myarray.

The computationally intensive part of the program has been highlighted and represents the section that will be executed on the Techila Workers in the distributed version.

Figure 19. The process flow in the locally executable program. The function libraryfunction is inside a for-loop, meaning the function will be called during each iteration. This means the shared library containing the function will need to be made available on the Techila Workers in the distributed version.

2.4.2. Distributed version

In the distributed version of this example, the shared library file will be placed in a Data Bundle, which will be transferred to all participating workers. The name of the shared library file will depend on what operating system you are using to compile the example material. If you`re compiling the example on Windows the name of the file will be library.dll.If you`re compiling on a Linux machine, the file will be liblibrary.so.

The value of the jobidx parameter is given as an input argument to the library function libraryfunction.The function will square the value of the input argument and store the result in a file named techila_output, which will be transferred to the Techila Server after the computational Job has been completed. After all Jobs in the Project have been completed, results can be downloaded from the Techila Server. The result files will be downloaded by using the techila_peach_nextFile function and post-processed by using a function that will read the result file (which will contain one integer) and store the value of the integer to the array myarray.

2.4.3. Local Control Code

The Local Control Code used in this example is in the file run_library.c. To view the entire source code, please open the file in an editor of your choosing. Most of the techila_peach function calls used in this example are similar as those described in earlier Chapters and are therefore not explained here. For more information on function calls not explained in this Chapter, please refer to the earlier Chapters the Tutorial section.

As the purpose of this example is to show functions can be used from shared libraries on the Techila Workers, the lines responsible for defining a Data Bundle and adding files to the Bundle are explained below.

techila_peach_newDataBundle(ph);

The techila_peach_newDataBundle function is used to define a new Data Bundle. This Data Bundle will be used to transfer the shared library to all participating Techila Workers. Files stored in the Bundle will be automatically copied to the same temporary working directory that contains the executable binary.

techila_peach_addDataFile(ph, "library.dll");

The techila_peach_addDataFile function is used to add files from the current working directory to a Data Bundle. In this example, the function shown above would add a file called library.dll to the Data Bundle. The function shown above will only be executed if the file library.dll exists in the current working directory.

techila_peach_addDataFile(ph, "liblibrary.so");
techila_peach_putExeExtras(ph, "Environment", "LD_LIBRARY_PATH;value=.");

Respectively, the functions shown above would add a file called liblibrary.so to the Data Bundle. In addition to adding the file, the environment variable LD_LIBRARY_PATH will be modified to contain the current working directory. The functions shown above will only be executed if the file liblibrary.so exists in the current working directory.

2.4.4. Techila Worker Code

The Techila Worker Code used in this example is in the file library_dist.c. To view the entire source code, please open the file in an editor of your choosing. The operations performed in the Techila Worker Code include the following fundamental parts, which are:

Accessing the function libraryfunction from the shared library
Passing the jobidx input argument to the library function

Code lines responsible for these operations have been explained below.

int jobidx = atoi(argv[1]);

int result =  libraryfunction(jobidx);

The value of the command line argument will be given to the function libraryfunction and the result will be stored in the result integer. This value will be then stored in the result file techila_output which will be transferred back to your computer.

2.4.5. Compilation

To compile all programs used in this example, please follow the steps below for the applicable operating system:

Launch a new command prompt/terminal

Change your current working directory to:

<full path to>\techila\examples\C\Tutorial\4_libraries

Compile the example material using the command applicable for your operating system:
- Windows,32-bit: make win32
- Windows,64-bit: make win64
- Linux,32-bit: make linux32
- Linux,64-bit: make linux64

After you have successfully compiled the programs, you can proceed with running the example programs as described below.

2.4.6. Running the example programs

To run the locally executable program, use the applicable command depending on your operating system:

Windows: local_program.exe
Linux: ./local_program

After executing the command the locally executable program will be executed. The libraryfunction function will be executed five times, once per iteration. The result of the function call will be stored in the array myarray and the return value of the function will also be displayed on the screen.

The generated output should resemble the one shown below.

Figure 20. Executing the locally executable program. The program will use a function from the shared library and will also display the result returned by the function each iteration.

To run the distributed version of the program, use the applicable command depending on your operating system:

Windows: run_library.exe
Linux: ./run_library

The computational Project will consist of five Jobs. During each Job, the function libraryfunction will be executed once. The return value of the function call will be stored in the result file, which will be transferred to the Techila Server

After the computational Jobs have been completed, the results will be downloaded from the Techila Server. When a result file has been downloaded, the program will display the locations of the downloaded result file, read the returned value from the file and store the value in the array myarray. Please note that the files that are displayed are only temporary files and will be automatically removed when the techila_unload command is executed later in the program. After all result files have been downloaded from the Techila Server, the elements stored in the array myarray will be displayed.

The distributed version should generate an output resembling the one shown below.

Figure 21. Executing the Local Control Code will create the computational Project. The locations of the temporary results files will be displayed alongside with the value of the result variable returned from each Job.