This document contains some of the most common questions and answers about using Techila Distributed Computing Engine.
1. General Questions
1.1. What is the Techila SDK?
Techila SDK is a library of software components that integrate applications to the environment.
For more information, see the following link:
1.2. How can I tell when the computations are running in Techila?
When a computational Project is created, you will see a status bar that shows the progress. The status bar will be updated automatically as Jobs are completed on the cloud Workers. You can also view active Projects in the Techila Web Interface.
1.3. Code runs very (too) quickly and without errors in Techila and I don’t see any results. What’s wrong?
If you are using try-catch statements in your code, it is possible that they are masking an actual computational error. So the first step is to check the code inside your try-blocks. Below MATLAB code snippet shows how to return the error stack for further debugging.
function debughelp=run_example()
items=1:10;
debughelp=cell(1,length(items));
cloudfor x=1:length(items)
if isdeployed
try
CodeThatGeneratesError
catch e
debughelp{x}=e;
end
end
cloudend
% print first error for reference.
firsterr=debughelp{1};
disp('Printing first error for reference')
disp(firsterr.message)
disp(firsterr.stack)
end1.4. Why does my code run slowly when using the Google free trial?
Google trials only allow for a very small amount of Techila Worker CPU cores to be deployed and used in computations. This lack of computing power is one typical reason that can limit the performance of distributed applications. The solution for this is to upgrade your Google account so you can request a CPU Core increase from Google.
1.5. What happens if Active Directory (AD) impersonation is not enabled but the user tries to use it anyway?
If the user tries to use AD impersonation, but the Techila Worker process fails to start the Job using the users AD account, then the Job will be launched using the Worker process account. So the Job does not immediately generate an error and all file access operations will be done by using the Worker process account.
1.6. My code generates error: "Project cancelled, because there are no more workers assigned!" What is wrong?
Is typically caused by errors that take place during the computational Jobs. Whenever a Job generates an error, it is removed from the Project. When all Workers generate an error → All workers are removed project → You will see this error/message.
To get more information about what actually went wrong during the computational Jobs you need to check out the log files. By default, you can find the project error logs in directory:
techila/logs/project_errors
2. Google Cloud Marketplace Questions
The questions under this heading are grouped to match the different steps in the Techila Configuration Wizard.
2.1. Techila in GCP Marketplace, Step 1 Help links
2.1.1. What is my login and password to the Techila Configuration Wizard?
Your login to the Techila Configuration Wizard is:
admin
Your password is displayed in the Google Cloud Deployment Manager. All passwords are unique, meaning your password will be different than the one shown in the below screencapture.
2.1.2. How long does it take to start up the Techila Server?
The Techila Server deployment will take several minutes. If you try to access the Techila Configuration Wizard before everything is ready, you might see a page saying "Techila Server deployment still in progress…". In this case, you need to wait a couple of seconds and refresh this page, it will not auto-refresh. In total, you should get a login screen to the Techila Configuation Wizard within 10 minutes of deployment.
2.1.3. My Browser Displays a Warning When Downloading Techila SDK. What Should I Do?
If your browser displays a warning stating that the TechilaSDK.zip is not commonly downloaded and may be dangerous, you can safely ignore the warning.
2.2. Techila in GCP Marketplace, Step 2 Help links
2.2.4. I Got an error code -4, what does this mean?
Error code -4 is an indication of a network connection issue between your computer and the Techila Server. This may also be accompanied with the following error message.
SEVERE main#realExecute Connection to the server failed: java.net.ConnectException: Connection timed out: connect'In this case, please check the following:
-
Verify that there are no firewalls blocking network connections from your computer to the Techila Server, ports 25001 and 25002.
-
Verify that your 'techila_settings.ini' file is configured to use the Techila Server’s IP address.
Steps for verifying that your 'techila_settings.ini' file is configured to use the Techila Server’s IP address can be found below:
-
On your computer, navigate to the 'techila' directory (i.e. to the location where you extracted the plugins package).
-
Open the 'techila_settings.ini' file with a text editor
-
Locate the hostname parameter and make a mental note what the IP address is.
-
Using your web browser, navigate to:
-
Check the IP address of the Techila Server VM.
The IP address in the 'techila_settings.ini' file must match the value displayed in the Google Cloud Console.
If the IP addresses do not match, re-download the Techila SDK from the Techila Configuration Wizard, extract it over your existing installation (If you have python, R or MATLAB running, close them before overwriting the files) and try the connection test again.
The screenshot below shows how to compare the IP addresses. Your IP addresses will differ from the ones shown in this example.
2.3. Techila in GCP Marketplace, Step 3 Help links
2.3.1. Why is the first Techila Project always slow?
When you start new Techila Workers, they do not have any programming language specific runtime components installed. They are essentially new VM instances, which only have the Techila Worker software installed.
When you run your first Project (Python, R or MATLAB), the required runtime components will be automatically transferred to the Techila Workers from the Techila Server. This transfer process can take between 1-5 minutes, depending on the programming language you are using. After the runtime components have been transferred to the Techila Worker, they will be stored on the Techila Worker’s hard disk. During subsequent Projects, the Techila Worker will use the runtime components stored on the hard disk, meaning there is no network transfer delay related to runtime component transfers.
If you shut down the Techila Workers, the runtime components will be deleted with the Techila Worker’s disk. This means that the runtime components will need to be transferred again when you start new Techila Workers.
2.3.2. How to manage large data amounts when using Techila in Google?
When working with large data amounts, it is recommeneded to use the Google cloud buckets for transferring data. Please see the following document for more details:
2.3.3. What do I need to do to stop billing?
If you delete the Techila Worker VM instances, the Techila Worker VM instances will be deleted, meaning they will not incur hourly fees. A description on how this can be done can be found in the following link:
If you want to stop ALL caused by Techila components running in Google Cloud Platform, you will need to delete the Techila deployment. Instructions for deleting the deployment can be found in the following link:
2.3.4. How can I increase my CPU quota?
CPU quota can be increased by making a quota increase request in the Google Cloud Console. Please see the following link for details:
2.3.5. How can I increase / decrease the number of Workers?
You can control the number of that are running by entering the desired number in the "Number of Worker Nodes" field in the Techila Configuration Wizard.
Example 1:
If you don’t have any Techila Workers deployed and want to start 10 Techila Workers, enter the value 10 in the field and click the Start button in the Techila Configuration Wizard.
Example 2:
If you have 10 Techila Workers deployed and want to increase the total count to 30, enter the value 30 in the field and click the Update button. This will start 20 additional Techila Workers, bringing the total amount of Techila Workers to 30.
Example 3:
If you have 30 Techila Workers deployed and want to decrease the total count to 5, enter the value 5 in the field and click the Update button. This will shut down 25 Techila Workers (from 30), bringing the total amount of Techila Workers to 5.
Example 4:
If you have 5 Techila Workers deployed and want shut down all Techila Workers, click the Stop button in the Techila Configuration Wizard. Workers (from 30), bringing the total amount of Techila Workers to 5.
2.3.6. What packages can I use in Techila?
I am using functions from various packages in my code. What packages are supported?
The following packages are available automatically.
Python 2
-
numpy, scipy, pandas, networkx, sklearn, patsy, statsmodels, pillow, numexpr
Python 3
-
numpy, scipy, pandas, networkx, patsy, pillow, sklearn (Linux only), statsmodels (Linux only), numexpr (Linux only)
R
-
plyr
Additionally, you can transfer any package from your computer to the Techila environment. Please see the Techila with R and Techila with Python documents for more information on how to transfer packages.
2.3.7. Workers Virtual Machines not deleted even when deployment removed?
If you deleted the deployment before shutting down the Techila Workers, then you need to manually delete the instance group in the Google console. Deleting the instance group will delete the instances.
Please note that you cannot simply delete the instances, because the auto scaling feature in the instance group will restart them. You will need to delete the instance group.
2.3.8. I am using the Google free trial, but I cannot start any Techila Workers. Why?
One potential cause for this could be that you have met your Google CPU quota.
The CPU core quota in the free trial can be quite small. If you have other virtual machines running in the environment, it can easily reduce the amount so that you won`t be able to deploy a 4 CPU core Techila Worker.
One option would be to try deploying a Techila Worker which has 2 CPU core (change the instance type using the dropdown menu).
Another option would be to remove any unnecessary virtual machines that are running in your google environment. This would free up CPU core quota for the Techila Worker.
Increasing the CPU core quota could also solve the CPU core quota problem, but this might not be possible in the free trial mode. More info about CPU quota requests here:
https://support.google.com/cloud/answer/6376374?hl=en https://cloud.google.com/free/docs/frequently-asked-questions#limitations
Instructions for making a Google quota increase request can be found here:
2.3.9. VMs in Google do not stop even when MATLAB is closed. Is this normal?
Yes, this is normal. The Techila Workers in Google are not linked to your specific MATLAB session and will not be shut down when you close MATLAB.
You can shut down the Techila Workers in the Techila Configuration Wizard by clicking the red stop button (on the same page you used to start Techila Workers).
You can also use the idle shutdown option to shut the Techila Worker capacity automatically when you no longer need it. When you set a timer, the timer will start from the last timestamp a Job was noticed on a Techila Worker. So it requires a Project to trigger the timer. If you start Techila Workers but no Jobs are processed on these, then the timer won`t be triggered. This is to avoid Workers from being shut down by the idle shutdown timer before the user has the chance to use them.
If you want to delete everything (including the Techila Server):
-
Stop the Techila Workers from the Techila Configuration Wizard by using the red stop button.
-
Delete the deployment
Please note that if you delete the deployment, you will need to download and configure the Techila SDK again after you setup a new deployment.
3. MATLAB Questions
3.1. I get a java.nio.file.AccessDeniedException when running techilainit(). What is causing this.
The error means that the system was unable to create the log files / temporary that are needed. I think the error is caused by the location of the techila directory (under Program Files) as this may require elevated permissions to write/create files.
There are a couple of different options on how to do this:
Option 1:
Move the techila directory from C:\Program Files\techila\ to your home directory. Having the techila directory in the home directory should solve the problem because it does not need elevated permissions to write files.
Also, after moving the techila directory, please run the installsdk command in MATLAB again to update the search path.
Option 2:
If you do not want to move the techila directory, you can edit the tempdir and logdir parameters in the techila_settings.ini file, which is located in the techila directory. Edit the values of the existing parameters to point to a location which does not require administrative permissions, for example to your home directory.
3.2. When running mbuild, I get an error and a message stating no supported compiler or SDK was found. What do I need to do?
The error means that no supported C compiler was found on your computer. The MATLAB compiler needs a C compiler in order to successfully compile executables.
The solution in this case would be to install a supported C compiler capable of compiling executables. You can find a list of supported compilers in the link displayed in the error message.
Typically, installing Visual Studio is a good way to get a C compiler. Different MATLAB versions support different Visual Studio versions. When installing Visual Studio, remember to perform the installation so that C compilers are also installed (basically you will need to check the compiler checkboxes during the Visual Studio installation process).
3.3. Can I run MATLAB applications in Techila if I’m using a student MATLAB version?
No, you cannot. This is because the student version of MATLAB indeed does not include support for the MATLAB compiler.
In order to use Techila with MATLAB, you need to have the MATLAB compiler.
3.4. How to list transferred files programmatically when using %cloudfor('datafile',..) parameter?
You can store the file names into a cell array and the pass the cell array to datafile parameter. You can build the cell array any way you want. I included an example below. This could be used to run the cloudfor example (3_transferring_datafiles) so that the input file list is built programmatically:
% Get list of files starting with input_ and ending in .mat
listf = dir('input_*.mat')
% create an input cell array for file names
inputlist=cell(1,size(listf,1))
% store the file names in the cell array
for x = 1:size(listf,1)
inputlist{x} = listf(x).name
end
% Build a array
cloudfor k=1:loops
%cloudfor('datafile',inputlist)
% Computational code would go here.
cloudend3.5. I got an error message about an unbalanced cloudfor-cloudend. What do I need to do?
Please check that you have matched all cloudfor-keywords with a cloudend-keyword.