24 Exchange of Data

IQdesktop is the working horse, providing the workflow tools, the computational power by the host system, and allowing the analyst to conduct the work to be done.

Storage of any data, models, or results should not happen on the virtual system and backup of files on the virtual system is not done. Transient storage is allowed. Several possibilities to efficiently and easily handle transient storage, long term storage, and backup exist:

Storage only outside the container via shared volumes (potentially slow when shares not on local computer)
Transient storage in IQdesktop with synchronization to the outside (recommended for server and cloud based deployment)

24.1 Shared volumes

The configuration file for IQdesktop allows to define a folder that is shared with the host computer’s file system. By default only one such shared volume is set up but multiple are possible. The shared folder can be accessed from IQdesktop as \IQDESKTOP\SHARE.

Note that on Windows it is only possible to share folders on local drives. Folders on network drives cannot be shared. On Linux and MacOS that limitation does not exist.

This approach is recommended when using IQdesktop on your local computer.

It can also be used easily when deploying IQdesktop on a local server. In this case different file servers might be mounted to the the host computer and selected folders can be shared with IQdesktop containers through the volume approach.

24.2 SMB Protocol

File servers that use the SMB protocol can be easily mounted from within IQdesktop and mounts to be done can be configured conveniently in the config file (see here).

Note that this only works on Linux host machines (on which docker is running). On Windows local computers the recommended exchange method is via Shared Volumes. This is a limitation of Windows/Docker, and not of IQdesktop.

The approach is as follows:

Define the servers and folders to be mounted in the configuration file (see example here).
Upon start of the container
- The /etc/fstab file is updated accordingly.
- Mounts are not done at start - they have to be exectued manually.
- Convenient command line commands are available to mount and unmount the server folders. The names of these commands is constructed as followd: mount_"LABEL", where "LABEL" should be replaced by the label that was provided to a certain mount point in the configuration file. Similarly, umount_"LABEL" is used to unmount a specific folder.
- Password for user on server will be asked and needs to be provided.

Requirements for this to work:

Host system must be a Linux system
The privileged mode needs to be enabled in the configuration file

24.3 AWS S3 Buckets

AWS S3 buckets can easily be mounted to IQdesktop. You will need to provide your AWS Access Key and your AWS Secret Access Key, either in the config file or by calling the command awskeys from the IQdesktop command line.

An AWS S3 bucket "s3bucketname" can then be mounted by typing mountS3 "s3bucketname".
It can be unmounted by typing umountS3 "s3bucketname".
In the commands above exchange "s3bucketname" with the bucket name that you have on AWS S3 and want to access.

After mounting, the mounted AWS S3 bucket filesystem will be available in: /IQDESKTOP/MOUNT/"s3bucketname".

24.4 Git

When running IQdesktop on a server, IntiQuan recommends to use the shared folder (folder, volumes) to share bare git repositories that allow to clone them into IQdesktop. The proposed destination of the cloned git repositories is ~/PROJECTS, where ~ is the IQdesktop user’s home folder.

Ongoing work can be saved by commiting and pushing changes to the outside of IQdesktop. In addition to saving work outside IQdesktop, this approach also allows collaboration between different (IQdesktop) users on the same project.

Convenient git command aliases are provided in IQdesktop:

gclone equals: git clone
gpull equals: git pull
gpush equals: git add .; git commit -m "-"; git push
gpush message equals: git add .; git commit -m "message"; git push

24.5 Subversion

Since version 0.3.0 subversion is available on IQdesktop, allowing to checkout, commit, etc. files located on an external subversion server. Standard command line calls to the subversion client svn apply.

24.6 Secure copy

The secure copy protocol (scp) can be used to exchange data between an IQdesktop container and another computer. The SSH server in the IQDesktop container needs to be enabled for that.

On Windows the use of the WinSCP software (obtain from here) can be used for this purpose.

To connect to an IQdesktop container with WinSCP:

Enter the IPADDRESS of the containers host computer
Change the Port number to the SSH port number on your container
Enter your username and password
Press “Login”

You will be connected to your home folder on the IQdesktop container and can transfer files back and forth in a secure manner.

24.7 Map as network drive

Your IQdesktop home folder can be mapped as a folder in your local computers explorer. This allows for an additional possibility of file exchange that does not require the use of the \IQDESKTOP\SHARE folder.

Linux and Mac users will be able to mount their IQdesktop home folder via sshfs from the file manager. That approach allows to access IQdesktop as if it were a folder on the local machine giving you immediate access to all results produced on IQdesktop.
On Windows sshfs is not available by default but can be installed and used as described below.

24.7.1 Installation and use of `sshfs` on Windows

The installation procedure is documented at this location. Below abbreviated instructions are given that should be sufficient for a successfull installation.

Note that this installation does require administrator rights.

Download WinFsp 2020 installer from github
Download SSHFS-Win 2020 installer from github
Run both installers, keeping all settings on default values

24.7.2 Mapping an IQdesktop container on Windows

Open the Windows Explorer. Right-click on This PC > select Map Network Drive and enter the address: \\sshfs\user@127.0.0.1!6900

Note that 127.0.0.1 should be used if you run IQdesktop on your local computer. If you want to access an IQdesktop instance that runs on a server you will need to provide that servers IP address or URL name. 6900 stands for your SSH port on your IQdektop instance. On local IQdesktop instances this is typically 6900 (but you can set this differently in the configuration file). On server installations your system administrator will provide you with the correct port number. Also exchange user with your username on your IQdesktop container.