Table of Contents
This guide contains information of interest to developers working with GridFTP. It provides reference information for application developers, including APIs, architecture, procedures for using the APIs and code samples.
Table of Contents
Features new in GT 6.0:
Features that continue to be supported from previous versions
Other Supported Features
Deprecated Features
Table 1.1. Tested Platforms
Operating System | Distribution | Version(s) | Architecture(s) |
---|---|---|---|
Linux | CentOS | 5, 6 | i386, x86_64 |
7 | x86_64 | ||
Fedora | 20, 21, 22 | i386, x86_64 | |
Red Hat Enterprise Linux | 5, 6 | i386, x86_64 | |
7 | x86_64 | ||
Scientific Linux | 5, 6 | i386, x86_64 | |
7 | x86_64 | ||
SUSE Linux Enterprise Server | 11SP3 | x86_64 | |
Debian | 6, 7, 8 | i386, amd64 | |
Ubuntu | 12.04LTS, 14.04LTS, 14.10, 15.04 | i386, amd64 | |
Mac OS X | 10.6-10.10 | i386, x86_64 | |
Solaris | OmniOS | r151006 | x86_64 |
Windows 7 | Cygwin | i386, x86_64 | |
MingW64 | i386, x86_64 |
While the above list includes platforms on which we have tested GridFTP, it does not imply support for a specific platform. However, we are interested in hearing reports of success or bug reports on any platform.
Protocol changes since GT 5.2
API changes since GT 5.2
Exception changes since GT 5.2
Schema changes since GT 5.2
Do you have a need to move large quantities of data rapidly and reliably to remote locations? Globus GridFTP is a software suite optimized for the gamut of data access issues — from bulk file transfer to the details of getting data out of complex storage systems within sites.
Are you concerned about authenticating and authorizing the users? Globus GridFTP supports various authentication and authorization mechanisms. In fact, it is easy to plugin in different authorization mechanism.
Do you need to move data in and out of a complex storage systems? In addition to the POSIX file systems, Globus GridFTP can move data in and out of HPSS and SRB. Capability to access other storage systems can be easily added by implementing a well-defined pluggable interface called Data Storage Interface (DSI).
Do you need transfer huge volume of data and you do not want to babysit the transfer? Use the GridFTP client Reliable File Transfer Service.
Do you need to protect the data while moving it to the remote location? Globus GridFTP provides support for data integrity and data encryption.
Do you need move your data to remote location that is far away and TCP limits the performance? Globus GridFTP supports UDT as an alternate transport protocol for UDT. GridFTP also allows you to use parallel TCP streams.
Do you want to move data from one location to many locations efficiently? Multicasting capability in Globus GridFTP allows you to do it.
Do you have a network link between source and destination that supports much higher data rate than the data rate supported by individual nodes at either end? GridFTP supports cluster-to-cluster data movement – coordinated data transfer using multiple computer nodes at source and destination.
Do you want to limit the resources used for each client connection on the server node? Globus GridFTP can be configured to run with GFork to limit the resource usage. Customized resource control modules can be plugged in easily.
Globus GridFTP has pervasive use in the e-Science Grid community. The high energy physics community in particular has been a huge user from the start. The Relativistic Heavy Ion Collider (RHIC) community in Brookhaven used Globus GridFTP to sustain 600 megabytes per second of data transfer (from Long Island, New York, to Japan) over 11 days.
Frequent large file transfer demands for the British Broadcasting Corporation (BBC) are met by GridFTP. Typical broadcast hour today requires 280 GB for all pre-processed media streams. "Everything in Gridcast is built using Globus Toolkit," said Terry Harmer, Technical Director at the Belfast e-Science Centre, in an interview (http://www.globusconsortium.org/journal/20050816/harmer.html) in '05 with the Globus Consortium Journal. "We use it as a means by which we create, define, and deploy services. We are big users of GridFTP."
Recently, US Department of Energy’s Advanced Photon Source user facility at Argonne transferred more than a terabyte of data (partitioned into lots of small files) to Australia using GridFTP at a rate 30 times faster than traditional data transfer mechanisms such as SCP. The Laser Interferometer Gravitational Wave Observatory (LIGO) project moved 1.5 Terabytes of data from University of Wisconsin at Milwaukee, USA to Hannover, Germany at a sustained rate of 80 MB/s.
Globus implemention of GridFTP is based on the following standards: RFC 959, RFC 2228, RFC 2389, IETF Draft MLST-16 , OGF GFD.020. In addition to these standards, some additional commands are implemented as well. The specification for one such command Data Channel Security Context (DCSC) is available at DCSC Specification.
Table of Contents
GridFTP represents a service that a host is providing. Therefore, the service must be listening on a port waiting for client to request access to that service. This is generally handled one of two ways:
The following list describes the process between the service listening for connection and an exchange of data taking place:
There are basically four important components of the exchange:
The following information is helpful if you want to use GridFTP to access data in DSIs (such as HPSS and SRB), and non-POSIX data sources.
Architecturally, the Globus GridFTP server can be divided into 3 modules:
In the GT 6.0 implementation, the data transform module and the DSI have been merged, although we plan to have separate, chainable, data transform modules in the future.
Note | |
---|---|
This architecture does NOT apply to the WU-FTPD implementation (GT3.2.1 and lower). |
The GridFTP protocol module is the module that reads and writes to the network and implements the GridFTP protocol. This module should not need to be modified since to do so would make the server non-protocol compliant, and unable to communicate with other servers.
The data transform functionality is invoked by using the ERET (extended retrieve) and ESTO (extended store) commands. It is seldom used and bears careful consideration before it is implemented, but in the right circumstances can be very useful. In theory, any computation could be invoked this way, but it was primarily intended for cases where some simple pre-processing (such as a partial get or sub-sampling) can greatly reduce the network load. The disadvantage to this is that you remove any real option for planning, brokering, etc., and any significant computation could adversely affect the data transfer performance. Note that the client must also support the ESTO/ERET functionality as well.
The Data Storage Interface (DSI) / Data Transform module knows how to read and write to the "local" storage system and can optionally transform the data. We put local in quotes because in a complicated storage system, the storage may not be directly attached, but for performance reasons, it should be relatively close (for instance on the same LAN).
The interface consists of functions to be implemented such as send (get), receive (put), command (simple commands that simply succeed or fail like mkdir), etc..
Once these functions have been implemented for a specific storage system, a client should not need to know or care what is actually providing the data. The server can either be configured specifically with a specific DSI, i.e., it knows how to interact with a single class of storage system, or one particularly useful function for the ESTO/ERET functionality mentioned above is to load and configure a DSI on the fly.
See Developing DSIs for GridFTP for more information.
Last Update: August 2005
Working with Los Alamos National Laboratory and the High Performance Storage System (HPSS) collaboration (http://www.hpss-collaboration.org), we have written a Data Storage Interface (DSI) for read/write access to HPSS. This DSI would allow an existing application that uses a GridFTP compliant client to utilize an HPSS data resources.
This DSI is currently in testing. Due to changes in the HPSS security mechanisms, it requires HPSS 6.2 or later, which is due to be released in Q4 2005. Distribution for the DSI has not been worked out yet, but it will probably be available from both Globus and the HPSS collaboration. While this code will be open source, it requires underlying HPSS libraries which are NOT open source (proprietary).
Note | |
---|---|
This is a purely server side change, the client does not know what DSI is running, so only a site that is already running HPSS and wants to allow GridFTP access needs to worry about access to these proprietary libraries. |
Last Update: August 2005
Working with the SRB team at the San Diego Supercomputing Center, we have written a Data Storage Interface (DSI) for read/write access to data in the Storage Resource Broker (SRB) (http://www.npaci.edu/DICE/SRB). This DSI will enable GridFTP compliant clients to read and write data to an SRB server, similar in functionality to the sput/sget commands.
This DSI is currently in testing and is not yet publicly available, but will be available from both the SRB web site (here) and the Globus web site (here). It will also be included in the next stable release of the toolkit. We are working on performance tests, but early results indicate that for wide area network (WAN) transfers, the performance is comparable.
When might you want to use this functionality:
Table of Contents
The Globus FTP Client library provides a convenient way of accessing files on remote FTP servers. In addition to supporting the basic FTP protocol, the FTP Client library supports several security and performance extensions to make FTP more suitable for Grid applications. These extensions are described in the GridFTP Protocol document.
In addition to protocol support for grid applications, the FTP Client library provides a plugin architecture for installing application or grid-specific fault recovery and performance tuning algorithms within the library. Application writers may then target their code toward the FTP Client library and, by simply enabling the appropriate plugins, easily tune their application to run it on a different grid.
All applications which use the Globus FTP Client API must include the
header file globus_ftp_client.h
and activate the and activate the
GLOBUS_FTP_CLIENT_MODULE.
To use the Globus FTP Client API, one must create an FTP Client handle. This structure contains:
The specifics of the connection caching and plugins are found in the "Handle Attributes" section of the API documentation.
Once the handle is created, one may begin transferring files or doing other FTP operations by calling the functions in the "FTP Operations" section of the API documentation. In addition to whole-file transfers, the API supports partial file transfers, restarting transfers from a known point, and various FTP directory management commands. All FTP operations may have a set of attributes, defined in the operationattr section, associated with them to tune various FTP parameters. The data structures and functions needed to restart a file transfer are described in the "Restart Markers" section of the API documentation. For operations which require the user to send to or receive data from an FTP server they must call the functions described in the "globus_ftp_client_data" section of the manual.
The globus_ftp_control library provides low-level services needed to implement FTP clients and servers. The API provided is protocol specific. The data transfer portion of this API provides support for the standard data methods described in the FTP Specification as well as extensions for parallel, striped, and partial data transfer.
For information on the internationalization API, see C Common Libraries API.
Table of Contents
globus-gridftp-server
quickstartTable of Contents
The globus-url-copy program is a command line tool for multi-protocol data movement. It supports gsiftp:// (GridFTP), ftp://, http://, https://, sshftp:// and file:/// protocol specifiers in the URL.
Criteria for determining if files differ when performing a sync transfer. The default sync level is 2. The available levels are:
The Globus Project does not provide an interactive client for GridFTP. Any normal FTP client will work with a GridFTP server, but it cannot take advantage of the advanced features of GridFTP. The interactive clients listed below take advantage of the advanced features of GridFTP.
There is no endorsement implied by their presence here. We make no assertion as to the quality or appropriateness of these tools, we simply provide this for your convenience. We will not answer questions, accept bugs, or in any way shape or form be responsible for these tools, although they should have mechanisms of their own for such things.
UberFTP was developed at the NCSA under the auspices of NMI and TeraGrid:
Table of Contents
The globus-gridftp-server program is a ftp server with support for GridFTP protocol extensions, including strong authentication, parallel data transfers, and parallel data layouts.
The list below contains the command-line options for the server, and also the name of the configuration file entry that implements that option. Note that any boolean option can be negated on the command line by preceding the specified option with -no- or -n. example: -no-cas
or -nf
.
Show usage information and exit.
This option can also be set in the configuration file as help
.
The default value of this option is FALSE
.
Show more usage information and exit.
This option can also be set in the configuration file as longhelp
.
The default value of this option is FALSE
.
Show version information for the server and exit.
This option can also be set in the configuration file as version
.
The default value of this option is FALSE
.
Show version information for all loaded globus libraries and exit.
This option can also be set in the configuration file as versions
.
The default value of this option is FALSE
.
Run under an inetd service.
This option can also be set in the configuration file as inetd
.
The default value of this option is FALSE
.
Run as a daemon. All connections will fork off a new process and setuid if allowed.
This option can also be set in the configuration file as daemon
.
The default value of this option is TRUE
.
Run as a background daemon detached from any controlling terminals.
This option can also be set in the configuration file as detach
.
The default value of this option is FALSE
.
Run over a connected ssh session.
This option can also be set in the configuration file as ssh
.
The default value of this option is FALSE
.
For statically compiled or non-GLOBUS_LOCATION standard binary locations, specify the full path of the server binary here. Only needed when run in daemon mode.
This option can also be set in the configuration file as exec
.
Change directory when the server starts. This will change directory to the dir specified by the chdir_to option.
This option can also be set in the configuration file as chdir
.
The default value of this option is TRUE
.
Directory to chdir to after starting. Will use / if not set. Note that this is the directory of the process, not the client’s home directory.
This option can also be set in the configuration file as chdir_to
.
Enable threaded operation and set the number of threads. The default is 0, which is non-threaded. When threading is required, a thread count of 1 or 2 should be sufficient.
This option can also be set in the configuration file as threads
.
Server will fork for each new connection. Disabling this option is only recommended when debugging. Note that non-forked servers running as root will only accept a single connection, and then exit.
This option can also be set in the configuration file as fork
.
The default value of this option is TRUE
.
Exit after a single connection.
This option can also be set in the configuration file as single
.
The default value of this option is FALSE
.
Path to become the new root after authentication. This path must contain a valid certificate structure, /etc/passwd, and /etc/group. The command globus-gridftp-server-setup-chroot can help create a suitable directory structure.
This option can also be set in the configuration file as chroot_path
.
Add levels together to use more than one.
4 = Disable changing process uid to authenticated user (no setuid) — DO NOT use this when process is started as root.
If not set uses level 2 for front ends and level 1 for data nodes. Note that levels 2 and 4 imply level 1 as well.
This option can also be set in the configuration file as auth_level
.
Only allow connections from these source ip addresses. Specify a comma separated list of ip address fragments. A match is any ip address that starts with the specified fragment. Example: 192.168.1. will match and allow a connection from 192.168.1.45. Note that if this option is used any address not specifically allowed will be denied.
This option can also be set in the configuration file as ipc_allow_from
.
Deny connections from these source ip addresses. Specify a comma separated list of ip address fragments. A match is any ip address that starts with the specified fragment. Example: 192.168.2. will match and deny a connection from 192.168.2.45.
This option can also be set in the configuration file as ipc_deny_from
.
Only allow connections from these source ip addresses. Specify a comma separated list of ip address fragments. A match is any ip address that starts with the specified fragment. Example: 192.168.1. will match and allow a connection from 192.168.1.45. Note that if this option is used any address not specifically allowed will be denied.
This option can also be set in the configuration file as allow_from
.
Deny connections from these source ip addresses. Specify a comma separated list of ip address fragments. A match is any ip address that starts with the specified fragment. Example: 192.168.2. will match and deny a connection from 192.168.2.45.
This option can also be set in the configuration file as deny_from
.
Use GSI security on ipc channel.
This option can also be set in the configuration file as secure_ipc
.
The default value of this option is TRUE
.
Set GSI authorization mode for the ipc connection. Options are: none, host, self or subject:[subject].
This option can also be set in the configuration file as ipc_auth_mode
.
The default value of this option is host
.
Allow clear text anonymous access. If server is running as root anonymous_user must also be set. Disables ipc security.
This option can also be set in the configuration file as allow_anonymous
.
The default value of this option is FALSE
.
Comma separated list of names to treat as anonymous users when allowing anonymous access. If not set, the default names of anonymous and ftp will be allowed. Use * to allow any username.
This option can also be set in the configuration file as anonymous_names_allowed
.
User to setuid to for an anonymous connection. Only applies when running as root.
This option can also be set in the configuration file as anonymous_user
.
Group to setgid to for an anonymous connection. If unset, the default group of anonymous_user will be used.
This option can also be set in the configuration file as anonymous_group
.
Allow sharing when using the supplied DN. A client connected with these credentials will be able to access any user for which sharing is enabled.
This option can also be set in the configuration file as sharing_dn
.
Full path to a directory that will contain files used by GridFTP to control sharing access for individual local accounts. The special variables $HOME and $USER can be used to create a dynamic path that is unique to each local account. This pathmust be writable by the associated account. The default path is $HOME/.globus/sharing/. This must refer to a path on the filesystem, not a path that is only accessible via a DSI plugin.
This option can also be set in the configuration file as sharing_state_dir
.
Allow a local user account to control its own sharing access via special GridFTP client commands. The user account must have filesystem write access to the sharing state dir.
This option can also be set in the configuration file as sharing_control
.
The default value of this option is TRUE
.
Sharing specific path restrictions. This completely replaces the normal path restrictions (-rp) when an account is being shared by a sharing-dn login.Follows normal path restriction semantics.
This option can also be set in the configuration file as sharing_rp
.
Comma separated list of usernames that are allowed to share unless matched in the user deny lists. If this list is set, users that are not included will be denied unless matched in the group allow list.
This option can also be set in the configuration file as sharing_users_allow
.
Comma separated list of usernames that are denied sharing even if matched in the user or group allow lists.
This option can also be set in the configuration file as sharing_users_deny
.
Comma separated list of groups whose members are allowed to share unless matched in the user or group deny lists. If this list is set, groups that are not included will be denied unless matched in the user allow list.
This option can also be set in the configuration file as sharing_groups_allow
.
Comma separated list of groups whose members will be denied sharing unless matched in the user allow list.
This option can also be set in the configuration file as sharing_groups_deny
.
Allow clients to be mapped to the root account.
This option can also be set in the configuration file as allow_root
.
The default value of this option is FALSE
.
Do not check if a user’s system account is disabled before allowing login.
This option can also be set in the configuration file as allow_disabled_login
.
The default value of this option is FALSE
.
Enable clear text access and authenticate users against this /etc/passwd formatted file.
This option can also be set in the configuration file as pw_file
.
Maximum concurrent connections allowed. Only applies when running in daemon mode. Unlimited if not set.
This option can also be set in the configuration file as connections_max
.
Disable all new connections. For daemon mode, issue a SIGHUP to the server process after changing the config file in order to not affect ongoing connections.
This option can also be set in the configuration file as connections_disabled
.
The default value of this option is FALSE
.
Custom message to be displayed to clients when the server is offline via the connections_disabled or connections_max = 0 options.
This option can also be set in the configuration file as offline_msg
.
A comma separated list of client commands that will be disabled.
This option can also be set in the configuration file as disable_command_list
.
Enable the GSI authorization callout framework, for callouts such as CAS.
This option can also be set in the configuration file as cas
.
The default value of this option is TRUE
.
Set the starting directory to the authenticated users home dir. Disabling this is the same as setting -home-dir /.
This option can also be set in the configuration file as use_home_dirs
.
The default value of this option is TRUE
.
Set a path to override the system defined home/starting directory for authenticated users. The special variable strings $USER and $HOME may be used. The authenticated username will be substituted for $USER, and the user’s real home dir will be substituted for $HOME. Be sure to escape the $ character if using these on the command line.
This option can also be set in the configuration file as home_dir
.
A comma separated list of full paths that clients may access. Each path may be prefixed by R and/or W, denoting read or write access, otherwise full access is granted. If a given path is a directory, all contents and subdirectories will be given the same access. Order of paths does not matter — the permissions on the longest matching path will apply. The special character ~ will be replaced by the authenticated user’s home directory, or the -home-dir option, if used. Note that if the home directory is not accessible, \~ will be set to /. By default all paths are allowed, and access control is handled by the OS. In a striped or split process configuration, this should be set on both the frontend and data nodes.
This option can also be set in the configuration file as restrict_paths
.
Do not verify that a symlink points to an allowed path before following. By default, symlinks are followed only when they point to an allowed path. By enabling this option, symlinks will be followed even if they point to a path that is otherwise restricted.
This option can also be set in the configuration file as rp_follow_symlinks
.
The default value of this option is FALSE
.
A comma separated list of ACL or event modules to load.
This option can also be set in the configuration file as acl
.
Log level. A comma separated list of levels from: ERROR, WARN, INFO, TRANSFER, DUMP, ALL. TRANSFER includes the same statistics that are sent to the separate transfer log when -log-transfer is used. Example: error,warn,info. You may also specify a numeric level of 1-255. The default level is ERROR.
This option can also be set in the configuration file as log_level
.
The default value of this option is ERROR
.
globus_logging module that will be loaded. If not set, the default stdio module will be used, and the logfile options apply. Built in modules are stdio and syslog. Log module options may be set by specifying module:opt1=val1:opt2=val2. Available options for the built in modules are interval and buffer, for buffer flush interval and buffer size, respectively. The default options are a 64k buffer size and a 5 second flush interval. A 0 second flush interval will disable periodic flushing, and the buffer will only flush when it is full. A value of 0 for buffer will disable buffering and all messages will be written immediately. Example: -log-module stdio:buffer=4096:interval=10
This option can also be set in the configuration file as log_module
.
Path of a single file to log all activity to. If neither this option or log_unique is set, logs will be written to stderr unless the execution mode is detached or inetd, in which case logging will be disabled.
This option can also be set in the configuration file as log_single
.
Partial path to which gridftp.(pid).log will be appended to construct the log filename. Example: -L /var/log/gridftp/ will create a separate log ( /var/log/gridftp/gridftp.xxxx.log ) for each process (which is normally each new client session). If neither this option or log_single is set, logs will be written to stderr unless the execution mode is detached or inetd, in which case logging will be disabled.
This option can also be set in the configuration file as log_unique
.
Log netlogger style info for each transfer into this file. You may also use the log-level of TRANSFER to include this info in the standard log.
This option can also be set in the configuration file as log_transfer
.
File access permissions of log files. Should be an octal number such as 0644.
This option can also be set in the configuration file as log_filemode
.
Disable transmission of per-transfer usage statistics. See the Usage Statistics section in the online documentation for more information.
This option can also be set in the configuration file as disable_usage_stats
.
The default value of this option is FALSE
.
Comma separated list of contact strings (host:port) for usage statistics receivers. The usage stats sent to a particular receiver may be customized by configuring it with a taglist (host:port!taglist) The taglist is a list of characters that each correspond to a usage stats tag. When this option is unset, stats are reported to usage-stats.globus.org:4810. If you set your own receiver, and wish to continue reporting to the Globus receiver, you will need to add it manually. The list of available tags follow. Tags marked * are reported by default.
*(e) START - start time of transfer *(E) END - end time of transfer *(v) VER - version string of GridFTP server *(b) BUFFER - tcp buffer size used for transfer *(B) BLOCK - disk blocksize used for transfer *(N) NBYTES - number of bytes transferred *(s) STREAMS - number of parallel streams used *(S) STRIPES - number of stripes used *(t) TYPE - transfer command: RETR, STOR, LIST, etc *(c) CODE - ftp result code (226 = success, 5xx = fail) *(D) DSI - DSI module in use *(A) EM - event modules in use *(T) SCHEME - ftp, gsiftp, sshftp, etc. (client supplied) *(a) APP - guc, rft, generic library app, etc. (client supplied) *(V) APPVER - version string of above. (client supplied) (f) FILE - name of file/data transferred (i) CLIENTIP - ip address of host running client (control channel) (I) DATAIP - ip address of source/dest host of data (data channel) (u) USER - local user name the transfer was performed as (d) USERDN - DN that was mapped to user id (C) CONFID - ID defined by -usage-stats-id config option (U) SESSID - unique id that can be used to match transfers in a session and transfers across source/dest of a third party transfer. (client supplied)
This option can also be set in the configuration file as usage_stats_target
.
Identifying tag to include in usage statistics data. If this is set and usage-stats-target is unset, CONFID will be added to the default usage stats data.
This option can also be set in the configuration file as usage_stats_id
.
Comma separated list of remote node contact strings.
This option can also be set in the configuration file as remote_nodes
.
When a server is configured for striped operation with the remote_nodes option, both a frontend and backend process are started even if the client does not request multiple stripes. This option will start backend processes only when striped operation is requested by the client, while servicing non-striped requests with a single frontend process.
This option can also be set in the configuration file as hybrid
.
The default value of this option is FALSE
.
This server is a backend data node.
This option can also be set in the configuration file as data_node
.
The default value of this option is FALSE
.
Size in bytes of sequential data that each stripe will transfer.
This option can also be set in the configuration file as stripe_blocksize
.
The default value of this option is 1048576
.
Number of number stripes to use per transfer when this server controls that number. If remote nodes are statically configured (via -r or remote_nodes), this will be set to that number of nodes, otherwise the default is 1.
This option can also be set in the configuration file as stripe_count
.
Stripe layout. 1 = Partitioned 2 = Blocked.
This option can also be set in the configuration file as stripe_layout
.
The default value of this option is 2
.
Do not allow client to override stripe blocksize with the OPTS RETR command
This option can also be set in the configuration file as stripe_blocksize_locked
.
The default value of this option is FALSE
.
Do not allow client to override stripe layout with the OPTS RETR command
This option can also be set in the configuration file as stripe_layout_locked
.
The default value of this option is FALSE
.
Size in bytes of data blocks to read from disk before posting to the network.
This option can also be set in the configuration file as blocksize
.
The default value of this option is 262144
.
Flush disk writes before sending a restart marker. This attempts to ensure that the range specified in the restart marker has actually been committed to disk. This option will probably impact performance, and may result in different behavior on different storage systems. See the manpage for sync() for more information.
This option can also be set in the configuration file as sync_writes
.
The default value of this option is FALSE
.
Set the default permissions for created files. Should be an octal number such as 0644. The default is 0644. Note: If umask is set it will affect this setting — i.e. if the umask is 0002 and this setting is 0666, the resulting files will be created with permissions of 0664.
This option can also be set in the configuration file as perms
.
Timeout in seconds for all disk accesses. A value of 0 disables the timeout.
This option can also be set in the configuration file as file_timeout
.
Port on which a frontend will listen for client control channel connections, or on which a data node will listen for connections from a frontend. If not set a random port will be chosen and printed via the logging mechanism.
This option can also be set in the configuration file as port
.
Hostname or IP address of the interface to listen for control connections on. If not set will listen on all interfaces.
This option can also be set in the configuration file as control_interface
.
Hostname or IP address of the interface to use for data connections. If not set will use the current control interface.
This option can also be set in the configuration file as data_interface
.
Hostname or IP address of the interface to use for ipc connections. If not set will listen on all interfaces.
This option can also be set in the configuration file as ipc_interface
.
Effectively sets the above control_interface, data_interface and ipc_interface options.
This option can also be set in the configuration file as hostname
.
Port on which the frontend will listen for data node connections.
This option can also be set in the configuration file as ipc_port
.
Time in seconds to allow a client to remain connected to the control channel without activity before authenticating.
This option can also be set in the configuration file as control_preauth_timeout
.
The default value of this option is 120
.
Time in seconds to allow a client to remain connected to the control channel without activity.
This option can also be set in the configuration file as control_idle_timeout
.
The default value of this option is 600
.
Idle time in seconds before an unused ipc connection will close.
This option can also be set in the configuration file as ipc_idle_timeout
.
The default value of this option is 900
.
Time in seconds before canceling an attempted ipc connection.
This option can also be set in the configuration file as ipc_connect_timeout
.
The default value of this option is 60
.
Enable protocol support for UDT with NAT traversal if the udt driver is available. Requires threads.
This option can also be set in the configuration file as allow_udt
.
The default value of this option is FALSE
.
Port range to use for incoming connections. The format is "startport,endport". This, along with -data-interface, can be used to enable operation behind a firewall and/or when NAT is involved. This is the same as setting the environment variable GLOBUS_TCP_PORT_RANGE.
This option can also be set in the configuration file as port_range
.
Message to display to the client before authentication.
This option can also be set in the configuration file as banner
.
File to read banner message from.
This option can also be set in the configuration file as banner_file
.
When this is set, the minimum allowed banner message will be displayed to unauthenticated clients.
This option can also be set in the configuration file as banner_terse
.
The default value of this option is FALSE
.
When this is set, the message set in the banner or banner_file option will be appended to the default banner message rather than replacing it.
This option can also be set in the configuration file as banner_append
.
The default value of this option is FALSE
.
Add an identifying string to the existing toolkit version. This is displayed in the default banner message, the SITE VERSION command, and usage stats.
This option can also be set in the configuration file as version_tag
.
Message to display to the client after authentication.
This option can also be set in the configuration file as login_msg
.
File to read login message from.
This option can also be set in the configuration file as login_msg_file
.
Data Storage Interface module to load. File and remote modules are defined by the server. If not set, the file module is loaded, unless the remote option is specified, in which case the remote module is loaded. An additional configuration string can be passed to the DSI using the format [module name]:[configuration string] to this option. The format of the configuration string is defined by the DSI being loaded.
This option can also be set in the configuration file as load_dsi_module
.
Comma separated list of ERET/ESTO modules to allow, and optionally specify an alias for. Example: module1,alias2:module2,module3 (module2 will be loaded when a client asks for alias2).
This option can also be set in the configuration file as allowed_modules
.
A comma separated list of drivers allowed on the network stack.
This option can also be set in the configuration file as dc_whitelist
.
A comma separated list of drivers allowed on the disk stack.
This option can also be set in the configuration file as fs_whitelist
.
A comma separated list of programs that the popen driver is allowed to execute, when used on the network or disk stack. An alias may also be specified, so that a client does not need to specify the full path. Format is [alias:]prog,[alias:]prog. example: /bin/gzip,tar:/bin/tar
This option can also be set in the configuration file as popen_whitelist
.
An option string to pass to the XIO Network Manager Driver, which will then be loaded for all data channel connections. This must be in the form "manager=module;option1=value;option2=value;". See the Network Manager documentation for more info.
This option can also be set in the configuration file as xnetmgr
.
A comma separated list of XIO drivers and options representing the default network stack. Format is of each driver entry is driver1[:opt1=val1;opt2=val2;…]. The bottom of the stack, the transport driver, is always first.
This option can also be set in the configuration file as dc_default
.
A comma separated list of XIO drivers and options representing the default disk stack. Format is of each driver entry is driver1[:opt1=val1;opt2=val2;…]. The bottom of the stack, the transport driver, is always first.
This option can also be set in the configuration file as fs_default
.
Path to directory holding configuration files that should be loaded. Files will be loaded in alphabetical order, and in the event of duplicate parameters the last loaded file will take precedence. Files with a . in the name (file.bak, file.rpmsave, etc.) will be ignored. Note that the main configuration file, if one exists, will always be loaded last.
This option can also be set in the configuration file as config_dir
.
Base path to use when config and log path options are not full paths. By default this is the current directory when the process is started.
This option can also be set in the configuration file as config_base_path
.
Sets options that make server easier to debug. Forces no-fork, no-chdir, and allows core dumps on bad signals instead of exiting cleanly. Not recommended for production servers. Note that non-forked servers running as root will only accept a single connection, and then exit.
This option can also be set in the configuration file as debug
.
The default value of this option is FALSE
.
This option can also be set in the configuration file as pidfile
.
The globus-gridftp-server-setup-chroot program creates a chroot directory tree that can be used for the globus-gridftp-server(8). This chroot contains a copy of essential POSIX devices in dev; hosts, group, passwd, and grid-security configuration files in etc; and a temporary file directory in tmp.
The -c CERT-DIR option copies certificate files from a different directory than the default [/etc/grid-security/certificates].
Table of Contents
Globus Toolkit does not provide a client with Graphical User Interface (GUI) but Globus Service provides a web GUI for GridFTP data movement. It has the following features: * With a one-click Globus Connect Personal installed on your local system, you can browse the local file system and transfer files and directories between the local system and remote GridFTP servers and between two remote GridFTP servers (third-party transfers). * Supports file system operations such as creating, deleting and renaming files and directories.
Supported Platforms: * Windows * Linux * Mac
Table of Contents
globus-gridftp-server
quickstartThe configuration interface for GridFTP is the admin tool, , which can be used with a configuration file and/or run-time options.
Note | |
---|---|
Command line options and configuration file options may both be used, but the command line overrides the config file. |
The configuration file for the GridFTP server is read from the following locations, in the given order. Only the first file found will be loaded:
-c <configfile>
command line option.
$GLOBUS_LOCATION/etc/gridftp.conf
/etc/grid-security/gridftp.conf
Options are one per line, with the format:
<option> <value>
If the value contains spaces, they should be enclosed in double-quotes ("). Flags or boolean options should only have a value of 0 or 1. Blank lines and lines beginning with # are ignored.
For example:
port 5000 allow_anonymous 1 anonymous_user bob banner "Welcome!"
For complete command documentation including all options, see .
This page includes information about general configuration of the GridFTP server. Security options are discussed here, and more advanced configuration is described here.
The following describes a typical GridFTP configuration of the front end (control channel) and back end (data channels). For other alternatives that provide greater levels of security, see Advanced Configuration.
By default, the data channel and control channel are separate socket connections within the same process. The client sends a command and waits to finish before issuing the next command. This is good for a single host, traditional-type user. If you have a single host and you want an ultra-reliable and light weight file transfer service, this is a good choice. This configuration is also good for testing purposes.
If the GridFTP server is behind a firewall:
Set the environment variable GLOBUS_TCP_PORT_RANGE:
export GLOBUS_TCP_PORT_RANGE=min,max
where min,max specify the port range that you have opened for the incoming connections on the firewall. This restricts the listening ports of the GridFTP server to this range. Recommended range is 1000 (e.g., 50000-51000) but it really depends on how much use you expect.
If you have a firewall blocking the outgoing connections and you have opened a range of ports, set the environment variable GLOBUS_TCP_SOURCE_RANGE:
export GLOBUS_TCP_SOURCE_RANGE=min,max
where min,max specify the port range that you have opened for the outgoing connections on the firewall. This restricts the GridFTP server to bind to a local port in this range for outbound connections. Recommended range is twice the range used for GLOBUS_TCP_PORT_RANGE, because if parallel TCP streams are used for transfers, the listening port would remain the same for each connection but the connecting port would be different for each connection.
Note | |
---|---|
If the server is behind NAT, the |
If the GridFTP client is behind a firewall:
Set the environment variable GLOBUS_TCP_PORT_RANGE
export GLOBUS_TCP_PORT_RANGE=min,max
where min,max specify the port range that you have opened for the incoming connections on the firewall. This restricts the listening ports of the GridFTP client to this range. Recommended range is 1000 (e.g., 50000-51000) but it really depends on how much use you expect.
If you have a firewall blocking the outgoing connections and you have opened a range of (local) ports, set the environment variable GLOBUS_TCP_SOURCE_RANGE:
export GLOBUS_TCP_PORT_RANGE=min,max
where min,max specify the port range that you have opened for the outgoing connections on the firewall. This restricts the GridFTP client to bind to a local port in this range for outbound connections. Recommended range is twice the range used for GLOBUS_TCP_PORT_RANGE, because if parallel TCP streams are used for transfers, the listening port would remain the same for each connection but the connecting port would be different for each connection.
Additional information on Globus Toolkit Firewall Requirements is available here.
There are many security options in GridFTP ranging from no security to higher security via GSI .
Anonymous mode (using the -aa option) allows any user with an FTP client to read and write (and delete) files that the server process can similarly access (it is also a quick way to test that your server works).
globus% globus-gridftp-server -aa Server listening at 127.0.0.1:58806
Warning | |
---|---|
When the server is run in this way, anyone who can connect to the server will posses all the same rights as the user that the process is run as (directly or via -anonymous-user). If using this mode intentionally for open access, it is best to run under a dedicated account with limited filesystem permissions. You can also use the option below to disable FTP commands such as STOR, ESTO, DELE, RDEL, RNTO, etc to make sure that users can only read from the server and not write to it. -disable-command-list <string> Where <string> represents a comma separated list of client commands that will be disabled. Default: not set. |
If you trust your network and want a minimal amount of security, you can run the globus-gridftp-server with clear text passwords. This security model is the one originally introduced in RFC959.
Warning | |
---|---|
We do not recommend it for long running servers open to the internet. |
To run the server in clear text password mode, we first need to create a password file dedicated to it. The format of the password file is the same as standard system password files; however, it is ill-advised to use a system password file. To create an entry in a GridFTP password file, run the following commands:
globus% touch pwfile globus% gridftp-password.pl >> pwfile Password:
This will ask you for a password and then create an entry in the password file for the current user name and the given password. Take a look at the file created. You will notice that the password you typed in is not in the file in a clear text form. We have run it though a one way hash algorithm before storing it in the file.
Simply start the server pointing it at the password file you just created.
globus% globus-gridftp-server -password-file /full/path/of/pwfile Server listening at 127.0.0.1:5555
This type of security introduces the sshftp control channel (frontend)
protocol. This is a very simple means of obtaining strong security on
the control channel only (the data channel is not authenticated).
With this approach, you can run a GridFTP transfer anywhere that you can
ssh. sshftp://
leverages the ubiquitous ssh/sshd programs to form
control channel connections much in the same way that inetd forms
connections. leverages the ubiquitous ssh/sshd programs to form control
channel connections much in the same way that inetd forms connections.
Every host that wishes to run a globus-gridftp-server
which can
accept sshftp://
connections must run the following command as root:
connections must run the following command as root:
globus% globus-gridftp-server-enable-sshftp
In the absence of root access, a user can configure the server to allow
sshftp://
connections for that user only with the following command:
connections for that user only with the following command:
globus% globus-gridftp-server-enable-sshftp -nonroot
The above command creates a file named sshftp in /etc/grid-security (if run as root) or in $HOME/.globus (if run as nonroot). You may edit this file to set gridftp commandline options or environment variables such as GLOBUS_TCP_PORT_RANGE, but you can also set those options in the config file.
In this case, a globus-gridftp-server
does not need to be
running. The server will be started via the sshd program. Therefore, the
hostname and port should be that of the sshd server. Run
globus-url-copy
just as you have before; simply change
ftp://
to to sshftp://
..
globus% globus-url-copy -v file:/etc/group sshftp://127.0.0.1/tmp/group globus% globus-url-copy -list sshftp://127.0.0.1/tmp/
This security option can be the most involved to set up, but provides the most security. It requires setting up GSI security as described in the GT Installation Guide here: Basic Security Configuration.
Once GSI has been set up (host and user credentials are valid, the gridmap file is updated and you’ve run grid-proxy-init to create a proxy certificate), you simply run the GridFTP server:
globus-gridftp-server
Note | |
---|---|
If run as |
Now you are ready to perform a GSI-authenticated transfer:
globus-url-copy <-s subject> src_url dst_url
Note | |
---|---|
The subject option is only needed if the server was not started as root. |
The server should generally be run as root in daemon mode, although it is possible to run it as a user (see below). When run as root you will need to have a host certificate.
Run the server:
globus-gridftp-server < -s | -S > <args>
where:
The following additional steps may be required when running as a user other than root (for more details, review Basic Security Configuration):
~/.gridmap
file, containing the DNs of any clients you wish to allow, mapped to the current username. file, containing the DNs of any clients you wish to allow, mapped to the current username.
grid-proxy-init
.
Note | |
---|---|
We also feature a user-configurable, super-server daemon plugin called GFork. Click here for more information. |
Set up xinetd/inetd config file
Note | |
---|---|
The service name used (gsiftp in this case) should be defined in
|
Here is a sample GridFTP server xinetd config entry in
/etc/xinetd.conf
: :
service gsiftp { instances = 100 socket_type = stream wait = no user = root env += GLOBUS_LOCATION=(globus_location) env += LD_LIBRARY_PATH=(globus_location)/lib server = (globus_location)/sbin/globus-gridftp-server server_args = -i log_on_success += DURATION nice = 10 disable = no }
Here is a sample gridftp server inetd config entry in
/etc/inetd.conf
(read as a single line)
gsiftp stream tcp nowait root /usr/bin/env env \ GLOBUS_LOCATION=(globus_location) \ LD_LIBRARY_PATH=(globus_location)/lib \ (globus_location)/sbin/globus-gridftp-server -i
Note | |
---|---|
On Mac OS X, you must set DYLD_LIBRARY_PATH instead of LD_LIBRARY_PATH in the above examples. |
Note | |
---|---|
You should NOT include USERID in the log lines. See High latency for GridFTP server connections for more information. |
globus-gridftp-server -i
Use the -i commandline option with globus-gridftp-server
:
globus-gridftp-server -i
launchd is used to start services in the system Mac OS X 10.4 (Tiger) and newer, and is also available as a port to FreeBSD.
Here is a sample configuration file for launchd to start the server:
<?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"> <plist version="1.0"> <dict> <key>Disabled</key> <false/> <key>Label</key> <string>globus.gridftp</string> <key>Program</key> <string>(globus_location)/sbin/globus-gridftp-server</string> <key>ProgramArguments</key> <array> <string>globus-gridftp-server</string> <string>-i</string> <string>-d</string> <string>ALL</string> <string>-l</string> <string>(globus_location)/var/gridftp.log</string> </array> <key>Sockets</key> <dict> <key>Listeners</key> <dict> <key>SockServiceName</key> <string>gsiftp</string> </dict> </dict> <key>inetdCompatibility</key> <dict> <key>Wait</key> <false/> </dict> <key>StandardErrorPath</key> <string>/dev/null</string> <key>EnvironmentVariables</key> <dict> <key>GLOBUS_LOCATION</key> <string>(globus_location)</string> <key>LD_LIBRARY_PATH</key> <string>(globus_location)/lib</string> </dict> </dict> </plist>
Provided that the configuration file is stored in
/System/Library/LaunchDaemons/globus.gridftp.plist
, the following
command starts the GridFTP server:, the following command starts the
GridFTP server:
launchctl load /System/Library/LaunchDaemons/globus.griftp.plist
The server will be started automatically with a system, unless the value
of the key Disabled is changed from <false/>
to
<true/>
.
Users are mapped to a local account on the server machine and file permissions are handled by the operating systems. In the anonymous mode, users that connect to the server will posses all the same rights as the user that the server process is run as (directly or via -anonymous-user).
In case of username/password authentication, the users are mapped to the uid corresponding to the username in the GridFTP password file and the access permissions for the users is same as that of the UID that they are mapped to. If SSH based authentication is used, upon successful authentication, SSHD maps users to a local account and the GridFTP server is run as the mapped local user. The access permissions are the same as that of the mapped local user.
If GSI is used, upon successful authentication an authorization callout is invoked to (a) verify authorization and (b) determine the local user id as which the request should be executed. This callout is linked dynamically. Globus GridFTP provides an implementation that supports a Globus "gridmapfile". Sites can also provide alternative implementations. Server does a setuid to the local user id as determined by the authorization callout and the access permissions are the same as that of the local user id.
GridFTP server provides an option to disable certain FTP commands:
-disable-command-list <string>
Where <string> represents a comma separated list of client commands that will be disabled. Default: not set.
The following is a quick guide to running the server and using the client:
Look through the list of options for globus-gridftp-server:
globus-gridftp-server --help
Start the server in anonymous mode (discussed more fully here):
globus-gridftp-server -control-interface 127.0.0.1 -aa -p 5000
where:
Run a two party transfer with client:
globus-url-copy -v file:///etc/group ftp://localhost:5000/tmp/group
Run 3rd party transfer:
globus-url-copy -v ftp://localhost:port/etc/group ftp://localhost:port/tmp/group2
Experiment with -dbg, and -vb options for debugging and checking the performance of your setup:
globus-url-copy -dbg file:///etc/group ftp://localhost:5000/tmp/group
globus-url-copy -vb file:///dev/zero ftp://localhost:5000/dev/null
where:
-dbg**:: A useful option when something is not working. It results in a GridFTP control channel protocol dump (along with other useful information) to stderr. If you understand the GridFTP protocol, or you have ambition to understand it, this can be a very useful tool to discover various problems in your setup such as overloaded servers and firewalls. When submitting a bug report or asking a question on the support email lists one should always send along the -dbg output.
Ctrl-c - Kill the server.
Note | |
---|---|
There are many possible options and configurations with
|
To enable file sharing using Globus Sharing, you have to add the Globus Sharing CA certificates to your trusted certificates directory (/etc/grid-security/certificates) and use -sharing-dn option in the server as follows:
globus% globus-gridftp-server -sharing-dn "/C=US/O=Globus Consortium/OU=Globus Online/OU=Transfer User/CN=__transfer__"
and use -sharing-rp option to restrict the file paths allowed for sharing:
globus% globus-gridftp-server -sharing-rp <path>
Table of Contents
The GridFTP server or client libraries do not read any environment variable directly, but the security and networking related variables described below may be useful.
Table of Contents
If you are having problems using the GridFTP server, try the steps listed below. If you have an error, try checking the server logs if you have access to them. By default, the server logs to stderr, unless it is running from inetd, or its execution mode is detached, in which case logging is disabled by default.
The command line options -d , -log-level, -L and -logdir can affect where logs will be written, as can the configuration file options log_single and log_unique. See globus-gridftp-server for more information on these and other configuration options.
Table 15.1. GridFTP Errors
Error Code | Definition | Possible Solutions |
---|---|---|
| This error message indicates that the GridFTP server doesn’t trust the certificate authority (CA) that issued your certificate. | You need to ask the GridFTP server administrator to install your CA certificate chain in the GridFTP server’s trusted certificates directory. |
| This error message indicates that your local system doesn’t trust the certificate authority (CA) that issued the certificate on the resource you are connecting to. | You need to ask the resource administrator which CA issued their certificate and install the CA certificate in the local trusted certificates directory. |
| This error message indicates one of the following: Certificate Revocation List (CRL) for the source or destination server CA at the client has expired or CRL for client CA has expired at source or destination server or CRL for source (destination) server CA has expired at | The tool available at http://dist.eugridpma.info/distribution/util/fetch-crl/ can be run in a crontab to keep the CRLs up to date. |
Verify that you can establish a control channel connection and that the server has started successfully by telnetting to the port on which the server is running:
% telnet localhost 2811 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. 220 GridFTP Server mldev.mcs.anl.gov 2.0 (gcc32dbg, 1113865414-1) ready.
If you see anything other than a 220 banner such as the one above, the server has not started correctly.
Verify that there are no configuration files being unexpectedly loaded from /etc/grid-security/gridftp.conf or $GLOBUS_LOCATION/etc/gridftp.conf. If those files exist, and you did not intend for them to be used, rename them to .save, or specify -c none on the command line and try again.
If you can log into the machine where the server is, try running the server from the command line with only the -s option:
$GLOBUS_LOCATION/sbin/globus-gridftp-server -s
The server will print the port it is listening on:
Server listening at gridftp.mcs.anl.gov:57764
Now try and telnet to that port. If you still do not get the banner listed above, something is preventing the socket connection. Check firewalls, tcp-wrapper, etc.
If you now get a correct banner, add -p 2811 (you will have to disable (x)inetd on port 2811 if you are using them or you will get port already in use):
$GLOBUS_LOCATION/sbin/globus-gridftp-server -s -p 2811
Now telnet to port 2811. If this does not work, something is blocking port 2811. Check firewalls, tcp-wrapper, etc.
If this works correctly then re-enable your normal server, but remove all options but -i, -s, or -S.
Now telnet to port 2811. If this does not work, something is wrong with your service configuration. Check /etc/services and (x)inetd config, have (x)inetd restarted, etc.
If this works, begin adding options back one at a time, verifying that you can telnet to the server after each option is added. Continue this till you find the problem or get all the options you want.
At this point, you can establish a control connection. Now try running globus-url-copy.
Once you’ve verified that you can establish a control connection, try to make a transfer using globus-url-copy.
If you are doing a client/server transfer (one of your URLs has
file:
in it) then try: in it) then try:
globus-url-copy -vb -dbg gsiftp://host.server.running.on/dev/zero file:///dev/null
This will run until you control-c the transfer. If that works, reverse the direction:
globus-url-copy -vb -dbg file:///dev/zero gsiftp://host.server.running.on/dev/null
Again, this will run until you control-c the transfer.
If you are doing a third party transfer, run this command:
globus-url-copy -vb -dbg gsiftp://host.server1.on/dev/zero gsiftp://host.server2.on/dev/null
Again, this will run until you control-c the transfer.
If the above transfers work, try your transfer again. If it fails, you likely have some sort of file permissions problem, typo in a file name, etc.
If the server has started correctly, and your problem is with a security failure or gridmap lookup failure, verify that you have security configured properly here.
If the server is running and your client successfully authenticates but has a problem at some other time during the session, please ask for help on gt-user@globus.org. When you send mail or submit bugs, please always include as much of the following information as possible:
If you run GridFTP servers via Xinetd and notice high latency for
connections and/or transfers, check if /etc/xinetd.conf
or the
gsiftp service configuration inside or the gsiftp service configuration
inside /etc/xinetd.d
is set to log USERID as follows: is set to log
USERID as follows:
log_on_success += USERID log_on_failure += USERID
Such a configuration tells Xinetd to log the remote user using the method defined in RFC 1413, which causes an ident client to attempt to query the machine that the connection is coming from before the service will run. Even when this succeeds, the response can’t be trusted, and more often than not it is rejected or simply dropped (which results in the longest delays) by the remote firewall.
Latency can be reduced by making sure Xinetd does not log the USERID.
Table of Contents
This document provides the specification and rationale for the DCSC (Data Channel Security Context) command that is implemented by Globus GridFTP server.
A user may have multiple credentials issued by different organizations that know nothing about the other. If a user tries to perform a 3rd party FTP transfer between these organizations' servers using DCAU (Data Channel Authentication), it will fail because DCAU uses the user’s credentials and each side does not have the CA certificate that issued the other side’s user credential. Thus, users are unlikely to be able to perform secure 3rd party transfers in large federated environments. We present the DCSC command as a way to enable DCAU in this scenario even if one side is a legacy server that knows nothing about DCSC. FTP servers that use SSH for user authentication also benefit from DCSC since it provides a common, interoperable context for DCAU.
The current DCAU protocol uses an SSL context that contains the logged in user’s credential. If two servers have different user credentials and do not have each others' CA certificates, a client can not perform a secure 3rd party transfer between them. If one of the servers supports DCSC, a client can tell it to both send and accept the user credential used by the other server, thus enabling DCAU where it previously was not possible. If both servers support DCSC, clients that desire higher security may specify a random, self signed certificate as the DCAU context.
The general format for DCSC is:
DCSC context type context specific blob]
Where context type is a case insensitive string and the blob is a string composed of only printable ASCII (32-126) characters, such as base64 encoding would produce. This document specifies the "P" and "D" context types.
A server that supports DCSC SHOULD include a line in its FEAT output so clients can discover the feature. The format of the FEAT line is:
DCSC context type ["," context type]
For example, "DCSC P,D" means a server supports the "P" and "D" DCSC types.
The allowed return codes for DCSC P and DCSC D are:
* 2xx Operation succeeded * 5xx Operation failed; see message for specific information
The "P" context type (short for proxy/PEM) is:
DCSC P base64 encoded blob
base64 encoded blob:
The PEM certificate(s) and key are already mostly in base64 encoding (the new line characters aren’t) and thus encoding them again in base64 wastes some space. However, this simplifies client and server implementation.
A DCSC "P" command will overwrite any previous request. A server SHOULD accept a DCSC "P" blob of at least 10KB in size.
The certificate (1) and key (2) MUST be used by the server as its client certificate and key for all of the current session’s subsequent DCAU connections. Additionally, the DCAU A (self) command MUST imply the subject of certificate (1) instead of the user’s log in credential. This MUST occur even for a DCAU A that is implied if no DCAU command was issued by a client.
The certificate in (1) MUST be self signed or verifiable using only intermediate and/or CA certificates in (3). If the certificate in (1) is not self signed, clients MUST send its full certificate chain, including the CA certificate, in (3). Servers SHOULD reject a request in which the certificate (1) is not self signed and can not be validated using only the certificates in (3).
A server MUST validate the remote party’s DCAU certificate using the combination of:
* The server's default CA certificates and signing policies * All self signed certificates given in (1) and (3)
The server must trust all self signed certificates presented in (1) and (3), whether they are marked as a CA or not, in addition to its default CA certificate list, for all subsequent DCAU connections. A server MUST NOT permanently trust the certificates in (1) and (3) or use them for anything other than validating DCAU connections used by the current session.
Clients SHOULD provide the server all of the intermediate and/or CA certificates needed to verify the remote party’s certificate by adding them to (3). Servers MUST NOT require signing policy files for any CA certificates in (3). If signing policies do exist for any CA certificates in (3), the server SHOULD still use and enforce them. The DCSC command does not provide a way to specify signing policies; it is expected that the server’s default CA certificates are already protected by signing policies.
The "D" context type stands for "default context". The command "DCSC D" will revert the context to whatever it was immediately after login.
In legacy mode (only one server supports DCSC), the server supporting DCSC holds two short term credentials - one from a CA it knows about, and one from a CA of the other party. A compromised server could thus leak both short term user credentials. This might be mitigated somewhat by using an independent proxy for DCSC. If both servers support DCSC, a random, self signed certificate SHOULD be used instead of a user credential for DCAU.
Clients SHOULD send the DCSC command over an encrypted control channel.
Table of Contents
The GridFTP server provides high speed remote access to data stores. There are many different types of data storage systems from standard file systems to arrays of magnetic tape. To allow GridFTP to be a transfer interface to as many data storage systems as possible the Data Storage Interface (DSI) was created.
The DSI presents a modular abstraction layer to a storage system. It consists of several function signatures and a set of semantics.
This document provides an introduction to the DSI and how to create one.
The set of interface functions that define the DSI can be found in
globus_gridftp_server.h
.
All type definitions starting with globus_gfs_storage_*()
are part
of the DSI interface.
An API is provided to the DSI author to assist in implementation. The most interesting parts of this API provide functions that abstract away the details of sending data across the data channel. The DSI author is not expected to know the intimate details of the data channel protocols involved in a GridFTP transfer. Instead this API provides functions for reading and writing data to and from the net.
FIXME - link to api doc
The following is a brief description of part of the DSI implementation process.
An FTP session is defined from the time a client is authorized to use the server until the time the connection is disconnected (disconnect can happen due to the client sending QUIT, error, or timeout, etc). In the lifetime of the session, the client issues various commands to the FTP server. Some of these commands require access to the storage system, and thus require action by the DSI. Whenever such a command is received, the server calls out to the appropriate DSI interface function requesting that the specific operation be performed.
The server passes a
globus_gfs_operation_t
data type as a parameter to all DSI request functions. When the DSI is finished performing that operation, it calls a corresponding
globus_gridftp_server_finished_<type>()
function, passing it this globus_gfs_operation_t
structure (and
whatever other data is needed for any given operation). This lets the
server know that the operation is completed and it can respond to the
client appropriately.
As an example we will look at how a simple unix file system DSI would
implement the stat
function.
The DSI’s function signature for stat
is:
void (*globus_gfs_storage_stat_t)( globus_gfs_operation_t op, globus_gfs_stat_info_t * stat_info, void * user_arg);
When it is called, the DSI is expected to:
stat_info→pathname
,
globus_gfs_stat_t
with that information,
globus_gridftp_server_finished_stat()
with that
structure.
static void globus_gfs_storage_example_stat( globus_gfs_operation_t op, globus_gfs_stat_info_t * stat_info, void * user_arg) { globus_gfs_stat_t stat_out; struct stat stat_in; stat(stat_info->pathname, &stat_in); stat_out.mode = stat_in.st_mode; stat_out.nlink = stat_in.st_nlink; stat_out.uid = stat_in.st_uid; stat_out.gid = stat_in.st_gid; stat_out.size = stat_in.st_size; stat_out.mtime = stat_in.st_mtime; stat_out.atime = stat_in.st_atime; stat_out.ctime = stat_in.st_ctime; stat_out.dev = stat_in.st_dev; stat_out.ino = stat_in.st_ino; stat_out.name = strdup(stat_info->pathname); globus_gridftp_server_finished_stat(op, GLOBUS_SUCCESS, &stat_out, 1); }
This is obviously a very basic example but it should serve for the purposes of understanding.
Every DSI must register itself with the Globus extensions module properly. This can be a tedious task yet must be done properly. For this reason, we created a distribution that provides a skeleton DSI upon which a developer can build.
The distribution includes a script to generate C stubs for a DSI with
all of the proper shared library hooks and names needed to work with the
globus-gridftp-server
. The DSI implementor must fill in the
stubbed-out functions with the necessary code specific to their needs.
% ./generate-stubs.sh dsi name flavor
This command will generate the c source file. "dsi name" is the string that will be associated with the DSI. It must be unique to your Globus installation. To load it into the server use the '-dsi ' option to the server.
% make
This will compile the DSI and create the dynamically loadable library.
To include additional compile dependencies or libraries, open
Makefile
and add them to the appropriate MACRO line. and add them to
the appropriate MACRO line.
% make install
This will copy the library to $GLOBUS_LOCATION/lib
, thereby making
it ready for use., thereby making it ready for use.
Table of Contents
GridFTP is a well known, extremely fast and efficient protocol for transferring data from one destination to another. Here we present how GridFTP can be used to transfer a single file to many destinations in a multicast/broadcast.
The purpose of this work is to efficiently transfer a single data set to many locations. Our goal is to use all available network cycles, effectively moving the total amount of data (destinations * file size) at network speeds. Ideally, we would like to transfer to all destinations in the same time it takes to transfer to a single destination.
Distributing the data to many endpoints is not difficult, and has been a feature of (a popular GridFTP client) for some time. It would be quite simple to have the client loop over the destination set and send to each destination in series. Of course, this would be quite slow and the transfer time would scale linearly with the data set.
In our architecture, destination servers are configured so that they can forward packets along to other endpoints. Thus, the GridFTP servers act as both servers receiving data and clients sending data. The server is set up as a tree such that the first destination writes the data to disk and then forwards it on to N more hosts (by default N is 2). This process is repeated until all destinations have received the data.
For further explanation, the architecture can be thought of as a directed graph. Each destination is a vertex with N edges connecting it to N other vertices. A spanning tree is formed connecting all vertices. The degree of any one vertex is a client -side configuration option.
The following image illustrates this:
In the image, data blocks are purple and are sent first from the client to a root destination. The root destination then forwards it on to two more servers.
The figure in the previous section shows 4 colored boxes. Orange represents client logic, blue is server logic, and as stated above, purple is for data block. The final box type is yellow and it is used to show globus XIO drivers.
More information on Globus XIO can be found here. For our purposes we can think of each XIO driver as a modular protocol interpreter that can be plugged in to an IO stack without involving the application using it. In this way, we can add functionality to an existing application without disturbing its tested code base.
Because the uses Globus XIO for all of its IO, we are able to forward
data at the block level. We achieve this by allowing the client to add a
new XIO driver, the gridftp_multicast
driver, to the GridFTP
server’s disk stack. Because of the modular driver abstraction that
Globus XIO provides as the GridFTP server writes data blocks to its file
system, the data blocks are first passed through the
gridftp_multicast
driver. As the gridftp_multicast
driver passes
the data block on to be written to disk, it also forwards the block on
to other GridFTP servers in the tree.
Using this approach to add the multicast functionality is minimally invasive to the tested and robust GridFTP server and is entirely modular. The driver is written to a well defined and clean abstraction. Enabling this feature is a simple matter of inserting the driver in the disk stack and passing the driver stack the destination list.
In addition to allowing for multicast, the gridftp_multicast
driver
and this architecture allow us to create a network overlay where many
GridFTP servers act as routers forwarding packets along to each other
until they get to the final destination where there are written to disk.
The advantage of this type of system is actively researched by
Phoebus.
The gridftp_multicast
driver can be configured to only forward data
along to the next server, and to not write it to disk. Furthermore, it
can be told to only forward to a single endpoint. When configured in
this way, we achieve the network overlay described above.
The following images illustrate this. In the first image we show the standard case where data is sent from a client to a server through the Internet. The routing is done by the Internet outside of the clients control.
In the next image we show how the gridftp_multicast
driver can route
data through the network via GridFTP servers. This allows the user to
have greater control over the network path which the data takes.
To show the effectiveness of this architecture we ran experiments on the UC TeraGrid. We show some of those results here.
In the first experiment, we leased nodes from the UC TeraGrid. 29 hosts
were designated as destinations and we ran gridftp_multicast
-enabled
GridFTP servers on them. 1 node was designated as the client node and
from it all transfers were started.
All transfers were performed with globus-url-copy
and a tcp
buffer size of 128KB. As a control group, we ran a transfer using
globus-url-copy
with the -f option. This caused the source
file to be sent to each endpoint in serial. We then transferred the
source to all destinations using this architecture.
The first graph shows the completion time of a multicast session against the number of destinations. The first line is when the transfers are performed in a serial fashion from the client. All other lines are multicast sessions performed using this architecture. Each line represents a different vertex degree in the spanning tree (ie, each server forwarding to a different number of destinations).
As expected, the results for the serial transfer scales linearly while the multicast sessions very slowly increase with more destinations.
The next graph shows the same experiment; but instead of graphing completion time, we graph the clients sending throughput. This is the size of the files being sent (1GB) divided by the time it takes for this file to reach all destinations.
The final graph shows the collective bandwidth of a transfer. The graphing function is (# of destinations * file size) / time.
We created another XIO driver that allows each endpoint in the session to buffer the data. This prevents stalls in sending data transfers due to the latency required to reach a leaf node, and gets data to the disks of nodes higher in the tree faster. Early results show slight improvements on a LAN, but we expect greater results when broadcasting across WANs.
The additions to the protocol are exceptionally minor. Every server in the tree (except for leaf nodes) becomes a client to another server, but that client speaks the standard GridFTP protocol. The only change needed is a command to add the driver to the file system stack, and that command has existed in the GridFTP server for some time.
The command is:
SITE SETDISKSTACK 1*{driver name[:driver options]},
The second parameter to the site command is a comma-separated list of driver names optionally followed by a colon (:) and a set of driver-specific URL-encoded options. From left to right, the driver names form a stack from bottom to top.
Adding the gridftp_multicast
driver to this list will enable the
multicast functionality. The set of options are the same as those
specified in the previous section. The only difference is that each url
in the urls= options must be url encoded.
The broadcast functionality can be used with globus-url-copy
. We
added the following option:
-mc filename
The file must contain a line separated list of destination urls. For example:
gsiftp://localhost:5000/home/user/tst1 gsiftp://localhost:5000/home/user/tst2 gsiftp://localhost:5000/home/user/tst4
The source url is specified on the command line as always. A single destination url may also be specified on the command line in addition to the urls in the file. An example globus-url-copy command is:
% globus-url-copy -MC multicast.file gsiftp://localhost/home/user/src_file
To transfer a single file to many destinations in a multicast/broadcast, use the new -mc option.
Note | |
---|---|
To use this option, the admin must enable multicasting. Click here for more information. |
Warning | |
---|---|
This option is EXPERIMENTAL |
globus-url-copy -vb -p 4 -mc filename source_url
The filename must contain a line-separated list of destination urls. For example:
gsiftp://localhost:5000/home/user/tst1 gsiftp://localhost:5000/home/user/tst3 gsiftp://localhost:5000/home/user/tst4
For more flexibility, you can also specify a single destination url on the command line in addition to the urls in the file. Examples are:
globus-url-copy -MC multicast.file gsiftp://localhost/home/user/src_file
or
globus-url-copy -MC multicast.file gsiftp://localhost/home/user/src_file gsiftp://localhost/home/user/dest_file1
Along with specifying the list of destination urls in a file, a set of
options for each url can be specified. This is done by appending a ?
to the resource string in the url followed by semicolon-separated key
value pairs. For example:
gsiftp://dst1.domain.com:5000/home/user/tst1?cc=1;tcpbs=10M;P=4
This indicates that the receiving host dst1.domain.com
will use 4
parallel stream, a tcp buffer size of 10 MB, and will select 1 host when
forwarding on data blocks. This url is specified in the will use 4
parallel stream, a tcp buffer size of 10 MB, and will select 1 host when
forwarding on data blocks. This url is specified in the -mc file as
described above.
The following is a list of key=value options and their meanings:
In addition to allowing multicast, function also allows for creating user-defined network routes.
If the local_write option is set to n
, then no data will be
written to the local disk, the data will only be forwarded on.
If the local_write option is set to n
and is used with the cc=1
option, the data will be forwarded on to exactly one location.
This allows the user to create a network overlay of data hops using each GridFTP server as a router to the ultimate destination. :leveloffset: 0
For security reasons the GridFTP server does not allow clients to load arbitrary xio drivers into the server. The GridFTP server admin must whitelist the driver individually. White-listing the mlink driver is done with the following parameter to the server:
-fs-whitelist file,gridftp_multicast
Notice that file
must also be specified. Without this option, the
file
driver is the default; however, if used, you must specifically
list it.