XCI-826 Testing Discussions

86 posts / 0 new
Last post

I am done with the "XSEDE Globus Connect Server v5.4 Installation Guide" (https://docs.google.com/document/d/18eft0Hm94xjV3sqe0UBVrgKi7w9ohaDy1jyz...)

As the result, I was able to create the three pairs of storage gateways and their collections as shown below.


 $ globus-connect-server whoami
Username      | Name         | ID                                   | Email
------------- | ------------ | ------------------------------------ | ----------------
jkm@xsede.org | Jonghoon Kim | ecb182f4-96b1-4cd1-83b5-d6ca9255d658 | jkm@illinois.edu


$ globus-connect-server storage-gateway list
Display Name                                    | ID                                   | Connector | High Assurance
----------------------------------------------- | ------------------------------------ | --------- | --------------
Expanse XSEDE Gateways Storage Gateway          | 57cbb913-1db6-4ab0-8552-b2fcaf983dbd | POSIX     | False
Expanse XSEDE Useer Storage Gateway             | 9fd69a73-a1d2-41d3-b3ba-71b97029f976 | POSIX     | False
Expanse XSEDE Guest Collections Storage Gateway | effafbab-f00b-474f-936b-45bc5bfef0e3 | POSIX     | False


$ globus-connect-server collection list
ID                                   | Display Name                                       | Owner         | Collection Type | Storage Gateway ID
------------------------------ | -------------------------------------------------- | ------------ | --------------- | ------------------------------------
0ba3ed73-79ce-4cce-92a5-5893c3c8a940 | Expansee XSEDE Gateways Collections                | jkm@xsede.org | mapped          | 57cbb913-1db6-4ab0-8552-b2fcaf983dbd
5705bbdd-7f0e-4e84-8673-7e3065b94ebe | Expanse XSEDE User Data Collection                 | jkm@xsede.org | mapped          | 9fd69a73-a1d2-41d3-b3ba-71b97029f976
8ba2076f-893c-4b00-84b9-747ccfe3b8f4 | Expanse XSEDE  Guest Collections for Users Sharing | jkm@xsede.org | mapped          | effafbab-f00b-474f-936b-45bc5bfef0e3


Now, Test1 (T1) is done from me.

First, I tested again for the update RPM xsede-oauth-mapfile and it worked.

Update the rpm and add the ‘MAP-RESOURCE:  "expanse.sdsc.xsede.org"’ to the map-config.json .
Running the ‘sudo  <rpm-path>/bin/xsede-oauth-mapfile.sh’ generated ‘/etc/grid-security/xsede-oauth-mapfile’ fille successfully.


Second, I am done with the "XSEDE Globus Connect Server v5.4 Installation Guide" (https://docs.google.com/document/d/18eft0Hm94xjV3sqe0UBVrgKi7w9ohaDy1jyz...)

As the result, three pairs of storage gateways and their collections as shown below, and I can access them via https://app.globus.org/ by searching 'xcidev1' collections.


$ globus-connect-server storage-gateway list
Display Name                                    | ID                                   | Connector | High Assurance
----------------------------------------------- | ------------------------------------ | --------- | --------------
xcidev1 XSEDE Gateways Storage Gateway          | 045b7ef5-9167-49fd-913b-5f44eacf166e | POSIX     | False
xcidev1 XSEDE Guest Collections Storage Gateway | 8eac372b-c6fb-40c5-b8b7-876cc85ead8d | POSIX     | False
xcidev1 XSEDE User Storage Gateway              | cbdbad47-54a3-4ef9-a87c-8109c6fae68b | POSIX     | False



$ globus-connect-server collection list
ID                                   | Display Name                                       | Owner         | Collection Type | Storage Gateway ID
------------------------------------ | -------------------------------------------------- | ------------- | --------------- | ------------------------------------
3e44567e-2a22-4391-a91a-8ebb36c259ed | xcidev1 XSEDE User Data Collection                 | jkm@xsede.org | mapped          | cbdbad47-54a3-4ef9-a87c-8109c6fae68b
4503dafa-2894-4105-a561-484139c82144 | xcidev1 XSEDE  Guest Collections for Users Sharing | jkm@xsede.org | mapped          | 8eac372b-c6fb-40c5-b8b7-876cc85ead8d
79847e56-77dd-40d1-b4a6-486baa3c3052 | xcidev1 XSEDE Gateways Collections                 | jkm@xsede.org | mapped          | 045b7ef5-9167-49fd-913b-5f44eacf166e



Note: for the authentication, I had to go through 2FA auth again for accessing the Gest collection.


TEST1 is done.


TEST2  (from the Testing Plan doc ) succeed.

  1. Transfer any file from your Primary Collection home directory to another existing XSEDE endpoint - OK
  2. Transfer any file from another existing XSEDE endpoint to your Primary Collection home - OK
  3. Attempt to transfer any file from another existing XSEDE endpoint to /tmp FAILS - OK (access failed)


TEST3  (from the Testing Plan doc ) 

Two issues.

1. the Prerequisites "Follow GCS Domain Guide" link is broken.

2. I ran the test3.3 command with existing ' /etc/pki/tls/certs/*.pem' files and got error as shown below. ( I didn't use sudo ) 

globus-connect-server endpoint domain update \
    --wildcard \
    --certificate-path /etc/pki/tls/certs/hostcert.pem \
    --private-key-path /etc/pki/tls/certs/hostkey.pem \
    --certificate-chain-path /etc/pki/tls/certs/incommon_rsa_chain.pem \
    --domain xcidev1.dyn.xsede.org \


PermissionError: [Errno 13] Permission denied: '/etc/pki/tls/certs/hostkey.pem'

The "endpoint domain update" command should be run as root (sudo) on xcidev1 so that it can access the "*.pem" files.


When I run it with 'sudo' I got this error.

Error: You must login first, before running this command!


I logged in as jkm as shown below.

$ globus-connect-server whoami
Username      | Name         | ID                                   | Email
------------- | ------------ | ------------------------------------ | ----------------
jkm@xsede.org | Jonghoon Kim | ecb182f4-96b1-4cd1-83b5-d6ca9255d658 | jkm@illinois.edu

I don’t know if this matters but in case it helps to debug.

According to the command examples from ‘4.2.2. Setup the endpoint’,  ‘4.3. Start the server’ and ‘4.4. Add Data Transfer Nodes to the endpoint’. at https://docs.globus.org/globus-connect-server/v5.4/#set-endpoint-as-managed , I used ‘sudo’ for 4.3 and 4.4 but not for 4.2.2.


Test3 succeeded.

It worked after login with 'sudo' and run the command.  Needed to run with root id. 


I ran Step A, trying to install the development repo, I do not get the error it says I should get.  From this doc:  https://software.xsede.org/development/repo/repoconfig.txt

Step B expects me to get an error: 

   You should receive a warning similar to this:
      warning: XSEDE-Development-config.centos-5-1.noarch.rpm: Header V3 DSA signature: NOKEY, key ID 20423dbb
   This is a known trust bootstrapping issue because until you complete this entire procedure,
      RPM doesn't trust the signer of this RPM.

Then suggests that I run steps C, D & E: 


c) Configure RPM to trust XSEDE's signature (PGP key) installed by the above package:
   rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-XSEDE-Development

d) If you wish to verify and surpress trust warnings for developer only signed packages, download:
   Verify the tar, untar it, and import the specific developer keys you want RPM to trust:
   rpm --import GPG-KEY-user-<portal_username>

e) To import a developers key into your GPG keyring so that you can independently verify packages:
   gpg --import GPG-KEY-user-<portal_username>

If I don't get the error, do I still need to run steps, C, D & E? 

Hey Susan,

That sentence should say "You may receive a warning.." because that warning only happens if your server doesn't already trust XSEDE's repository. It's just a warning, so no worries if you don't get it. You can proceed.

So following on with Steps C-E, and some of them failed.  If I didn't get the error message, I should be told to jump to the next section. 

[root@vm034 susan]# yum install  https://software.xsede.org/development/repo/repos/XSEDE-Development-conf...
Loaded plugins: fastestmirror
XSEDE-Development-config.centos-7-1.noarch.rpm                                              | 4.7 kB  00:00:00
Examining /var/tmp/yum-root-C7CqI_/XSEDE-Development-config.centos-7-1.noarch.rpm: XSEDE-Development-config.centos-7-1.noarch
Marking /var/tmp/yum-root-C7CqI_/XSEDE-Development-config.centos-7-1.noarch.rpm to be installed
Resolving Dependencies
--> Running transaction check
---> Package XSEDE-Development-config.centos.noarch 0:7-1 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

 Package                              Arch        Version   Repository                                        Size
 XSEDE-Development-config.centos      noarch      7-1       /XSEDE-Development-config.centos-7-1.noarch      2.1 k

Transaction Summary
Install  1 Package

Total size: 2.1 k
Installed size: 2.1 k
Is this ok [y/d/N]: y
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : XSEDE-Development-config.centos-7-1.noarch                                                      1/1
  Verifying  : XSEDE-Development-config.centos-7-1.noarch                                                      1/1

  XSEDE-Development-config.centos.noarch 0:7-1



[root@vm034 susan]#    rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-XSEDE-Development
[root@vm034 susan]# rpm --import GPG-KEY-user-slitzing
error: GPG-KEY-user-slitzing: import read failed(2).
[root@vm034 susan]# gpg --import GPG-KEY-user-slitzing
gpg: directory `/root/.gnupg' created
gpg: new configuration file `/root/.gnupg/gpg.conf' created
gpg: WARNING: options in `/root/.gnupg/gpg.conf' are not yet active during this run
gpg: keyring `/root/.gnupg/secring.gpg' created
gpg: keyring `/root/.gnupg/pubring.gpg' created
gpg: can't open `GPG-KEY-user-slitzing': No such file or directory
gpg: Total number processed: 0

Hi Susan,

You don't need to import a developer's PGP key. This is for special situations that don't apply here.

Because this is XSEDE repo documentation I've created a JIRA issue to improve that documentation. That issue shouldn't need to be resolved move forward with GCS.

Moving onto the next set of installation steps, I don't know whether I'm supposed to do something with this or if it's just for my information.  I ran the install and it succeeded and the files exist in /usr/local but am I supposed to copy/move them to /etc/grid-security?  If not, why are you telling me this now? 


Install package:

     $ yum install xsede-oauth-mapfile

The RPM installs files under /usr/local/share/utils/xsede_oauth_mapfile/ by default,
except as noted below. The default location for the generated mapfile is:


I agree.  That was confusing to me as well.

For a newly installed system, '/etc/grid-security/xsede-oauth-mapfile' doesn't exist yet.

 It gets generated and copied by  '/usr/local/share/utils/xsede_oauth_mapfile/bin/xsede-oauth-mapfile.sh' and it comes later.  

This should be mentioned either here or at the cron section. 

When you finish configuring and run the xsede-oauth-mapfile tool it will generate the mapfile in that location.

You're right, it's not important to say that up-front. Sorry about the confusion.

Susan, the next release will only mention the default location in the configuration step where it's useful and not in the install step when it's confusing. Thanks for the feedback!

Further down in the same document, this statement is very confusing: 


If you have an etc/xsede-oauth-mapfile-config.json from a previous install, copy it to
this install etc/ directory, otherwise create it by copying the newly RPM/TAR installed

Where am I supposed to copy it to?   'this install etc/ directory' I guess, but that is really vague.  I had to read that a few times to get it.  

Yeah, I had to guess through it as well.  It was confusing.   

Now I went through it, I also see it may not be easy to describe it clearer with short (compressed) sentences. 


in that same document, I am editing the bin/xsede-oauth-mapfile.sh to set MAP_FILE and MAP_FILE_BASE.  The default shows the file xsede-oauth-mapfile be used for the MAP_FILE.  I have an xsede-oauth-mapfile.json.  Do I specify the .json extension in the .sh file?  Or do I need to convert the .json file to a mapfile with no extension?  Or does .sh know to lop it off?  

Ok - now I see that I should run by hand what is shown in the docs being run in cron, namely the .sh file I just edited.  Here is the head of that file showing my changes: 

cat bin/xsede-oauth-mapfile.sh

# The base install directory where mapfile generation is run from
# The mapfile generated from XDCDB in the current working directory
# Other locally managed mapfile entries
# The real map_file used by services


Running the command gives this error: 


[root@vm034 xsede_oauth_mapfile]# bin/xsede-oauth-mapfile.sh
cp: cannot stat ‘xsede-oauth-mapfile’: No such file or directory

As it should because that file doesn't exist yet: 

[root@vm034 xsede_oauth_mapfile]# find .


You should not change the default value of MAP_FILE in bin/xsede-oauth-mapfile.sh. It should point to /etc/grid-security/xsede-oauth-mapfile.

The next release of xsede-oauth-mapfile will correct those instructions. You can see the new unreleased instructions here:


Where did your xsede-oauth-mapfile.json file come from? I don't see anywhere in the packages or documentation where an xsede-oauth-mapfile.json file is created or referenced.

The RPM and TAR packages provide an etc/xsede-oauth-mapfile-config-template.json configuration file that can be copied to create the etc/xsede-oauth-mapfile-config.json file. The bin/xsede-oauth-mapfile.sh program will generate what the MAP_FILE variable points to which by default is in called /etc/grid-security/xsede-oauth-mapfile. That file has no extension by default.

Maybe I'm following an out of date document.  Here are the instructions I followed: 

### Configure xsede-oauth-mapfile

If you have an etc/xsede-oauth-mapfile-config.json from a previous install, copy it to
this install etc/ directory, otherwise create it by copying the newly RPM/TAR installed

Edit etc/xsede-oauth-mapfile-config.json and set:

    "XA-AGENT": "spacct",
    "XA-RESOURCE": "<XDCDB resource name> or <other fully qualified API client hostname>",
    "MAP-RESOURCE": "<XDCDB resource name>",
    "XA-API-KEY": "<your API-KEY>"

Where "XA-RESOURCE" identifies the Client ID (XDCDB Resource Name above) that is retrieving
the mappings, and "MAP-RESOURCE" is which XSEDE resource mappings you are retrieving. On a
production resource both of these are normally the same, but in a testing or other situation
a testing server / XA-RESOURCE may be accessing mappings for a production MAP-RESOURCE.

The API-KEY may be from a previous xsede-oauth-mapfile configuration or a new one
in the previous step.

Set the permissions for the config for read-only by root to keep the API-KEY private:

    $ chmod 0600 etc/xsede-oauth-mapfile-config.json

In bin/xsede-oauth-mapfile.sh:
 * Set MAP_FILE to where you want your production mapfile
 * Set MAP_FILE_BASE to your base xsede-oauth-mapfile installation directory

If you have additional local mappings that don't come from XDCDB, place them in
/etc/grid-security/xsede-oauth-mapfile.local, or customize the LOCAL_FILE variable
in bin/xsede-oauth-mapfile.sh to point to your local mappings.

XDCDB generated mappings and local mappings will be combined into the MAP_FILE.



Those are the correct instructions that you were asked to follow. However, your previous posting was about setting the MAP_FILE variable and I decided those instructions were confusing and could be improved. So I've improved them and gave you a GitHub pointer to the improved instructions. Those improved instructons haven't been packaged and released yet.

In step 1-g in the XSEDE Globux Connect Server Install guide, I am instructed to go to the Globus page and create groups.    Should I use my xsede login or my globus login?  Would be nice to be sure.  I used my Xsede login and it hasn't choked yet. Also, do I need to create an entire new group for the Activity Admins and Watchers, or can they be subgroups of the main  Bridges-GCS Administrators group? 

Testing Plan Test 1 Step 3. says: Skip the “Create Globus Groups” step.

If you still want to do this outside the testing plan, then you should login using your XSEDE identity which should be the same identity you used to define the endpoint.

Test 4 from Test Plan Doc

It succeeds!


We had some failures and those were by getting access link URL from Gateways' mapped collection instead of its guest collection.

Test 5 from Test Plan Doc

It succeeds!


There was a failure and it was because I (user) created a share dir in '/home/jkm' instead of '/local/xsede-guestdata-storage/jkm'.

When it got correctly, other users were able to access according to thier read/write permissions. 


In 2.1.1 of the XSEDE GCS Installation Guide, step 4 says "Use the XSEDE identity mapping script that uses the XSEDE mapfile". Where is this script and what is it called?

We clarified bullet 4 to say "Use xsede-oauth-mapfile supplied mapfile lookup json configuration". The json is shown in the example command below.

In step 4.6 of the GCS Installation Guide, when I run globus-connect-server endpoint show I am getting an error. I have something wrong with the certificate.


[enstrom@xcidev3 ~]$ globus-connect-server endpoint show
[instance:139697371672192] NetworkError on request
Error contacting the GCS Manager API
SSL error connecting to 17d6b.8540.data.globus.org: CERTIFICATE_VERIFY_FAILED
This may be because the trust roots for your system don't support the
Let's Encrypt CA or another virtual host is configured to use the same
server name as the GCS Manager.

I am trying to run Test 2.

enstrom@xsede.org ux454971


The file /etc/grid-security/xsede-oauth-mapfile does contain "enstrom@xsede.org ux454971"

So, it isn't terribly clear from your comment, but I tried it too, and got what I assume is the same error you got:

Connecting to "xcidev3 XSEDE User Data Collection" fails becuase authentication fails, because 

Command Failed: Error (login)
Endpoint: xcidev3 XSEDE User Data Collection (420584ef-055f-4dc6-a55e-66ce4966c727)
Server: m-2c388b.9f559.8443.data.globus.org:443
Message: Login Failed
Details: 530-Login incorrect. : GlobusError: v=1 c=LOGIN_DENIED
530-GridFTP-Message: Identity set contains an identity from an allowed domain, but it does not map to a valid username for this connector
530-GridFTP-JSON-Result: {
	"DATA_TYPE": "result#1.0.0",
	"code": "permission_denied",
	"detail": {
		"DATA_TYPE": "invalid_user#1.0.0",
		"usernames": [
	"has_next_page": false,
	"http_response_code": 403,
	"message": "Identity set contains an identity from an allowed domain, but it does not map to a valid username for this connector"
530 End. 

This is happening because the test plan told you to configure xsede-oauth-mapfile as if you were expanse.sdsc.xsede.org, but SDSC uses 'uxDDDDDD' (where D is a digit) logins, and xcidev3 has logins that correspond to the XSEDE portal logins. This can be worked around either by creating a ux454971 user on xcidev3 for purposes of the test, or, easier, adding a line with "enstrom@xsede.org enstrom" to /etc/grid-security/xsede-oauth-mapfile (IIRC you don't need to replace the existing line, just add the new one)

For test#4, I need to use the globus file manager to access "xcidev3 Gateway Data Collection". I am getting an "Identity set contains an identity from an allowed domain, but it does not map to a valid username for this connector." error. In the xsede-oauth-mapfile I have "enstrom@xsede.org enstrom". I can access the other collection I created "xcidev3 XSEDE User Data Collection".

The collection I can't access was created with
globus-connect-server collection create \
  256c0b1d-030e-4514-a8d4-f9855e71ac38 \
  / "xcidev3 Gateway Data Collection" \
  --organization 'xcidev3 RACD Test' \
  --contact-email help@xsede.org \
  --description "This collection was created for RACD testing on server xcidev3" \
  --keywords XSEDE,"RACD Testing",xcidev3,"Science Gateways" \
  --allow-guest-collections \


The one I can access was created with
globus-connect-server collection create \
    484fcfaf-ca36-46d3-be00-5730388c0b32 \
    / "xcidev3 XSEDE User Data Collection" \
    --organization 'RACD Testing' \
    --contact-email help@xsede.org \
    --description "Collection for XSEDE users to access data on xcidev3" \
    --keywords XSEDE,xcidev3,'RACD Testing' \



Log in to post comments