XCI-826 Testing Discussions

This is feedback from the chapter 1 of the Installation Guide’ (XSEDE Globus Connect Server v5.4)’.

I’ve never done this, so I get them by just reading it at this point.

• For the ‘Installation Guide’ (XSEDE Globus Connect Server v5.4) , there is “...using the globus-connect-server54 installation…” phrase in the very first section.  Is the “globus-connect-server54” correct?   (54 or 5.4 or none?)
•  
•  For the ‘Installation Guide’ (XSEDE Globus Connect Server v5.4) , about the 1.a, it seems suddenly appeared without an anchor point from the installation doc. It would be helpful to point it if applicable like 1.b did with “.. In Section “4.2.2. Setup the Endpoint ..”.
•  
•  For the ‘Installation Guide’ (XSEDE Globus Connect Server v5.4), about the 1.d and 1.e, I was wondering if this has an anchor point or is this after a certain step?   If applicable, it would be helpful to point to it or comment on it.

This is feedback from the chapter 2 of the ‘Installation Guide’ (XSEDE Globus Connect Server v5.4)

• At the beginning of chapter 2, right after the dotted lines, there is a phrase “Refer to the Design for XSEDE SP Deployment of Globus Connect Server 5.4 for a more in-depth…” .  When I clicked the link, it led to the “5.1.2.1 Primary access - For members of XSEDE-allocated projects“ section of another doc (Design for XSEDE SP Deployment of Globus Connect Server 5.4) .   For sometime, I got confused when I was scrolling down from the 5.1.2.1 section.    It would be better to show contents from the 5.1.2  section.  As I am reading, I was wondering about the term “collection” and it turned out that it was under the 5.1.2 section of the other doc.   I also found the term “storage gateway” which is good to understand ahead. 
•  
• At the beginning of Chapter 2, right before 2.3.1 , there is a phrase “The configuration of a storage gateway and mapped collection for Science Gateways is very similar to that from Option 1 ... “.   It made me wonder what is the ‘Option 1’.  Does it mean option at 2.2 or the previous option (guest collection) ? Probably good to update the word.

This is a question.

In the Globus Connect Server v5.4 - Integrated Testing Plan doc ( https://docs.google.com/document/d/1YaqxkS-AtIin0VYl5Pt_OkHW5LJJw1SLqbmg... ) , at the "Testing Setup" section, I see the "/local/xsede-*-storage/*" paths are listed. However, I don't have access permission to the "/local/" path.
Is this fine to proceed or do I need access permission to those paths?

In the 'https://docs.globus.org/globus-connect-server/v5/#install_section' , at 4.2.1-6 , I got stuck because I don't know what to enter for the 'Client Secrets' Description field.   I think it would be helpful to tell a user what to enter for it in our 'XSEDE Globus Connect Server v5.4 Installation Guide' doc in the '1. Creating an Endpoint' section after a.  
What should I enter for it?  Where do I get it?

I tried empty or a string, I get an error message "Failed to get client secret. Please try again."

This is just to record my procedure here. ( only the steps that are not totally clear to me if it's a step must do or an optional step. )  

• NEWLY ADDED  -----------------
•  The instruction https://software.xsede.org/development/xsede-oauth-mapfile/latest/INSTALL ( from 1.e step of the installation guide ) is a bit confusing.  I had to read multiple times to make more sense.  I hope it makes better sense for the admins.   Here is my confusion list:
  •  From "### RPM Install" step , I see that 'xsede-oauth-mapfile' rpm got installed under '/usr/local/share/utils/xsede_user_mapfile/' as stated. However it also says the default location for the mapfile is '/usr/local/etc/grid-security/xsede-oauth-mapfile' .  Where should it be?
  • From '### Configure xsede-oauth-mapfile' step,  I found the 'etc/xsede-oauth-mapfile-config-template.json' file from the 'xsede-oauth-mapfile' rpm installed in the previous step.  I got confused with the path for a while.   It would be helpful to mention that the template .json file is from the 'xsede-oauth-mapfile' rpm installed in the previous step. 
  • Question, the template json file looks as below.  Do I use the RESOURCE "bridges.psc.xsede" or other?
    •      "XA-AGENT": "spacct",
          "XA-RESOURCE": "bridges.psc.xsede",
          "XA-API-KEY": "012345abcde012345abcde"
    • As I read back to the "

      XSEDE Globus Connect Server v5.4 Integrated Testing Plan" doc , at step Test1.2 , I see "On xcidev1 pretend to be expanse.sdsc.xsede.org". Does this mean set the "XA-RESOURCE" to 'expanse.sdsc.xsede.org' ?  

  •  At step "## Request API Access Key" , it would be helpful to rephrase the explanation to " obtain one by clicking the "Generate APIKEY" link and follow the instructions at https://xsede-xdcdb-api.xsede.org/.   I had to read it and click links for a while.
  • Where should I locate the 'etc/xsede-oauth-mapfile-config.json' file?    The same path  as the template json file from the rpm installed?  I am asking it because the path ' /usr/local/etc/grid-security/xsede-oauth-mapfile' is also mentionoed in the 'RPM Installl' step and get confused. 

• DONE ANSWERED ---------------
• At https://docs.globus.org/globus-connect-server/v5/#install_section, the "4.3. Start the server" and the "4.4. Add Data Transfer Nodes to the endpoint" seem to use the same command.   After 4.3, the server is running so I didn't run it again at 4.4.

• At the "4.5. Log into the endpoint" , it looks like an optional step.  I ran the command, went to the link and enter the "Native App Authorization Code" from the web link.  The command displayed "You have successfully logged into GCS endpoint ...".  Not sure what to do after that, so record it here.

• At the "4.7. Set the endpoint as managed", I ran the command but got an error.   Thus, I requested adding the endpoint ID to the XSEDE Globus subscription via help@xsede.org according to step '1.d' in "XSEDE Globus Connect Server v5.4 Installation Guide".  (this is just testing so a little confused )

About step b at https://software.xsede.org/development/repo/repoconfig.txt.  (from the 1.e Installation Guide),  when I run the "rpm -i ..." , I get an error as below.  

$ rpm -i http://software.xsede.org/development/repo/repos/XSEDE-Development-confi...
error: can't create transaction lock on /var/lib/rpm/.rpm.lock (Permission denied) 

I thought to try 'sudo' but I am following the instruction.

Let me know how to handle this.

This is at step 2.1.1  Creating a Storage Gateway' from the GCS Install Guide.

I am getting errors for creating the Storage Gateway.

What I did:

1. I created a cmd fille 'StorageGatewayCreate.sh' as below:

• globus-connect-server storage-gateway create \
      posix "XSEDE Expanse Primary Storage Gateway" \
      --domain xsede.org \
      --authentication-timeout-mins $((60 * 24 * 11)) \
      --identity-mapping file:/usr/local/share/utils/xsede_oauth_mapfile/etc/gcs-mapfile-lookup.json \
      --restrict-paths file:/home/jkm/TestCmds/InstallGuide_2/StorageGatewayCreate/path_restrictions.json

2. Run it without sudo

• $ ./StorageGatewayCreate.sh

ERROR1:  PermissionError: [Errno 13] Permission denied: '/usr/local/share/utils/xsede_oauth_mapfile/etc/gcs-mapfile-lookup.json'

3.  Run it with sudo

• sudo ./StorageGatewayCreate.sh

ERROR2: Error: You must login first, before running this command!

4. Login

• $ globus-connect-server login localhost
  You are already logged into a GCS endpoint.

5. Run it again with sudo

• sudo ./StorageGatewayCreate.sh

ERROR: Error: You must login first, before running this command!

I am still getting error even I am already logged in.

I am done with 2.1 and I am at step 2.2  creating Guest Collections from the GCS Install Guide.

Q1:

For the command to create a storage gateway for the Guest, what should I set for “<sharing group>”?   

This question also is for the Science Gateways.

Q2:

Also, can I use the same “path-restrictions.json” file I used for creating the Primary section for the Guestt?   

This question also is for the Science Gateways.

Btw, I really have to work on the other project at NCSA at least today and tomorrow, so I will focus back on this after that. (I told Shava about it)

It seems that this Test1 is the most important and takes the longest time to go through. I think the most critical part is done so I probably can finish T1 by Monday and all the rest by Tuesday. Let me know if I guessed wrong about the T2,T3,T4 time line.

I saw Lee’s comment about not using ‘sudo’ from a certain point, and I see that step 1.d is added for that in ‘XSEDE Globus Connect Server v5.4 Installation Guide’ doc.

Does this mean I should have not used ‘sudo’ for “yum install xsede-oauth-mapfile” as well at the  https://software.xsede.org/development/xsede-oauth-mapfile/latest/INSTALL page?

Progress!

After reading comments about not using 'sudo', I went through again from the step 1.d of "XSEDE Globus Connect Server v5.4 Installation Guide". 

It worked this time for creating the Storage gateway(2.1.1) and Collection (2.1.2) for Primary access as shown below:

( notice that I didn't need the sudo. )

=========

$ globus-connect-server whoami
Username      | Name         | ID                                   | Email
------------- | ------------ | ------------------------------------ | ----------------
jkm@xsede.org | Jonghoon Kim | ecb182f4-96b1-4cd1-83b5-d6ca9255d658 | jkm@illinois.edu

==========

$ globus-connect-server storage-gateway list
Display Name                        | ID                                   | Connector | High Assurance
----------------------------------- | ------------------------------------ | --------- | --------------
Expanse XSEDE Useer Storage Gateway | 4806fe58-7f80-4ae2-8ba6-169edee49c3e | POSIX     | False

=============

$ globus-connect-server collection list
ID                                   | Display Name                       | Owner         | Collection Ty
pe | Storage Gateway ID
------------------------------------ | ---------------------------------- | ------------- | -------------
-- | ------------------------------------
d0710bd9-ad1c-4da7-bf65-0f59f020ad65 | Expanse XSEDE User Data Collection | jkm@xsede.org | mapped
   | 4806fe58-7f80-4ae2-8ba6-169edee49c3e$ globus-connect-server collection list
ID                                   | Display Name                       | Owner         | Collection Ty
pe | Storage Gateway ID
------------------------------------ | ---------------------------------- | ------------- | -------------
-- | ------------------------------------
d0710bd9-ad1c-4da7-bf65-0f59f020ad65 | Expanse XSEDE User Data Collection | jkm@xsede.org | mapped
   | 4806fe58-7f80-4ae2-8ba6-169edee49c3e

At step “### Setup to generate using cron” of https://software.xsede.org/development/xsede-oauth-mapfile/latest/INSTALL ,  it would be helpful to add a comment about what the user can expect as the result of running the command.    For example,  “If the command succeeds, the result file will be generated as “/etc/grid-security/xsede-oauth-mapfile” and it will contain peoples’ email id lines. This also means it succeeds to access XDCDB API.”   This way if it didn’t work the user will know what to do next. I was wondering if I am allowed to run it manually for a test before setting it for cron.

At step 2.2 of "XSEDE Globus Connect Server v5.4 Installation Guide" ,  I got an error by running the creating Storage-gateway for the Guest collection shown below.

CMD:

globus-connect-server storage-gateway create \
   posix "Expanse XSEDE Guest Collections Storage Gateway" \
   --domain xsede.org \
   --identity-mapping \ file:/usr/local/share/utils/xsede_oauth_mapfile/etc/gcs-mapfile-lookup.json \
   --restrict-paths file:/home/jkm/TestCmds/InstallGuide_2/2_Guest/path_restrictions.json \
   --allow-guest-collections \
   --user-allow jkm

OUTPUT:

Usage: globus-connect-server storage-gateway create posix [OPTIONS]
                                                          DISPLAY_NAME

Error: no such option: --allow-guest-collections

The spelling looks correct for the "--allow-guest-collections". 

Do you see any issue with it?

FYI, 

path_restrictions.json
{
  "DATA_TYPE": "path_restrictions#1.0.0",
  "read_write": [
    "/local/xsede-guestdata-storage/"
  ],
  "none": [
    "/tmp/"
  ]
}