The Information Publishing Framework "IPF" (1) is a generic framework used by resource operators to gather and publish dynamic resource information in GLUE 2 serialized format. IPF can publish four types of resource information: 1) Batch system configuration and queue contents, 2) Software Modules available from the command line, 3) Network services information, and 4) Batch scheduler job events.
Campuses and other research resource operators that are not affiliated with ACCESS-CI are welcome to use IPF. All information published using IPF is available to portals and to science gateways through ACCESS Operations APIs.
This document describes how to install and configure IPF.
IPF is a Python program that gathers resource information, formats it in a GLUE2 standard format (5), and publishes it to a RabbitMQ service. IPF is configured to run one or more “workflows” each defining the steps that IPF executes to collect, format, and publish a specific type of resource information.
Each IPF workflow consists of a series of "steps" with inputs, outputs, and dependencies. The steps for each workflow are defined
in one or more JSON formatted files. Workflow JSON files can incorporate other workflow JSON files: for example,
the <resource>_services_periodic.json workflow contains one step, which is the <resource>_services.json workflow.
IPF workflows are typically defined by JSON files under $IPFETCPATH/ipf/workflow/, particularly in $IPFETCPATH/ipf/workflow/glue2.
To run a workflow execute the ipf_workflow program passing it a workflow definition file argument, like this:
$INSTALL_DIR/ipf-VERSION/ipf/bin/ipf_workflow <workflow.json>
Workflow JSON files are specified relative to $IPFETCPATH, for example:
ipf_workflow sysinfo.json
ipf_workflow glue2/<resource>_services.json
Part of workflow configuration includes generating $IPFETCPATH/ipf/init.d scripts to run a workflow periodically. These scripts are usually copied to the system /etc/init.d directory during installation.
The following workflows are recommended for the listed scenarios.
Batch System workflow (compute workflow): clusters with batch systems
Software Module workflow: clusters with command line software modules
Network Accessible Services workflow: clusters with with edge services like openssh
Batch Scheduler Job Event workflow (activity workflow): clusters that want to support live job event subscriptions
Before installing IPF operators should register their cluster resource in CiDeR (4). While IPF is capable of publishing information for resources not in CiDeR, ACCESS needs resource descriptions in CiDeR to complement the information published with IPF.
Identify a single server to run IPF -- a single IPF instance can be used to publish information for multiple resources.
To install IPF on a cluster that presents publicly as multiple resources please review this document: Publishing Software for multiple Resources from a single IPF deployment
Decide what installation method to use:
If you already have an older IPF create a backup of the /etc/ipf working configurations:
$ tar -cf ipf-etc-yyyymmdd.tar /etc/ipf
These dependencies are encoded in the RPM.
There are two recommended ways to install IPF: you can use pip install, or you can install from ACCESS RPMs on software.operations.access-ci.org.
Installing IPF from RPMs will put it in the directories /usr/lib/python-<VERSION>/site-packages/ipf, /etc/ipf, /var/ipf).
To install to an alternate location we recommend using pip.
To install using pip, you need to have the pip package installed in an appropriate version of Python (3.6+). We recommend using venv to manage Python installations. More information on venv is available at https://packaging.python.org/guides/installing-using-pip-and-virtual-environments/
Depending on how many python versions are in place on your system, "pip" may or may not refer to the python 3 version. "pip3" should always unambiguously refer to a python3 version of pip.
Once you have a Python 3.6 environment (whether venv or not), to install execute:
$ pip3 install ipf
When installing via pip: unlike in an RPM install, the files get installed relative to your Python installation (whether in a virtualenv or system Python). Notably, ipfconfigure and ipfworkflow end up in the virtualenv's bin directory, and the location IPF expects to find as its IPFETCPATH (/etc/ipf in an RPM install) is relative to the Python site-packages directory.
You can find your site-packages path for the Python you used for the pip install with:
$ python -c 'import sysconfig; print(sysconfig.get_paths()["purelib"])'
When running any IPF commands by hand in a pip install, you will need to set the environment variable IPFETCPATH. Its value should be the site-packages directory referenced above, plus "/etc/ipf". For a system Python, this might look something like "/usr/lib/python3.6/site-packages/etc/ipf".
If you have run ipfconfigure to set up your workflows, and chosen the recommended base directory, your workflow definitions will have the appropriate IPFETC_PATH defined in them.
If you wish to have the workflows run as a user other than the one that performed the pip install, you will have to do so manually.
Note(s): - The RPM will automatically create an "xdinfo" account that will own the install and that will execute the workflows via sudo.
Steps: 1) Download the latest RPM from the IPF releases at GitHub: https://github.com/access-ci-org/ipf/releases
2) Install ipf
$ yum install ./ipf-1.8.1-%REL.noarch.rpm
Note(s): - The RPM will automatically create an "xdinfo" account that will own the install and that will execute the workflows via sudo.
Steps: 1) Configure ACCESS-CI/XSEDE RPM repository trust. For a production version use Production Repo Trust Instructions. For a development/testing version use Development Repo Trust Instructions.
2) Install ipf
$ yum install ipf
The JSON files that have been written by previous runs of ipfconfigure would, in some previous versions of IPF get overwritten by subsequent runs of ipfconfigure. This is no longer the case--previous versions get backed up, not overwritten. They will not be erased by removing OR updating the package (nor will the service files copied to /etc/init.d be erased).
To perform the update to the latest RPM distribution of ipf:
To make configuration easier, an ipf_configure script is
provided in the bin directory (in /usr/bin if you installed RPMs,
otherwise in $INSTALL_DIR/ipf-VERSION/ipf/bin). This script will
generate workflow definition files and example init files.
If you intend to publish software module information via the extmodules workflow, set the environment variable MODULEPATH to point to the location of the module files before running ipfconfigure. If you intend to publish the service workflow set SERVICEPATH to point to the location of the service definition files before running ipfconfigure (more on this below). As of IPF v 1.7, ipf_configure accepts command line parameters to tell it which workflows to configure, and with which options.
An invocation of ipf_configure on a resource that has installed IPF using RPM might look like:
/usr/bin/ipfconfigure --rpm --resourcename
These options mean:
--rpm IPF was installed using RPM; this lets us know where files should be on disk
--resource_name The name of your resource. To find your resource name, go to "https://operations.access-ci.org/resources/access-allocated" to find your resource, and use the "Global Resource ID" value.
--workflows Comma delimited list of workflows to configure. Values can include: compute, activity, extmodules, services --publish Necessary if you wish to configure your workflow to publish to ACCESS's AMQP service for inclusion in Information Services
--amqp_certificate The path to the certificate to use to authenticate with ACCESS’s AMQP
--amqp_key The path to the key for your certificate
--modulepath The MODULEPATH where the modulefiles for software publishing are found. If not specified $MODULEPATH from the user environment will be used.
--lmodcachefile The location of an lmod cache file that contains exactly the set of modules you wish to publish. If you do not specify an lmodcachefile, IPF will fall back to its traditional behavior of walking the MODULEPATH.
--scheduler The batch scheduler in use on your resource. Typically slurm.
--slurmctl_log The path (including filename) of your slurmctl log file, used in the activity workflow.
Other common options:
--amqpusername If not using certificates to authenticate, use these to specify --amqppassword username and password
--pip IPF was installed using “pip install”
For a full list of command line options, please try
$ ipf_configure ----help
ipf_configure should be run as the user that will run the
information gathering workflows
You must always specify --resource_name, and you should use the "Global Resource ID" from https://operations.access-ci.org/resources/access-allocated except
The preferred way to authenticate is via an X.509 host certificate and key. You can place these files wherever you like, but the default locations are /etc/grid-security/xdinfo-hostcert.pem and /etc/grid-security/xdinfo-hostkey.pem. These files must be readable by the user that runs the information gathering workflows.
Submit an ACCESS ticket to authorize your server to publish ACCESS's RabbitMQ services. If you will authenticate via X.509, include the output of 'openssl x509 -in path/to/cert.pem -nameopt RFC2253 -subject -noout' in your ticket. If you will authenticate via username and password, state that and someone will contact you.
Note: The xdinfo user as created by the ipf rpm installation has /bin/nologin set as its shell by default. This is because for most purposes, the xdinfo user doesn't need an interactive shell. However, for some of the initial setup, it is easiest to use the xdinfo user with an interactive shell (so that certain environment variables like MODULEPATH can be discovered.) Thus, it is recommended that the configuration steps are run after something like the following:
$ sudo -u xdinfo -s /bin/bash --rcfile /etc/bashrc -i
$ echo $MODULEPATH
$ echo $SERVICEPATH
Execute:
$ ipf_configure \<command line options shown above\>
If you encounter any errors or the script does not cover your situation, Please submit an ACCESS ticket.
When the script exits, the etc/ipf/workflow/glue2/ directory will contain a set of files RESOURCENAME.json that describe the information gathering workflows you have configured and etc/ipf/init.d will contain ipf-RESOURCE_NAME_ files which are the init scripts you have configured.
As root, copy the etc/ipf/init.d/ipf-RESOURCE_NAME-* files into /etc/init.d. Your information gathering workflows can then be enabled, started, and stopped in the usual ways. You may need to perform a 'chkconfig --add' or equivalent for each service.
Since IPF 1.5, you have the ability to filter the Compute workflow by queue/partition, if you are using SLURM. This is primarily intended for sites that need to consider GPU nodes as separate resources for the purposes of computing utilization, etc.
To set parameters of Queues/Partitions for filtering, it is simplest to manually edit the workflow configuration files.
For example, on can set a params: partitions: inside the compute workflow in the ExectutionEnvironmentsStep as below:
{ "name": "ipf.glue2.slurm.ExecutionEnvironmentsStep", "params": { "partitions": "+GPU -RM -LM -XLM -DBMI" } }
This will filter such that only the nodes in the GPU partition are considered for the Execution Environment. A similar parameter, "queues" will filter by queue: { "name": "ipf.glue2.slurm.ExecutionEnvironmentsStep", "params": { "queues": "+DBMI-GPU -DBMI -RM -RM-shared -RM-small -LM -XLM -GPU -GPU-shared" } }
These parameters can be mixed and matched, and can be applied at other steps such as ComputingSharesStep and AccelerationEnvironmentsStep.
When filters are applied at the ExecutionEnvironmentsStep, that constrains the set of nodes considered for total CPU stats (TotalPhysicalCPUs, etc.)
When applied at ComputingSharesStep, it constrains which jobs are counted when calculating Used/Available CPUs or Accelerators.
When applied at the AccelerationEnvironmentsStep, it constrains the set of nodes considered for total Accelerator stats.
It is necessary for Torque to log at the correct level in order for IPF to be able to parse the messages that it uses in determining state. Furthermore, the logging level of Torque can be confusing, as it has both a loglevel and a logevents (which is a bitmask). It is important to set log_events to 255 to ensure that all types of events are logged. You can check the setting on your Torque installation by using "qmgr".
The SLURM logs must be in a directory that is local to and accessible to
the IPF installation. ipf_configure allows one to set this
location for the workflow.
Define \
The Name field refers to the GLUE2 Primary protocol name supported by the service endpoint, as seen in the table below.
Each service can have multiple capabilities, but each line is a key/value pair, so to publish multiple capabilities for a service, have a line that starts "Capability =" for each value. (see example below). The Capability field describe what services the endpoint URL supports.
Valid Capability values are listed in the table below. Please edit your service.conf files to include appropriate Capability values.
A key/value pair for SupportStatus in your service.conf file will override the default, which is the support status of your service as published in CiDeR.
Globus Connect Server (GCS) services should not be registered as network services. ACCESS discovers these automatically from Globus.
A table of valid Name, Version and Capability values:
Name Version Capability
org.globus.openssh 5.y.z login.remoteshell
login.remoteshell.gsi
Sample Service publishing file: #%Service1.0################################################################### ## ## serviceinfofiles/org.openssh-x.y.z.conf ##
Name = org.openssh
Version = 8.x
Endpoint = delta.ncsa.access-ci.org
Capability = login.shell.local-password-mfa
Capability = login.shell.pubkey
SupportStatus = production
The IPF Software Module workflow publishes information about locally installed software available through modules or Lmod. IPF tries to make intelligent inferences from the system installed modules files when it publishes software information. There are some easy ways, however, to add information to your module files that will enhance/override the information otherwise published.
The modules workflow, as of IPF 1.8 has two methods for discovering the modules you wish to publish. The recommended method, for any site using Lmod, is to point the workflow at an lmod cache file that represents exactly what you wish to publish. It will then publish every module in the spiderT table from the cache file, except modules listed in the hiddenT table.
If you are not using Lmod, or do not wish to use lmod cache files, the workflow will fall back to the traditional method of walking the MODULEPATH. The workflow then traverses your MODULEPATH and infers fields such as Name and Version from the directory structure/naming conventions of the module file layout. Specifically, IPF will traverse the tree under each directory in your MODULEPATH, and assume that, in each leaf directory, the filename equates to the module version, and the directory name equates to the module name. We are aware that this is not necessarily true in all cases, and there is an open issue in Github discussing potential future ways to better discover name and version. The issue can be found at: https://github.com/access-ci-org/ipf/issues/2 For the moment, if you have module files that are being misidentified, the easiest way forward is to include the fields (such as Name and Version) in the file as described below.
Depending on the exact workflow steps, fields such as Description may be blank, or inferred from the stdout/stderr text of the module. However, the following fields can always be explicitly added to a module file:
Name:
Version:
Description:
URL:
Category:
Keywords:
SupportStatus:
SupportContact:
Each field is a key: value pair. The IPF workflows are searching the whole text of each module file for these fields. They may be placed in a module-whatis line, or in a comment, and IPF will still read them.
The URL field should point to local user documentation.
The Category field may contain one or more comma separated fields of
science or software categories. ACCESS's official fields of science are
listed at https://operations-api.access-ci.org/wh2/allocations/v1/fos/?format=html&hierarchy=true.
Some recommended software categories are: data, compiler, language,
debugger, profiler, optimization, system, utilities. Category values are
discoverable at https://support.access-ci.org/tags.
The Keywords field may contain any other desired keywords.
The SupportStatus field should be: development, testing, or production.
The SupportContact field must contain either:
a one-liner JSON blob such as the default value:
[{"GlobalID":"support.access-ci.org","Name":"ACCESS Support","Description":"ACCESS Support","ShortName":"ACCESS","ContactURL":"https://support.access-ci.org"}]
another URL that returns a JSON document formatted exactly like the one shown above (but with different values)
ipf_configure offers the opportunity to define a default SupportContact that is published for every module
that does not define its own. By default, this value is as above:
`[{"GlobalID":"support.access-ci.org","Name":"ACCESS Support","Description":"ACCESS Support","ShortName":"ACCESS","ContactURL":"https://support.access-ci.org"}]`
Example:
#%Module1.0#####################################################################
##
## modulefiles/accessusage/1.5.2
##
module-whatis "Description: ACCESS Allocation Usage lookup tool"
module-whatis "URL: http://software.operations.access-ci.org/production/accessusage/latest/"
module-whatis "Category: System tools"
module-whatis "Keywords: information"
module-whatis "SupportStatus: testing"
However, IPF would read these just as well if they were:
#%Module1.0#####################################################################
##
## modulefiles/accessusage/1.5.2
##
#module-whatis "Description: Allocation Usage lookup tool"
# "URL: http://software.operations.access-ci.org/production/accessusage/latest/"
# Random text that is irrelevant "Category: System tools"
# module-whatis "Keywords: information"
module-whatis "SupportStatus: testing"
With this in mind, it is recommended that you add these fields to relevant module files. The decision on whether to include them as module-whatis lines (and therefore visible as such to local users) or to include them as comments is up to each operator.
1) To test the extended attribute modules workflow, execute:
# service ipf-RESOURCE_NAME-glue2-extmodules start
This init script starts a workflow that periodically gathers (every hour by default) and publishes module information containing extended attributes.
The log file is in /var/ipf/RESOURCENAMEmodules.log (or $INSTALLDIR/ipf/var/ipf/RESOURCENAME_extmodules.log) and should contain messages resembling:
2013-05-30 15:27:05,309 - ipf.engine - INFO - starting workflow extmodules
2013-05-30 15:27:05,475 - ipf.publish.AmqpStep - INFO - step-3 - publishing representation ApplicationsOgfJson of Applications expanse.sdsc.access-ci.org
2013-05-30 15:27:05,566 - ipf.publish.FileStep - INFO - step-4 - writing representation ApplicationsOgfJson of Applications expanse.sdsc.access-ci.org
2013-05-30 15:27:06,336 - ipf.engine - INFO - workflow succeeded
If any of the steps fail, that will be reported and an error message and stack trace should appear. Typical failures are caused by the environment not having specific variables or commands available.
This workflow describes your modules as a JSON document containing GLUE v2.0 Application Environment and Application Handle objects. This document is published to the ACCESS RabbitMQ service in step-3 and is written to a local file in step-4. You can examine this local file in /var/ipf/RESOURCENAMEapps.json. If you see any errors in gathering module information, please submit an ACCESS ticket.
2) To test the compute workflow, execute:
# service ipf-RESOURCE_NAME-glue2-compute start
This init script starts a workflow that periodically gathers (every minute by default) and publishes information about your compute resource. This workflow generates two types of documents. The first type describes the current state of your resource. This document doesn't contain sensitive information and ACCESS makes it available without authentication. The second type describes the queue state of your resource, contains sensitive information (user names), and will only be made available to authenticated ACCESS users.
The log file is in /var/ipf/RESOURCENAMEcompute.log (or $INSTALLDIR/ipf/var/ipf/RESOURCENAME_compute.log) and should contain messages resembling:
2013-05-30 15:50:43,590 - ipf.engine - INFO - starting workflow sge_compute
2013-05-30 15:50:45,403 - ipf.publish.AmqpStep - INFO - step-12 - publishing representation PrivateOgfJson of Private expanse.sdsc.access-ci.org
2013-05-30 15:50:45,626 - ipf.publish.FileStep - INFO - step-14 - writing representation PrivateOgfJson of Private expanse.sdsc.access-ci.org
2013-05-30 15:50:45,878 - ipf.publish.AmqpStep - INFO - step-11 - publishing representation PublicOgfJson of Public expanse.sdsc.access-ci.org
2013-05-30 15:50:46,110 - ipf.publish.FileStep - INFO - step-13 - writing representation PublicOgfJson of Public expanse.sdsc.access-ci.org
2013-05-30 15:50:46,516 - ipf.engine - INFO - workflow succeeded
Typical failures are caused by the execution environment not having specific commands available. Review the environment setup in the init script.
You can examine /var/ipf/RESOURCENAMEcompute.json (or $INSTALLDIR/ipf/var/ipf/RESOURCENAMEcompute.json) to determine if the description of your resource is accurate. You can also exampine /var/ipf/RESOURCENAMEactivities.json ( or $INSTALLDIR/ipf/var/ipf/RESOURCENAMEactivities.json) to determine if the description of the jobs being managed by your resource is correct.
3) To test the activity workflow, execute:
# service ipf-RESOURCE_NAME-glue2-activity start
This init script starts a long-running workflow that watches your scheduler log files and publishes information about jobs as they are submitted and change state.
The log file is in /var/ipf/RESOURCENAMEactivity.log (or $INSTALLDIR/ipf/var/ipf/RESOURCENAME_activity.log) and should contain messages resembling:
2013-05-30 16:04:26,030 - ipf.engine - INFO - starting workflow pbs_activity
2013-05-30 16:04:26,038 - glue2.pbs.ComputingActivityUpdateStep - INFO - step-3 - running
2013-05-30 16:04:26,038 - glue2.log - INFO - opening file 27-6930448 (/opt/pbs6.2/default/common/reporting)
2013-05-30 16:05:50,067 - glue2.log - INFO - reopening file 27-6930448 (/opt/pbs6.2/default/common/reporting)
2013-05-30 16:05:50,089 - ipf.publish.AmqpStep - INFO - step-4 - publishing representation ComputingActivityOgfJson of ComputingActivity 1226387.user.resource.access-ci.org
2013-05-30 16:05:50,493 - ipf.publish.AmqpStep - INFO - step-4 - publishing representation ComputingActivityOgfJson of ComputingActivity 1226814.user.resource.access-ci.org
2013-05-30 16:06:12,109 - glue2.log - INFO - reopening file 27-6930448 (/opt/pbs6.2/default/common/reporting)
2013-05-30 16:06:12,361 - ipf.publish.AmqpStep - INFO - step-4 - publishing representation ComputingActivityOgfJson of ComputingActivity 1226867.user.resource.access-ci.org
2013-05-30 16:06:12,380 - ipf.publish.AmqpStep - INFO - step-4 - publishing representation ComputingActivityOgfJson of ComputingActivity 1226868.user.resource.access-ci.org
2013-05-30 16:06:12,407 - ipf.publish.AmqpStep - INFO - step-4 - publishing representation ComputingActivityOgfJson of ComputingActivity 1226788.user.resource.access-ci.org
2013-05-30 16:06:12,428 - ipf.publish.AmqpStep - INFO - step-4 - publishing representation ComputingActivityOgfJson of ComputingActivity 1226865.user.resource.access-ci.org
2013-05-30 16:06:12,448 - ipf.publish.AmqpStep - INFO - step-4 - publishing representation ComputingActivityOgfJson of ComputingActivity 1226862.user.resource.access-ci.org
...
You can look at the activity information published in /var/ipf/RESOURCENAMEactivity.json (or $INSTALLDIR/ipf/var/ipf/RESOURCENAME_activity.json). This file contains the sequence of activity JSON documents published.
4) To test the Network Service (services) workflow, execute:
# service ipf-RESOURCE_NAME-glue2-services start
This init script starts a workflow that periodically gathers (every hour by default) and publishes service information from the service definition files.
The log file is in /var/ipf/RESOURCENAMEservices.log (or $INSTALLDIR/ipf/var/ipf/RESOURCENAME_services.log) and should contain messages resembling:
2013-05-30 15:27:05,309 - ipf.engine - INFO - starting workflow s
2013-05-30 15:27:05,475 - ipf.publish.AmqpStep - INFO - step-3 - publishing representation ASOgfJson of AbstractServices expanse.sdsc.access-ci.org
2013-05-30 15:27:05,566 - ipf.publish.FileStep - INFO - step-4 - writing representation ASOgfJson of AbstractServices expanse.sdsc.access-ci.org
2013-05-30 15:27:06,336 - ipf.engine - INFO - workflow succeeded
If any of the steps fail, that will be reported and an error message and stack trace should appear. Typical failures are caused by the environment not having specific variables or commands available.
This workflow describes your modules as a JSON document containing GLUE v2.0 Application Environment and Application Handle objects. This document is published to the ACCESS RabbitMQ services in step-3 and is written to a local file in step-4. You can examine this local file in /var/ipf/RESOURCENAMEapps.json. If you see any errors in gathering module information, please submit an ACCESS ticket.
The log files described above will grow over time. The logs are only needed for debugging, so they can be deleted whenever you wish. logrotate is a convenient tool for creating archival log files and limiting the amount of files kept. One note is that the ipf workflows keep the log files open while they run, so you should either use the copytruncate option or have a post-rotate statement to restart the corresponding ipf service.
There are several files associated with IPF, both pre and post configuration that help determine how workflows are configured and how and when they run.
Depending on how you installed IPF, the IPF source tree can be found in various locations. If you used RPM to install, the source code can be found at /usr/lib/python3.6/site-packages/ipf/. Note, however, that the $IPFETCPATH dir for an RPM install is /etc/ipf (that is to say, inside the canonical /etc directory)
If you used pip to install, the source code can be found in your python's sitepackages/ipf/ directory. To find the sitepackages directory for your python, you can run:
$python -c 'import sysconfig; print(sysconfig.get_paths()["purelib"])'
The pip-installed IPFETCPATH path is this directory above, with /etc/ipf/ added to the end, e.g. /usr/lib/python3.6/site-packages/etc/ipf
If you used a tarball or git checkout, you should be able to find the source code on your filesystem yourself.
IPF ships with pre-configuration workflow templates for the various schedulers. They can be found in the IPFETCPATH directory in the $IPFETCPATH/workflow/templates directory. The template files as currently constructed are: genericpublish.json genericprint.json glue2/lmod.json glue2/serviceremotepublish.json glue2/moabpbscompute.json glue2/extmodules.json glue2/extmodulesremote.json glue2/condorcompute.json glue2/abstractservice.json glue2/pbsactivity.json glue2/sgeactivity.json glue2/ipfinfopublish.json glue2/sgecompute.json glue2/pbscompute.json glue2/slurmactivity.json glue2/modules.json glue2/slurmcompute.json glue2/openstackcompute.json glue2/catalinapbs_compute.json
These workflow templates are automatically used in workflow configuration via the ipf_configure script.
After you configure your workflows, the configured workflows will be
represented by json files in $IPFETCPATH/workflow and/or
$IPFETCPATH/workflow/glue2. There will be a json file for each configured
workflow, and various workflows will have a second "periodic" workflow
that calls the other at a configurable delay. These workflow files can
be invoked by $INSTALLDIR/ipf-VERSION/ipf/bin/ipfworkflow
\
The ipfconfigure script generates an init script for each workflow you configure, and puts them in $IPFETCPATH/init.d/ Each script runs one workflow, on a periodic basis (using the periodic workflows described above). It is important to note that the scripts in $IPFETC_PATH/init.d are NOT automatically configured to be run by the operating system: this must be done manually, typically by being copied into the system /etc/init.d directory as part of the installation and configuration process documented in this file. Thus, to see what is actually being used to invoke the workflows in production, one must find the operating system location of init scripts, and see what is configured there.