Venkat,
Auditing client usage, server logging, and the analysis scripts all look great.
We should also collect all the relevant logs in the central xci-metrics.xsede.org. I propose these additions to this design:
- Daily log rotation of all logs used in auditing (or weekly if you feel that is more appropriate)
- Daily sync'ing/scp'ing of all rotated logs to the xci-metrics.xsede.org server
- Duplicate installation of the analysis tools on xci-metrics.xsede.org. If you prefer, the design can just stipulate that all the scripts will be maintained in XSEDE's svn repo, and deploying a duplicate copy on xci-metrics.xsede.org can be part of the xsede-metrics implementation design rather than the design of this activity.
This will give us central long term repository for metrics data from all our services. We'll also be able to use those analysis tools on usage logs from other servers.
Thanks,
JP
Delivery Effort Stage:
I think this activity (XCI-185) should focus on the scripts to compute the metrics. Central collection of logs should be covered in the XCI-187 design.
As a side-note, if the XCI-187 design proposes scp'ing log files around, I'll raise an objection. XSEDE should be using the standard syslog protocol for central log collection. Opening scp access to a central log server is risky.
+1 on using standard remote syslog methods for log transfer/collection. Is XCI-185 dependent on completion of XCI-187?
I recommend that XCI-185 not dependent on completing XCI-187. However, I do think the XCI-185 design should specify 1) whether usage/audit logs will be collected centrally, 2) what the proposed central service is, and 3) what protocols/methods will be used to send the logs to that location.
Implementing getting the logs to the central server could be done with each service, or could be done as a separate activity, especially if the central service isn't ready yet.
I think we have flexibility to do what makes the most sense.
I agree that setting up a central server, or using an existing one, is a separate activity. However, do you think it's pertinent to a specific service's logging and usage tracking design to mention if the logs/audit records will only exist on that server or if they will also be collected in a specific central service, and how they will be transmitted to that location? I'm just suggesting a couple of sentences stating whether the logs/audit record will be sent or collected in a central service, which one, and the protocol/method used to get them there.
About scp'ing log files around. The primary need is to copy historical log files to the new central log server. I have lots of them around and I don't want to loose them. If all our new usage collection can remote log on the fly, then we won't need to scp long term. Incidentally, there are very secure methods to scp files where a specific client credential can only scp (not ssh login) to a single directory. There are HTTPS POST alternatives that could be used.
Thanks,
JP
Hi JP,
I can specify a general statement along the following lines, but beyond this, I wonder how any specifics can be mentioned, not knowing the central repo design.
"Filtered auditd logs and 'last -f' output will be generated on a daily basis and sent to the central repository being designed as part of XCI-187. XCI-187 will specify the mode of collection of this filtered raw data."
Also, the four scripts that comprise XCI-185 don't use any syslog files.
Thanks,
venkat
Sounds good. Thanks.
Venkat,
One small detail that I think should be in the design document. The scripts should be maintained in a public source repository. If you already have one, mention where. If you don't have one yet, I recommend a new repo https://software.xsede.org/svn/xci/source/usage-analysis/.
JP