Ubuntu Wiki

• Launchpad Entry: cloud-server-n-distributed-logging

• Created:

• Contributors:

• Packages affected:

Summary

Release Note

Rationale

User stories

As a UEC admin I can view all logs from every physical system include in my UEC deployment in one central place so that I can easily find out and debug why some components of my UEC deployment are failing.

Assumptions

Design

Central logging is built around rsyslog with the relp protocol to support reliable message delivery. Puppet recipes are used to configure all rsyslog daemons to connect to their relevant peers.

Implementation

See Work Items section of the blueprint whiteboard.

Test/Demo Plan

Unresolved issues

BoF agenda and discussion

UDS Natty discussion

Distributed logging for physical deployment (for UEC and more)


User story:
  As a UEC admin I can view all logs from every physical system include in my UEC deployment in one central place so that I can easily find out and debug why some components of my UEC deployment are failing.
  
 * flume:
   - java component
   - centrally configured (redundant via zookeeper) 
   - agents / forwarders / sinks / management server
   - not packaged -- low complexity to do so
   
 * central syslog: rsyslog, relp (MIR librelp), rsyslog output modules (sql Database, hdfs).
   - rsyslog
     * agent -> * can use [ udp / tcp / relp ] to forward
       * RELP Reliable Event Logging Protocol - http://www.rsyslog.com/doc/relp.html
         - supports reliable delivery end-to-end
     * "sinks" have output plugins
       * existing plugins: sql & files
     * centralized configs == config management, i.e. puppet
     * TLS enabled for TCP
     
 * Reconoiter
   - Created by OmniTI 
   - Log aggregation and introspection
   - Highly Polished
   - can be used for reporting and data presentation
   - experimental debian packaging. http://github.com/skorgu/reconnoiter-debian
   - Reasonable dependencies -- python, pcre, postgres, etc. (for the web ui)
   - Minimal dependencies for the agents.
   
   
 * real-time debugging
   - tail + grep on log "sink" -- for UEC, CLC
     - tail -f /var/log/uec/* | grep $INSTANCE_ID
     - highlighting tools with UEC knowledge -- swatch, etc.
     - Web interface for watching/querying logs
       - leverage text searching rather than grep
       - http://www.linuxjournal.com/content/centralized-logging-web-interface
       
 * scribe:
   - c++ binary
   - built on top of thrift
   
 * Swatch
 * 
 
ACTIONS:
 * Support relp in main (MIR librelp)
 * write puppet recipes to automatically configure rsyslog
 * Integrate in UEC:
   * configure central rsyslog on the ClC
   * configure aggegrator rsyslog on the CC
   * configure central logging via rsyslog on the NC, SC, Walrus
   * use syslog for all UEC components
   * write a script (grep++) to automatically track messages related to an InstanceId
 * package Reconoiter in ubuntu to use it for reporting and presentation

CategorySpec