Configuring Syslog For Data Center Use.
Revision 1.0
Colin Bitterfield
August 3, 2001
Contact Me
This paper is intended to assist a data center manager in setting up a centralized syslog server. There are a variety of commercial packages that deal with security and troubleshooting; however the use of the syslog facilities is common to all UNIX systems and most network equipment. The configurations defined here are tested in the Solaris 8 environment.
What is Syslog?
Is a well-defined program that allows you to take an action; (record to a file, send to a user, or forward to another machine) when an event takes place on your server. An event is a condition defined by a program that generates a message. Messages are defined by facility and level. Each level consists of all of its messages and all of the more severe ones. A facility is considered to be a service like the kernel, email, printer, or network. The level is the severity from (0-7), (Emergency to Debug.)
Why Centralize it?
In a complex environment like a data center, the manager of the data center may not have access to "root" on the boxes or even a login. A centralized server provides to security functions in addition to the normal monitoring. 1. It provides a separate server with different logins (limited access) to monitor all levels of security. 2. It prevents a "hacker" on a compromised server from deleting all traces of the compromise. The central server provides an easier method of finding problems. It provides one place to check. It provides a simple method for archival of logs.
What do I need?
* A Server running Solaris 8. The processor of the machine is not as critical as the disk space.
* 128 MB RAM
* At least 18GBs of free disk space (Preferably Mirrored/RAID5)
Notes on the Configuration Files
The configuration files for syslog are based on M4 macros, so spaces are different then tabs.
How do I configure Syslog?
There are two parts to this.
The local /etc/syslog.conf (What gets logged)
Client
#ident"@(#)syslog.conf1.598/12/14 SMI"/* SunOS 5.0 */ # # Copyright (c) 1991-1998 by Sun Microsystems, Inc. # All rights reserved. # # syslog configuration file. # # This file is processed by m4 so be careful to quote (`') names # that match m4 reserved words.Also, within ifdef's, arguments # containing commas must be quoted. # # Local Logging *.err;kern.notice;auth.notice/dev/sysmsg *.err;kern.debug;daemon.notice;mail.crit/var/adm/messages # *.alert;kern.err;daemon.erroperator *.alertroot # *.emerg* # auth.notice/var/log/authlog mail.info/var/log/mail # # #Local Messages local0.debug/var/log/local0 local1.debug/var/log/local1 local2.debug/var/log/local2 local3.debug/var/log/local3 local4.debug/var/log/local4 local5.debug/var/log/local5 local6.debug/var/log/local6 local7.debug/var/log/local7 # # # Remote Logging *.err@loghost *.notice@loghost *.alert@loghost mail.info@loghost #Local Messages local0.debug@loghost local1.debug@loghost local2.debug@loghost local3.debug@loghost local4.debug@loghost local5.debug@loghost local6.debug@loghost local7.debug@loghost
Client Rotation script for cron.
(Backup /usr/lib/newsyslog and replace with this)
#! /bin/sh # # Copyright(c) 1997, by Sun Microsystems, Inc. # All rights reserved. # #ident @Z%newsyslog1.397/03/31 SMI # LOG=messages cd /var/adm test -f $LOG.2 && mv $LOG.2 $LOG.3 test -f $LOG.1 && mv $LOG.1 $LOG.2 test -f $LOG.0 && mv $LOG.0 $LOG.1 mv $LOG$LOG.0 cp /dev/null $LOG chmod 644$LOG # # Deal with Mail LOG=mail cd /var/log test -f $LOG.2 && mv $LOG.2 $LOG.3 test -f $LOG.1 && mv $LOG.1 $LOG.2 test -f $LOG.0 && mv $LOG.0 $LOG.1 mv $LOG$LOG.0 cp /dev/null $LOG chmod 644$LOG # LOGDIR=/var/log LOCALS="local0 local1 local2 local3 local4 local5 local6 local7" if test -d $LOGDIR then cd $LOGDIR for LOG in $LOCALS; do if test -s $LOG then test -f $LOG.6 && mv $LOG.6$LOG.7 test -f $LOG.5 && mv $LOG.5$LOG.6 test -f $LOG.4 && mv $LOG.4$LOG.5 test -f $LOG.3 && mv $LOG.3$LOG.4 test -f $LOG.2 && mv $LOG.2$LOG.3 test -f $LOG.1 && mv $LOG.1$LOG.2 test -f $LOG.0 && mv $LOG.0$LOG.1 mv $LOG$LOG.0 cp /dev/null $LOG chmod 644$LOG fi done sleep 40 fi # kill -HUP `cat /etc/syslog.pid`
Server Configuration(syslog.conf)
kern.debug/var/log/kernel mail.info/var/log/mail user.info/var/log/user lpr.notice/var/log/lpr cron.notice/var/log/cron *.err;kern.debug;daemon.notice;mail.crit/var/adm/messages # Config for other Local Syslogs #Local Messages local0.debug/var/log/local0 local1.debug/var/log/local1 local2.debug/var/log/local2 local3.debug/var/log/local3 local4.debug/var/log/local4 #Applications local5.debug/var/log/local5 # CISCO Routers local6.debug/var/log/local6 # Cisco Switchs local7.debug/var/log/local7
Server Configuration Cron Rotation Script
#! /bin/sh # # Copyright(c) 1997, by Sun Microsystems, Inc. # All rights reserved. # #ident @Z%newsyslog1.397/03/31 SMI # LOG=messages cd /var/adm test -f $LOG.2 && mv $LOG.2 $LOG.3 test -f $LOG.1 && mv $LOG.1 $LOG.2 test -f $LOG.0 && mv $LOG.0 $LOG.1 mv $LOG$LOG.0 cp /dev/null $LOG chmod 644$LOG # # LOGDIR=/var/log LOCALS="kernel user lpr mail cron local0 local1 local2 local3 local4 local5 loca l6 local7" if test -d $LOGDIR then cd $LOGDIR for LOG in $LOCALS; do if test -s $LOG then test -f $LOG.6 && mv $LOG.6$LOG.7 test -f $LOG.5 && mv $LOG.5$LOG.6 test -f $LOG.4 && mv $LOG.4$LOG.5 test -f $LOG.3 && mv $LOG.3$LOG.4 test -f $LOG.2 && mv $LOG.2$LOG.3 test -f $LOG.1 && mv $LOG.1$LOG.2 test -f $LOG.0 && mv $LOG.0$LOG.1 mv $LOG$LOG.0 cp /dev/null $LOG chmod 644$LOG fi done sleep 40 fi # kill -HUP `cat /etc/syslog.pid`
Notes on Configurations:
In the preceding examples the facility "uucp" and "news" were purposefully omitted because they were not in use in the test lab. If you use news or uucp, you would need to make appropriate changes to the configurations and scripts.
The facility local5 was setup to be the application log facility. You need to add the following to the various application start and stop scripts.
Start:
logger -p local5.info "Application STARTed normally"
Stop:
logger -p local5.info "Application STOPped normally"
Error
logger -p local5.err "Application failed to start"
Required Basic Machine Preparation:
1. Setup NTP on everything. (Accurate time is required)
2. Deativate all unnecessary services (i.e. telnet)
3. Install openssh
4. Limit access to machine. Users on client machines should not be users on this machine. Do not put on naming service like NIS, NIS+ or LDAP. Each operator should have his or her own account.
5. Harden the machine as much as possible
6. Make sure that /var/logs & /var/adm are on their own slices or change the control files.
Sample CISCO Configuration File:
logging facility local6
logging 192.168.1.10 (Ip of your syslog server)
! ntp configurable
ntp clock-period 17180345
ntp server 206.40.88.30
ntp server 192.5.5.250
Sample NTP Configuration [2]
File /etc/inet/ntp.conf
server 16.1.0.4 (This will work if you are on the Internet)
Future Considerations:
· Restricting remote messages to server only starting syslogd with the –t option.
· Setting up a standards document for Application installation and configuration.
Appendix:
Facility Definitions:[1]
|
Keyword
|
Description
|
Symbol in sys/syslog.h
|
syslog
number
|
|
kern
|
Kernel
|
LOG_KERN
|
0
|
|
user
|
User Processes
|
LOG_USER
|
1
|
|
mail
|
Electronic Mail
|
LOG_MAIL
|
2
|
|
daemon
|
Background System Processes
|
LOG_DAEMON
|
3
|
|
auth
|
Authorization
|
LOG_AUTH
|
4
|
|
syslog
|
System Logging
|
LOG_SYSLOG
|
5
|
|
lpr
|
Printing
|
LOG_LPR
|
6
|
|
news
|
Usenet News
|
LOG_NEWS
|
7
|
|
uucp
|
Unix-to-Unix Copy Program (uucp)
|
LOG_UUCP
|
8
|
|
sys9 – sys14
|
Reserved for System (defined only on Cisco router)
|
Not Defined
|
9 - 14
|
|
cron
|
Daemon to Execute Scheduled Commands
|
LOG_CRON
|
15
|
|
local0 – local7
|
For Local Use
|
LOG_LOCALn
|
16 - 23
|
Table 2: Severity Definitions [1]
|
Keyword for
syslog.conf
|
Keyword for
Cisco Router
|
Symbol in syslog.h
|
syslog
number
|
|
emerg
|
emergencies
|
LOG_EMERG
|
0
|
|
alert
|
alerts
|
LOG_ALERT
|
1
|
|
crit
|
critical
|
LOG_CRIT
|
2
|
|
err
|
errors
|
LOG_ERR
|
3
|
|
warning
|
warnings
|
LOG_WARNING
|
4
|
|
notice
|
notifications
|
LOG_NOTICE
|
5
|
|
info
|
informational
|
LOG_INFO
|
6
|
|
debug
|
debugging
|
LOG_DEBUG
|
7
|
References :
[1] "Log Consolidation with syslog" by Donald Pitts, December 23, 2000
http://www.sans.org/infosecFAQ/unix/syslog.htm
[2] Understanding and using the Network Time Protocolby Ulrich Windl, et al.
http://www.eecis.udel.edu/~ntp/ntpfaq/NTP-a-faq.htm
Analysis of Log Files:
"Automated Analysis of Cisco Log Files", Copyright © 1999, Networking Unlimited, Inc. All Rights Reserved
http://www.networkingunlimited.com/white007.html
Further Research:
(These URLs are reference by [1]). I have reviewed them that they are worthy of a read.
1] "File Formats - syslog.conf(4)." 22 January 1997. URL:
http://www.bama.ua.edu/cgi-bin/man-cgi?syslog.conf+4 (23 December 2000).
[2] Internet Engineering Task Force. "Security Issues in Network Event Logging (syslog)." 23 October 2000. URL: http://www.ietf.org/html.charters/syslog-charter.html (23 December 2000).
[3] Carnegie Mellon University. "Manage logging and other data collection mechanisms." 18 October 2000. URL: http://www.cert.org/security-improvement/practices/p092.html (23 December 2000).
[4] Orebaugh, Angela. "Securing Solaris." October 2, 2000. URL: http://www.sans.org/infosecFAQ/sec_solaris.htm(23 December 2000).
[5] Bezroukov, Nikolai. "Softpanorama University Pages: Solaris Hardening and Security." URL: http://www.softpanorama.org/Security/sos.shtml(23 December 2000).
[6] "Maintenance Commands - syslogd(1M)." 27 February 1997. URL: http://www.bama.ua.edu/cgi-bin/man-cgi?syslogd+1M (23 December 2000).
[7] "Tucows Linux Man Pages syslogd.8." 12 October 1998. URL: http://howto.tucows.com/man/man8/syslogd.8.html (23 December 2000).
[8] Carnegie Mellon University. "Configure firewall logging and alert mechanisms." CERT Security Improvement Modules. 2 August 1999. URL: http://www.cert.org/security-improvement/practices/p059.html (23 December 2000).
[9] Adiscon. "EventReporter." URL: http://www.eventreporter.com/en/Product/Integrate-NT-Event-Log-into-Unix-Syslogd.asp (23 December 2000).
[10] Kiwi Enterprises. "Kiwi’s Software." 23 December 2000. URL: http://www.kiwi-enterprises.com/products.htm (23 December 2000).
[11] Cisco Systems. "Logging." Improving Security on Cisco Routers. URL:http://www.cisco.com/warp/public/707/21.html#logging (23 December 2000).
[12] Cisco Systems. "Troubleshooting Commands." 19 December 2000. URL: http://www.cisco.com/univercd/cc/td/doc/product/software/ios120/12cgcr/fun_r/frprt3/frtroubl.htm (23 December 2000).
[13] Cisco Systems. "Managing the System." URL: http://www.cisco.com/univercd/cc/td/doc/product/software/ios111/mods/1mod/1cbook/1csysmgt.htm (23 December 2000).
[14] Cisco Systems. "Logging and Counter Caveats." Characterizing and Tracing Packet Floods Using Cisco Routers. URL: http://www.cisco.com/warp/public/707/22.html#3f (23 December 2000).
[15] Carnegie Mellon University. "Using newsyslog to rotate files containing logging messages on systems running Solaris 2.x." 2 March 2000. URL: http://www.cert.org/security-improvement/implementations/i041.09.html (23 December 2000).
[16] Carnegie Mellon University. "Understanding system log files on a Solaris 2.x operating system." 2 March 2000. URL: http://www.cert.org/security-improvement/implementations/i041.12.html (23 December 2000).
[17] Reed, Darren. "Nsyslogd." URL:http://coombs.anu.edu.au/~avalon/nsyslog.html (23 December 2000).
[18] Scheidler, Balázs. "syslog-ng." 6 November 2000. URL: http://www.balabit.hu/products/syslog-ng (23 December 2000).
[19] "Core FreeSoft." URL: http://www.core-sdi.com/english/freesoft.html(23 December 2000).
[20] Conover, Matt. "Index of /files/SRS." URL: http://www.w00w00.org/files/SRS/ (23 December 2000).
[21] Stokely, Celeste. "Celeste’s Tutorial On Solaris 2.x Modems & Terminals." 20 May 2000. URL: http://www.stokely.com/unix.serial.port.resources/modem.html (23 December 2000).
[22] Spitzner, Lance. "Watching Your Logs." 19 July 2000. URL: http://www.enteract.com/~lspitz/swatch.html (23 December 2000).
[23] Rowland, Craig. "Psionic Logcheck Version 1.1.1." 10 May 2000. URL: http://www.psionic.com/abacus/logcheck/(23 December 2000).
|