System administration

Client Workstations

• Installation
  • Operating system
  • Applications
  • Policy configuration
• Maxims
  • Automate
  • Avoid pre-installed systems
• Update operating system and applications
• Configure network settings (DHCP)

Servers

• Hardware
  • Internal expansion space
  • Upgrade path
  • Input and output
  • Rack mounting
  • Exclusive front access
  • Hot swapping
  • RAID storage
• Maintenance policy
• Active backup plan
• Installation at the computer centre
• Specialised software
• Remote access
• Separation of operational network from the administration network

Rack-mounted servers

Services

• Open architecture
• Distinguish products vs protocols
• Provision the service independently from the server (e.g. mail.company.gr)
• Insist on locating the service in the computing centre
  • Protected power
  • Cooling and humidity control
  • Fire suppression
  • Physical security
  • Network capacity
• Can have multiple servers for the same service
• Need mechanism for service monitoring
• Expand or organise into one server for each service

Debugging Techniques

• Understand problem
• Find the problem's cause (not the symptom)
• Fix the problem
• Use appropriate tools

• Successive elimination of possible causes (e.g. removing drivers)
• Successive refinement of possible causes (e.g. traceroute)

• ping
• traceroute / tracert
• nslookup
• strace / truss / apispy
• grep / locate / regedit
• "Use the source Luke"

The Fix-Once Principle

• Fix the problem, not the symptom
• ... for ever ...
• ... without re-inventing the wheel ...
• ... on all hosts ...
• Automation plays a key role in satisfying the above

Namespaces

• Users
• Hosts
• Network addresses
• Files
• User groups

• Name types
  • Formulaic (e.g. pc01, pc02, dspin, gvour, ptsah, gpap02)
  • Thematic (e.g. thira, naxos, paros, kriti, thasos, samos, ithaki)
  • Functional (e.g. finance, admin, guest)
  • Free; first come first served
• Longevity policy (when are entries removed?)
• Scope policy (where is this namespace to be used? - in terms of geography and services)

• Adding new names
• Changing names
• Removing names
• All the above should preferably be centralised

Disaster Planning and Recovery

• Acts of god (e.g. earthquake, fire, flood)
• Human (e.g. power loss, terrorist act, bulldozer)

Actions:

• Minimize possibility of damage (e.g. using rack-mounted components makes servers less susceptible to earthquake damage, an uninterruptible power supply (UPS) will guard against power fluctuations)
• Preparedness
  • Disaster recovery plan
  • Backup data centre

Change Management

• Use version control tools (e.g. CVS, RCS)
• Keep a detailed change log
• Run regression changing after each change
• Notify affected parties before and after the change

RCS Log Example

Manual Change Log Example

Thu Dec 04 14:20:52 EST 2003
To correct wireless dhcp problems download and install fix KB826942

Tue Dec 24 10:16:13 EET 2002
Enable time serving for Windows clients in smb.conf
 time server = True
From Windows execute: net time /set

Thu Dec  5 09:24:38 EET 2002
build kernels
run mergemaster
make installkernel KERNCONF=SPITI
cp /vol/obj/vol/src/sys/OFFICE/kernel /usr/export/boot

Wed Dec  4 15:15:08 EET 2002
In /usr/src run build and install
co /sys/dev/ata/ata-disk.c
rebuild kernel

Mon Dec  2 12:50:17 EET 2002
In /vol/ports/databases/p5-DBD-SQLite/work/DBD-SQLite-0.21
run perl getsqlite.pl 2.7.3 after manually downloading the file)
After that make, make install

Thu Nov 28 10:02:53 EET 2002
pkg_delete newt-0.50.13 dhcpconf-0.7.0
pkg_delete tcl-8.2.3
make install in /vol/ports/databases/sqlite
(also installed tcl-8.3.5)

Sun Nov 17 13:21:29 EET 2002
Run cvsup -g -L 2 -z supfile
Run cvsup -g -L 2 -z ports-supfile
Install ports/p5-DBI
Install ports/p5-DBD-SQLite

Upgrades

1. Existing services
   1. Services offered by the server
   2. Service users
   3. Software offering the services
2. Ensure each software element works in the new release
3. Plan testing procedure for each service
4. Create back-out plan
5. Choose maintenance interval
6. Notify users
7. Run test plan on existing server
8. Upgrade (with an assistant)
9. Repeat tests
10. If tests fail follow back-out plan
11. Notify customers

Upgrading TAXISnet

Centralising and Decentralising

• Reasons to move towards a direction
• Problem we want to solve
• Try to centrally manage as many elements as realistically possible
• Increasing centralisation increases the need for handling special cases
• Once a technology becomes widely adopted it also becomes a target for centralised management
• A move towards centralised management is like any other change or upgrade, but
• ... it offers us a unique chance to make a good impression
• Listen to customer concerns, but remember that IT should have the last word

• Distributed systems
• Consolidate services on less machines
• System administration
• Infrastructure decisions
• Procurement
• Support

The Helpdesk Service

• What is being supported?
• Who is being supported?
• When is support being provided?
• Where is support being provided?
• What is the expected response time?

• Registering new requests
• Assigning a ticket to each request
• Informs management and customers on the progress of each request
• Supports an issue lifecycle
• Allows searching for issues using various criteria

Example of a Problem Entry Form (Bugzilla)

Search Form

Search Results

Customer Support

1. Greeting
2. Learn about the problem
3. Reproduce the problem and agree with the customer
4. Try the problem
5. Locate problem resolution proposals
6. Select solution
7. Execute solution
8. Verify solution
9. Agree with customer that problem has been solved

Organising the Data Centre

• Physical access and security
• Rack-mounting
• Fire protection
• Natural disaster protection
• Network provisioning
• Raised floor (strong + ramp access)
• Power supply (emergency cuto-off)
• Air conditioning
• Temperature, access, and power monitoring
• Labeling
  • Machines
  • Jacks
  • Cables
• Tools
• Workbench
• Storage space
• Cart

/>
Data center

Seen outside a data centre:

ACHTUNG!!! Das machine is nicht fur gefingerpoken und mittengrabben. Ist easy schnappen der springenwerk, blowenfusen und corkenpoppen mit spitzensparken. Ist nicht fur gewerken by das dummkopfen. Das rubbernecken sightseeren keepen hands in das pockets. Relaxen und vatch das blinkenlights!!!

Network Management

• Flat (all elements are at the same level-3)
• According to the physical layout of equipment
• According to functional requirements

• An important element is the "intermediate distribution frame (IDF)
• Termination point for all floor's machines
• Label each frame with building, floor, and serial number
• Label each connection with room number and jack identifier
• Example 1/4/2/12A: building 1, floor 4, frame 2, room 12, jack A

In large buildings a main distribution frame (MDF), often located at the data centre, acts as a central hub.

Basic principles:

• Simple architecture
• Reliable implementation
• Labeling and documentation
• Use the best wiring you can afford
• UPS at the distribution frames
• Consistently locate IDFs on each floor
• Use clear demarcation points for network providers
• Utilise open protocols (IETF, IEEE)
• Adopt simple routing
• Use specialised routers (not general-purpose computers)
• Minimise number of vendors
• Avoid bleeding-edge technology

Distribution frame. On the bottom one can see the UPS units.

Email Services

• Transfer
• Delivery
• Mailing lists

• Simple architecture based on open standard protocols (SMTP, IMAP, POP)
• Proper consistent naming and management
• Service monitoring (postmaster account)
• Personal data protection policy
• Security
• User education
• Spam and virus handling

Backups

• Data is (accidentally) deleted
• The main storage medium fails
• As a corporate data repository

• Full backup (contains a complete copy of the data - also called level-0)
• Incremental backup (often multiple levels - also called level-n)

Ensure that:

• Backup copies are stored in a different, but safe location (off-site)
• Periodically the ability to recover a single file or a complete disk is verified

References

• Mark Burgess. Principles of Network and System Administration. John Wiley and Sons, 2001.
• Tina Darmohray, editor. Job Descriptions for System Administrators. Short Topics in System Administration. USENIX Association, Berkeley, CA, USA, 1997.
• Thomas A. Limoncelli and Christine Hogan. The Practice of System and Network Administration. Addison-Wesley, 2001.

Discussion Topics

1. Outline the services offered by your organisation's (or the University's) data centre
2. Provide an example where the symptom, rather than the cause is corrected
3. Describe the naming policy and the management procedures for the .gr domain

• Contents

 Last change: Tuesday, January 17, 2006 4:57 pm
 Unless otherwise expressly stated, all original material on this page created by Diomidis Spinellis is licensed under a Creative Commons Attribution-Share Alike 3.0 Greece License.