Sep 162015
 

Stolen from Reddit, source

1) Set up a KVM hypervisor.

2) Inside of that KVM hypervisor, install a Spacewalk server. Use CentOS 6 as the distro for all work below. (For bonus points, set up errata importation on the CentOS channels, so you can properly see security update advisory information.)

3) Create a VM to provide named and dhcpd service to your entire environment. Set up the dhcp daemon to use the Spacewalk server as the pxeboot machine (thus allowing you to use Cobbler to do unattended OS installs). Make sure that every forward zone you create has a reverse zone associated with it. Use something like “internal.virtnet” (but not “.local”) as your internal DNS zone.

4) Use that Spacewalk server to automatically (without touching it) install a new pair of OS instances, with which you will then create a Master/Master pair of LDAP servers. Make sure they register with the Spacewalk server. Do not allow anonymous bind, do not use unencrypted LDAP.

5) Reconfigure all 3 servers to use LDAP authentication.

konfucjusz

6) Create two new VMs, again unattendedly, which will then be Postgresql VMs. Use pgpool-II to set up master/master replication between them. Export the database from your Spacewalk server and import it into the new pgsql cluster. Reconfigure your Spacewalk instance to run off of that server.

7) Set up a Puppet Master. Plug it into the Spacewalk server for identifying the inventory it will need to work with. (Cheat and use ansible for deployment purposes, again plugging into the Spacewalk server.)

8) Deploy another VM. Install iscsitgt and nfs-kernel-server on it. Export a LUN and an NFS share.

9) Deploy another VM. Install bakula on it, using the postgresql cluster to store its database. Register each machine on it, storing to flatfile. Store the bakula VM’s image on the iscsi LUN, and every other machine on the NFS share.

10) Deploy two more VMs. These will have httpd (Apache2) on them. Leave essentially default for now.

11) Deploy two more VMs. These will have tomcat on them. Use JBoss Cache to replicate the session caches between them. Use the httpd servers as the frontends for this. The application you will run is JBoss Wiki.

12) You guessed right, deploy another VM. This will do iptables-based NAT/round-robin loadbalancing between the two httpd servers.

13) Deploy another VM. On this VM, install postfix. Set it up to use a gmail account to allow you to have it send emails, and receive messages only from your internal network.

14) Deploy another VM. On this VM, set up a Nagios server. Have it use snmp to monitor the communication state of every relevant service involved above. This means doing a “is the right port open” check, and a “I got the right kind of response” check and “We still have filesystem space free” check.

15) Deploy another VM. On this VM, set up a syslog daemon to listen to every other server’s input. Reconfigure each other server to send their logging output to various files on the syslog server. (For extra credit, set up logstash or kibana or greylog to parse those logs.)

16) Document every last step you did in getting to this point in your brand new Wiki.

17) Now go back and create Puppet Manifests to ensure that every last one of these machines is authenticating to the LDAP servers, registered to the Spacewalk server, and backed up by the bakula server.

18) Now go back, reference your documents, and set up a Puppet Razor profile that hooks into each of these things to allow you to recreate, from scratch, each individual server.

19) Destroy every secondary machine you’ve created and use the above profile to recreate them, joining them to the clusters as needed.

20) Bonus exercise: create three more VMs. A CentOS 5, 6, and 7 machine. On each of these machines, set them up to allow you to create custom RPMs and import them into the Spacewalk server instance. Ensure your Puppet configurations work for all three and produce like-for-like behaviors.

Do these things and you will be fully exposed to every aspect of Linux Enterprise systems administration. Do them well and you will have the technical expertise required to seek “Senior” roles. If you go whole-hog crash-course full-time it with no other means of income, I would expect it would take between 3 and 6 months to go from “I think I’m good with computers” to achieving all of these — assuming you’re not afraid of IRC and google (and have neither friends nor family …).

Jul 172015
 

Intro

It took me a while to figure out optimal configuration for the tape library with two streamers used with Bacula backup software.

Exact model of tape library in use is Quantum Scalar i40 with two LTO5 streamers. It is hooked up directly to the main NFS server (so heavy backup traffic goes via localhost only) – server that runs bacula-sd and bacula-fd services only. Bacula director runs on separate, dedicated backup server.

Currently there are around 20 other servers connected to this system as clients, with various daily Incremental, weekly Differential and monthly Full backup level jobs scheduled for execution.

Some additional info about this setup in previous post – click here. Config files below:

insta-24

 


Relevant config files from Backup server


/etc/bacula/bacula-dir.conf

Director {  
  Name = prod-backup-dir
  QueryFile = "/etc/bacula/scripts/query.sql"
  WorkingDirectory = "/var/lib/bacula"
  PidDirectory = "/var/run/bacula"
  Password = "xxxxx"
  Messages = Daemon
  DirAddress = prod-backup.domain.com
  Maximum Concurrent Jobs = 20
}
@/etc/bacula/JobDefs/JobDefs.conf
@|"sh -c 'cat /etc/bacula/Job/*'"
@|"sh -c 'cat /etc/bacula/FileSet/*'"
@|"sh -c 'cat /etc/bacula/Schedule/*'"
@|"sh -c 'cat /etc/bacula/Clients-enabled/*'"
@|"sh -c 'cat /etc/bacula/Storage/*'"
@|"sh -c 'cat /etc/bacula/Pool/*'"
Catalog {
  Name = MyCatalog
  dbaddress = prod-db.domain.com ;
  dbname = "bacula"; dbuser = "bacula"; dbpassword = "xxxxx"
}
Messages {
  Name = Standard
  mailcommand = "/usr/lib/bacula/bsmtp -h prod-mailhub.domain.com -f \"\(Bacula\) \<%r\>\" -s \"Bacula: %t %e of %c %l\" %r"
  operatorcommand = "/usr/lib/bacula/bsmtp -h prod-mailhub.domain.com  -f \"\(Bacula\) \<%r\>\" -s \"Bacula: Intervention needed for %j\" %r"
  mail = [email protected] = all, !skipped            
  operator = [email protected] = mount
  console = all, !skipped, !saved
  append = "/var/lib/bacula/log" = all, !skipped
  catalog = all
}
Messages {
  Name = Daemon
  mailcommand = "/usr/lib/bacula/bsmtp -h localhost -f \"\(Bacula\) \<%r\>\" -s \"Bacula client %c job %n exit code %e  \" %r"
  mail = [email protected] = all, !skipped            
  console = all, !skipped, !saved
  append = "/var/lib/bacula/log" = all, !skipped
}
Console {
  Name = prod-backup-mon
  Password = "xxxxxxxxxxx"
  CommandACL = status, .status
}

Example job definition /etc/bacula/Job/Studies2010-1.conf

#----------------------------------
Job {
  Name = Studies2010-1
  Type = Backup
  Client = nfs-prod-fd
  Schedule = MonthlyCycle
  Messages = Daemon
  FileSet = Studies2010-1
  Level = Full
  Pool = lto5-pool
  Priority = 12
  Max Run Time = 1555200 # default limit is 6 days, 518400sec. bumped 3x just in case
  Spool Data = yes
  Spool Attributes = yes

}
#----------------------------------

Example fileset, /etc/bacula/FileSet/Studies2010-1.conf

#-------------------------------------------
FileSet {
  Name = "Studies2010-1"
  Include {
    Options {
      signature = MD5
      compression=GZIP5
      noatime=yes
      aclsupport = yes
      wilddir = "/export/studies/201007*"
      wilddir = "/export/studies/201008*"
     	    }
    Options {
      RegexDir = ".*"
      exclude = yes
	    }
    File = "/export/studies"
          }
}

Example Schedule, /etc/bacula/Schedule/MonthlyCycle3.conf

Schedule {
  Name = MonthlyCycle3
  Run = Level=Full Pool=lto5-pool 3rd fri at 23:30
}

Tape library, storage definition:

Storage {
  Name = TapeLibrary
  Address = prod-tapelib.comain.com
  SDPort = 9103
  Password = "xxxxxx"
  Device = QuantumScalar-I40
  Media Type = LTO-5
  Autochanger = yes
  Maximum Concurrent Jobs = 4
}

Pool of tapes defined here:

Pool {
  Name = lto5-pool
  Pool Type = Backup
  Volume Retention = 6 months
  Recycle = yes
  AutoPrune = yes
  Recycle = yes
  Label Format = LTO5
  Storage = TapeLibrary
}

 

Relevant config files from Tape Library server

 

Note that I spool data before saving to the tape – this prevents tape “shoe shine” during Incremental/Differential backups.

 

  
Storage { 
  Name = TapeLibrary
  WorkingDirectory = "/var/spool/bacula"
  Pid Directory = "/var/run"
}
Autochanger {
  Name = QuantumScalar-I40
  Device = Drive0
  Device = Drive1
  Changer Device = /dev/changer
  Changer Command = "/usr/libexec/bacula/mtx-changer %c %o %S %a %d"
}
Device {
  Name = Drive0
  Drive Index = 0
  Media Type = LTO-5
  Archive Device = /dev/nst0
  AutomaticMount = yes
  AlwaysOpen = yes
  RemovableMedia = yes
  RandomAccess = no
  AutoChanger = yes
  Alert Command = "sh -c 'smartctl -H -l error %c'"  
  Maximum Changer Wait = 600
  Maximum Rewind Wait = 600
  Maximum Open Wait = 600
  Spool Directory = /var/spool/bacula/Spool
  Maximum Spool Size = 45G
  Maximum Concurrent Jobs = 2
}
Device {
  Name = Drive1
  Drive Index = 1
  Media Type = LTO-5
  Archive Device = /dev/nst1
  AutomaticMount = yes
  AlwaysOpen = yes
  RemovableMedia = yes
  RandomAccess = no
  AutoChanger = yes
  Alert Command = "sh -c 'smartctl -H -l error %c'"
  Maximum Changer Wait = 600
  Maximum Rewind Wait = 600
  Maximum Open Wait = 600
  Spool Directory = /var/spool/bacula/Spool
  Maximum Spool Size = 45G
  Maximum Concurrent Jobs = 2
     }
Messages {
  Name = Standard
  director = prod-backup-dir = all
}
Director {
  Name = prod-backup-dir
  Password = "xxxxxxxx"
}
Director {
  Name = prod-backup-mon
  Password = "xxxxxxxxxx"
  Monitor = yes
}

Thoughts

Implementing Bacula driven backup solution requires some time and effort – but what you get in the end is sophisticated, enterprise grade backup system, capable of backing up TBs of data in organised and efficient manner.

Used in conjunction with Monitoring system it offers fully automated backup solution, with minimal operator effort required. Routine tasks boil down to: