## Chapter 6. Maintenance¶

Because of SUSE Manager's unique environment, its users are provided with abilities not available to any other Novell Customer Center customers. In addition, SUSE Manager itself also requires maintenance. This chapter discusses the procedures that should be followed to carry out administrative functions outside of standard use, as well as to apply patches to SUSE Manager.

## 6.1. Managing SUSE Manager with spacewalk-service¶

Since SUSE Manager consists of a multitude of individual components, Novell provides a command-line tool that allows you to stop, start, or retrieve status information from the various services in the appropriate order: spacewalk-service. This tool accepts all of the typical commands:

    /usr/sbin/spacewalk-service start
    /usr/sbin/spacewalk-service stop
    /usr/sbin/spacewalk-service restart
    /usr/sbin/spacewalk-service reload
    /usr/sbin/spacewalk-service enable
    /usr/sbin/spacewalk-service disable
    /usr/sbin/spacewalk-service status

Use spacewalk-service to shut down and bring up the entire SUSE Manager and retrieve status messages from all of its services at once.

## 6.2. Updating SUSE Manager¶

If any critical updates are made to SUSE Manager, they will be released in the form of a patch for SUSE Manager. Find a generic description on how to apply patches in Procedure 6.1, “Updating a SUSE Manager Server”. Note that specific instructions may apply depending on the patch.

For SUSE Manager systems that may be connected to the Internet, the best method for applying these patches is using zypper or YaST Online Update. Proper registration at Novell Customer Center is mandatory for the system to receive updates. For details, refer to Section “Installation and Setup” (, ↑Quick Start). SUSE Manager systems that may not be connected to the Internet (disconnected setup) will receive updates from an internal update server instead.

Procedure 6.1. Updating a SUSE Manager Server ¶

As soon as SUSE Manager is up and running and the database is configured, updating the server requires more than executing zypper patch (or running YaST Online Update alternatively).

The steps below describe the generic procedure, but depending on the patch, specific instructions may apply.

| Read Patch Advisory
Before applying any updates, make sure to read the patch advisory. Additional configuration steps may be required to apply certain updates, especially if they involve the database. In such cases, the advisory will contain specific and detailed information about necessary steps that may be required.

1. Log in as `root` user to the SUSE Manager server.

2. Stop the Spacewalk service:

       spacewalk-service stop

   If the SUSE Manager database is running on the same machine as the SUSE Manager server, this command also stops the SUSE Manager database instance.

3. Apply the patch using either zypper patch or YaST Online Update. For more information about zypper or YaST Online Update, refer to Section “Updating Packages on SLE” (↑Reference Guide).

4. If the patch includes an update of the database schema, proceed as follows (otherwise skip the substeps below):

   1. If the SUSE Manager database is running on the same machine as your SUSE Manager server, start the database instance with

          /etc/init.d/oracle-xe start

   2. Upgrade the database schema with

          spacewalk-schema-upgrade

5. Start the Spacewalk service:

       spacewalk-service start

| Restart of Services and Applications
Services affected by a package update are not automatically restarted after the update—you need to restart them manually to avoid failures. Also execute zypper ps to check for any applications that still use old code. Restart those applications.

## 6.3. Backing Up SUSE Manager¶

Backing up SUSE Manager can be done in several ways. Regardless of the method chosen, the associated database also needs to be backed up. For the stand-alone database, consult your organization's database administrator. For the embedded database, refer to Section 6.4, “Using DB Control” for a complete description of this process and the options available.

Here are the minimum files and directories Novell recommends backing up:

- `/rhnsat/` - embedded database only (never to be backed up while the database is running - refer to Section 6.4.2, “Backing up the Database”)

- `/etc/sysconfig/rhn/`

- `/etc/rhn/`

- `/etc/sudoers`

- `/etc/tnsnames.ora`

- `/srv/www/htdocs/pub/`

- `/var/spacewalk/packages/1` - custom RPMs

- `/root/.gnupg/`

- `/root/ssl-build/`

- `/etc/dhcp.conf`

- `/tftpboot/`

- `/var/lib/cobbler/`

- `/var/lib/rhn/kickstarts/`

- `/srv/www/cobbler`

- `/var/lib/nocpulse/`

If possible, back up `/var/spacewalk/`, as well. In case of failure, this will save lengthy download time. Since `/var/spacewalk/` (specifically `/var/spacewalk/packages/NULL/`) is primarily a duplicate of the package repository, it can be regenerated with `mgr-ncc-sync`. Novell recommends the entire `/var/spacewalk/` tree be backed up. In the case of disconnected satellites, `/var/spacewalk/`must be backed up.

Backing up only these files and directories would require reinstalling the SUSE Manager ISO RPMs and re-registering SUSE Manager. In addition, packages would need to be resynchronized using the `mgr-ncc-sync` tool. Finally, you would have to reinstall the `/root/ssl-build/\ rhn-org-httpd-ssl-key-pair-`MACHINE_NAME-VER-REL`.noarch.rpm`.

Another method would be to back up all of the files and directories mentioned above but reinstall the SUSE Manager without re-registering it. During the installation, cancel or skip the registration and SSL certificate generation sections.

The final and most comprehensive method would be to back up the entire machine. This would save time in downloading and re-installing but would require additional disk space and back up time.

Regardless of the back-up method used, when you restore SUSE Manager from a back-up, you must run the following command to schedule the recreation of search indexes the next time the rhn-search service is started:

    rcrhn-search cleanindex

## 6.4. Using DB Control¶

SUSE Manager with embedded database requires a utility for managing that database: DB Control. This command line utility allows you to do everything from make, verify, and restore backups to obtain database status and restart it when necessary. You must be the oracle user to invoke DB Control. To begin, switch to the `oracle` user:

    su - oracle

Next, issue the following command:

    db-control option

### 6.4.1. DB Control Options¶

DB Control offers many command line options. To use them, as user `oracle` insert the option and the appropriate value, if needed, after the db-control command.

Table 6.1. DB Control Options

Option | Description
--- | ---
`help` | Lists these `db-control` options with additional details.
`backup `DIRNAME`` | Backs up the database to the directory specified.
`examine `DIRNAME`` | Examines the contents of a backup directory. Returns the timestamp of backup creation and reports on its contents.
`extend` | Increase the tablespace
`gather-stats `PCT`` | Gather statistics on the database objects. `PCT` is the percentage of rows to estimate (the default is 15%).
`report` | Reports on current usage of database space.
`report-stats` | Reports on segments with stale or empty statistics.
`restore `DIRNAME`` | Restores the database from backup kept in DIRNAME. Database must be stopped for this command to run successfully.
`start` | Starts the database instance. This can also be accomplished by issuing the service oracle start command as root.
`shrink-segments` | Shrinks SUSE Manager Oracle database segments with significant amounts of free space.
`status` | Shows the current status of the database, either "running" or "offline".
`stop` | Stops the database instance. This can also be accomplished by issuing the service oracle stop command as root.
`tablesizes` | Show space report for each table
`verify `DIRNAME`` | Verifies the contents of the backup kept in DIRNAME. This command runs a checksum of each of the files kept in the backup.

Database statistics are collections of data that describe more details about the database and the objects in the database. These statistics are used by the query optimizer to choose the best execution plan for each SQL statement. Because the objects in a database can be constantly changing, statistics must be regularly updated so that they accurately describe these database objects. Statistics are maintained automatically by Oracle. However, if your database has performance issues after a significant amount of data changes, consider performing manual gathering of statistics.

After deleting large amount of data, use the `segment-shrink` feature to reclaim fragmented free space in an Oracle Database segment. The benefits of `segment-shrink` are compaction of data that leads to better cache utilization and the compacted data requires fewer blocks to be scanned in full table scans, which both lead to better performance.

### 6.4.2. Backing up the Database¶

It is recommended performing nightly backups of the embedded database and moving the resulting directory to another system via NFS, SCP, FTP, etc. Preferably, this backup system resides off-site. To conduct a backup, shut down the database and related services first by issuing the following command as root:

    /usr/sbin/spacewalk-service stop

Then switch to the `oracle` user and issue this command to initiate the backup:

    db-control backup DIRNAME

Backup files are stored in the directory specified. Note that this is a cold backup; the database must be stopped before running this command. This process takes several minutes. The first backup is a good indicator of how long subsequent backups will take.

Once the backup is complete, return to `root` user mode and restart the database and related services with the following command:

    /usr/sbin/spacewalk-service start

You should then copy that backup to another system using rsync or another file-transfer utility. It is strongly recommended scheduling the backup process automatically using cron jobs. For instance, back up the system at 3 a.m. and then copy the backup to the separate repository (partition, disk, or system) at 6 a.m.

### 6.4.3. Verifying the Backup¶

Backing up the embedded database is useful only if you can ensure the integrity of the resulting backup. DB Control provides two methods for reviewing backups, one brief, one more detailed. To conduct a quick check of the backup's timestamp and determine any missing files, issue this command as oracle:

    db-control examine DIRNAME

To conduct a more thorough review, including running a checksum of each of the files in the backup, issue this command as oracle:

    db-control verify DIRNAME

### 6.4.4. Restoring the Database¶

DB Control makes embedded database restoration a relatively simple process. As in the creation of backups, you will need to shut down the database and related services first by issuing the following commands in this order as `root`:

    /usr/sbin/spacewalk-service stop

Then switch to the oracle user and issue this command, including the directory containing the backup, to begin the restoration:

    db-control restore DIRNAME

This not only restores the embedded database but first verifies the contents of the backup directory using checksums. Once the restoration is complete, return to `root` user mode and restart the database and related services with these commands in this order:

    /usr/sbin/spacewalk-service start

## 6.5. Cloning SUSE Manager with the embedded database¶

You may limit outages caused by hardware or other failures by cloning SUSE Manager with Embedded Database in its entirety. The secondary server can be prepared for use if the primary fails. To clone SUSE Manager, perform these tasks:

1. Install SUSE Manager with embedded database on a separate machine, skipping the SSL Certificate generation step.

2. Back up the primary SUSE Manager database daily using the commands described in Section 6.4.2, “Backing up the Database”. If this is done, only changes made the day of the failure will be lost.

3. Establish a mechanism to copy the backup to the secondary SUSE Manager and keep these repositories synchronized using a file transfer program such as rsync. If you're using a SAN, copying isn't necessary.

4. Use DB Control's `restore` option to import the duplicate data.

5. If the primary SUSE Manager fails, transfer the SSL key pair RPM package in `/root/ssl-build` from the primary to the secondary SUSE Manager, and install the package. This ensures that clients can authenticate with and securely connect to the secondary server.

6. Change DNS to point to the new machine or configure your load balancer appropriately.

## 6.6. Establishing Redundant SUSE Manager server with stand-alone database¶

In keeping with the cloning option available to SUSE Manager with embedded database, you may limit outages on SUSE Manager servers with stand-alone database by preparing redundant SUSE Manager servers. Unlike cloning a SUSE Manager with embedded database, redundant SUSE Manager servers with stand-alone database may be run as active, as well as standby. This is entirely up to your network topology and is independent of the steps listed here.

To establish this redundancy, first install the primary SUSE Manager server normally, except the value specified in the Common Name field for the SSL certificate must represent your high-availability configuration, rather than the hostname of the individual server. Then:

1. Prepare the stand-alone database for failover using Oracle's recommendations for building a fault-tolerant database. Consult your database administrator.

2. Install SUSE Manager with stand-alone database on a separate machine, skipping the database configuration, database schema, SSL certificate, and bootstrap script generation steps. Include the same Novell Customer Center account and database connection information provided during the initial SUSE Manager install and register the new SUSE Manager server.

   If your original SSL certificate does not take your high-availability solution into account, you may create a new one with a more appropriate Common Name value now. In this case, you may also generate a new bootstrap script that captures this new value.

3. After installation, copy the following files from the primary to the secondary SUSE Manager:

   - `/etc/rhn/rhn.conf`

   - `/etc/tnsnames.ora`

4. Copy and install the server-side SSL certificate RPMs from the primary SUSE Manager to the secondary. Refer to the Sharing Certificates section of the Client Configuration Guide for precise instructions. Remember, the Common Name value must represent the combined SUSE Manager solution, not a single machine's hostname.

   If you generated a new SSL certificate during SUSE Manager installation that included a new Common Name value, copy the SSL certificate RPMs from the secondary to the primary server and redistribute the client-side certificate. If you also created another bootstrap script, you may use this to install the certificate on client systems.

5. If you did not create a new bootstrap script, copy the contents of `/srv/www/htdocs/pub/bootstrap/` from the primary server to the secondary. If you did generate a new one, copy that directory's contents to the primary SUSE Manager.

6. Turn off the Task Engine on the secondary server with the following command:

       rctaskomatic stop

   You may use custom scripting or other means to establish automatic start-up/failover of the Task Engine on the secondary server. Regardless, it will need to be started upon failover.

7. Share channel package data (by default located in `/var/spacewalk`) between the SUSE Manager servers over some type of networked storage device. This eliminates data replication and ensures a consistent store of data for each SUSE Manager.

8. Share cache data (by default located in `/var/cache/rhn`) between the SUSE Manager servers over some type of networked storage device. This eliminates data replication and ensures a consistent store of cached data for each server.

9. Make the various SUSE Manager servers available on your network via Common Name and a method suiting your infrastructure. Options include round-robin DNS, a network load balancer, and a reverse-proxy setup.

## 6.7. Changing the SUSE Manager Hostname¶

If you need to change the hostname or IP address of the SUSE Manager server, the `spacewalk-utils` package contains the `spacewalk-hostname-rename` script.

To use the spacewalk-hostname-rename script, you must first ensure that you know your SSL CA passphrase by performing the following command:

    openssl rsa -in path/RHN-ORG-PRIVATE-SSL-KEY

Then enter passphrase when prompted.

spacewalk-hostname-rename requires one mandatory argument, which is the IP address of the SUSE Manager server, regardless of whether the IP address will change along with the hostname or not.

The usage of spacewalk-hostname-rename is as follows:

    spacewalk-hostname-rename <ip address> [ --ssl-country=<country> \
    --ssl-state=<state> --ssl-org=<organization/company>\
    --ssl-orgunit=<department> --ssl-email=<email address> \
    --ssl-ca-password=<password>]

Changing the hostname requires to generate a new SSL certificate. If not all necessary options are passed at the command-line (as in the above example), they will be asked interactively via series of prompts.

## 6.8. Conducting SUSE Manager-Specific Tasks¶

### 6.8.1. Deleting Users¶

Because of the isolated environment in which SUSE Manager operates, SUSE Manager customers have been granted the ability to delete users. To access this functionality, click Users in the top navigation bar of the SUSE Manager website. In the resulting User List, click the name of the user to be removed. This takes you to the User Details page. Click the delete user link at the top-right corner of the page.

Figure 6.1. User Deletion

A confirmation page appears explaining that this removal is permanent. To continue, click Delete User at the bottom-right corner of the page.

The Organization Administrator role must be removed from the user's profile before deleting the user from SUSE Manager. Failing to do so causes the delete operation to fail.

The Organization Administrator role may be removed by any Organization Administrator (provided they are not the sole Organization Administrator for the organization) by clicking on the Users tab and then visiting the Details sub-tab.

Figure 6.2. User Delete Confirmation

### 6.8.2. Configuring SUSE Manager Search¶

SUSE Manager administrators may want to configure certain search options to customize search results for their own optimization requirements.

SUSE Manager search results can be customized via the `/etc/rhn/search.rhn-search.conf` file. The following list defines the search configuration and their default values in parentheses.

- search.index_work_dir : Specifies where Lucene indexes are kept (`/usr/share/rhn/search/indexes`)

- search.rpc_handlers : semi-colon separated list of classes to act as handlers for XMLRPC calls.

      (filename>index:com.redhat.satellite.search.rpc.handlers.IndexHandler,
      db:com.redhat.satellite.search.rpc.handlers.DatabaseHandler,
      admin:com.redhat.satellite.search.rpc.handlers.AdminHandler)

- search.max_hits_returned : maximum number of results which will be returned for the query (`500`)

- search.connection.driver_class : JDBC driver class to conduct database searches (`oracle.jdbc.driver.OracleDriver`)

- search.score_threshold : minimum score a result needs to be returned back as query result (`.10`)

- search.system_score_threshold : minimum score a system search result needs to be returned back as a query result (`.01`)

- search.errata_score_threshold : minimum score a patch search result needs to be returned back as a query result (`.20`)

- search.errata.advisory_score_threshold : minimum score a patch advisory result needs to be returned back as a query result (`.30`)

- search.min_ngram : minimum length of n-gram characters. Note that any change to this value requires clean-index to be run, and doc-indexes need to be modified and rebuilt) (`1`)

- search.max_ngram : maximum length of n-gram characters. Note that any change to this value requires clean-index to be run, and doc-indexes need to be modified and rebuilt) (`5`)

- search.doc.limit_results : type `true` to limit the number of results both on search.score_threshold and restrict max hits to be below search.max_hits_returned; type `false` means to return all documentation search matches (`false`)

- search.schedule.interval : input the time in miliseconds to control the interval with which the SearchServer polls the database for changes; the default is 5 minutes (`300000`).

- search.log.explain.results : used during development and debugging. If set to true, this will log additional information showing what influences the score of each result. (`false`)

## 6.9. Automating Synchronization¶

Manually synchronizing the SUSE Manager repository with Novell Customer Center can be a time-intensive task. United States business hours tend to be the peak usage time for Novell Customer Center, so synchronization at that time may be slow. For these reasons, Novell encourages you to automate synchronization at other times to better balance load and ensure quick synchronization. Continental United States business hours are roughly 8:00 AM to 9:00 PM EST (UTC -5), due to four time zones, Monday through Friday. These hours may vary seasonally by one hour. Further, Novell strongly recommends that synchronization occur randomly for best performance.

This automation can be set easily by the addition of a simple cron job. To do this, edit the crontab as root:

    crontab -e

This opens the crontab in a text editor, by default Vi. Another editor can be used by first changing the `EDITOR` variable, like so: export EDITOR=emacs.

Once opened, use the first five fields (minute, hour, day, month, and weekday) to schedule the synchronization. Remember, hours use military time. Edit the crontab to include random synchronization, like so:

    0 1 * * * perl -le 'sleep rand 9000' && mgr-ncc-sync >/dev/null \
    2>/dev/null

This particular job will run randomly between 1:00 a.m. and 3:30 a.m. system time each night and redirect `stdout` and `stderr` from cron to prevent duplicating the more easily read message from mgr-ncc-sync. Options other than `--email` can also be included. Once you exit from the editor, the modified crontab is installed immediately.

## 6.10. Implementing PAM Authentication¶

As security measures become increasingly complex, administrators must be given tools that simplify their management. For this reason, SUSE Manager supports network-based authentication systems via Pluggable Authentication Modules (PAM). PAM is a suite of libraries that helps system administrators integrate SUSE Manager with a centralized authentication mechanism, thus eliminating the need for remembering multiple passwords.

SUSE Manager supports LDAP, Kerberos, and other network-based authentication systems via PAM. To enable SUSE Manager to use PAM and your organization's authentication infrastructure, follow the steps below.

To ensure that PAM authentication functions properly, install the pam-devel package.

Set up a PAM service file (usually `/etc/pam.d/susemanager`) and have SUSE Manager use it by adding the following line to `/etc/rhn/rhn.conf`:

    pam_auth_service = susemanager

This assumes the PAM service file is named `susemanager`.

To enable a user to authenticate against PAM, select the checkbox labeled Pluggable Authentication Modules (PAM). It is positioned below the password and password confirmation fields on the Create User page.

As an example, for a Red Hat Enterprise Linux 5 i386 system, to authenticate against Kerberos one could put the following in `/etc/pam.d/susemanager`:

    #%PAM-1.0
    auth        required      pam_env.so
    auth        sufficient    pam_krb5.so no_user_check
    auth        required      pam_deny.so
    account     required      pam_krb5.so no_user_check

Please note that changing the password on the SUSE Manager website changes only the local password on the SUSE Manager, which may not be used at all if PAM is enabled for that user. In the above example, for instance, the Kerberos password will not be changed.

## 6.11. Enabling Push to Clients¶

In addition to allowing client systems to regularly poll SUSE Manager for scheduled actions, you may enable it to immediately initiate those tasks on provisioning-entitled systems. This bypasses the typical delay between scheduling an action and the client system checking in to retrieve it. This support is provided by the OSA dispatcher (`osad`).

OSA dispatcher is a service that periodically runs a query that checks the SUSE Manager server to see if there are any commands to be executed on the client. If there are, it sends a message through `jabberd` to the `osad` instances running on the clients.

SSL must be employed between SUSE Manager and its clients systems for this feature to work. If the SSL certificates are not available, the daemon on the client system fails to connect.

To take advantage of this feature, you must first configure your firewall rules to allow connections on the required port(s), as described in Section 3.2, “Additional Requirements”.

Then you must install the `osa-dispatcher` package, which can be found in the SUSE Manager software channel. Once installed, start the service on as `root` using the command:

    rcosa-dispatcher start

Finally, install the `osad` package on all client systems to receive pushed actions. The package can be found within the Tools child channel for the systems on SUSE Manager.

Do not install the `osad` package on the SUSE Manager server, as it will conflict with the `osa-dispatcher` package.

Once installed, start the service on the client systems as root using the command:

    rcosad start

Like other services, rcosa-dispatcher and rcosad accept stop, restart, and status commands, as well.

Keep in mind, this feature depends on the client system recognizing the fully qualified domain name (FQDN) of SUSE Manager. This name and not the IP address of the server must be used when configuring the YaST Online Update.

Now when you schedule actions from SUSE Manager on any of the push-enabled systems, the task will begin immediately rather than wait for the system to check in.