Cisco Prime Infrastructure NCS 1.3 can not start DB

I ran into a problem recently where our Cisco Prime Infrastructure Manager (v1.3) wouldn't load the web interface.  I could ping the system, I could SSH into the CLI, but no response from the Web Management Interface.

After a bit if diagnosis from the CLI, using the following commands:

ncs status (the result was similar to the following)

Health Monitor is running, with an error.
initHealthMonitor(): can not start DB
Reporting Server is Stopped
Ftp Server is Stopped
Database server is stopped
Tftp Server is Stopped
Matlab Server is Stopped
NMS Server is stopped.
SAM Daemon is not running ...
DA Daemon is not running ...
Syslog Daemon is not running ...

ncs stop
ncs cleanup

ncs start (the system still wouldn't start)

Starting Network Control System...
Unable to verify hardware.
This may take a few minutes...
Health Monitor is already running.
Failure during Network Control System startup.  Check launchout.log for details.

I figured I could have done a ncs DB reinitdb but why would I want to lose all of our data.

Thanks to 42wim's blog I found the solution.  It has to do with an Oracle DB issue.

http://blog.42.be/2013/02/cisco-prime-infrastructure-ncs-12-and.html

Expert from 42wim's Post.

This was the solution (increasing the recovery db size):

[root@ncs oracle]# su - oracle
[oracle@ncs ~]$ ls
base  coracleenv  dbPasswd.pwd  oracleenv  oraInventory  templates  utils
[oracle@ncs ~]$ . oracleenv
[oracle@ncs ~]$ sqlplus '/as sysdba'
SQL*Plus: Release 11.2.0.2.0 Production
Copyright (c) 1982, 2010, Oracle.  All rights reserved.
Connected to an idle instance.

SQL> startup nomount
ORACLE instance started.

Total System Global Area 4275781632 bytes
Fixed Size                  2233336 bytes
Variable Size            2986347528 bytes
Database Buffers         1275068416 bytes
Redo Buffers               12132352 bytes
SQL> show parameter db_recovery_file_dest_size

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
db_recovery_file_dest_size           big integer 100G
SQL> alter system set db_recovery_file_dest_size=120G scope=both;

System altered.

SQL> show parameter db_recovery_file_dest_size;

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
db_recovery_file_dest_size           big integer 120G
SQL> shutdown
ORA-01507: database not mounted


ORACLE instance shut down.
SQL> quit

Still the error, but now some free space to do something

ORA-19815: WARNING: db_recovery_file_dest_size of 128849018880 bytes is 85.07% used, and has 19241095680 remaining bytes available.


Next ran ncs cleanup on the NCS CLI

ncs/admin# ncs cleanup
===================================================
Starting Cleanup:
===================================================
Removing all files in backup staging directory
Removing all Matlab core related files
Removing all older log files
Cleaning older archive logs
Cleaning database backup and all archive logs
Cleaning database
Stopping database
Starting database
Starting database clean
Completed database clean
Stopping database
===================================================
Completed Cleanup
===================================================
ncs/admin#

And rebooted the system.

Oracle DB came up normally and NCS was happily running again.