Skip to main content

Java Application server crashing with null on Sun Solaris

I have a very old Sun Solaris Server running on on SPARC platform
# cat /etc/release
                       Solaris 10 6/06 s10s_u2wos_09a SPARC
           Copyright 2006 Sun Microsystems, Inc.  All Rights Reserved.
                        Use is subject to license terms.
                             Assembled 09 June 2006

Running an old java application server on it
$ asadmin version
Unable to communicate with admin server, getting version locally.
Version = Sun Java System Application Server Platform Edition 8.2
Command version executed successfully.

We had a planned power shut at our office so we needed to shut the servers down for few hours, after the shut down window, I started the servers - servers boot successfully but while starting the  application server i was getting output "null" the server even was not writing any logs to the server.log file

asadmin> start-domain
Starting Domain domain1, please wait.
Log redirected to /xyz/SUNWappserver/domains/domain1/logs/server.log.
null
$

It was a nightmare that I could not even get clue of the problem so that i can google and resolve, additionally not being an application and linux guy it only worsen the situation and it took me days to reach the at the unexpected root cause of the problem that was crashing the application and not allowing even to write logs.

The issue with application server was that it had exhausted inode limits of the file system. And I confirmed it with this command.

$ df -oi / 
Filesystem             iused   ifree  %iused  Mounted on
/dev/md/dsk/d0       2486848 0 100%   /

To resolve the issue off course I  needed to find the unwanted files and delete them to free up inode capacity for the file system. And for that my main target was all log folders in the servers. Exploring the log folders I found millions of small log files which were created since the server was installed and running for years.
After finding and deleting millions of log files in different log folders in the server (mainly in  /xyz/SUNWappserver/domains/domain1/logs and /var/logs) I reduced the inode usage below 30% and after that the applications was started like a charm.
During this troubleshooting I learnt importance of the backup and how to backup domain :-)

Comments

Popular posts from this blog

How to find out what VMTools version will come with ESXi

Before installing an VMWare ESXi server i was little curious that what VMTools version will come after the ESXi have been installed and any VM has been created.
Looking around found a link on VMWare worth to share and keep handy while working on the deployment of VMWare especially before upgrading and existing VMWare infrastructure to newer one you would like to know the VMTools version, its support with your Operating Systems in Guest OS and/or read upon open or resolved caveats by identifying the VMTools version prior the installation and choosing the right version for your environment.

Below information has been collected from the link here

# The ESXi server mapping is only to show that the particular version of # Tools ships with that particular ESXi server build number, but the Tools # can work with a greater range of ESXi versions. # # Column 1: Tools version on NGC/VI Client # Column 2: ESXi server version.'esx/0.0' indicates that the tools version # is not yet bundled …

what is transparent bridging

The term bridging refers to a technology in which a device known as a bridge connects two or more LAN segments. Bridges are OSI Data Link layer, or Layer 2, devices that were originally designed to connect two network segments. Multiport bridges were introduced later to connect more than two network segments, these devices analyze the frames as they come in and make forwarding decisions based on information in the frames themselves.

Bridges learn the location of the network stations without any intervention from a network administrator or any manual configuration of the bridge software. This process is commonly referred to as self−learning.

When a bridge is turned on and begins to operate, it examines the MAC addresses located in the headers of frames passed through the network. As the traffic passes through the bridge, the bridge builds a table of known source addresses, assuming the port from which the bridge received the frame is the port to which the device is a sending device is…