Skip to main content

Java Application server crashing with null on Sun Solaris

I have a very old Sun Solaris Server running on on SPARC platform
# cat /etc/release
                       Solaris 10 6/06 s10s_u2wos_09a SPARC
           Copyright 2006 Sun Microsystems, Inc.  All Rights Reserved.
                        Use is subject to license terms.
                             Assembled 09 June 2006

Running an old java application server on it
$ asadmin version
Unable to communicate with admin server, getting version locally.
Version = Sun Java System Application Server Platform Edition 8.2
Command version executed successfully.

We had a planned power shut at our office so we needed to shut the servers down for few hours, after the shut down window, I started the servers - servers boot successfully but while starting the  application server i was getting output "null" the server even was not writing any logs to the server.log file

asadmin> start-domain
Starting Domain domain1, please wait.
Log redirected to /xyz/SUNWappserver/domains/domain1/logs/server.log.
null
$

It was a nightmare that I could not even get clue of the problem so that i can google and resolve, additionally not being an application and linux guy it only worsen the situation and it took me days to reach the at the unexpected root cause of the problem that was crashing the application and not allowing even to write logs.

The issue with application server was that it had exhausted inode limits of the file system. And I confirmed it with this command.

$ df -oi / 
Filesystem             iused   ifree  %iused  Mounted on
/dev/md/dsk/d0       2486848 0 100%   /

To resolve the issue off course I  needed to find the unwanted files and delete them to free up inode capacity for the file system. And for that my main target was all log folders in the servers. Exploring the log folders I found millions of small log files which were created since the server was installed and running for years.
After finding and deleting millions of log files in different log folders in the server (mainly in  /xyz/SUNWappserver/domains/domain1/logs and /var/logs) I reduced the inode usage below 30% and after that the applications was started like a charm.
During this troubleshooting I learnt importance of the backup and how to backup domain :-)

Comments

Post a Comment

Popular posts from this blog

How to import Putty Saved Connections to mRemoteNG

Just started using mRemoteNG and its being very cool to connect to different remote connection with different protocols e.g Window Remote Desktop, VNC to Linux, SSH, HTTP connection etc. from a single application. As new user I configured some remote desktop connection which was quite easy to figure out. But when I wanted to add SSH connections, it came in my mind to import all of the saved connections in the putty. But I couldn't figure it out how can it be done, though it was quite easy and here are the steps. Open your mRemoteNG Create a folder if you want segregation of multiple networks Create a new connection Enter the IP address of remote server under connection in Config pane Under the config pane, select protocol " SSH version 2 ".  Once you select protocol to SSH version 2 you are given option to import putty sessions, as shown in the snap below. In the above snap, I have imported CSR-AWS session from my saved sessions in Putty.
1 comment
Read more

BGP Local Preference Controlling Outbound Traffic in BGP

In BGP, Local Preference is used to control the outbound traffic path. It helps you decide which egress point (exit point) should be used when you have multiple connections to external networks, such as ISPs. Local Preference is an attribute that is local to your AS and is shared with all iBGP peers but not with eBGP neighbors. Higher Local Preference = More preferred outbound path. Example Scenario : You have two external links: ISP1 (via CE1) and ISP2 (via CE2). You want traffic to prefer ISP1 for all outbound traffic. Network Topology : CE1 (connected to ISP1): 10.0.1.1/30 CE2 (connected to ISP2): 10.0.2.1/30 iBGP Router (Internal) connected to both CE1 (10.0.1.2/30) and CE2 (10.0.2.2/30). Configuration on CE1 (Higher Local Preference) : Create a route map to set the local preference to 200 for routes learned from CE1: route-map SET_LOCAL_PREF permit 10 set local-preference 200 In the BGP configuration for CE1, apply this route map to the neighbor: router bgp 65001 ne...
Post a Comment
Read more

BGP MED: Managing Inbound Traffic with Multi-Exit Discriminator

The Multi-Exit Discriminator (MED) is used in BGP to control inbound traffic into your AS. It tells a neighboring AS which entry point into your network it should prefer when there are multiple links between your AS and the neighboring AS. The lower the MED value , the more preferred the path. MED is only honored between the same neighboring AS . Example Scenario : You are connected to ISP1 via two routers, CE1 and CE2 , and want to control which router ISP1 uses to send traffic into your AS. Network Topology : CE1 (connected to ISP1): 10.0.1.1/30 CE2 (connected to ISP1): 10.0.2.1/30 iBGP Router (Internal) connected to both CE1 (10.0.1.2/30) and CE2 (10.0.2.2/30). Configuration on CE1 (Lower MED, More Preferred) : Create a route map to set the MED to 50 for CE1: route-map SET_MED permit 10 set metric 50 Apply this route map to the neighbor in the BGP configuration for CE1: router bgp 65001 neighbor 10.0.1.1 remote-as 65000 neighbor 10.0.1.1 route-map SET_MED out Configuratio...
Post a Comment
Read more