Skip to main content

Java Application server crashing with null on Sun Solaris

I have a very old Sun Solaris Server running on on SPARC platform
# cat /etc/release
                       Solaris 10 6/06 s10s_u2wos_09a SPARC
           Copyright 2006 Sun Microsystems, Inc.  All Rights Reserved.
                        Use is subject to license terms.
                             Assembled 09 June 2006

Running an old java application server on it
$ asadmin version
Unable to communicate with admin server, getting version locally.
Version = Sun Java System Application Server Platform Edition 8.2
Command version executed successfully.

We had a planned power shut at our office so we needed to shut the servers down for few hours, after the shut down window, I started the servers - servers boot successfully but while starting the  application server i was getting output "null" the server even was not writing any logs to the server.log file

asadmin> start-domain
Starting Domain domain1, please wait.
Log redirected to /xyz/SUNWappserver/domains/domain1/logs/server.log.
null
$

It was a nightmare that I could not even get clue of the problem so that i can google and resolve, additionally not being an application and linux guy it only worsen the situation and it took me days to reach the at the unexpected root cause of the problem that was crashing the application and not allowing even to write logs.

The issue with application server was that it had exhausted inode limits of the file system. And I confirmed it with this command.

$ df -oi / 
Filesystem             iused   ifree  %iused  Mounted on
/dev/md/dsk/d0       2486848 0 100%   /

To resolve the issue off course I  needed to find the unwanted files and delete them to free up inode capacity for the file system. And for that my main target was all log folders in the servers. Exploring the log folders I found millions of small log files which were created since the server was installed and running for years.
After finding and deleting millions of log files in different log folders in the server (mainly in  /xyz/SUNWappserver/domains/domain1/logs and /var/logs) I reduced the inode usage below 30% and after that the applications was started like a charm.
During this troubleshooting I learnt importance of the backup and how to backup domain :-)

Comments

Popular posts from this blog

What is Cisco Supervisor Engine?

Supervisor Engine is a module that is installed in the Cisco Chassis-based Catalyst Switches or Routers. Supervisor engine contains nearly all the same components of a fixed Cisco Switches or Routers. These Supervisor engines come in a variety of different types with different functionalities and are installed in the Switches/Router Chassis as per requirements of the network types.

Benefits of Supervisor Engines
By installing Latest Supervisor Engines in your existing investments (Switches and Routers) you can scale system performance and integrate next-generation services into your Networks.
Within a single multilayer switch chassis, two supervisor modules with integrated route processors can be used to provide hardware redundancy. If an entire supervisor module fails, the other module can pick up the pieces and continue operating the switch.
The supervisor engine contains the following integrated daughter cards that perform forwarding and routing and provide the protocols supported …

GNS3 Docker Error while creating node: Docker has returned an error: Cannot connect to host docker:80

Error while creating node: Docker has returned an error: Cannot connect to host docker:80 ssl:False [No such file or directory]

After adding docker template for Alpine Linux in gns3, you get above mentioned message when you want to use alpine linux in GNS3.

To get rid of this message you have to install Docker by following below link
curl -fsSL https://get.docker.com/ | sh

If you do not have curl installed then instal curl first with below command.apt-get install curl
After installing Docker you need to add your user name in the docker group with the following command. $ sudo usermod -aG docker your_username

Verify if the docker service is started with following command$ service docker status
If docker is not started then start with following command $ sudo service docker start
Logout from GNS3 Virtual Machines and log back. Start gns3 and use alpine linux.

a file I/O error has occurred while accessing vmware converter

While converting physical Windows 7 machine to Virtual machine of infrastructure type, I got this error. The error seems it is unable to read/write source or destination datastore.

I have installed VMware-converter-en-6.2.0-8466193 on Windows 7 physical machine with option locally selected. (not at server/client option)

All of my ESXi servers are connected to the vCenter Server, so I had to use vCenter Server's IP address to send this physical machine to the virtual world.

The issue i found was with the dns resolution to the vCenter Server's hostname. Since I am not using the same dns server on the Windows 7 client machine. So I updated the host entries manually for the vCenter Server's hosname to it IP address.

After adding dns eteries to the hostfile of windows 7, I am not getting this "a file I/O error has occurred while accessing vmware converter" and the migration has started.