Skip to main content

Graylog is restarting stuck with disk full

Graylog is restarting...
There is no Graylog web application running at the moment, please reload this page in a minute. It can take up to 1-2 minutes until all services are running properly. In case this is a permanent error, check the following:

Check if all services are running - sudo graylog-ctl status shows an overview of all running services
Check for errors in log files - Relevant services write log files here: /var/log/graylog/*/current
Ask for help - If there is no way to fix the issue ask for help:


I got this error on my Gray-log server, upon troubleshooting I found that the disk was 100% full and was unable to start elastic search mongodb and etcd while checking gray-log server status with command
#graylog-ctl status

Solution to this problem was obvious that I have to clean some disk space to get gray-log working again but what file should I delete was my next thought!

Upon googling I found that I could safely delete the old log files of elastic search to free up the space.

So I stopped gray-log server with

$sudo graylog-ctl stop

My gray-log installation path for elasticsearch logs was at

root@graylog:/var/opt/graylog/data/elasticsearch/graylog/nodes/0/indices#

Listed the files at this path

root@graylog:/var/opt/graylog/data/elasticsearch/graylog/nodes/0/indices# ls -al

drwx------ 7 graylog graylog 4096 Aug 12  2016 graylog_0
drwx------ 7 graylog graylog 4096 Aug  3  2017 graylog_1

I deleted one old log folder "graylog_0" which had consumed disk space of around 5 GB inside it.

root@graylog:/var/opt/graylog/data/elasticsearch/graylog/nodes/0/indices# rm -R graylog_0/

After deleting the log folder I restarted the graylog server

root@graylog:~# graylog-ctl start

Now I can access graylog server, all my configuration and dashboards are in place and working good. But I am getting an error for etcd (for clustering of node) of database corruption, a type of file "wal" is not accessible.

Since this is the only of my node and not a cluster configuration, I deleted the etcd folder and reconfigured the graylog server.

Delete the etcd folder here

root@graylog:~#/var/opt/graylog/data/rm -R etcd

root@graylog:~#/var/opt/graylog/data/graylog-ctl reconfigure

Now i can see the working status of all service with graylog as below

root@graylog:/var/opt/graylog/data/etcd/member# graylog-ctl status
run: elasticsearch: (pid 4437) 21s; run: log: (pid 876) 1059s
run: etcd: (pid 4272) 25s; run: log: (pid 891) 1059s
run: graylog-server: (pid 4490) 20s; run: log: (pid 857) 1059s
run: mongodb: (pid 4314) 23s; run: log: (pid 890) 1059s
run: nginx: (pid 4515) 20s; run: log: (pid 856) 1059s





Comments

Popular posts from this blog

What is Cisco Supervisor Engine?

Supervisor Engine is a module that is installed in the Cisco Chassis-based Catalyst Switches or Routers. Supervisor engine contains nearly all the same components of a fixed Cisco Switches or Routers. These Supervisor engines come in a variety of different types with different functionalities and are installed in the Switches/Router Chassis as per requirements of the network types.

Benefits of Supervisor Engines
By installing Latest Supervisor Engines in your existing investments (Switches and Routers) you can scale system performance and integrate next-generation services into your Networks.
Within a single multilayer switch chassis, two supervisor modules with integrated route processors can be used to provide hardware redundancy. If an entire supervisor module fails, the other module can pick up the pieces and continue operating the switch.
The supervisor engine contains the following integrated daughter cards that perform forwarding and routing and provide the protocols supported …

GNS3 Docker Error while creating node: Docker has returned an error: Cannot connect to host docker:80

Error while creating node: Docker has returned an error: Cannot connect to host docker:80 ssl:False [No such file or directory]

After adding docker template for Alpine Linux in gns3, you get above mentioned message when you want to use alpine linux in GNS3.

To get rid of this message you have to install Docker by following below link
curl -fsSL https://get.docker.com/ | sh

If you do not have curl installed then instal curl first with below command.apt-get install curl
After installing Docker you need to add your user name in the docker group with the following command. $ sudo usermod -aG docker your_username

Verify if the docker service is started with following command$ service docker status
If docker is not started then start with following command $ sudo service docker start
Logout from GNS3 Virtual Machines and log back. Start gns3 and use alpine linux.

a file I/O error has occurred while accessing vmware converter

While converting physical Windows 7 machine to Virtual machine of infrastructure type, I got this error. The error seems it is unable to read/write source or destination datastore.

I have installed VMware-converter-en-6.2.0-8466193 on Windows 7 physical machine with option locally selected. (not at server/client option)

All of my ESXi servers are connected to the vCenter Server, so I had to use vCenter Server's IP address to send this physical machine to the virtual world.

The issue i found was with the dns resolution to the vCenter Server's hostname. Since I am not using the same dns server on the Windows 7 client machine. So I updated the host entries manually for the vCenter Server's hosname to it IP address.

After adding dns eteries to the hostfile of windows 7, I am not getting this "a file I/O error has occurred while accessing vmware converter" and the migration has started.