Skip to main content

Graylog is restarting stuck with disk full

Graylog is restarting...
There is no Graylog web application running at the moment, please reload this page in a minute. It can take up to 1-2 minutes until all services are running properly. In case this is a permanent error, check the following:

Check if all services are running - sudo graylog-ctl status shows an overview of all running services
Check for errors in log files - Relevant services write log files here: /var/log/graylog/*/current
Ask for help - If there is no way to fix the issue ask for help:


I got this error on my Gray-log server, upon troubleshooting I found that the disk was 100% full and was unable to start elastic search mongodb and etcd while checking gray-log server status with command
#graylog-ctl status

Solution to this problem was obvious that I have to clean some disk space to get gray-log working again but what file should I delete was my next thought!

Upon googling I found that I could safely delete the old log files of elastic search to free up the space.

So I stopped gray-log server with

$sudo graylog-ctl stop

My gray-log installation path for elasticsearch logs was at

root@graylog:/var/opt/graylog/data/elasticsearch/graylog/nodes/0/indices#

Listed the files at this path

root@graylog:/var/opt/graylog/data/elasticsearch/graylog/nodes/0/indices# ls -al

drwx------ 7 graylog graylog 4096 Aug 12  2016 graylog_0
drwx------ 7 graylog graylog 4096 Aug  3  2017 graylog_1

I deleted one old log folder "graylog_0" which had consumed disk space of around 5 GB inside it.

root@graylog:/var/opt/graylog/data/elasticsearch/graylog/nodes/0/indices# rm -R graylog_0/

After deleting the log folder I restarted the graylog server

root@graylog:~# graylog-ctl start

Now I can access graylog server, all my configuration and dashboards are in place and working good. But I am getting an error for etcd (for clustering of node) of database corruption, a type of file "wal" is not accessible.

Since this is the only of my node and not a cluster configuration, I deleted the etcd folder and reconfigured the graylog server.

Delete the etcd folder here

root@graylog:~#/var/opt/graylog/data/rm -R etcd

root@graylog:~#/var/opt/graylog/data/graylog-ctl reconfigure

Now i can see the working status of all service with graylog as below

root@graylog:/var/opt/graylog/data/etcd/member# graylog-ctl status
run: elasticsearch: (pid 4437) 21s; run: log: (pid 876) 1059s
run: etcd: (pid 4272) 25s; run: log: (pid 891) 1059s
run: graylog-server: (pid 4490) 20s; run: log: (pid 857) 1059s
run: mongodb: (pid 4314) 23s; run: log: (pid 890) 1059s
run: nginx: (pid 4515) 20s; run: log: (pid 856) 1059s





Comments

Popular posts from this blog

a file I/O error has occurred while accessing vmware converter

While converting physical Windows 7 machine to Virtual machine of infrastructure type, I got this error. The error seems it is unable to read/write source or destination datastore.

I have installed VMware-converter-en-6.2.0-8466193 on Windows 7 physical machine with option locally selected. (not at server/client option)

All of my ESXi servers are connected to the vCenter Server, so I had to use vCenter Server's IP address to send this physical machine to the virtual world.

The issue i found was with the dns resolution to the vCenter Server's hostname. Since I am not using the same dns server on the Windows 7 client machine. So I updated the host entries manually for the vCenter Server's hosname to it IP address.

After adding dns eteries to the hostfile of windows 7, I am not getting this "a file I/O error has occurred while accessing vmware converter" and the migration has started.

Connection control operation failed for disk 'ide1:0'

I was getting this error while removing Operating System ISO image mounted on the Virtual Machine.

What worked for me, is
1. Uncheck the "Connected and Connect at power on" from Device Status.
2. Then Change the Device type from "Datastore ISO File to Client Device" Radio Button
3. and press OK to save the changes.

Note:- I was able to remove the mounted ISO only by directly logging to the ESXi at https://esxi-ip-address/ui

where it asks

"The guest operating system has locked the CD-ROM door and is probably using the CD-ROM, which can prevent the guest from recognizing media changes. If possible, eject the CD-ROM from inside the guest before disconnecting. Disconnect anyway and override the lock?"

You need to select yes to eject the CD-ROM and then remove the ISO file successfully.

GNS3 Docker Error while creating node: Docker has returned an error: Cannot connect to host docker:80

Error while creating node: Docker has returned an error: Cannot connect to host docker:80 ssl:False [No such file or directory]

After adding docker template for Alpine Linux in gns3, you get above mentioned message when you want to use alpine linux in GNS3.

To get rid of this message you have to install Docker by following below link
curl -fsSL https://get.docker.com/ | sh

If you do not have curl installed then instal curl first with below command.apt-get install curl
After installing Docker you need to add your user name in the docker group with the following command. $ sudo usermod -aG docker your_username

Verify if the docker service is started with following command$ service docker status
If docker is not started then start with following command $ sudo service docker start
Logout from GNS3 Virtual Machines and log back. Start gns3 and use alpine linux.