Skip to main content

Graylog is restarting stuck with disk full

Graylog is restarting...
There is no Graylog web application running at the moment, please reload this page in a minute. It can take up to 1-2 minutes until all services are running properly. In case this is a permanent error, check the following:

Check if all services are running - sudo graylog-ctl status shows an overview of all running services
Check for errors in log files - Relevant services write log files here: /var/log/graylog/*/current
Ask for help - If there is no way to fix the issue ask for help:


I got this error on my Gray-log server, upon troubleshooting I found that the disk was 100% full and was unable to start elastic search mongodb and etcd while checking gray-log server status with command
#graylog-ctl status

Solution to this problem was obvious that I have to clean some disk space to get gray-log working again but what file should I delete was my next thought!

Upon googling I found that I could safely delete the old log files of elastic search to free up the space.

So I stopped gray-log server with

$sudo graylog-ctl stop

My gray-log installation path for elasticsearch logs was at

root@graylog:/var/opt/graylog/data/elasticsearch/graylog/nodes/0/indices#

Listed the files at this path

root@graylog:/var/opt/graylog/data/elasticsearch/graylog/nodes/0/indices# ls -al

drwx------ 7 graylog graylog 4096 Aug 12  2016 graylog_0
drwx------ 7 graylog graylog 4096 Aug  3  2017 graylog_1

I deleted one old log folder "graylog_0" which had consumed disk space of around 5 GB inside it.

root@graylog:/var/opt/graylog/data/elasticsearch/graylog/nodes/0/indices# rm -R graylog_0/

After deleting the log folder I restarted the graylog server

root@graylog:~# graylog-ctl start

Now I can access graylog server, all my configuration and dashboards are in place and working good. But I am getting an error for etcd (for clustering of node) of database corruption, a type of file "wal" is not accessible.

Since this is the only of my node and not a cluster configuration, I deleted the etcd folder and reconfigured the graylog server.

Delete the etcd folder here

root@graylog:~#/var/opt/graylog/data/rm -R etcd

root@graylog:~#/var/opt/graylog/data/graylog-ctl reconfigure

Now i can see the working status of all service with graylog as below

root@graylog:/var/opt/graylog/data/etcd/member# graylog-ctl status
run: elasticsearch: (pid 4437) 21s; run: log: (pid 876) 1059s
run: etcd: (pid 4272) 25s; run: log: (pid 891) 1059s
run: graylog-server: (pid 4490) 20s; run: log: (pid 857) 1059s
run: mongodb: (pid 4314) 23s; run: log: (pid 890) 1059s
run: nginx: (pid 4515) 20s; run: log: (pid 856) 1059s





Comments

Popular posts from this blog

What is Cisco Supervisor Engine?

Supervisor Engine is a module that is installed in the Cisco Chassis-based Catalyst Switches or Routers. Supervisor engine contains nearly all the same components of a fixed Cisco Switches or Routers. These Supervisor engines come in a variety of different types with different functionalities and are installed in the Switches/Router Chassis as per requirements of the network types.

Benefits of Supervisor Engines
By installing Latest Supervisor Engines in your existing investments (Switches and Routers) you can scale system performance and integrate next-generation services into your Networks.
Within a single multilayer switch chassis, two supervisor modules with integrated route processors can be used to provide hardware redundancy. If an entire supervisor module fails, the other module can pick up the pieces and continue operating the switch.
The supervisor engine contains the following integrated daughter cards that perform forwarding and routing and provide the protocols supported …

Telnet Client is Disabled Enable it to use it from this application

This is a Cisco Network Assistant (CNA) error when you want to telnet any Cisco Devices from within CNA.
Solution to remove this error is related to Windows 7 or Windows Vista. First of all you need to Add TELNET program from Control Panel, which by Default is not added in fresh installation of Windows 7.For adding and using Telnet Program in Windows 7, Click Start > Control Panel > Uninstall a Program > Turn Windows Features On and OFF > Scroll to TELNET and Select it > Press OK After Adding Telnet Program into the Windows 7 you need to Copy Telent.exe from C:\Windows\System32 folder and paste it to C:\Windows\SysWOW64\ Now you should not get the same error.

GNS3 Docker Error while creating node: Docker has returned an error: Cannot connect to host docker:80

Error while creating node: Docker has returned an error: Cannot connect to host docker:80 ssl:False [No such file or directory]

After adding docker template for Alpine Linux in gns3, you get above mentioned message when you want to use alpine linux in GNS3.

To get rid of this message you have to install Docker by following below link
curl -fsSL https://get.docker.com/ | sh

If you do not have curl installed then instal curl first with below command.apt-get install curl
After installing Docker you need to add your user name in the docker group with the following command. $ sudo usermod -aG docker your_username

Verify if the docker service is started with following command$ service docker status
If docker is not started then start with following command $ sudo service docker start
Logout from GNS3 Virtual Machines and log back. Start gns3 and use alpine linux.