• Demand analysis
• Implementation phase
• New solution
• postmodel

download logs for more than a year to prepare to be imported to ES at one time. Although it is cut through Redis, it is found that the log information is accumulated in the Redis cache after the problem.

Demand analysis

Now there are download logs of VSFTPD and Pureftpd in the ES server. Different indexes are used, so they do not interfere with each other. However, if you need to count the download volume, you need to count different indexes separately, so you need to set the indexes of the two to the same data, delete all data of PureFTPD in ES, and then re -import the log into the index of VSFTPD.

Implementation stage

Switch to Logstash file directory 
 # CD /HOME/logstash/logstash-1.5.3/etc 
 Open the logstash configuration file to change the Type to the same configuration file of VSFTPD, restart the logstash service

Analysis

is such a simple thing, which leads to about 5 hours without a new log to ES for other servers, and then start to check the problem. Positioning to the Logstash Indexer error, the wrong report is 503; the amount of data input is too large. As a result, it is unable to work. The input is Redis, which means that the amount of data in Redis is too large to process.

problem solving

Without other solutions, you can only abandon the log of PureFTPD and stop the logstash of PureFTPD. Go to the Redis server and restart the Redis, and abandon these 4 hours of logs.

redis-server /etc/redis.conf

The log of part of the portal that has been imported is useless, you need to delete it

Delete the data without deleting indexes 
 #CURL -XDELETE 'http://192.168.201.145:9200/logstash -vsftp_log-2017*/_query' -d #'{ 
 # "Query": { 
 # "MATCH": {"ftptype": "Pure-ftpd"} 
 #} 
 #} ' 

 If there is a space index in it, you need to delete the corresponding index

Restart the PureFTPD Logstash, and the log does not need to be brushed from scratch, only the way to store the new log.

New solution

The log is not reconciled, but I ca n’t brush into the redis cache at one time, or it will hang up again, and you ca n’t do it directly to the ES through the cache, and you dare not try it casually. Therefore, the solution in the folding method can be solved to the redis cache. The solution is to write a shell script to brush the log file from the beginning to 10-20 lines per second to the new file, and set the input of logstash to the new file, so that the slow brush effect can be achieved.

• Create a shell script

	#!/bin/bash
	
	cd /home/logstash/logstash-1.5.3/pure_solve # Switch directory
	
	toltalCount=`cat pureftpd.log |wc -l` # How many lines are there in the statistical log
	
	startLine=1	# From the beginning to brush
	
	while (($startLine < $toltalCount))		# cyclic brush
	do
	        endLine=`expr $startLine + 20`	# step value is 20
	        sed -n "$startLine,$endLine p" pureftpd.log >> pureftpd_new.log  # Start writing
	        startLine=$endLine		# mobile steps
	        echo $startLine > sincedb	# Where did the record brush? 

 Set the Input Path of Logstash Shipper to the new /home/logstash/logstash-1.5.3/pure_solve/pureFTPD_NEW.LOG, start the logstash
	done

• execute script

post -book

Today is the last day I went to work a year ago. I brushed a blog. After get off work, I am going home to go home, oh!