分类导航

Linux 发布时间：2022-04-01 发布网站：大佬教程 code.js-code.com

大佬教程收集整理的这篇文章主要介绍了linux – 从bash脚本启动时Logstash shutdown停止，大佬教程大佬觉得挺不错的，现在分享给大家，也给大家做个参考。

概述

我写了一个bash脚本,它在指定的文件夹中找到CSV文件,并使用正确的配置文件将它们管道到logstash中.但是,当运行此脚本时,我遇到以下错误,说关闭进程停止,导致无限循环,直到我用ctrl c手动停止它： [2018-03-22T08:59:53,833][INFO ][logstash.runner ] StarTing Logstash {"logstash.versi

我写了一个bash脚本,它在指定的文件夹中找到CSV文件,并使用正确的配置文件将它们管道到logstash中.但是,当运行此脚本时,我遇到以下错误,说关闭进程停止,导致无限循环,直到我用ctrl c手动停止它：

[2018-03-22T08:59:53,833][INFO ][logstash.runner          ] StarTing Logstash {"logstash.version"=>"6.2.3"}
[2018-03-22T08:59:54,211][INFO ][logstash.agent           ] successfully started Logstash API endpoint {:port=>9600}
[2018-03-22T08:59:57,970][INFO ][logstash.pipeline        ] StarTing pipeline {:pipeline_id=>"main","pipeline.workers"=>2,"pipeline.batch.size"=>125,"pipeline.batch.delay"=>50}
[2018-03-22T08:59:58,116][INFO ][logstash.pipeline        ] Pipeline started succesfully {:pipeline_id=>"main",:thread=>"#<Thread:0xf6851b3 run>"}
[2018-03-22T08:59:58,246][INFO ][logstash.agent           ] Pipelines running {:count=>1,:pipelines=>["main"]}
[2018-03-22T08:59:58,976][INFO ][logstash.outputs.file    ] opening file {:path=>"/home/kevin/otrs_customer_user"}
[2018-03-22T09:00:06,471][WARN ][logstash.shutdownwatcher ] {"inflight_count"=>0,"stalling_thread_info"=>{["LogStash::Filters::CSV",{"separator"=>";","columns"=>["IOT","OID","SUM","XID","change_by","change_time","city","company","company2","create_by","create_time","customer_id","email","fax","first_name","id","inst_city","inst_first_name","inst_last_name","inst_street","inst_zip","last_name","login","mobile","phone","phone2","street","title","valid_id","varioCustomerId","zip"],"id"=>"f1c74146d6672ca71f489aac1b4c2a332ae515996657981e1ef44b441a7420c8"}]=>[{"thread_id"=>23,"name"=>nil,"current_call"=>"[...]/logstash-core/lib/logstash/util/wrapped_synchronous_queue.rb:90:in `read_batch'"}]}}
[2018-03-22T09:00:06,484][ERROR][logstash.shutdownwatcher ] The shutdown process appears to be stalled due to busy or blocked plugins. check the logs for more information.
[2018-03-22T09:00:11,438][WARN ][logstash.shutdownwatcher ] {"inflight_count"=>0,"current_call"=>"[...]/logstash-core/lib/logstash/util/wrapped_synchronous_queue.rb:90:in `read_batch'"}]}}

当我用bash logstash -f xyz.config<手动运行相同的文件和相同的配置时myfile.config它可以根据需要工作,并且进程得到正确终止.在bash脚本中我基本上使用的是确切的命令,我遇到了上面的错误. 我也注意到问题似乎是随机的,而不是每次都在同一个文件和配置上. 我的配置包括一个stdin输入一个csv过滤器和用于测试json格式的输出到一个文件(也删除了stdout {}). 有没有人知道为什么我的进程在脚本执行期间停止？或者如果没有,是否有可能告诉logstash在停止时关闭？示例配置：

input {
  stdin {
    id => "${LS_FILE}"
  }

}
filter {
    mutate {
        add_field => { "foo_type" => "${foo_TYPE}" }
        add_field => { "[@Metadata][LS_FILE]" => "${LS_FILE}"}
    }
    if [@Metadata][LS_FILE] == "contacts.csv" {
      csv {
          separator => ";"
          columns =>
          [
            "IOT","kundenid"              
          ]
      }
      if [kundenid]{
        mutate {
            update => { "kundenid" => "n-%{kundeniD}" }
        }
        }
    }
}
output {   
       if [@Metadata][LS_FILE] == "contacts.csv" {
           file{
                path => "~/contacts_file"
                codec => json_lines
            }
       }
}

示例脚本：

LOGSTASH="/customer/app/logstash-6.2.3/bin/logstash"

    for file in $(find $TARGETPATH -name *.csv) # Loop each file in given path
    do
        if [[ $file = *"foo"* ]]; then
            echo "ImporTing $file"
            export LS_FILE=$(basename $filE)
            bash $LOGSTASH -f $CFG_FILE < $file  # StarTing logstash
            echo "file $file imported."
        fi
    done

我在bash脚本中导出环境变量,并将它们设置为logstash配置中的元数据,以便为不同的输入文件执行一些条件.文件中JSON的输出仅用于测试目的.