Mark Needham

Thoughts on Software Development

logstash not picking up some files

with 3 comments

We’re using logstash to collect all the logs across the different machines that we use in various environments and had noticed that on some of the nodes log files which we’d told the logstash-client to track weren’t being collected.

We wanted to check what the open file descriptors of logstash-client were so we first had to grab its process id:

$ ps aux | grep logstash
logstash     19896  134  9.1 711404 187768 ?       Ssl  09:13   0:06 java -Xms128m -Xmx256m -jar /var/apps/logstash/logstash-1.1.1-rc2-monolithic.jar agent -f /etc/logstash/logstash-client.conf
root     19910  0.0  0.0   7624   936 pts/1    S+   09:13   0:00 grep --color=auto logstash

And then list the open file descriptors:

$ ls -alh /proc/19896/fd
lr-x------ 1 root root 64 2012-09-07 09:16 9 -> /var/log/syslog
...

That seemed to be restricted to 50 files for some reason so we also tried ‘lsof':

$ lsof -p 19896
COMMAND   PID USER   FD      TYPE             DEVICE SIZE/OFF    NODE NAME
root    20230 root  txt       REG                8,1    39584   22895 /var/log/syslog
...

Either way we weren’t seeing most of the files we were supposed to be tracking so we put some print statements into the ruby-filewatch gem (which is included in the logstash jar) and redeployed the jar to see if we could figure out what was going on.

Eventually we narrowed it down to the watch.rb file’s _discover_file method which was making a call to Dir.glob and returning an empty array even for some paths which definitely existed.

114
115
116
117
    def _discover_file(path, initial=false)
      Dir.glob(path).each do |file| # if Dir.glob is empty the file doesn't get watched!
        next if @files.member?(file)
        next unless File.file?(file)

logstash 1.1.1 uses the JRuby 1.6.7 interpreter so we installed that locally to check if we could replicate the problem but we didn’t really end up getting anywhere so we ended up writing some code to work around the problem.

The beginning of the _discover_file method now looks like this:

114
115
116
117
118
119
120
121
122
123
124
125
126
    def _discover_file(path, initial=false)
      globbed_dirs = Dir.glob(path)
      @logger.debug("_discover_file_glob: #{path}: glob is: #{globbed_dirs}")
      if globbed_dirs.empty? && File.file?(path)
        globbed_dirs = [path]
        @logger.debug("_discover_file_glob: #{path}: glob is: #{globbed_dirs} because glob did not work")
      end
      globbed_dirs.each do |file|
        next if @files.member?(file)
        next unless File.file?(file)
 
        @logger.debug("_discover_file: #{path}: new: #{file} (exclude is #{@exclude.inspect})")
    ...

With this hack we can still ensure that a file will be watched even if Dir.glob returns an empty array.

Be Sociable, Share!

Written by Mark Needham

September 7th, 2012 at 11:49 pm

  • Jordan Sissel

    Tracking this bug here:  https://logstash.jira.com/browse/LOGSTASH-591 – will likely include your workaround even if I can’t reproduce this.

  • http://twitter.com/jabley James Abley

    Did you raise it on the JRuby list / IRC? I’ve found the community really helpful and the committers tend to have excellent turnaround for fixing issues.

  • http://www.markhneedham.com/blog Mark Needham

    @twitter-8145762:disqus I couldn’t actually replicate the problem with JRuby on my machine so I wasn’t sure how exactly to report it!