NotImplementedError: block device detection unsupported or native support failed to load

[mattstibbs]

Issue

I am encountering the following error when trying to use the file input to watch /var/log/syslog and /var/log/auth.log

I have run the following on the log files:

setfacl -m u:logstash:r /var/log/{syslog,auth.log}

The following exception stack trace is from /var/log/logstash/logstash.err

NotImplementedError: block device detection unsupported or native support failed to load
       blockdev? at org/jruby/RubyFileTest.java:67
         device? at /opt/logstash/vendor/bundle/jruby/1.9/gems/filewatch-0.6.2/lib/filewatch/helper.rb:67
  _sincedb_write at /opt/logstash/vendor/bundle/jruby/1.9/gems/filewatch-0.6.2/lib/filewatch/tail.rb:230
   sincedb_write at /opt/logstash/vendor/bundle/jruby/1.9/gems/filewatch-0.6.2/lib/filewatch/tail.rb:203
        teardown at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-file-0.1.9/lib/logstash/inputs/file.rb:151
     inputworker at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.0.rc3-java/lib/logstash/pipeline.rb:203
     synchronize at org/jruby/ext/thread/Mutex.java:149
     inputworker at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.0.rc3-java/lib/logstash/pipeline.rb:203
     start_input at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.0.rc3-java/lib/logstash/pipeline.rb:171

If I run interactively using
sudo -u logstash /opt/logstash/bin/logstash agent -f /etc/logstash/conf.d/ --verbose

I get the same exception stack trace plus this:

The error reported is:
  Bad file descriptor - Bad file descriptor

I have a number of file inputs in the pipeline, but it seems only be this file input that causes a problem. Removing this file input allows it to start fine.

Environment Details

root@logstash1:/# lsb_release -d
Description:    Ubuntu 14.04.1 LTS

root@logstash1:/# /opt/logstash/bin/logstash -V
logstash 1.5.0-rc3

root@logstash1:/# java -version
java version "1.8.0_45"
Java(TM) SE Runtime Environment (build 1.8.0_45-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.45-b02, mixed mode)

[mattstibbs]

I have now significantly simplified the pipeline to try and narrow this problem down. Pipeline is now:

input {
  file {
    sincedb_path => [ "/var/tmp/log.sincedb" ]
    path => [ "/var/tmp/log.log" ]
    type => "syslog"
  }
}
output {
  stdout { codec => rubydebug }
}

I have also changed permissions on /var/tmp with:

sudo chmod 777 -R /var/tmp/

Running logstash with the following still generates the original error:

sudo -u logstash /opt/logstash/bin/logstash agent -f /etc/logstash/conf.d/ --verbose

Stuck as to where to go next!

[mattstibbs]

Having started with a fresh install of Ubuntu, I am getting the same problem but am now noticing the following error intermittently as well as the reported error.

I can't find any pattern as to what causes which to appear, although it seems that if I only use RC2 I only get the below error (and not the block device error)

NotImplementedError: stat.st_dev unsupported or native support failed to load
       dev_major at org/jruby/RubyFileStat.java:188
  _discover_file at /opt/logstash/vendor/bundle/jruby/1.9/gems/filewatch-0.6.1/lib/filewatch/watch.rb:150
            each at org/jruby/RubyArray.java:1613
  _discover_file at /opt/logstash/vendor/bundle/jruby/1.9/gems/filewatch-0.6.1/lib/filewatch/watch.rb:132
           watch at /opt/logstash/vendor/bundle/jruby/1.9/gems/filewatch-0.6.1/lib/filewatch/watch.rb:38
            tail at /opt/logstash/vendor/bundle/jruby/1.9/gems/filewatch-0.6.1/lib/filewatch/tail.rb:68
             run at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-file-0.1.6/lib/logstash/inputs/file.rb:133
            each at org/jruby/RubyArray.java:1613
             run at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-file-0.1.6/lib/logstash/inputs/file.rb:133
     inputworker at /opt/logstash/lib/logstash/pipeline.rb:174
     start_input at /opt/logstash/lib/logstash/pipeline.rb:168

Also confirmed not an issue when reverting to logstash-1.4.2 using same config on same machine

[suyograo]

@mattstibbs I am unable to reproduce this :( It is weird because we have many folks hitting this issue -- #2962, #3033

ubuntu@ip-10-0-1-9:/opt/logstash$ lsb_release -d
Description:    Ubuntu 14.04.1 LTS
ubuntu@ip-10-0-1-9:/opt/logstash$ java -version
java version "1.8.0_45"
Java(TM) SE Runtime Environment (build 1.8.0_45-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.45-b02, mixed mode)
ubuntu@ip-10-0-1-9:/opt/logstash$ bin/logstash version
logstash 1.5.0-rc3
ubuntu@ip-10-0-1-9:/opt/logstash$ bin/logstash -e 'input { file { path => "/var/log/syslog" start_position => "beginning" } }'
Logstash startup completed
{
       "message" => "Apr 30 07:00:49 ip-10-0-1-9 dhclient: DHCPREQUEST of 10.0.1.9 on eth0 to 10.0.1.1 port 67 (xid=0x54d2a924)",
      "@version" => "1",
    "@timestamp" => "2015-04-30T21:22:40.079Z",
          "host" => "ip-10-0-1-9",
          "path" => "/var/log/syslog"
}

[mattstibbs]

@suyograo :(

All my previous tests have been on digitalocean droplets. Have just followed below steps and am getting exactly same error:

1. Download Ubuntu 14.04.2 Server iso from http://www.ubuntu.com/download/server
2. Create standard Ubuntu install using VirtualBox (RAM 512Mb, HDD 16GB).
3. During startup create a user as instructed and then log into machine as that user
4. Run following commands in order:

sudo add-apt-repository -y ppa:webupd8team/java
sudo apt-get update
sudo apt-get -y install oracle-java8-installer
reboot

wget http://download.elastic.co/logstash/logstash/packages/debian/logstash_1.5.0-rc3-1_all.deb
sudo dpkg -i logstash_1.5.0-rc3-1_all.deb
reboot

cd /opt/logstash
sudo bin/logstash -e 'input { file { path => "/var/log/syslog" start_position => "beginning" } }'

• Ubuntu 14.04.2 LTS
• java version "1.8.0_45"
• logstash 1.5.0-rc3

[suyograo]

@mattstibbs Thanks, I was able to reproduce -- we'll look into this.

ubuntu@ip-10-0-1-10:/opt/logstash$ lsb_release -d
Description:    Ubuntu 14.04.2 LTS
ubuntu@ip-10-0-1-10:/opt/logstash$ java -version
java version "1.8.0_45"
Java(TM) SE Runtime Environment (build 1.8.0_45-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.45-b02, mixed mode)
ubuntu@ip-10-0-1-10:/opt/logstash$ bin/logstash version
logstash 1.5.0-rc3
ubuntu@ip-10-0-1-10:/opt/logstash$