I'm trying to use logstash to extract metrics from my logs, and then send those metrics with statsdb.

Because statsdb sends "live data" there's no point reading old logs and send them. So I don't need logstash to read all the existing and available data when it restarts. I just want it to process what's appended to the files.

I tried to play with the settings it still reads all the data when:

input {
  file
  {
      path => "./dbg*.log"
      sincedb_path => ""
      start_position => "end"
  }
}

Is there a way to skip the existing data when starting logstash ?

UPDATE:

I had a look at the bug report and I'm considering raising an issue with logstash. I found those two that are relevant, but they don't seem to be resolved or working on windows:

start_position refers to data within a single log file, so it isn't of any use re. older logs files.

If your logfiles have a naming format based on date, you should be able to exclude them from logstash using the exclude parameter and a wildcard pattern.

http://logstash.net/docs/1.4.0/inputs/file#exclude

If you files don't have a naming pattern, you could also just zip older files and then exclude *.gz

find ./ -mtime +1 | xargs -n1 gzip

will zip all files older than 1 day.

You may also want to consider managing the key string used by Redis. This will allow you to move the keys from Redis so that they don't get into Elasticsearch

http://www.nightbluefruit.com/blog/2014/03/managing-logstash-with-the-redis-client/