How to Configure Filebeat to Send Logs to Elasticsearch

Learning how to configure Filebeat to send logs to Elasticsearch is a vital skill for anyone managing modern applications, servers, or cloud platforms. Filebeat is a simple, efficient tool designed to collect, process, and forward log files to Elasticsearch, where logs can be stored, searched, analyzed, and visualized via Kibana. A robust log pipeline improves observability, accelerates troubleshooting, and enhances security monitoring.

Why Configure Filebeat to Send Logs to Elasticsearch?

• Centralized Logging: Aggregate logs from multiple systems into a single searchable location, simplifying management and investigation.
• Real-Time Monitoring: Gain immediate insights into system health, performance, and security through Elasticsearch and Kibana dashboards.
• Alerting & Automation: Enable automated alerting when issues or anomalies occur, with integrations for further automation.
• Compliance & Auditing: Meet regulatory and operational requirements by maintaining reliable, queryable logs.
• Scalable and Flexible: Filebeat supports numerous log sources and integrates seamlessly with the entire Elastic Stack.

Prerequisites:How to Configure Filebeat to Send Logs to Elasticsearch

• Access to an Elasticsearch cluster (local or remote); credentials if authentication is required.
• Basic understanding of your system’s log files and which logs should be shipped.
• Administrator/root privileges to install Filebeat and modify configuration files on the target system.
• YAML file editing experience (used for filebeat.yml configuration).

Installing Filebeat

1. Download Filebeat:
   Visit the Elastic website and download Filebeat for your OS (Linux, Windows, MacOS).
2. Install Filebeat:
   • Linux: Use package managers (apt, yum), or extract from a tarball.
   • Windows: Run the installer or extract from a zip file.
   • MacOS: Use brew install filebeat or download from Elastic.
3. Verify Installation: Check Filebeat is installed with:

   filebeat version

How to Configure Filebeat to Send Logs to Elasticsearch

The essential step in how to configure Filebeat to send logs to Elasticsearch is editing the filebeat.yml configuration file. This file outlines which logs to collect, where to send them, and additional options for filtering and enrichment.

Step 1: Define Log Inputs

Specify the paths or files Filebeat should monitor. Example configuration:

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/syslog
    - /var/log/auth.log
    # Add more paths as needed

Each item under paths is a file or directory Filebeat will monitor in real time.

Step 2: Configure Output to Elasticsearch

Direct Filebeat to your Elasticsearch cluster by updating the output section:

output.elasticsearch:
  hosts: ["http://localhost:9200"]   # Use the Elasticsearch host/IP address
  username: "YOUR_ELASTIC_USERNAME"  # Omit if no authentication set up
  password: "YOUR_ELASTIC_PASSWORD"  # Omit if no authentication set up
  # Optional: Configure SSL if needed
  # ssl.certificate_authorities: ["/path/to/ca.pem"]

For remote clusters or cloud deployments, replace localhost with the cluster’s address. For a secured cluster, provide user credentials or API key.

Step 3: (Optional) Connect Filebeat to Kibana

Setting up Kibana allows Filebeat to load dashboards for instant log visualization:

setup.kibana:
  host: "localhost:5601" # Set to your Kibana URL and port
setup.dashboards.enabled: true

Step 4: (Optional) Enable Filebeat Modules

Use modules for predefined log patterns—great for common software such as Nginx, Apache, MySQL, and system logs.

# Enable system module for basic Linux logs
filebeat.modules:
- module: system
  syslog:
    enabled: true
  auth:
    enabled: true

Enable only the modules you actually need for efficiency.

Step 5: (Optional) Add Processors & Filtering

Filebeat supports processors to modify logs (e.g., add fields, mask sensitive data, drop events based on conditions).

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~

See Elastic documentation for more advanced processor settings.

Starting Filebeat

1. Load Kibana Dashboards (optional):

   sudo filebeat setup --dashboards

   This step is only required if you want prebuilt dashboards in Kibana.

2. Enable and Start Filebeat:
   • Linux: sudo systemctl enable filebeat and sudo systemctl start filebeat
   • Windows: Start from the Services panel or by running Start-Service filebeat in PowerShell.

Verification: Confirming Log Delivery

• Go to Kibana and open the Discover tab.
• Select the filebeat-* index pattern.
• Check for incoming log entries from your configured sources.
• On the Filebeat host, check Filebeat logs for errors:

  sudo tail -f /var/log/filebeat/filebeat

Troubleshooting Tips

• Check syntax in filebeat.yml. YAML format is sensitive to indentation and spacing.
• Ensure Elasticsearch is running and accessible from the Filebeat host.
• Review connectivity, authentication credentials, and firewall settings.
• Use filebeat test output to check connection to Elasticsearch.
• Look for errors in Filebeat logs, typically at /var/log/filebeat/filebeat (Linux) or in the installation directory (Windows).

Best Practices for Filebeat and Elasticsearch

• Keep Filebeat updated to benefit from the latest features and bug fixes.
• Use minimal inputs and modules necessary for your environment to optimize resource usage.
• Regularly rotate and prune old logs from Elasticsearch indices to manage storage.
• Set up Index Lifecycle Management (ILM) in Elasticsearch for log retention and automatic deletion.
• Monitor Filebeat and Elasticsearch resource consumption and adjust configuration as needed.
• Review security: Use SSL/TLS for data in transit and strict authentication.

Frequently Asked Questions

1. Can Filebeat monitor multiple file paths?
   Yes. List all log files or directories under paths: in your filebeat.inputs section.
2. What if Elasticsearch requires SSL?
   Configure the ssl.certificate_authorities option under output.elasticsearch in filebeat.yml.
3. Where is the Filebeat config file located?
   • Linux: /etc/filebeat/filebeat.yml
   • Windows: In the Filebeat installation directory.
4. How can I verify Filebeat is sending logs?
   Use Kibana’s Discover tab and look for the filebeat-* index. You can also check Elasticsearch indices with REST API.
5. Can Filebeat send logs to more than one Elasticsearch cluster?
   No. Filebeat supports a single output (one cluster); use Logstash or third-party tools for multiple outputs.
6. Does Filebeat handle log rotation?
   Yes. Filebeat detects rotated files and follows the new files automatically.
7. How do I filter which logs are sent?
   Use Filebeat processors, include/exclude patterns, or route logs via Logstash for advanced filtering.
8. Can Filebeat enrich or modify events?
   Yes. Add fields, remove sensitive data, or drop certain events using processors in the configuration.
9. What are Filebeat modules?
   Pre-configured log collectors and parsers for popular platforms and applications.
10. How can I start Filebeat automatically on system boot?
    Enable the service: sudo systemctl enable filebeat on Linux; set startup type to “Automatic” on Windows.
11. Does Filebeat support Docker?
    Yes. You can run Filebeat in a Docker container using the official image and a mapped configuration.
12. What happens if Elasticsearch is unavailable?
    Filebeat buffers events locally (within limits) and retries sending logs when connectivity returns.
13. What authentication methods are available?
    Basic username/password, API keys, or SSL certificates (depending on your Elasticsearch security setup).
14. Can I add custom fields to all logs?
    Yes. Use the fields: entry under each input or globally for all events.
15. Where does Filebeat write its own logs?
    • Linux: /var/log/filebeat/filebeat
    • Windows: logs folder in the Filebeat installation directory
16. Does Filebeat use a lot of CPU or memory?
    No. Filebeat is designed to be lightweight, even on high-volume log sources.
17. Is it safe to upgrade Filebeat?
    Yes, but back up filebeat.yml and test your new version before upgrading in production.
18. How do I limit Filebeat’s bandwidth usage?
    Use bulk_max_size, output queues, and throttling options in the config.
19. Can Filebeat parse JSON log lines?
    Yes. Use the json.keys_under_root setting under an input to extract fields from JSON logs.
20. Is there a way to test Filebeat configuration?
    Run filebeat test config to check syntax and filebeat test output to check connectivity.

Conclusion

Mastering how to configure Filebeat to send logs to Elasticsearch positions you to centralize logging, boost observability, and streamline problem-solving across any environment. With a few simple configuration steps and best practices, you can transform raw log files into live insights inside the Elastic Stack. Start with basic monitoring, and explore advanced Filebeat features such as modules, processors, and custom pipelines for even more robust log management.

Whether you’re a system administrator, developer, security engineer, or IT student, understanding this workflow is a key foundational skill for modern infrastructure operations. Configure, ship, analyze—and ensure your logs are always working for you!

How to Configure Filebeat Output to Elasticsearch (Official Elastic Documentation)