Grafana Loki Log Aggregation

Vibe Prompt

“Help me set up Grafana Loki + Promtail to collect logs from every Pod in a Kubernetes cluster, and query them in Grafana.”

What is Loki?

Loki is a horizontally scalable, highly available log aggregation system inspired by Prometheus. Unlike traditional log stacks such as ELK, Loki filters logs by labels rather than indexing every log line. This design keeps storage costs low while still enabling powerful, label‑based queries.

Why Use Loki?

  • Cost Efficiency: Loki stores raw log streams without full-text indexing, reducing storage by roughly 90 % compared to Elasticsearch.
  • Seamless Grafana Integration: A single Grafana instance can display metrics from Prometheus and logs from Loki side‑by‑side.
  • Operational Simplicity: Promtail, Loki’s lightweight agent, automatically discovers and streams logs from Kubernetes containers.
  • Business Value: Faster debugging, reduced storage spend, and unified observability reduce mean time to resolution (MTTR) and improve uptime.

How to Deploy Loki, Promtail, and Grafana with Docker Compose

Docker Compose File

services:
  loki:
    image: grafana/loki:latest
    ports:
      - "3100:3100"
    command: -config.file=/etc/loki/local-config.yaml
    volumes:
      - ./loki-config.yaml:/etc/loki/local-config.yaml

  promtail:
    image: grafana/promtail:latest
    volumes:
      - /var/log:/var/log
      - ./promtail-config.yaml:/etc/promtail/config.yaml
    command: -config.file=/etc/promtail/config.yaml

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    environment:
      GF_SECURITY_ADMIN_PASSWORD: admin
    depends_on:
      - loki

Step‑by‑Step Implementation

  1. Create a working directory

    mkdir loki-demo && cd loki-demo

  2. Add the docker-compose.yml file
    Paste the YAML above into docker-compose.yml.

  3. Create Loki configuration (loki-config.yaml)

    auth_enabled: false
server:
  http_listen_port: 3100
ingester:
  wal:
    enabled: true
schema_config:
  configs:
    - from: 2020-10-15
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 168h
storage_config:
  boltdb_shipper:
    active_index_directory: /data/loki/index
    cache_location: /data/loki/cache
    shared_store: filesystem
  filesystem:
    directory: /data/loki/chunks
limits_config:
  enforce_metric_name: false
  reject_old_samples: true
  reject_old_samples_max_age: 168h
chunk_store_config:
  max_look_back_period: 0s
table_manager:
  retention_deletes_enabled: true
  retention_period: 720h

  4. Create Promtail configuration (promtail-config.yaml)

    server:
  http_listen_port: 9080
  grpc_listen_port: 0
positions:
  filename: /tmp/positions.yaml
clients:
  - url: http://loki:3100/loki/api/v1/push
scrape_configs:
  - job_name: kubernetes-pods
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_app]
        target_label: app
      - source_labels: [__meta_kubernetes_namespace]
        target_label: namespace
      - source_labels: [__meta_kubernetes_pod_name]
        target_label: pod
      - source_labels: [__meta_kubernetes_pod_container_name]
        target_label: container
      - source_labels: [__meta_kubernetes_pod_container_name]
        target_label: job
      - source_labels: [__meta_kubernetes_pod_container_name]
        target_label: __path__
        replacement: /var/log/containers/*${1}.log

  5. Start the stack

    docker compose up -d

  6. Verify services

    • Loki: http://localhost:3100/ready → should return {"status":"success"}
    • Promtail: logs should appear in the Docker logs (docker logs promtail)
    • Grafana: http://localhost:3000 → login with admin / admin

Promtail Config Explained

scrape_configs:
  - job_name: system
    static_configs:
      - targets: [localhost]
        labels:
          job: varlogs
          __path__: /var/log/*log
  • job_name: Identifier for the scrape job.
  • static_configs: Explicitly lists log file paths.
  • labels: Attach metadata to each log stream.
    • job: Logical grouping.
    • __path__: The actual file path pattern Promtail watches.

In Kubernetes, Promtail uses service discovery (kubernetes_sd_configs) to automatically discover pod logs, eliminating the need for static paths.

LogQL – Loki’s Query Language

LogQL blends PromQL’s label filtering with powerful log content filtering.

| Query | What It Does | Why It Matters | |-------|--------------|----------------| | {app="my-app"} | Select all logs tagged with app=my-app. | Quickly isolate logs for a specific microservice. | | {app="my-app"} |= "error" | Filter those logs to lines containing the string “error”. | Spot error events without scanning every line. | | {app="my-app"} |= "error" != "heartbeat" | Exclude lines that also contain “heartbeat”. | Reduce noise from routine health checks. | | {app="my-app"} |~ "error|exception|failed" | Regex match for multiple patterns. | Catch a broader set of failure indicators. | | {app="my-app"} | json | status >= 500 | Parse JSON logs and filter by status code. | Detect HTTP 5xx responses in structured logs. |

Example: Correlating Metrics and Logs

{namespace="production", pod=~"api-server.*"} 
  |= "error" 
  != "timeout" 
  | json 
  | line_format "{{.message}}"
  • Step 1: Find pods in the production namespace whose names match api-server.*.
  • Step 2: Filter for log lines containing “error” but not “timeout”.
  • Step 3: Parse JSON to extract structured fields.
  • Step 4: Display only the message field for readability.

Grafana Explore Workflow

  1. Open Grafana → Navigate to Explore.
  2. Select Data Source → Choose Loki.
  3. Enter LogQL Query → Paste or build your query.
  4. Adjust Time Range → Use the time picker to focus on relevant periods.
  5. Inspect Results → Review log lines, apply filters, or export data.

Tips for Efficient Exploration

  • Use label filters first to narrow the dataset before applying content filters.
  • Leverage |=, !=, |~ operators for string matching.
  • Combine | json with | line_format to extract and display specific fields.
  • Save frequently used queries as Dashboards for quick access.

Key Takeaways

  • ✅ Loki is a Prometheus‑inspired log system that indexes only labels, not full text.
  • ✅ LogQL merges PromQL label syntax with log content filtering, enabling expressive queries.
  • ✅ Seamless Integration: One Grafana instance can display metrics from Prometheus and logs from Loki side‑by‑side.
  • ✅ Cost Advantage: Loki’s storage cost is roughly 1/10 of Elasticsearch’s for similar workloads.
  • ✅ Operational Simplicity: Promtail auto‑discovery in Kubernetes eliminates manual configuration.

Prometheus vs. Loki

| Feature | Prometheus | Loki | |---------|------------|------| | Data Type | Numerical metrics | Text logs | | Indexing | Label + TSDB | Label + raw log stream | | Query Language | PromQL | LogQL (PromQL‑based) | | Storage Cost | Low | Very low (no full‑text index) | | Primary Use | Monitoring & alerting | Debugging & auditing |

Common Pitfalls & Troubleshooting

| Issue | Symptom | Fix | |-------|---------|-----| | Promtail cannot find logs | No logs appear in Grafana | Verify __path__ matches actual log file locations; check container log directories. | | Loki returns “no data” | Query yields empty result | Ensure labels in query match those emitted by Promtail; use label_values(app) to list available labels. | | High memory usage | Loki container crashes | Tune ingester and chunk_store_config settings; consider increasing max_look_back_period. | | Slow query performance | LogQL queries take seconds | Add more labels to narrow search; avoid |~ regex on large streams. | | Grafana data source error | “Unable to connect to Loki” | Confirm Loki is listening on port 3100; check network policies in Kubernetes. |

Code Example: Sending Logs from a Python Service

import logging
import json
import requests

# Configure logger
logger = logging.getLogger("my-app")
logger.setLevel(logging.INFO)
handler = logging.StreamHandler()
formatter = logging.Formatter('%(asctime)s %(levelname)s %(message)s')
handler.setFormatter(formatter)
logger.addHandler(handler)

# Sample log entry
log_entry = {
    "timestamp": "2024-07-04T12:00:00Z",
    "level": "ERROR",
    "message": "Failed to connect to database",
    "status": 500
}
logger.info(json.dumps(log_entry))

When running inside a Kubernetes pod, Promtail will capture this log and forward it to Loki. In Grafana, you can query {app="my-app"} | json | status >= 500.

Loki: A Lightweight Log System

Loki’s design philosophy is to be resource‑efficient while still providing powerful observability. By indexing only labels, Loki keeps the write path fast and the storage footprint small.

Loki vs. ELK Stack

| Aspect | Loki | ELK (Elasticsearch + Logstash + Kibana) | |--------|------|----------------------------------------| | Indexing | Label‑only | Full‑text | | Storage Cost | Low | High | | Query Language | LogQL | Lucene | | Grafana Integration | Native | Requires plugin | | Operational Overhead | Minimal | Significant (Elasticsearch cluster, Logstash pipelines) | | Use Case | Microservice logs, structured logs | Enterprise log analytics, SIEM |

Next Chapter Preview: OpenTelemetry

Logs provide a snapshot of what happened, but to understand how an event propagated through a distributed system, you need distributed tracing. The upcoming chapter will introduce OpenTelemetry, the open source standard for collecting traces, metrics, and logs across services. You’ll learn how to instrument your code, export traces to Loki or Tempo, and visualize end‑to‑end request flows in Grafana. This knowledge will empower you to pinpoint latency bottlenecks, trace root causes, and deliver faster, more reliable services.

Unlock Full Tutorial

This chapter is paid content. Join the project to unlock over 5000 words of deep analysis, including 10+ god-tier Prompts and real Source Code examples!

💳 View Payment Plans (NT$ 1999)Already purchased? Click here to login