IntermediateSystem-Design
30 min
Design a Centralized Logging Platform
LoggingIngestionSearchAlerting
Advertisement
Interview Question
Design a centralized logging system for microservices with ingestion, indexing, querying, and alerting at scale.
Key Points to Cover
- Ingestion: log shippers, forwarders, batching, backpressure
- Indexing: schema, storage, hot vs warm tiers
- Querying: full-text search, filters, aggregations
- Alerting: thresholds, anomaly detection, integrations
- Retention: compaction, TTLs, archiving to cold storage
Evaluation Rubric
Scalable ingestion & backpressure mgmt25% weight
Efficient indexing & tiering25% weight
Powerful querying & alerting25% weight
Retention & archiving strategy25% weight
Hints
- 💡Index only what’s queryable to control costs.
Common Pitfalls to Avoid
- ⚠️**Ignoring Log Volume Growth:** Underestimating future log volumes leading to performance bottlenecks and unexpected infrastructure costs.
- ⚠️**Lack of Standardization:** Inconsistent log formats across microservices makes searching and correlation extremely difficult.
- ⚠️**Over-reliance on Full-Text Search:** Not utilizing structured logging and proper indexing, leading to slow and inefficient queries.
- ⚠️**Ineffective Alerting:** Generating too many noisy alerts or missing critical issues due to poorly defined alert rules.
- ⚠️**Neglecting Security & Access Control:** Exposing sensitive log data without proper authorization, leading to security vulnerabilities.
Potential Follow-up Questions
- ❓How do you prevent cardinality blowups?
- ❓How do you ensure logs aren’t tampered with?
Advertisement