LESSON 36 · INTERMEDIATE4 min read

Guided Project 2: High-Throughput Log Analytics & Binary Search Index

Engineer an in-memory log indexing and search engine. Parse structured timestamps, index log records with binary search bounds, and filter anomalies with sliding windows.

OpenJDK 21+ (LTS)Free to readModule 9 of 15

YOUR NEXT SMALL WINS

By the end, you’ll be able to…

Construct an in-memory log event search engine using Java 21 Records and custom comparator sorting.
Implement sub-millisecond timestamp range queries using binary search lower and upper bounds.
Detect error spikes using a sliding window frequency monitor.

01Project Architecture & Log Event Schema

In this second guided project, we engineer an enterprise-grade In-Memory Log Analytics Engine. Production services generate millions of log lines per minute. Centralized observability platforms need to filter, search, and detect anomalies across massive timestamp intervals in real time.

We model our log events with Java 21 Records, storing Unix epoch millisecond timestamps, log levels (INFO, WARN, ERROR), service identifiers, and message payloads.

example.js☕Java

Ctrl + ↵

Output

Click Run or press Ctrl+Enter

Output will appear here after you run the code.

02Sub-Millisecond Range Queries via Binary Search

Log events arrive in chronological order. Because the master log list is sorted by timestamp, querying all log events between startTime and endTime does not require an O(N) linear scan over millions of records.

Instead, we implement Lower Bound and Upper Bound binary searches to locate the exact index range [startIndex, endIndex) in O(log N) time. The sublist is returned instantaneously as a memory-efficient slice.

example.js☕Java

Ctrl + ↵

public class LogIndex {
    private final List<LogEvent> logs = new ArrayList<>();
    
    public void ingest(LogEvent event) {
        logs.add(event); // Chronological ingestion
    }
    
    // Range query: O(log N + M) where M is number of matching events
    public List<LogEvent> queryRange(long start, long end) {
        int fromIndex = lowerBound(start);
        int toIndex = upperBound(end);
        if (fromIndex >= toIndex) return List.of();
        return Collections.unmodifiableList(logs.subList(fromIndex, toIndex));
    }
    
    private int lowerBound(long target) {
        int l = 0, r = logs.size();
        while (l < r) {
            int m = l + (r - l) / 2;
            if (logs.get(m).timestamp() >= target) r = m;
            else l = m + 1;
        }
        return l;
    }
    
    private int upperBound(long target) {
        int l = 0, r = logs.size();
        while (l < r) {
            int m = l + (r - l) / 2;
            if (logs.get(m).timestamp() > target) r = m;
            else l = m + 1;
        }
        return l;
    }
}

Output

Click Run or press Ctrl+Enter

Output will appear here after you run the code.

03Sliding Window Anomaly Detection

To detect denial-of-service attacks or cascading system failures, our engine incorporates an Error Spike Detector using a Sliding Window.

Given a threshold (e.g. more than 50 ERROR events in any 60-second window), the detector slides across the chronological stream, counting error events in real time. If the count exceeds the threshold, it triggers an alert with the exact time interval.

example.js☕Java

Ctrl + ↵

Output

Click Run or press Ctrl+Enter

Output will appear here after you run the code.

04Why This Matters in Production

This project combines divide-and-conquer binary search with sliding window stream analysis to solve the exact problems encountered by platforms like Elasticsearch, Datadog, and Prometheus.

By indexing sorted timestamps, range query latency is slashed from hundreds of milliseconds to under 20 microseconds, allowing real-time dashboards to query terabytes of operational telemetry with zero lag.

05Production Architecture & Engineering Trade-Offs

Building an industrial-grade production project requires rigorous engineering beyond passing unit test suites. Real-world systems must handle partial network failures, malformed input streams, concurrency race conditions, and graceful shutdown sequences without data corruption or memory leaks.

When developing this project, ensure that domain boundaries are clearly defined, errors are modeled with typed exceptions or sealed result types rather than generic runtime errors, and performance critical paths are profiled to verify zero unnecessary heap allocations and predictable latency profiles.

Furthermore, real-world deployments mandate structured observability: instrument latency percentiles (p50, p95, p99) using Micrometer metrics, log contextual correlation identifiers across asynchronous thread boundaries, and implement health-check readiness probes for Kubernetes container orchestration.

Finally, validate defensive resource cleanup: all file descriptors, network sockets, and database connections must be managed via try-with-resources blocks to guarantee deterministic reclamation even when unhandled exceptions occur during critical processing loops.

LET IT CLICK

The ideas worth keeping.

Chronological time-series data allows binary search to find arbitrary time windows in O(log N) time.
Sliding windows detect frequency anomalies over streaming events in linear O(N) overall time.
Use subList views and unmodifiable wrappers to return query results without copying overhead.

TURN UNDERSTANDING INTO SOMETHING YOU CAN DO

Make it yours.

Type and test your code right here in the interactive workspace. Run your solution, check the output, and compare with the worked guide when ready.

Exercise 1: Adding Service Filter to Log Query

Extend queryRange to accept an optional String serviceName, filtering the binary-searched sublist using Java 21 Stream pipelines.

your_turn_1.java☕Java

Ctrl + ↵

Output

Click Run or press Ctrl+Enter

Output will appear here after you run the code.

Explore a worked solution

one_possible_solution.javaJava

public List<LogEvent> queryRangeService(long start, long end, String service) {    return queryRange(start, end).stream()        .filter(event -> service == null || event.service().equals(service))        .toList();}

Binary search isolates the candidate time window in O(log N), and the stream filter filters by service.

Exercise 2: Million-Event Benchmark Test

Write a test that generates 1,000,000 log events and verifies that a queryRange query executes in under 1 millisecond.

your_turn_2.java☕Java

Ctrl + ↵

Output

Click Run or press Ctrl+Enter

Output will appear here after you run the code.

Explore a worked solution

one_possible_solution.javaJava

long start = System.nanoTime();var results = engine.queryRange(500_000, 501_000);long durationUs = (System.nanoTime() - start) / 1000;assertTrue(durationUs < 1000); // Sub-millisecond execution!

Logarithmic binary search executes in ~20 operations, completing in microseconds regardless of heap size.

A SMALL QUESTION. A USEFUL NEXT STEP.

What stayed with you?

Check the ideas while they’re fresh. Your answers stay in this page and feedback appears as you go.

A LITTLE ACTIVE RECALL0 / 3 answered

01Why does the log analytics engine achieve sub-millisecond range queries over millions of events?

ABecause chronological logs are pre-sorted by timestamp, allowing binary search to locate boundaries in O(log N) time.BBecause it compresses files with GZIP.CBecause it skips reading the data.

02How does the sliding window error detector maintain error counts in O(1) amortized time per event?

ABy incrementing errorCount on incoming right events and decrementing on outgoing left events as the window slides.BBy running a full count query every millisecond.CBy rebooting the server.

03What is the return type of logs.subList(fromIndex, toIndex)?

AA memory-efficient view of the underlying list without allocating a duplicate array buffer.BA copy of the entire operating system RAM.CAn array of strings.

⚡CODING CHALLENGE

🔒Required to Mark Lesson Complete

Exercise 1: Adding Service Filter to Log Query

Extend queryRange to accept an optional String serviceName, filtering the binary-searched sublist using Java 21 Stream pipelines.

EXPECTED OUTPUT

Build successful. (Java 15)
[Process exited normally]

Main.javaJava

Ctrl + ↵

Execution Console

Click "Run & Verify" (or press Ctrl + Enter) to test your code against the challenge.

Keep exploring.

The official references behind this lesson, when you want a second angle or the full details.

MODULE 09 · PRACTICAL SELF-CHECK

Pause. Put the pieces together.

You’ve reached the end of Sorting Algorithms, Binary Search on Answer & Log Analytics Project. Before moving on, use this task to check what you can do independently.

Deliver Project 2: High-Performance Log Analytics Search Engine.

Try it in your own environment and write down the result. This is a self-check; reading completion does not evaluate your code.

Review this module

One lesson closer.

Reading is a start. Trying it yourself makes the connection.

Solve the hands-on coding challenge in the knowledge section to verify your code and unlock lesson completion.