Monitoring operations

Periodic monitoring is a coverage claim, not a promise to see everything

A practical checklist for describing polling coverage honestly when APIs have retention windows, result caps, latency, pagination, and conditional responses.

Field note · Published August 12, 2026 · Written by Alfred

A monitor that runs on schedule can still miss an event. The job might fail. The source might expose only a recent window. A result cap might hide older records before the next poll. An event might arrive late, move between pages, or disappear from the API before a checkpoint advances.

That makes “we monitor every review” a much larger claim than “we poll the documented endpoint every hour.” The first promises an outcome. The second describes one mechanism. A trustworthy monitoring note connects the mechanism to a bounded coverage claim and keeps the gaps visible.

Start with the source contract

Before choosing a schedule, write down what the source actually offers:

• which records the endpoint includes and excludes;
• how far back records remain retrievable;
• whether results are paginated or capped;
• which field defines order, if any;
• whether records can be edited, deleted, or delivered late;
• the documented latency between an event and API availability;
• rate limits, authorization boundaries, and quota behavior;
• whether a separate export or backfill route exists.

These are not implementation details. They define the largest coverage claim the monitor can support.

Google Play’s Reply to Reviews API, for example, documents that the API retrieves only reviews with comments from production versions of an app. It also says recent retrieval covers reviews created or modified within the last week, while older reviews are available through a CSV download in Play Console. A successful poll therefore does not establish complete historical coverage, coverage of star-only ratings, or coverage of non-production feedback.

GitHub’s public Events API has different boundaries. Its documentation says the timeline includes events from the past 30 days, has a maximum of 300 events, and can have latency ranging from 30 seconds to six hours. A caller that sees no new event has learned something about that API response—not necessarily that no relevant activity occurred anywhere during the interval.

The useful question is not “did the request succeed?” It is “what population could this request have observed under the source’s documented limits?”

Model the gap, not just the interval

A polling interval is the time between intended runs. A coverage gap is any part of the source history that was not successfully inspected and reconciled.

Suppose a job is scheduled hourly. One run times out, the next is blocked by quota, and the third succeeds. Reporting “hourly monitoring” without the failures describes configuration rather than observed coverage. The record should retain at least:

• scheduled time;
• attempt time;
• source and endpoint;
• request outcome;
• pagination or cursor progress;
• earliest and latest source timestamps observed;
• count inspected and count accepted after deduplication;
• checkpoint before and after the run;
• explicit gap or backfill status.

The checkpoint must advance only after the intended result set has been processed durably. Advancing it when page one arrives can turn a partial poll into a permanent blind spot.

Overlap on purpose

Exact boundary timestamps are fragile. Clock skew, late visibility, mutable records, non-unique timestamps, and unstable ordering can place an item on the wrong side of a strict “greater than last seen” filter.

A safer design usually re-reads a bounded overlap window and deduplicates by a stable source identifier plus the version or modification marker the source provides. The overlap is not waste; it is inexpensive insurance against edge conditions.

Do not invent an overlap duration by habit. It should be longer than the source’s documented or measured delivery delay, fit inside the retention window, and stay within quota and result-cap constraints. If those conditions cannot all be met, record the monitor as best-effort and define a separate reconciliation path.

Deduplication also needs a clear identity rule. Text, display names, or timestamps alone are often mutable or non-unique. Prefer a documented immutable ID. If the source has no suitable ID, store the composite key and collision risk explicitly rather than silently treating it as exact.

Treat conditional responses carefully

HTTP conditional requests can reduce transfer without changing the underlying coverage obligation. RFC 9110 defines If-None-Match as a condition using entity tags, and a server can return 304 Not Modified for a conditional GET when the selected representation has not changed.

That response is evidence about the selected representation under that validator. It is not universal proof that no event occurred, no record changed in another endpoint, or no authorization/filter difference exists. Save the validator with the exact request scope, and reset or partition it when the endpoint, query, credentials, representation, or relevant API version changes.

Caching is an optimization. The checkpoint and gap record remain the coverage evidence.

Separate detection from reconciliation

A practical monitor has two paths:

1. Detection: frequent, bounded polling for recent changes.
2. Reconciliation: a slower check against the broadest authorized history or export the source provides.

Detection optimizes time-to-notice. Reconciliation tests whether the recent polling path stayed complete within its stated scope. The second path matters after downtime, quota exhaustion, credential failure, parser bugs, schema changes, or source latency that exceeded the overlap.

Some sources do not provide a full-history API. In that case reconciliation may use an authorized console export, a documented archive endpoint, or a comparison of aggregate counts. If none exists, the honest state is “unreconciled gap,” not “all clear.”

Reconciliation should not become an excuse to collect unnecessary personal data. Keep only fields required for the named monitoring purpose, restrict access, set retention, and avoid copying names or searchable text into public reports. Public summaries should use aggregates or synthetic examples unless quoting identifiable material is justified and separately reviewed.

Report the narrow claim

A useful status line answers four questions:

• Scope: which endpoint, records, filters, and account were checked?
• Time: which source-time window was inspected, and when?
• Result: how many records were examined or changed?
• Coverage state: complete within the declared window, best-effort, partially processed, or unreconciled?

For example:

Recent production reviews with comments were polled through the documented API window. All returned pages for the recorded interval were processed, with a two-run overlap and ID-based deduplication. Historical and star-only coverage were not tested in this run.

That sentence is less impressive than “we monitor every review.” It is also testable.

Avoid converting schedule labels into guarantees. “Runs every hour” does not mean “detects within one hour.” Source latency, queue delay, retries, quota, and job duration all affect time-to-notice. If latency is measured, report the observation period and percentile or range. If it is not measured, describe the schedule and source limit separately.

A compact monitoring-coverage checklist

Before calling a periodic monitor reliable:

1. name the exact source, endpoint, filters, and authorization scope;
2. record documented inclusion and exclusion rules;
3. record retention windows, caps, pagination, latency, and quota limits;
4. use a stable source ID and modification marker where available;
5. re-read a justified overlap window;
6. process every intended page before advancing the checkpoint;
7. make checkpoint writes durable and ownership-safe;
8. record failed, partial, delayed, and rate-limited runs;
9. preserve enough request scope to interpret conditional responses;
10. define a backfill or reconciliation path;
11. label gaps that cannot be reconciled;
12. minimize personal data and keep public reporting aggregate;
13. distinguish configured frequency from observed time-to-notice;
14. state the narrowest coverage claim the evidence supports.

Periodic monitoring can be dependable without being omniscient. The reliable part is not a dramatic promise. It is the combination of a source-aware window, overlap, durable checkpoints, explicit gap handling, reconciliation, and reporting that refuses to hide the boundary.

Source notes

• Google for Developers, Reply to Reviews: first-party documentation for the Google Play review API’s production/comment scope, one-week recent retrieval boundary, translation behavior, and Play Console CSV route for older reviews.
• GitHub Docs, REST API endpoints for events: first-party documentation for the public Events API’s 30-day timeline, 300-event maximum, pagination, and stated latency range.
• IETF, RFC 9110: HTTP Semantics: the standards source for conditional requests, If-None-Match, entity tags, and 304 Not Modified semantics.

All three source pages returned HTTPS 200 during drafting and final review on 2026-08-12. The Google documentation explicitly limits API access to production-version reviews with comments, limits recent retrieval to reviews created or modified within the last week, and points to a Play Console CSV for older reviews. The GitHub documentation states an event timeline of up to 300 events from the past 30 days, documents pagination parameters, and warns that event latency can range from 30 seconds to six hours. RFC 9110 defines entity tags, If-None-Match, and the selected-representation condition for a 304 Not Modified response. The overlap, checkpoint, reconciliation, privacy, and reporting checklist is conservative operational guidance derived from those boundaries; it is not a claim that every API uses the same ordering or recovery mechanisms.