event_date is a claim about the world. coded_at is a claim about when we coded it. A backtest
that filters only on event_date can include events learned later. To keep your own copy current, see
Keep a copy in sync.
The as-of claim we do make
Filtering on coding time selects a coding-time cohort, not a replay of the public API at that time. The coding timestamp is useful for latency research, but an event can be coded a little before it becomes available through the API.- The coding timestamp is set on every event, once, when the event is coded, and never revised.
- Every event links the articles it was derived from (
top_articles, and its Story throughprimary_story_url). Nothing is coded from a source we do not cite, so you can go read the evidence that was available at the time.
How to ask for it
Use the ordinary date window first. Theobserved_* filters remain accepted for specialist
coding-time work, but new integrations should not use them to poll for new rows.
as_of on /api/v2/events returns 400, deliberately. Event metrics are not versioned over time,
so as_of could not return a reproducible point-in-time result there. Use observed_start /
observed_end to bound when an Event was coded; that is not an as-of replay. as_of is accepted
where the data has dated versions: /macro/observations, /screening/match, /intelligence/gpr,
/intelligence/posture, the political-offices endpoints, the Situation endpoints and the entity
dossier. Each operation’s reference says what as_of means there.Why recent days keep filling
event_date is when an event happened, not when it was reported. Reporting can arrive a day or more
after the event: local and non-English outlets often publish the next day, and for some places that is
the only coverage. So the counts for recent dates keep growing after those dates end. Events that
appear later are often the better-evidenced ones, so do not drop them to tighten a latency number.
Is a window complete?
Every Event and Story list response says how complete its dates are, inmeta:
meta.coverage.window_completeistrueonly when every date you asked for is served and complete.meta.coverage.partial_dateslists served dates that are still filling. Their rows are included, and more can arrive or change. Recent dates stay partial for some days after they end, so compare windows whose dates are not inpartial_dates.meta.settled_atis when the oldest snapshot behind the response was built.
Keep a copy in sync
Events and Stories have no change feed: the activity feed does not include them. A copy stays current by re-reading recent dates and replacing what it holds.1
Walk a window once
Page through a date window with
cursor until pagination.next_cursor is null, and store each
record by its id. Send the same parameters on every page: a page sent with different filters
answers 400 CURSOR_STALE with details.reason filters_changed. If one of the walk’s dates is
re-served before you finish, the next page also answers 400 CURSOR_STALE. Either way, start the
walk again from the first page.2
Re-read recent dates on every run
Re-walk today and the two dates before it on each run: nearly every late Event is coded within
two days of its date (measured).
Once a day, also re-walk any older date that a response still lists in
meta.coverage.partial_dates, until a read no longer lists it.3
Replace whole records by id
Overwrite each stored record with the one you just read. Do not compare
updated_at to find
changes: entity links, article counts and metric values change without it moving.4
Look up ids that did not come back
Fetch a stored id that a re-walked date no longer returns, one at a time, with
GET /api/v2/events/{id} or GET /api/v2/stories/{id}:A filter on a value that is recomputed, such as
significance_min, is the usual reason for a
200: a record leaves your results when its value drops below your floor.5
Call a window final only when it says so
Treat a window as final only when
meta.coverage.window_complete is true.Python
How to use this in a backtest
1
Filter on when we coded it, not on when it happened
For a signal you claim to have had at time T, bound
observed_end at T. Filtering on
event_date alone silently imports hindsight.2
Set your decision lag deliberately
If your strategy acts on same-day information, model a decision point at T+24h and expect to see
most, not all, of the day’s events. If it can wait, T+48h captures nearly all of them.
3
Do not treat a thin day as a quiet day
A day with an unusually low event count near the present is more likely still filling in than
genuinely calm. Re-pull recent windows before drawing a conclusion about them, and prefer
windows that are at least 48 hours old.
4
Check the file's own as-of stamp
Every bulk file carries an
as_of value: the oldest point in time any of its dates reflects, so
one fresh day cannot vouch for a stale month.
