16 Years of 1-Minute ES Data: Where I Got It and How I Cleaned It

Summary

• Where the data comes from: I bought it from Databento. The dataset is CME’s official feed, CME Globex MDP 3.0 (code GLBX.MDP3). The instrument is the ES (E-mini S&P 500 futures) continuous contract ES.c.0, 1-minute bars, from June 2010 to September 2026.
• The prices themselves are clean: no missing values, no duplicate timestamps, no impossible highs or lows, and every price sits on the 0.25 tick.
• The real problem is which days you can’t use. I dropped three kinds of days:
  1. Days around a contract roll (66 rolls): old and new contract prices don’t line up, and on expiration Friday there’s no RTH data at all;
  2. Holidays and half days (128 days, 5 of which overlap with a roll): the stock market is closed and only futures trade, or the market closes early at 13:00;
  3. Days with bad data (14 days): a big chunk missing from the feed, or an overnight session stretched over several nights by a holiday.
• Result: 3,924 trading days left, losing only about 2% of full trading days. Every study on this site uses this data.

Where the data comes from

CME Globex MDP 3.0 is the Chicago Mercantile Exchange’s official market data feed. Every futures and options contract that trades on Globex comes through it. Databento has this data from June 6, 2010 onward and charges by usage. Downloading the 1-minute bars cost me about $20.

Here’s exactly what I downloaded:

Why a “continuous contract”

ES isn’t one contract. There’s one for every quarter: ESZ6 expires in December 2026, ESH7 in March 2027, and so on. Each contract is the front month only for the few months before it expires.

To study 16 years of price action, you have to stitch these contracts together end to end. ES.c.0 is the continuous contract Databento stitches for you: it always follows the nearest expiry and rolls to the next one after expiration. That automatic roll turns out to be the biggest trap.

The prices themselves are clean

I checked all of the roughly 5.5 million 1-minute bars:

The real problem: which days you can’t use

1. Around contract rolls (66 rolls)

ES.c.0 only switches to the new contract on the Sunday evening after expiration Friday. But the old contract stops trading at 9:30 a.m. on expiration Friday. So:

• There’s no RTH data at all on expiration Friday;
• When the code looks for Monday’s overnight session, it searches back to Thursday evening and splices “Thursday evening (old contract) + Sunday evening (new contract)” into one overnight session. Old and new contracts usually differ by 10 to several dozen points, once by as much as 224 points, so the spliced overnight high and low are fake.

What I did: for every roll, delete everything from Thursday evening to Monday’s close after the roll. Monday is dropped entirely; Tuesday is kept in full.

One special case: in June 2026, expiration day fell on Juneteenth (Friday 6/19, market closed), so expiration moved up to Thursday and the contract rolled that same evening. I handled it separately.

2. Holidays and half days (128 days)

5 of these fall inside a roll’s deletion window and are already gone with the roll, so the funnel below counts only 123 for this group.

• Stock market closed, futures open for half a day: Martin Luther King Jr. Day, Presidents’ Day, Memorial Day, July 4, Labor Day, Thanksgiving, Juneteenth. The stock market isn’t open at all on these days, and volume is only 1–10% of normal;
• Stock market closes early at 13:00: the day after Thanksgiving, July 3, Christmas Eve. Only half a session, not comparable with a full trading day.

3. Days with bad data (14 days)

Cleaning results

Limitations

• The week before expiration uses the expiring contract, whose volume is only about 1/4 of the new one. It barely matters for price structure, but exclude those days when you study volume;
• Prices are not back-adjusted, so price jumps at each roll. Percentages are fine, but don’t compare absolute point moves across different years;
• Data before May 21, 2017 comes from the exchange’s historical files, without Databento’s own capture timestamps. It hardly matters for 1-minute bars, but it’s worth knowing;
• 23 days are missing a few RTH bars: 19 are in expiration week, when the old contract trades thinly and some minutes have no trades, so no bar (1–9 bars short per day); the other 4 are the March 2020 circuit breakers, when trading halted for 15 minutes (13–14 bars short per day). The prices are fine; just those few minutes are missing;
• 46 days have incomplete overnight data. RTH is normal on those days, so I didn’t drop them. Filter them out separately if you study the overnight session.

How I use it

Every study on this site is built on this cleaned data, and I won’t repeat the details in each post. The cleaning rules are in one script; when new data comes in, I just rerun it.

How to reproduce it

Please write Python code that downloads historical 1-minute bars for ES futures (S&P 500 E-mini) from Databento. Requirements:
1. Use the official databento library; dataset GLBX.MDP3 (CME Globex MDP 3.0), symbol ES.c.0, stype continuous (continuous contract), schema ohlcv-1m.
2. Date range from 2010-06-06 to today.
3. First call metadata.get_cost to look up and print the cost of this download. Only download after I type y to confirm, so I don't get charged by mistake.
4. Ask me for the API key at runtime with getpass. Don't put it in the code or any file.
5. Save the result as CSV, keeping the original columns such as ts_event, instrument_id, open, high, low, close, volume, with timestamps in UTC.
6. After downloading, print the total row count and the timestamps of the first and last rows so I can check them.
7. Add comments to every step, and tell me what I need to install first (pip install databento).

I have a CSV of 1-minute ES futures bars (Databento's ES.c.0 continuous contract, timestamps in UTC, with an instrument_id column). Please write Python code to clean it, following these rules:
1. Convert timestamps to US Eastern time (US/Eastern), handling daylight saving time automatically. RTH = 09:30–16:15 (1-minute bars from 09:30 to 16:14, 405 bars); ETH = previous trading day 16:15 to 09:29 today.
2. First check data quality and print the results: missing values, duplicate timestamps, rows where the high is below the open/close or the low is above the open/close, and prices not on the 0.25 tick.
3. Remove data around contract rolls: find where instrument_id changes (the roll), and delete everything from 16:15 on the last trading day before expiration to 16:15 on the first trading day after the roll. Expiration is usually the third Friday; if that day is a holiday, expiration moves up a day, so handle that case separately.
4. Remove holidays and half days: days whose last RTH bar is earlier than 13:30 (stock market closed with only futures trading, or an early 13:00 close). Delete the previous evening's overnight session along with them.
5. Remove days whose overnight session spans two or more nights (for example Good Friday or a state funeral, when stocks are closed but futures trade), and the first day after any fully missing day.
6. Write out the cleaned 1-minute data, and also build 5-minute bars in US Eastern time (open = first bar's open, high = max, low = min, close = last bar's close, volume = sum).
7. Finally verify and print: how many trading days are left, whether every day has a complete RTH session (405 one-minute bars), and whether any day's overnight session mixes two contracts.
Please add comments to every step explaining why it's done that way.

(Note: in October 2026 I downloaded the April–September 2026 data as well. The date range and numbers in this post reflect the merged data.)