Skip to main content

Think Stats Exploratory Data Analysis in Python

Allen B. Downey

Section 12.1 Electricity

As an example of time-series data, we’ll use a dataset from the U.S. Energy Information Administration -- it includes total electricity generation per month from renewable sources from 2001 to 2024. Instructions for downloading the data are in the notebook for this chapter.
🔗
The following cell downloads the data, which I downloaded September 17, 2024 from https://www.eia.gov/electricity/data/browser/.
🔗
Listing 12.1.1. Python Code
🔗
filename = "Net_generation_for_all_sectors.csv"
download("https://github.com/AllenDowney/ThinkStats/raw/v3/data/" + filename)
After loading the data, we have to make some transformations to get it into a format that’s easy to work with.
🔗
Listing 12.1.2. Python Code
🔗
elec = (
    pd.read_csv("Net_generation_for_all_sectors.csv", skiprows=4)
    .drop(columns=["units", "source key"])
    .set_index("description")
    .replace("--", np.nan)
    .transpose()
    .astype(float)
)
In the reformatted dataset, each column is a sequence of monthly totals in gigawatt-hours (GWh). Here are the column labels, showing the different sources of electricity, or "sectors".
🔗
Listing 12.1.3. Python Code
🔗
$ elec.columns
Index(['Net generation for all sectors', 'United States',
       'United States : all fuels (utility-scale)', 'United States : nuclear',
       'United States : conventional hydroelectric',
       'United States : other renewables', 'United States : wind',
       'United States : all utility-scale solar', 'United States : geothermal',
       'United States : biomass',
       'United States : hydro-electric pumped storage',
       'United States : all solar',
       'United States : small-scale solar photovoltaic'],
      dtype='object', name='description')
The labels in the index are strings indicating months and years -- here are the first 12.
🔗
Listing 12.1.4. Python Code
🔗
$ elec.index[:12]
Index(['Jan 2001', 'Feb 2001', 'Mar 2001', 'Apr 2001', 'May 2001', 'Jun 2001',
       'Jul 2001', 'Aug 2001', 'Sep 2001', 'Oct 2001', 'Nov 2001', 'Dec 2001'],
      dtype='object')
It will be easier to work with this data if we replace these strings with Pandas Timestamp objects. We can use the date_range function to generate a sequence of Timestamp objects, starting in January 2001 with the frequency code "ME", which stands for "month end", so it fills in the last day of each month.
🔗
Listing 12.1.5. Python Code
🔗
$ elec.index = pd.date_range(start="2001-01", periods=len(elec), freq="ME")
elec.index[:6]
DatetimeIndex(['2001-01-31', '2001-02-28', '2001-03-31', '2001-04-30',
               '2001-05-31', '2001-06-30'],
              dtype='datetime64[ns]', freq='ME')
Now the index is a DataTimeIndex with the data type datetime64[ns], which is defined in NumPy -- 64 means each label uses 64 bits, and ns means it has nanosecond precision.
🔗
🔗
PrevTopNext