Read CSV To Dictionary Using Pandas In Python

Working with CSV files is a common task in data analysis and manipulation. While Pandas DataFrames are great for most operations, sometimes you need to convert your CSV data into a Python dictionary for specific use cases.

In my decade-plus experience as a Python developer, I’ve found that knowing how to transform CSV data into dictionaries can significantly streamline your workflow.

In this article, I’ll show you multiple methods to read a CSV file into a dictionary using Pandas in Python.

Table of Contents

Read CSV to Dictionary Using Pandas in Python

Now, I will show you the methods to read CSV to a dictionary using Pandas in Python, along with examples.

Read Pandas GroupBy Without Aggregation Function in Python

Method 1: Use pandas.read_csv() and to_dict()

The easiest approach is to use Python Pandas’ built-in methods: read_csv() to load the data and to_dict() to convert it to a dictionary.

import pandas as pd

# Read the CSV file into a DataFrame
df = pd.read_csv('sales_data.csv')

# Convert DataFrame to dictionary
data_dict = df.to_dict()

print(data_dict)

When you run this code with a sales data CSV containing quarterly sales figures for different states, you’ll get a dictionary where each column becomes a key and its values are stored in a nested dictionary with indices as keys.

The output will look something like this:

{
    'State': {0: 'California', 1: 'Texas', 2: 'New York', 3: 'Florida'},
    'Q1_Sales': {0: 45000, 1: 32000, 2: 28000, 3: 30000},
    'Q2_Sales': {0: 52000, 1: 38000, 2: 31000, 3: 42000}
}

You can refer to the screenshot below to see the output:

Check out Python Pandas Write to Excel

Method 2: Customize Dictionary Format with to_dict() Parameters

Python to_dict() method in Pandas is quite flexible. You can specify different orientations to structure your dictionary in various ways.

import pandas as pd

df = pd.read_csv('customer_data.csv')

# Records orientation - list of dictionaries
records_dict = df.to_dict(orient='records')

# List orientation - dictionary of lists
list_dict = df.to_dict(orient='list')

# Index orientation - dictionary with index as outermost key
index_dict = df.to_dict(orient='index')

print("Records orientation:")
print(records_dict)

For a customer dataset with columns like ‘Name’, ‘Age’, and ‘State’, the ‘records’ orientation produces a list of dictionaries, where each dictionary represents a row:

# Records orientation
[
    {'Name': 'John Smith', 'Age': 34, 'State': 'California'},
    {'Name': 'Emily Johnson', 'Age': 28, 'State': 'Texas'},
    {'Name': 'Michael Davis', 'Age': 42, 'State': 'New York'}
]

You can refer to the screenshot below to see the output:

This format is particularly useful when you need to process each record individually.

Read Create Plots Using Pandas crosstab() in Python

Method 3: Use a Specific Column as Dictionary Keys

Often, you’ll want to use a specific column’s values as the keys in your Python dictionary. This is especially useful when you have a unique identifier column.

import pandas as pd

# Read the CSV file
df = pd.read_csv('products.csv')

# Set a specific column as the index
df.set_index('ProductID', inplace=True)

# Convert to dictionary
products_dict = df.to_dict(orient='index')

print(products_dict)

With a product’s CSV containing ‘ProductID’, ‘Name’, and ‘Price’, this will give you:

{
    'P001': {'Name': 'Laptop', 'Price': 899.99},
    'P002': {'Name': 'Smartphone', 'Price': 499.99},
    'P003': {'Name': 'Tablet', 'Price': 299.99}
}

You can refer to the screenshot below to see the output:

This makes it easy to look up product information by ID.

Check out Drop the Header Row of Pandas DataFrame

Method 4: Read CSV Directly into Dictionary using DictReader

For large files or when you don’t need Pandas’ advanced features, you can use Python’s built-in csv.DictReader:

import csv

# Open the CSV file
with open('employee_data.csv', 'r') as file:
    # Create a dictionary reader
    reader = csv.DictReader(file)

    # Convert to list of dictionaries
    employees = list(reader)

print(employees)

For employee data with columns like ‘ID’, ‘Name’, and ‘Department’, you’ll get:

[
    {'ID': '1001', 'Name': 'Alice Williams', 'Department': 'Marketing'},
    {'ID': '1002', 'Name': 'Bob Jones', 'Department': 'Engineering'},
    {'ID': '1003', 'Name': 'Carol Brown', 'Department': 'Finance'}
]

This method is more memory-efficient for very large files as it doesn’t load the entire dataset into memory at once.

Method 5: Convert CSV to Dictionary with Custom Processing

Sometimes you need to apply transformations while converting CSV data to a dictionary in Python:

import pandas as pd

# Read CSV file
df = pd.read_csv('sales_tax.csv')

# Process data during conversion
sales_tax_dict = {}
for index, row in df.iterrows():
    state = row['State']
    # Convert percentage string to float
    tax_rate = float(row['TaxRate'].strip('%')) / 100
    sales_tax_dict[state] = tax_rate

print(sales_tax_dict)

This example reads state sales tax data and converts percentage strings like “7.25%” to float values (0.0725):

{
    'California': 0.0725,
    'Texas': 0.0625,
    'New York': 0.0400,
    'Florida': 0.0600
}

The custom processing approach is versatile when you need to transform data during the conversion process.

Handle Large CSV Files

When working with large CSV files, you can read the file in chunks using pandas to manage memory usage:

import pandas as pd

# Initialize an empty dictionary
big_data_dict = {}

# Read the file in chunks
for chunk in pd.read_csv('large_dataset.csv', chunksize=10000):
    # Process each chunk and update the dictionary
    for index, row in chunk.iterrows():
        # Using the ID column as the key
        big_data_dict[row['ID']] = row.to_dict()

This approach allows you to process millions of rows without running out of memory.

Practical Application: Data Analysis with Dictionary Structure

Let’s look at a real-world example where converting a CSV to a dictionary is particularly useful:

import pandas as pd

# Read US stock market data
df = pd.read_csv('stock_data.csv')

# Organize by stock symbol
df.set_index('Symbol', inplace=True)
stocks_dict = df.to_dict(orient='index')

# Calculate average price
for symbol, data in stocks_dict.items():
    avg_price = (data['Open'] + data['Close']) / 2
    stocks_dict[symbol]['AvgPrice'] = avg_price

print(stocks_dict['AAPL'])

This example reads stock market data, organizes it by stock symbol, and calculates the average price for each stock.

Converting Dictionary Back to CSV

After processing your dictionary, you might want to save it back to a CSV file:

import pandas as pd

# Convert dictionary back to DataFrame
processed_df = pd.DataFrame.from_dict(stocks_dict, orient='index')

# Save to CSV
processed_df.to_csv('processed_stocks.csv')

This creates a clean workflow for reading, processing, and saving data.

In this tutorial, I have explained methods to read CSV to a dictionary using Pandas in Python: use pandas.read_csv() and to_dict(), customize dictionary format with to_dict() parameters, specific column as dictionary keys, read CSV directly into dictionary using DictReader, and convert CSV to dictionary with custom processing.