AI-powered search and chat for the LessWrong forum

The dataset is a CSV file containing all text & embeddings used.

Download it here.

Search was created with OpenAI Embeddings (text-embedding-ada-002).

First, we loop over the essays and generate embeddings for each chunk of text.

Then in the app we take the user's search query, generate an embedding, and use the result to find the most similar essays from the forum.

The comparison is done using cosine similarity across our database of vectors.

Our database is a Postgres database with the pgvector extension hosted on Supabase.

Results are ranked by similarity score and returned to the user.

Chat builds on top of search. It uses search results to create a prompt that is fed into GPT-3.5-turbo.

This allows for a chat-like experience where the user can ask questions about the forum and get answers.

Here's a quick overview of how to run it locally.

1. Set up OpenAI

You'll need an OpenAI API key to generate embeddings.

2. Set up Supabase and create a database

Note: You don't have to use Supabase. Use whatever method you prefer to store your data. But I like Supabase and think it's easy to use.

There is a schema.sql file in the root of the repo that you can use to set up the database.

Run that in the SQL editor in Supabase as directed.

I recommend turning on Row Level Security and setting up a service role to use with the app.

3. Clone repo

git clone https://github.com/bryantanjw/lesswrong-gpt.git

4. Install dependencies

npm i

5. Set up environment variables

Create a .env.local file in the root of the repo with the following variables:

OPENAI_API_KEY=

NEXT_PUBLIC_SUPABASE_URL=
SUPABASE_SERVICE_ROLE_KEY=

6. Run scraping script

npm run scrape

This pulls the top 5000 posts from the LessWrong website, using their GraphQL endpoint, and saves them to a json file. For simplicity and cost, this project only later embeds the top 300 posts.

7. Run embedding script

npm run embed

This reads the json file, generates embeddings for each chunk of text, and saves the results to your database.

There is a 100ms delay between each request to avoid rate limiting.

This process will take 2-3 hours.

8. Run app

npm run dev

This project was inspired by McKay Wrigley's Wait But Why GPT