Skip to content

sqlite: add serialize() and deserialize() to DatabaseSync#62579

Open
thisalihassan wants to merge 1 commit intonodejs:mainfrom
thisalihassan:sqlite-serialize-deserialize
Open

sqlite: add serialize() and deserialize() to DatabaseSync#62579
thisalihassan wants to merge 1 commit intonodejs:mainfrom
thisalihassan:sqlite-serialize-deserialize

Conversation

@thisalihassan
Copy link
Copy Markdown
Contributor

@thisalihassan thisalihassan commented Apr 3, 2026

Add serialize() and .deserialize() methods to DatabaseSync, wrapping the sqlite3_serialize and sqlite3_deserialize. These allow extracting an in-memory database as a Uint8Array and loading one back enabling snapshots, cloning, etc..

Implemenation Notes:

serialize() wraps the SQLite allocated buffer directly in a V8 BackingStore to avoid copying the entire database. #ifdef V8_ENABLE_SANDBOX falls back to a copy path since that configuration forbids external backing stores. deserialize() must copy once because SQLite requires sqlite3_malloc64allocated memory.

Wrote test cases with Claude & GPT 5.4

Refs: #62575

@nodejs-github-bot
Copy link
Copy Markdown
Collaborator

Review requested:

  • @nodejs/sqlite

@nodejs-github-bot nodejs-github-bot added c++ Issues and PRs that require attention from people who are familiar with C++. needs-ci PRs that need a full CI run. sqlite Issues and PRs related to the SQLite subsystem. labels Apr 3, 2026
Add database.serialize() and database.deserialize() methods to
DatabaseSync, wrapping the sqlite3_serialize and sqlite3_deserialize
C APIs. These allow extracting an in-memory database as a Uint8Array
and loading one back, enabling snapshots, cloning, and transfer of
databases without filesystem I/O.

Refs: nodejs#62575
@thisalihassan thisalihassan force-pushed the sqlite-serialize-deserialize branch from 9b9212f to 52f29ac Compare April 3, 2026 22:28
}

unsigned char* buf =
static_cast<unsigned char*>(sqlite3_malloc64(byte_length));
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where is this freed?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sqlite3_deserialize() takes ownership of the buffer when passed SQLITE_DESERIALIZE_FREEONCLOSE

sqlite3_deserialize docs:
"If the SQLITE_DESERIALIZE_FREEONCLOSE bit is set in F, then SQLite will invoke sqlite3_free() on the serialization buffer when the database connection closes. If the SQLITE_DESERIALIZE_RESIZEABLE bit is set, then SQLite will try to increase the buffer size using sqlite3_realloc64() if writes on the database cause it to grow larger than M bytes."

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A comment to that effect would be helpful.

@codecov
Copy link
Copy Markdown

codecov bot commented Apr 3, 2026

Codecov Report

❌ Patch coverage is 75.30864% with 20 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.70%. Comparing base (e419afd) to head (52f29ac).
⚠️ Report is 8 commits behind head on main.

Files with missing lines Patch % Lines
src/node_sqlite.cc 75.30% 11 Missing and 9 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #62579      +/-   ##
==========================================
- Coverage   89.71%   89.70%   -0.01%     
==========================================
  Files         695      695              
  Lines      214441   214522      +81     
  Branches    41062    41077      +15     
==========================================
+ Hits       192384   192438      +54     
+ Misses      14123    14122       -1     
- Partials     7934     7962      +28     
Files with missing lines Coverage Δ
src/node_sqlite.h 80.64% <ø> (ø)
src/node_sqlite.cc 80.68% <75.30%> (-0.14%) ⬇️

... and 40 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

c++ Issues and PRs that require attention from people who are familiar with C++. needs-ci PRs that need a full CI run. sqlite Issues and PRs related to the SQLite subsystem.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants