- black box testing: HTTP client like
reqwest
- embedded test module
- good for testing private structs
- external tests folder
- good for integration test at identical level of using crate
- doc test
-
Test should be decoupled from app aside of objective target
-
We need to run our App as a background task
- tokio::spawn takes a future and hands it over to the runtime for polling, without waiting for its completion
- port 0: OS scans available port to bind to app
std::net::TCpListener: returns listener with ip, port info
- can try parameterized test with
rstestcrate
- roll-my-own parametrised test stops as soon as one test fail + we don't know which it was
- can extract url-encoded data from req body or send url-encoded data as res
- 10 extractors per handler fn
impl<T> FromRequest for Form<T>
where
T: DeserializeOwned + 'static,
{
type Error = actix_web::Error;
async fn from_request(req: &HttpRequest, payload: &mut Payload) -> Result<Self, Self::Error> {
// Omitted stuff around extractor configuration (e.g. payload size limits)
match UrlEncoded::new(req, payload).await {
Ok(item) => Ok(Form(item)),
// The error handler can be customised.
// The default one will return a 400, which is what we want. Err(e) => Err(error_handler(e))
}
}
}UrlEncodeddoes serde
serde_urlencoded::from_bytes::<T>(&body).map_err(|_| UrlencodedError::Parse)- thanks to
monomorphizationeven with generics, serde is not slow - all information required to ser/de a specific type are available at
compile_time - Josh Mcguigan:
Understanding Serde
#[derive(serde::Deserialize)] pub struct FormData {
email: String,
name: String,
}
// Let's start simple: we always return a 200 OK
async fn subscribe(_form: web::Form<FormData>) -> HttpResponse {
HttpResponse::Ok().finish()
}- before calling
subscribe,Form::from_requestdeserialize body into FormData - if
Form::from_requestfails, 400 BAD REQUEST, else 200 OK
- Postgres: exhaustive docs, easy to run locally and CI via Docker, well-supported within the Rust ecosystem
- crates
- tokio-postgres
- sqlx
- diesel
- When do we realize mistake?
- disel and sqlx detect most of mistakes at compile-time
- disel: CLI rust code gen
- sqlx: proc macro to connect DB at compile-time + query validation
- disel and sqlx detect most of mistakes at compile-time
- disel support query builder (DSL)
- Threads are for working in parallel, Async is for waiting in parallel
- sqlx, tokio-postgres support async
- disel supports only sync
| Crate | Compile-time safety | Query interface | Async |
|---|---|---|---|
| tokio-postgres | N | SQL | Y |
| sqlx(chosen) | Y | SQL | Y |
| diesel | Y | DSL | N |
cargo install --version="~0.6" sqlx-cli --no-default-features --features rustls,postgres
- init_db.sh
- Cargo.toml
[dependencies.sqlx]
version = "0.6"
default-features = false
features = [
"runtime-tokio-rustls",
"macros",
"postgres",
"uuid",
"chrono",
"migrate",
]- PgConnection
- organize files
src/
configuration.rs
lib.rs
main.rs
routes/
mod.rs
health_check.rs
subscriptions.rs
startup.rs
// src/configuration.rs
pub fn get_configuration() -> Result<Settings, config::ConfigError> {
let settings = config::Config::builder()
.add_source(config::File::new("configuration", config::FileFormat::Yaml))
.build()?;
settings.try_deserialize::<Settings>()
}- web::Data wraps connection in
Atomic Reference Counted pointer
- the
web::Data<T>cast any value to the typeT(equivalent to dependency injection)
// src/routes/subscriptions.rs
pub async fn subscribe(
_form: web::Form<FormData>,
_connection: web::Data<PgConnection>,- replace PgConnection to PgPool for sharing mut ref
// src/main.rs
let connection_pool = PgPool::connect(&configuration.database.connection_string())
.await
.expect("Failed to connect to Postgres.");
..
// src/startup.rs
pub fn run(listener: TcpListener, db_pool: PgPool) -> Result<Server, std::io::Error> {
let db_pool = web::Data::new(db_pool);
let server = HttpServer::new(move || {
App::new()
.route("/health_check", web::get().to(health_check))
.route("/subscriptions", web::post().to(subscribe))
.app_data(db_pool.clone())
..
// src/routes/subscriptions.rs
pub async fn subscribe(form: web::Form<FormData>, pool: web::Data<PgPool>) -> HttpResponse {
sqlx::query!(..)
.execute(pool.get_ref())
.await;async fn spawn_app() -> TestApp {
let listener = TcpListener::bind("127.0.0.1:0").expect("Failed to bind random port");
let port = listener.local_addr().unwrap().port();
let address = format!("http://127.0.0.1:{}", port);
let configuration = get_configuration().expect("Failed to read configuration.");
let connection_pool = PgPool::connect(&configuration.database.connection_string())
.await
.expect("Failed to connect to Postgres.");
let server = run(listener, connection_pool.clone()).expect("Failed to bind address");
let _ = tokio::spawn(server);
TestApp {
address,
db_pool: connection_pool,
}
}- Solutions
- wrap the whole test in a SQL transaction and rollback at the end of it
- no way to capture that connectino in a SQL tx context
- spin up a brand-new logical database for each integration test
- slower but easier
- create a new logical db with a unique name
- run db migrations on it.
// tests/health_check.rs
configuration.database.database_name = uuid::Uuid::new_v4().to_string();
let connection_pool = configure_database(&configuration.database).await;
..
async fn configure_database(config: &zero2prod::configuration::DatabaseSettings) -> PgPool {
let mut connection = PgConnection::connect(&config.connection_string_without_db())
.await
.expect("Failed to connect to Postgres.");
connection
.execute(format!(r#"CREATE DATABASE "{}";"#, config.database_name).as_str())
.await
.expect("Failed to create database.");
let connection_pool = PgPool::connect(&config.connection_string())
.await
.expect("Failed to connect to Postgres.");
sqlx::migrate!("./migrations")
.run(&connection_pool)
.await
.expect("Failed to migrate the database.");
connection_pool
}- what happens if we lose connection to the database?
- Does sqlx::PgPool try to automatically recover?
- what happens if an attacker tries to pass malicious payloads in the body of the POST like large payloads or SQL injection
- Sometimes experience is enough to transform an unknown unknown into a known unknown
- impossible to reproduce outside of live environment
- the system is pushed outside of its usual operating conditions
- multiple components experience failures at the same time
- a change is introduced that moves the system equilibrium(e.g. tuning a retry policy)
- no changes have been introduced for a long time (e.g. app have not been restarted for weeks and memory leaks)
- Observability is about being able to ask arbitrary questions about your environment
without having to know ahead of time what you wanted to ask
- instrument our app to collect high-quality telemetry data
- access to tools and systems to efficiently slice, dice and manipulate the data to find answers to our questions
- trace: lowest level, extremely verbose
- debug
- info
- warn
- error: serious failures that might have user impact
fn fallible_operation() -> Result<String, String> { ... }
pub fn main() {
match fallible_operation() {
Ok(success) => {
log::info!("Operation succeeded: {}", success);
}
Err(err) => {
log::error!("Operation failed: {}", err);
}
}
}// src/routes/startup.rs
let server = HttpServer::new(move || {
App::new()
.wrap(Logger::default())
..- global decision that app are uniquely positioned to do =>
logcrate- file, print, send to remote over HTTP(e.g. ElasticSearch)
- it provides Log trait instead of how to record log
pub trait Log: Sync + Sned {
fn enabled(&self, metadata: &Metadata) -> bool;
fn log(&self, record: &Record);
fn flush(&self);
}- should call
set_loggerat main to use log records => useenv_logger env_loggerto print all log records to terminal- format:
[<timestamp> <level> <module path>] <log message>
- format:
# Cargo.toml
[dependencies]
env_logger = "0.9"// src/main.rs
async fn main() -> std::io::Result<()> {
// `init` does call `set_logger`, so this is all we need to do.
// We are falling back to printing all logs at info-level or above
// if the RUST_LOG environment variable has not been set.
env_logger::Builder::from_env(Env::default().default_filter_or("info")).init();- print trace-level logs (default: RUST_LOG=info)
RUST_LOG=trace cargo run- add log as a dependency
#! Cargo.toml
[dependencies]
log = "0.4"- success => log::info!()
- failure => log::error!()
- make it observable like customer is reporting by email
- our log should include id (email) info
- add UUID to each log::info!
- rewrite all upstream components in the req processing pipeline
- change the sign of all downstream fn.s calling from subscribe handler
- each sub-routines has
- duration
- context
- trying to represent tree-like processing pipeline
- Logs are the wrong abstraction
- expand upon logging-style diag with additional info
let request_span = tracing::info_span!(
"Adding a new subscriber",
%request_id,
subscriber_email = %form.email,
subscriber_name = %form.name
);
let _request_span_guard = request_span.enter();- info_span! gets multiple arguments => structured info
%prefix: implementDisplayfor logging.enter(): as long not dropped, all downstream spans and log events will be registered as children- like Rust' RAII(Resource Acquisition Is Initialization)
- tracing sign
->: enter the span<-: exit the span--: close the span
//! src/routes/subscriptions.rs
use tracing::Instrument;
// ..
let query_span = tracing::info_span!("Saving new subscriber details in the database");
match sqlx::query!(/* */)
.execute(pool.get_ref())
.instrument(query_span)
.awaitRUST_LOG=trace cargo run+curl -i -X POST -d 'email=test@hotmail.com&name=tester' http://127.0.0.1:8000/subscriptions
[2023-03-21T13:36:51Z TRACE tracing::span::active] -> Saving new subscriber details in the database;
[2023-03-21T13:36:51Z INFO sqlx::query] INSERT INTO subscriptions (id, …; rows affected: 1, rows returned: 0, elapsed: 747.553ms
INSERT INTO
subscriptions (id, email, name, subscribed_at)
VALUES
($1, $2, $3, $4)
[2023-03-21T13:36:51Z TRACE tracing::span::active] <- Saving new subscriber details in the database;
[2023-03-21T13:36:51Z TRACE tracing::span] -- Saving new subscriber details in the database;
[2023-03-21T13:36:51Z TRACE tracing::span::active] <- Adding a new subscriber;
[2023-03-21T13:36:51Z TRACE tracing::span] -- Adding a new subscriber;
- env_logger -> tracing-subscriber
# Cargo.toml
tracing-subscriber = { version = "0.3", features = ["registry", "env-filter"] }- Layer: build a processing pipeline for spans data
- Registry: collects and stores span data exposed to any layer wrapping it
- tracing_subscriber::filter::EnvFilter -> discards spans based on log levels
- tracing_bunyan_formatter::JsonStorageLayer -> processes spans data and stores the associated metadata in JSON; propagate context from parent to children
- tracing_bunyan_formatter::BunyanFormatterLayer -> builds on top of JsonStorageLayer and outputs log records in bunyan-compatible JSON format
# Cargo.toml
tracing-bunyan-formatter = "0.3"- it provides duration:
elapsed_millisecond
//! src/main.rs
let env_filter = EnvFilter::try_from_default_env().unwrap_or(EnvFilter::new("info"));
let formatting_layer = BunyanFormattingLayer::new("zero2prod".into(), std::io::stdout);
let subscriber = Registry::default()
.with(env_filter)
.with(JsonStorageLayer)
.with(formatting_layer);
set_global_default(subscriber).expect("Failed to set subscriber.");- record every time a tracing event happens
# Cargo.toml
tracing-log = "0.1"//! src/main.rs
LogTracer::init().expect("Failed to set logger.");- cargo-udeps (Unused Dependencies) automatically remove redundant crates
cargo install cargo-udeps
cargo +nightly udeps