A middleware cannot decide with the request's services, wait on a store, or answer a request with a status #1281

The problem

Middleware::create is synchronous, is given no services, and is asked inside the synchronous streaming paths the Fastly adapter needs. A middleware whose decision needs the key-value store or the network cannot decide there. An action can only change a body, so a middleware cannot answer a request with a status, which a well known file made from a service's answer needs when it has nothing to give, because an empty robots.txt tells crawlers to crawl everything.

The change

A middleware may answer Middleware::prepare with a future. The request's handler runs it before the document is processed, as the middleware's module with the permissions it declares through required_permissions, and what it answers, a Send value, is kept under the middleware's name for create to take. A preparation may answer a Refusal instead, with a status, an optional Retry-After and a message, and the request is answered with that status, no document is processed, and nothing is stored or served from the store.

fn prepare<'a>(&'a self, call: ModuleCall<'a>, context: &'a MiddlewareContext<'a>) -> Option<Preparation<'a>> {
    Some(Box::pin(async move {
        let lines = self.lines_for(call, context.request_host).await?;
        Ok(Box::new(lines) as Box<dyn Any + Send>)
    }))
}

fn create(&self, context: &MiddlewareContext<'_>) -> MiddlewareAction {
    let lines: Vec<String> = context.prepared.take(MODULE).unwrap_or_default();
    MiddlewareAction { stream: Some(Box::new(AppendLines::new(lines))), ..MiddlewareAction::pass() }
}

A middleware that prepares nothing is asked create as before, and nothing of it changes. A preparation runs once per phase the document goes through, fetch and serve on a document fetched for the request, serve alone on a copy read from the store, in the order the entry names the middleware. The first refusal ends the preparations of a document, and its response is marked private and unstorable. In the fetch phase a preparation must not depend on the reader, as create must not, because its answer may be stored with the document. Only the answer travels to the streaming paths, so create, the plan and every streaming path are as they were.

Where it is

Draft on the fork, jwrosewell#18, on top of the creating rule (fork pull request 17). robots.txt as a middleware on this interface is a later change in the stack.