It would be nice if DefaultAuthorizationManagerFactory could apply authorization rules automatically, like for certain authorities that are always required.
@Bean 
AuthorizationManagerFactory<Object> authorizationManagerFactory() {
    return DefaultAuthorizationManagerFactory.withAuthorities("FACTOR_PASSWORD", "FACTOR_X509");
}
These would then be applied to all authorization managers relating to authenticated users. That is, permitAll, denyAll, and anonymous are not affected.
Note, given https://github.com/spring-projects/spring-security/issues/17932, I've updated the suggested static factory method to avoid a collision.
Comment From: therepanic
Hi, @jzheaux. May I attempt to resolve this issue?
Comment From: rwinch
@therepanic Thank you for volunteering. The issue is yours
I think that this can be done now by having DefaultAuthorizationManagerFactory combine AllAuthoritiesAuthorizationManager and the existing AuthorizationManager using AuthorizationManagers.allOf(new AuthorizationDecision(false), AuthorizationManager...)
Comment From: rwinch
@therepanic Sorry I realized that it was ready to go as I wasn't thinking correctly. Please see update above