CredentialStore Backed Expression Resolution

The following is a link to an early discussion on the wildfly-dev mailing list.

Expression Resolver Discussion

[ ] Engineering

[x] QE

Within the WildFly Elytron project new utilities will be added to handle the generation of SecretKey instances and the encryption and decryption of clear text values. These utilities will support transfer of the values using versioned Base64 encoded tokens which will be described in more detail below.

Within WildFly Core the general ExpressionResolver support will be updated to detect the availability of an expression resolver capability exposing a runtime API for expression resolution.

Also within WildFly Core the Elytron subsystem will be updated with new resources as required, resources will also contain operations both for key management and operations for the encryption of clear text strings for use as expressions in the management model.

Additionally all documentation lives within the WildFly project so updates will be applied there accordingly.

As a core management feature this enhancement will be applicable to all installation types.

The overall intent of this enhancement is for the Elytron subsystem to provide a configurable expression resolver resource to make use of an AES SecretKey loaded from a CredentialStore to decrypt expressions in the management model which have been previously encrypted using the same key. To meet this requirement some additional functionality will also be required to hande the management of the keys.

It is a requirement within this enhancement that multiple secret keys can be used for the decryption of expressions, these keys could be loaded from the same credential store or from separate credential stores - this relates to the "chicken and egg" problem which will be described later.

The initial integration of this enhancement will not support specifying custom java.security.Provider instances as the key type and transformation will be using pre-defined defaults - however the expression resolver will be initialised after the Elytron subsystem has performed initialisation and registration of the providers.

This enhancement will exclusively make use of AES javax.crypt.SecretKey instances, although it is possible to avoid the use of a javax.crypto.SecretKeyFactory as the keys can be generated using random key material directly using the javax.crypto.spec.SecretKeySpec class we should still use the SecretKeyFactory in case certified providers are added for key generation. The generated SecretKey will be stored in a CredentialStore.

Presently WildFly Core supports an expression resolver that can delegate to a Vault configuration and if that is not available fall back to use either system properties or environment variables. This will be updated to make use of the CapabilityRegistry and attempt to lookup a capability using a predefined constant org.wildfly.controller.expression-resolver which exposes an expression resolver runtime API. The expression resolver looked up using a capability will be used after first attempting to resolve the expression as a Vault expression to resolve any expression.

A new singleton resource will be added to the elytron subsystem called expression=encryption the purpose of this resource is to contain the configuration both for the encryption and decryption of values. As a singleton we know only one instance of this resource can be defined within the subsystem reducing some of the complexity multiple instances would cause. The resource will contain the definition of one or more resolvers to decrypt the inline expressions.

The expression=encryption resource will register it’s expression resolver capability with the CapabilityRegistry making it available for runtime resolution of expressions.

The expression=encryption resource will contain three attributes, the first attribute being resolvers which will be used to define one or more resolvers to handle the encryption and decryption of values. Each resolver will support the following configuration:

The format of the expressions resolved using the WildFly Elytron expression resolver will be ${ENC:Resolver:ENCRYPTED_DATA} where Resolver is a reference to the specific resolver defined in the expression=encryption resource and ENCRYPTED_DATA is the data to be decrypted encoded using Base64.

The second attribute on the expression=encryption resource will be default-resolver which will be used to optionally specify which resolver is the default, the expression can then be simplified to ${ENC:ENCRYPTED_DATA}.

As expressions are already widely in use with the application server there is a small possibility that users may have already defined expressions with a prefix of ENC: expecting it to be resolved as a system property. The expression=encryption resource will also have a third attribute prefix attribute which will allow an alternative prefix to ENC to be specified.

TODO - I need to double check if the format should be ENC: or ENC:: the former may be easier to accidentially interpret as a system property with a default value.

To simplify the first iteration of this enhancement the only supported transformation will be AES/CBC/PKCS5Padding, this has been selected as one of the transformations all JVMs are required to support. Additionally as the values to be encryopted will be of varying lengths padding is required as the values may not fit neatly into the block size for AES encryption.

Additional requirements such as the generation of an initialisation vector will be handled by the underlying Cipher implementation, different providers may choose to handle this differently so we will not second guess their requirements.

Although the individual resources are not supporting references to custom security providers it should be possibly for alternatives such as BouncyCastle to be registered in the JVM and used.

A common problem when working with application servers is the desire to protect any credentials contained within the configuration, however an application server is often installed in such a way that enables it to be run without requiring direct user interaction. This means that when values are protected the application server installation needs access to everything at once including any secrets to decrypt or access encrypted values.

In the past password based encryption has been used to add some protection, however this still needs an initial secret which would be hard coded into the source code of an open source project as somearbitrarycrazystringthatdoesnotmatter which is easily accessible to anyone.

The use of password based encryption with a public secret offers some protection in that someone glancing at a configuration file would find it harder to remember the Base64 representation but as the secret is public if anyone is able to capture the masked password they could take it away and decrypt at their leisure. Overall the encryption is not offering anything more than Base64 encoding with some additional padding could offer. Password based encryption could be updated so that the secret can also be specified in the configuration but this still leaves the situation that on gaining access to the configuration all of the required information is still present to decrypt the values.

The additional resources in this RFE still suffer from similar issues but also offer alternative options for some mitigation.

The new secret-key-credential-store offers a mechanism that the initial secret used in the configuration can be handled completely independently of both the configuration and source. Should a malicious actor gain access to all or part of the configuration file they will still not have access to enough information to decrypt the inlined expressions even with access to the source.

By moving the initial secret into it’s own credential store this does provide an opportunity for the filesystem level access permissions to be defined independently, ideally the only account which should be able to access the file is the account the application server uses to run. No other users which can access the system should have access to the file containing the credential store.

Of course if a user is able to access both the configuration and the credential store they will have sufficient information to decrypt the values within the encrypted expressions. A further mitigation could be to only use this secret key to protect the password to a second credential store, this second credential store could in turn contain the secret key for the remaining expression in the management model. If this second credential store was backed by a hardware security module it would make it a lot harder for a malicious actor to decrypt the values in the configuration themselves.

These nice to have requirements are outside the scope of the current enhancement, they are added here based on discussions and thoughts during development to identify further enhancements we could add.

It would also be beneficial to support Public / Private Key Pairs, in this case a public key from the server can be used to encrypt the value leaving it decryptable using the private key, this will have a benefit that giving a user the ability to encrypt a value does not give them the ability to decrypt that value. If we are to support private key encryption the credential store does not presently support the storage of private keys unless they are either paired with their public key or are associated with an X509 certificate - individual private key storage may become desirable.

We should consider deprecating the --add operation for the credential-store command on the Elytron tool and instead adding an --add-password operation to being this in alignment with the operations being used for keys. As passwords are not generated equivalent import / export operations would not be required.

It would be nice to cross reference subsystem managed security providers for the expression resolver, however this component needs to be usable at the start of Stage.RUNTIME so there will be a limit as to how many subsystem managed resources can be depended upon.

A lot of the arguments passed into the command line tool are repeated on each invocation, a configuration file containing these to avoid repetition may be beneficial - I suspect however that may be an independent enhancement.

The import operations could be enhanced to also support the import of a raw SecretKey encoded using Base64, after decoding the Base64 representation we could check if the resulting size is appropriate for a SecretKey as only fixed sizes are supported and we can also detect the missing header.

In domain mode we will not support propagating secret keys ourselves, an administrator will be required to ensure the correct keys are available on each host. As a separate RFE we could consider if it would make sense for an application server to be able to access credentials in the credential store of it’s host controller or even from the central host controller.

This enhancement will not support the retrieval of plain text strings from the credential store, this enhancement is specifically adding support for decrypting reversibly encrypted attribute expression values. Other than the support within the expression resolver to decrypt values at runtime we will not be providing any tooling to decrypt the previously encrypted tokens.

Automatic encryption of attribute values will not be supported via this enhancement, as multiple steps are required that would be better performed within enhancements to the management tooling - each of which would require special consideration based on their own user interfaces. A big risk when using multiple steps is if intermediate representations of the management model are persisted as these intermediate representations could contain the clear text values.

As with other CredentialStore use cases no automatic replication of the store or it’s entries are supported with this enhancement.

This enhancement will not add support for migrating expressions to a different credential, however if support for multiple expression resolvers is added at a later point there may be opportunities to support migration.

This enhancement is only in relation to expression resolution within the application server’s management model - this does not extend to any other descriptors or configuration files.

Expression resolution will only be supported against attributes that already support expression resolution, this enhancement will not perform a review of which attributes support expression resolution and will not be changing any attributes to support expression resolution.

Custom expression resolver implementations are outside of the scope of this RFE, adding custom implementations which potentially reference a credential store could be a future enhancement.

Resolved expressions are a deliberate decision to move values from the model to an alternative expression resolver, it is not possible to determine the expression resolution capabilities of a slave. Where credential store backed expression resolution is in place transformers will not reject sending those expressions to the slave. However as the expression=encryption resource is not supported on the slave that will by itself fail transformation and be rejected.