C#Bot Release 3.0.0.0

Features

Upgrade .NET to version 6

The major change for this release is upgrading .NET to version 6, as well as updating all serverside dependencies to their latest versions.

Some notable library increments

One of the big themes for .NET 6 is performance, with some exciting developments such as EF Core performing 70% faster on industry standard benchmark code when using version 6 compared to 5.

Improvements

  • Updated built in containers to use Node 16 LTS for building the clientside.
  • Added support for JWT access token encryption, with support for either X509 certificates or RSA keys.
  • Switched all JSON serialisation and deserialisation to use System.Text.Json.
    • Global JSON options for API endpoints can be configured in Startup.cs.
  • Created async CSV output formatter that will that can be used to return CSV data from a controller.
    • To use this formatter place the [Produces("text/csv")] attribute on your controller endpoint and return the data you want to be transformed in any IActionResult implementation.
  • Removed GraphQL.EntityFramework library. All operations that were performed by this library are now performed internally by C#Bot. There is no differences to the API with this change.
  • Improved error handling for exceptions that have occurred in GraphQL functions. Errors are now better passed back to the controller to be logged.
  • Change backing GraphQL models for creating users to use the actual model class instead of a registration model DTO. With this changed the entity GraphQL registration models have been removed.
  • GraphQL update queries can now optionally choose which fields to update instead of setting unset fields back to defaults.
  • Moved group names to const strings in a new UserGroups class instead of using magic strings.
  • Moved authorization policy names into const strings in StaticIdentityConstants.
  • Added pruning of expired OpenIddict tokens as a scheduled Hangfire job. This job will run every hour by default.
  • Moved UseOpenIddict from Startup.cs to DbContext.cs.
  • Made internal server class members visible to the serverside test framework.
  • Made export code not have to buffer results in the clientside and instead download directly in the browser.
    • Previously this used a POST request with a body that had conditions inside of it. Now the conditions are stored in a base64 encoded query parameter of a GET request.
  • Ported ByChained class into C#Bot code after it was removed from Selenium.
  • Added a customisable ErrorController to serve an error page for unhanded server exceptions in the production environment.
  • Improvements to form designer UI.
  • TileOptions have been refactored into separate files to make custom options easier to build.
  • Correctly serialise email attachments being sent through background jobs.
  • Added a refresh callback to useAsync hook in clientside.
  • LeavePageConfirmation component added.
  • Two Factor options are now listed in all user CRUD tables.
  • User can now temporarily view their password by hovering over the icon in password fields.
  • Added developer considerations to `SECURITY.MD’ to be more in line with OWASP Standards.
  • Maximum password length set to 64 characters in accordance with OWASP password security requirements.
  • Certain API tests have been moved to Serverside tests for increased performance and less overhead.
    • BatchUpdateTests
    • GraphQLTests
    • InvalidEntityTests
  • Certain Selenium tests have been moved to Jest tests for increased performance and less overhead.
    • ResetPassword.test
    • EntityAttributeList.test

Resolved defects

  • Better error handling for exceptions while in production code. Instead of routing to the clientside with a 200 OK it will now route to an error endpoint with the appropriate status code.
  • Correctly get the claim types when extracting built in claims from the authorization claims principal when logging request data.
  • Prevent resolving singleton services per request when serving the clientside and instead resolve them once on launch.
  • Remove duplicate claims being embedded in ClaimsPrincipals that have been generated from refreshed user cookies.
  • Added some missing nullable contexts.

Migration Path

Npgsql Date Times

The latest version of Npgsql changes how date times are handled by the database. Previously, any model with a DateTime attribute would be stored in the PostgreSQL database using the timestamp type. With this release the the column type has changed to timestamptz instead. The original timestamp type does not contain any timezone information, where the new timestamptz explicitly contains a UTC timestamp. Npgsql will always store data in this column with the UTC timezone.

This is a breaking change if you are storing a DateTime instance in the database that is not representing a UTC datetime. For any columns that are not storing UTC datetimes, you should place a [Column(TypeName="timestamp")] on the database model in your code.

When PostgreSQL updates a database column from timestamp to timestamptz, it will update all the values to be local times in the current timezone but since the old type lacks timezone information, the timezone will be wrong. To safely migrate data that is in UTC format already, perform the following steps.

  • Create a new migration with dotnet ef migrations add Net6
  • In the generated migration add the following code at the top of the Up and Down methods.
    migrationBuilder.Sql("SET TimeZone='UTC';");
    
  • Update the database with dotnet ef database update.

For more information on this change please read the the following documents

And for a detailed look into the topic of storing timestamps, take a look at Storing UTC is not a silver bullet.

Npgsql Trigrams & Fuzzy Matching

The Npgsql.EntityFrameworkCore.PostgreSQL.Trigrams and Npgsql.EntityFrameworkCore.PostgreSQL.FuzzyStringMatch plugins have been integrated into the main Npgsql package. Any dependencies on these projects can be safely ignored.

Adding JWT Encryption Keys

In C#Bot the access token for JWT tokens was sent as plain text. In line with changes to OpenIddict, JWT access tokens are now encrypted by default.

By default the encryption key will be generated in memory. This however will mean that the key will change on server restart, and more importantly, will not be safe in multi tenanted environments. To ensure that the key remains the same we must generate a static key and pass it to the application.

To perform this we will need to generate two 2048 bit RSA keys.

openssl genrsa -out signing.pem 2048
openssl genrsa -out encryption.pem 2048

Now we can embed the keys into our appsettings configuration. In this case the encryption key is embedded directly inside the configuration where the signing key is referencing a file path. Either method can be used for each key, the differences in this scenario are for example purposes only.

	<CertificateSetting>
		<!-- % protected region % [Configure Certificate] on begin -->
		<EncryptionKey>
			<KeyType>RSA</KeyType>
			<Data>
				-----BEGIN RSA PRIVATE KEY-----
				MIIEowIBAAKCAQEAqmRZFe97n17NVad/I/+s3jBadzCy0jtcQydPGlN2P29KWGX9
				...
				MJLzm1VBflsTDtQ5n65zlqU2YJSIhB6X4owx4RiYGtA67UtEDIoG
				-----END RSA PRIVATE KEY-----
			</Data>
		</EncryptionKey>
		<SigningKey>
			<KeyType>RSA</KeyType>
			<FilePath>/path/to/signing.pem</FilePath>
		</SigningKey>
		<JwtBearerAudience>resource-server</JwtBearerAudience>
		<!-- % protected region % [Configure Certificate] end -->
	</CertificateSetting>

Like all appsettings variables, these can be configured from environment variables as well.

CertificateSetting__EncryptionKey__KeyType="RSA"
CertificateSetting__EncryptionKey__Data="-----BEGIN RSA PRIVATE KEY-----...."
CertificateSetting__SigningKey__KeyType="RSA"
CertificateSetting__SigningKey__FilePath="/path/to/signing.pem"

Finally, instead of using RSA keys, X509 certificate files can be used as well. There is no preference to which key type is used. X509 certs can only be provided using file paths and have an optional password field in the case they are password protected.

<EncryptionKey>
	<KeyType>X509</KeyType>
	<FilePath>/path/to/x509/cert.pfx</FilePath>
	<Password>certificate-password-here</Password>
</EncryptionKey>

System.Text.Json API serialisation

With this release we have changed the JSON serialisation and deserialisation for the REST API endpoints from Newtonsoft.Json to System.Text.Json.

All bot written code has been checked to ensure that API functionality remains the same, however custom code will need to be checked as well. For further guidance please see the official migration guide.

The main thing to look out for will be custom serialisation attributes that need to be changed, for example [JsonProperty("MyFieldName")] will need to be changed to [JsonPropertyName("MyFieldName")].

The Newtonsoft.Json library is still included in the project (and used in many places still), so any code that is not involved in serialising or deserialising from the API will remain unaffected.

GraphQL Type Mapping Registration

Previously when adding any custom GraphQL types you needed to register it as a singleton service in Startup.cs and also add an entry to the GraphQL type registry in the same location. With the latest version of the GraphQL.NET library, this has changed so the type registration is now done in the constructor of Schema.cs instead. The singleton service registration remains the same as before however.

Scalar Graph Type Changes

Due to updates to the GraphQL.NET library the handling for scalar value parsing has changed. When creating mutation types, if you have any fields that use the following types, please update them accordingly.

Old Type New Type
FloatGraphType DeserializableFloatGraphType
IntGraphType DeserializableIntGraphType
BooleanGraphType DeserializableBooleanGraphType
DateTimeGraphType DeserializableDateTimeGraphType

ByChained Namespace

The ByChained class has been removed from Selenium and has been introduced into C#Bot code instead. Any class that uses ByChained will need to remove the OpenQA.Selenium.Support.PageObjects import and replace it with an import to SeleniumTests.Utils.