http://blog.peterritchie.com/Peter Ritchie's Blog20232023-11-30T18:55:01ZPeter Ritchiehttp://blog.peterritchie.com/posts/contributing-to-many-git-reposA tool to help contributing to many Git repos2023-11-30T00:00:00Z<p><img src="../assets/contributing-to-many-git-repos.jpg" class="img-fluid" alt="Source code from many sources" /></p>
<p>I've contributed to many Git repos over the years. Doing this means I work in a code base for a little while, switch to another, and often eventually switch back.</p>
<h2 id="collaborating-with-others">Collaborating with Others</h2>
<p>In the repos that I work in, many have multiple contributors. The contributions to those repos can be prolific, and if the repo is using a workflow that uses feature or topic branches, branches come and go quite often. <code>git fetch</code> by default (or with no other options) gets all branches so you'll have other team members' branches after a fetch--which can be used to do a deep dive on a PR.</p>
<table class="table">
<thead>
<tr>
<th>You could choose not to use the <code>git fetch</code> defaults and have it only get a particular branch. This can typically be done with <code>git fetch origin main</code> (depending on how you've named your remotes and your branches.)</th>
</tr>
</thead>
</table>
<p>I work with many organizations and rarely is there one repo (yes, I know, there's this thing called a "monorepo"; but I find that organizations that can make this work need to be very technically savy, with products/technologies geared towards developers, and only a few of the organizations I work with are at that level.) With remote work being what it is (I'm often working at a different time than other contributors), when I return to work with an organization's code, I usually need to update several repos.</p>
<p>|Why not do a <code>git pull</code> instead of <code>git fetch</code>?|
|:-:|
What I'm contributing to, what I may be reviewing, and whether I'm connected, are variable enough that I've built a habit only to pull when I'm ready to merge and deal with potential conflicts. If I have conflicts, I must resolve them (or abort: <code>git merge --abort</code> or <code>git fetch origin</code> and <code>git reset --hard origin</code>) before doing anything else. This means I must commit to resolving those conflicts before switching to another branch to review or work with it. (Yes, I could re-clone in a different place, but frequent-fetch>abort>clone in terms of effort and risk.)}</p>
<h1 id="a-tool-to-help">A tool to help</h1>
<p>When I re-start work (or maybe I'm coming off a vacation), going to each repo dir to perform <code>git fetch</code> is tedious. I've developed a Powershell script to do that. I'll walk through the script after the code (commented code available <a href="https://github.com/peteraritchie/pri.powershell/blob/main/git/fetch-all.ps1">here</a>.)</p>
<pre><code class="language-powershell">using namespace System.IO;
param (
[switch]$WhatIf,
[switch]$Verbose,
[switch]$Quiet
)
$currentDir = (get-location).Path;
if($Verbose.IsPresent) {
$VerbosePreference = "Continue";
}
function Build-Command {
$expression = 'git fetch';
if($Quiet.IsPresent) {
$expression += ' -q';
}
if($Verbose.IsPresent) {
$expression += ' -v --progress';
}
if($WhatIf.IsPresent) {
$expression += ' --dry-run';
}
$expression += " origin";
return $expression;
}
foreach($item in $([Directory]::GetDirectories($currentDir, '.git', [SearchOption]::AllDirectories);)) {
$dir = get-item -Force $item;
Push-Location $dir.Parent;
try {
Write-Verbose "fetching in $((Get-Location).Path)...";
$expression = Build-Command;
Invoke-expression $expression;
} finally {
Pop-Location;
}
}
</code></pre>
<p>First, I'm translating the PowerShell idioms <code>WhatIf</code>, <code>Verbose</code>, and <code>Quiet</code> to common Git options <code>--dry-run</code>, <code>--verbose</code> (<code>-v</code>), and <code>--quiet</code> (<code>-q</code>). The <code>Build-Command</code> builds up the expression we want to use to invoke git. I've included the <code>--progress</code> option with <code>git fetch</code> to display progress when<code>-Verbose</code> is specified. Next, I'm looping through all directories, looking for a <code>.git</code> directory. I'm using <code>System.IO.GetDirectories</code> instead of <code>Get-ChildItem</code> because it's much faster. For each directory that contains a <code>.git</code> subdirectory, Git <code>fetch</code> is invoked. This allows me to fetch several Git repos within the hierarchy of the current directory.</p>
<table class="table">
<thead>
<tr>
<th style="text-align: center;">Organizaing Code Locally</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: center;">I work with my code (spikes, libraries, experiments, etc.), open-source projects, and multiple clients. All these diverge from one another at one level in my directory structure. e.g. I may have a <code>src</code> subdiretory in my home directory; and <code>oss</code>, <code>experiments</code>, and <code>client</code> subdirectories within <code>src</code>, so I can choose to fetch from all the repos recursively in each of those subdirectories--if I'm returning to work on an OSS project after being away from OSS for a while, I just <code>fetch-all.ps</code> within the <code>oss</code> subdirectory.</td>
</tr>
</tbody>
</table>
<p>By default (or with no other options), <code>git fetch</code> does not delete corresponding local branches that have been removed from a remote. So, new branches will be downloaded, but those that were removed will remain.</p>
<table class="table">
<thead>
<tr>
<th>To also remove local branches removed from the remote, you can include a purge option with <code>git fetch</code>: <code>git fetch --prune</code> or <code>git fetch -p</code>.</th>
</tr>
</thead>
</table>
<p>If I'm reviewing a PR, I don't necessarily want removed remote branches to be removed locally <em>all the time</em>. So, I like pruning separately from fetching. The following is the script for that (other than Build-Command, it has the same structure and flow as <code>fetch-all.ps1</code> (so I won't walk through this snippet.)</p>
<pre><code class="language-PowerShell">using namespace System.IO;
param (
[switch]$WhatIf,
[switch]$Verbose
)
$currentDir = (get-location).Path;
if($Verbose.IsPresent) {
$VerbosePreference = "Continue";
}
function Build-Command {
$expression = 'git remote';
if($Verbose.IsPresent) {
$expression += ' -v';
}
$expression += ' prune';
if($WhatIf.IsPresent) {
$expression += ' --dry-run';
}
$expression += ' origin';
return $expression;
}
foreach($item in $([Directory]::GetDirectories($currentDir, '.git', [SearchOption]::AllDirectories);)) {
$dir = get-item -Force $item;
Push-Location $dir.Parent;
try {
Write-Verbose "pruning in $((Get-Location).Path)...";
$expression = Build-Command;
Invoke-expression $expression;
} finally {
Pop-Location;
}
}
</code></pre>
<p>Separating pruning from fetching also allows me to prune at a wider scope than fetching. e.g. <code>c:\Users\peter\src\client .\fetch-all.ps1</code> and <code>c:\Users\peter\src .prune-all.ps1</code>.</p>
<p>I look forward to your feedback and comments.</p>
<p><img src="../assets/contributing-to-many-git-repos.jpg" class="img-fluid" alt="Source code from many sources"></p>http://blog.peterritchie.com/posts/entity-framework-in-aspireEntity Framework in .NET Aspire2023-11-29T00:00:00Z<p><img src="../assets/entity-framework-in-aspire.jpg" class="img-fluid" alt="A path through the infrastructure" /></p>
<blockquote class="blockquote">
<p>.NET Aspire is an opinionated, cloud ready stack for building observable, production ready, distributed applications.</p>
</blockquote>
<p><a href="https://learn.microsoft.com/en-us/dotnet/aspire/get-started/aspire-overview">.NET Aspire</a> is currently in preview and focuses on <em>simplifying the developer experience</em> with <em>orchestration</em> and <em>automatic service discovery</em> features. There's a huge potential for .NET Aspire beyond this initial valuable feature set.</p>
<p>Being in preview, .NET Aspire may not yet support all the scenarios or workloads you may be comfortable with. It's an opinionated framework, which means differences of opinion are natural and expected. Currently, one of those opinions seems to be a focus on containers. The sample solutions that the new <code>dotnet</code> templates provide are a great example of the benefits of containerization. The .NET Aspire starter solution that <code>dotnet new --use-redis-cache --output AspireStarter</code> generates, out of the box, is something that, when debugged, will download, run, and utilize a Docker Redis image. (I've worked with teams where getting each member productive in a development environment has ended up being days of work.) The AppHost component of a .NET Aspire solution codifies abstract aspects of the architectural decisions that automates the generation and deployment of a development environment<(!--and configuration provides the details from future decisions about other environments-->.)</p>
<p>A container focus is empowered by .NET Aspire's <em>orchestration</em> features. An independent orchestration responsibility enables better separation of <em>release and deploy</em> concerns from <em>build and test</em> concerns; <em>shifting right</em> those decisions that release and deploy depend on. (i.e., the ability to develop, execute, and evaluate solutions are discernibly <em>left</em> of release and operation.) Containers are an established method of componentizing a distributed system with independent servers (sometimes called "tiers.") This provides flexibility to deploy and execute in a development environment even before architectural decisions about a production topology have been considered. For example, debugging the .NET Aspire starter app automatically spins up a Redis container in Docker, but it's extremely unlikely that's how it will be deployed in production. In production, will there be one only Redis instance? If you have many instances, what sort of gateway or reverse proxy to that pool of instances will be utilized? Will it be on-prem or cloud? Will it be Azure, AWS, or Google Cloud? The beauty of Aspire's orchestration feature is that it doesn't matter yet; you can configure orchestration to <em>figure it out at run-time</em>, one environment at a time!</p>
<p>But, with every decision comes compromise. Technologies that depend on the physical resources that come from those decisions (that we're now effectively deferring) introduce some challenges with some existing software development idioms. A chicken-and-egg situation: if how to connect to physical resources may only be known at run-time, what happens to design-time technologies that depend on that connection information?</p>
<p>One popular technology in .NET, Entity Framework, suffers one of those challenges in .NET Aspire (Possibly only in <em>code-first</em> scenarios. Many Entity Framework examples detail adding Entity Framework support to an existing component (resource, like console app, ASP.NET Core web API, Razor app, etc.), creating a circular dependency between its project and the existence of an executing database (i.e., a valid database connection string.) In database-first, you have an existing application with existing physical databases and practices to utilize them in a development environment. With .NET Aspire, developers are <em>shifted left</em> from the decisions that provide the resources that things like <code>migrations add <migration-name></code> and <code>dotnet ef database update</code> require to function properly.</p>
<p>To be clear, the way .NET Aspire works is that the orchestration (AppHost) executes, figures out the various connection strings, and overrides the <code>appsettings</code> by setting environment variables before running the other components. The premise behind this means that at run-time, whatever is in <code>appsettings</code> is ignored. <code>dotnet ef</code> command doesn't execute at run-time; it effectively runs at design-time and gets its configuration from <code>appsettings</code>, so it's out of sync with reality.</p>
<p>The basic guidance is to <em>abstract those types of dependencies as .NET Aspire resources</em>. Nothing new conceptually, but this might be an application of the principles of abstraction at a level less commonly applied. Refining that guidance to using Entity Framework: <em>the database should be an independent resource</em>. Independent resources are modeled in .NET as either separate projects or separate solutions. Luckily, an <a href="https://github.com/dotnet/aspire-samples/tree/main/samples/eShopLite">.NET Aspire sample</a> addresses this. Let's look into the details.</p>
<p>The structure of the eShopLite sample overlaps with the .NET Aspire starter <code>dotnet new</code> template. It has a Blazor web frontend, a web API, an Aspire AppHost, and an Aspire service defaults project. Additionally, there is a shopping cart service (BasketService), and the catalog database (CatalogDb) project is an abstraction of the database resource.</p>
<p>The CatalogDb looks very similar to what you'd end up with following <a href="https://learn.microsoft.com/en-us/aspnet/core/tutorials/first-web-api?view=aspnetcore-8.0&tabs=visual-studio">Tutorial: Create a web API with ASP.NET Core</a>: an ASP.NET Core web API that leverages Entity Framework, and is effectively a gateway to a backend database. Although, that tutorial uses Entity Framework <em>in-memory</em> rather than via PostgreSQL. The way eShopLite supports Entity Framework is through the CatalogDb project. CatalogDb is like a stub project to the rest of the solution: Aspire doesn't execute it, but CatalogService depends upon it for the database model classes and <code>DbContext</code> (utilized more like a class library.) Nothing connects to the CatalogDb <em>web API</em>. The CatalogDb project contains all the Entity Framework design-time details and references, allowing you to utilize Entity Framework's features like <code>migrations add <migration-name></code> and <code>dotnet ef database update</code>. The target of Entity Framework operations like migration add and database update would depend on the configuration in <code>appsettings.json</code>. Initialization/seeding of the data is handled in <code>CatalogDbInitializer</code> within CatalogDb, as well as migrations at run-time (startup). CatalogDb <code>appsettings</code> connection strings must be in sync with the run-time values for <code>ef</code> commands to work.</p>
<p>In summary, if you want to utilize Entity Framework in a basic .NET Aspire application, <a href="https://learn.microsoft.com/en-us/aspnet/core/tutorials/first-web-api?view=aspnetcore-8.0&tabs=visual-studio">adding a project</a> to contain the entity models, context, and Entity Framework references and supporting a database engine container is a recommended place to get started. I suspect this guidance may be refined as .NET Aspire evolves.</p>
<p>I'm still wrapping my head around how .NET Aspire can support other non-containerized workloads like Azure SQL. Still, a containerized design melds nicely with the idea of independent resources (or nodes) in .NET Aspire. .NET Aspire also helps to more clearly delineate concerns like design, build, test, release, and deploy. As with .NET Aspire, containerization is an easier starting point for someone interested in distributed applications.</p>
<p>I look forward to how .NET Aspire evolves.</p>
<p><img src="../assets/entity-framework-in-aspire.jpg" class="img-fluid" alt="A path through the infrastructure"></p>http://blog.peterritchie.com/posts/etags-in-aspdotnet-coreETags in ASP.NET Core2023-06-28T00:00:00Z<p><img src="../assets/farside-etags-in-asp-dot-net.jpg" class="img-fluid" alt="lots of things going on at the same time, in the style of Farside" /></p>
<p>A good software architect doesn't just provide expectations of structure; they also work with developers to give feedback and guidance for implementation. It's easy to say, "Use ETags for entity concurrency control in a Web API," it's another to empower teams to accomplish the objectives of entity versioning.</p>
<p>To review: entity-tags (Etags) are a method of implementing Optimistic Concurrency Control. Optimistic Concurrency Control is a means to avoid distributed locking in situations when two or more potentially concurrent operations rarely interfere with each other. You can see cases like this on the Web when multiple processes or people are not normally working on the same data simultaneously. With the Web, there are rare situations where a single process or single person can (usually inadvertently) modify data from two places at the same time. It's rare case like this where the low overhead of optimistic concurrency can avoid accidental overwrites.</p>
<p>Entity-tags are a moniker of a particular incarnation of an entity. The tag is opaque, so it shouldn't need to be interpretable by a requestor to your service. With opaque data, you want to make the value itself as unobvious as possible.</p>
<p>The value, of course, could be an incrementing integer if you could reliably and efficiently increment an integer in a distributed environment (remember, we're addressing the possibility of two distributed transactions interfering with one another, the same transactions mechanisms that would be used to increment an integer.) But, before choosing to increment an integer (an ordinal number), consider <a href="https://www.rfc-editor.org/rfc/rfc9110#field.etag">RFC 9110 ETags</a> and why ordinal version numbers are not specified.</p>
<p>| If you think an ordinal number will work, do you need entity-tags at all?</p>
<p>A time-stamp is something to consider, in which case, prefer the <code>Last-Modified</code> header field validator. Or in conjunction with entity-tags. If a time-stamp is reliable, <code>Last-Modified</code> offers better interoperability options than re-inventing the wheel. Also, be thoughtful when considering time-stamps, especially their granularity; per-second time-stamp granularity can only partially solve the problem of concurrent writes.</p>
<p>So, how do you reliably generate an entity-tag value? The first thing to consider is what you want to accomplish. Do you want to prevent accidental overwrites, or do you want entity versioning? If you said, "I want entity versioning," to what end? If a client gets version 1, and another client updates it to version 2, what action do you want to perform when the first client requests to update the entity? You don't need <em>versioning</em> to prevent that first client from updating the entity. If you want to merge with version 2, you probably want versioning; in this case, you can stop reading now; I won't get into detail like that in this post.</p>
<p>If we're interested in preventing accidental overwrites, on the server side, we only really care about the current entity and the basis for the current request to update it. It doesn't matter if the basis is the previous version or ten versions behind; we only care that it's not based on the current version.</p>
<p>Another thing to consider is that entity-tags are also used in HTTP caching, which requires that an entity-tag be unique per encoding (e.g., a gzipped response should have a different entity-tag than a non-gzipped response.) The encoding value is often postfixed to the entity-tag to make it unique per encoding. But be careful to parse that out when checking for semantically identical entities. That's out of the scope of this post.</p>
<p>With an understanding of those constraints, a common method of generating an entity-tag is to use a hash of the entity representation.</p>
<p>Let's look at an example controller that tries to isolate the implementation detail of how the entity-tag is calculated. For my examples, I'm choosing to use controllers over minimal APIs; the controller class attributes make some of what is required easier. For clarity, my examples are stripped of error responses unrelated to conditional requests and exception middleware. For complete source, see <a href="https://github.com/peteraritchie/Examples.Etag">this repo</a>.</p>
<pre><code class="language-csharp">[ApiController] [Route("[controller]")]
public class AppointmentController : ControllerBase
{
[HttpGet(Name = "GetAppointmentRequests")]
[ProducesResponseType(typeof(WebCollectionElement<AppointmentRequestDto>[]),
StatusCodes.Status200OK, MediaTypeNames.Application.Json)]
public async Task<IActionResult> GetMany(CancellationToken cancellationToken = default)
{
var resource = appointmentRequestService.GetRequests(cancellationToken);
List<WebCollectionElement<AppointmentRequestDto>> items = new();
foreach (var (dto, guid, concurrencyToken) in await resource.ToListAsync(cancellationToken: cancellationToken))
{
items.Add(
new WebCollectionElement<AppointmentRequestDto>(dto, Url.Action(nameof(GetById),
new { id = guid })!, etag: concurrencyToken));
}
return base.Ok(items);
}
[HttpGet("{id}", Name = "GetAppointmentRequest")]
[ProducesResponseType(typeof(AppointmentRequestDto), StatusCodes.Status200OK, MediaTypeNames.Application.Json)]
[ProducesResponseType(typeof(AppointmentRequestDto), StatusCodes.Status304NotModified)]
public async Task<IActionResult> GetById(Guid id, [FromHeader(Name = "If-None-Match")] string? ifNoneMatch,
CancellationToken cancellationToken = default)
{
var (resource, concurrencyToken) = string.IsNullOrWhiteSpace(ifNoneMatch)
? await appointmentRequestService.GetRequest(id, cancellationToken)
: await appointmentRequestService.GetRequest(id, ifNoneMatch, cancellationToken);
HttpContext.Response.Headers.Add(HeaderNames.ETag, concurrencyToken);
return Ok(resource);
}
[HttpPost(Name = "CreateAppointmentRequest")]
[Consumes(MediaTypeNames.Application.Json)]
[ProducesResponseType(StatusCodes.Status201Created)]
public async Task<IActionResult> Create([FromBody] AppointmentRequestDto appointmentRequest,
CancellationToken cancellationToken = default)
{
var (id, concurrencyToken) = await appointmentRequestService.CreateRequest(appointmentRequest, cancellationToken);
HttpContext.Response.Headers.Add(HeaderNames.ETag, concurrencyToken);
return CreatedAtAction(nameof(GetById), routeValues: new { id }, value: null);
}
[HttpPut("{id}", Name = "ReplaceAppointmentRequest")]
[Consumes(MediaTypeNames.Application.Json)]
[ProducesResponseType(StatusCodes.Status204NoContent)]
public async Task<IActionResult> Replace(Guid id, [FromBody] AppointmentRequestDto appointmentRequest,
[FromHeader(Name = "If-Match")] string? ifMatch, CancellationToken cancellationToken = default)
{
var concurrencyToken = string.IsNullOrWhiteSpace(ifMatch)
? await appointmentRequestService.ReplaceRequest(
id, appointmentRequest, cancellationToken)
: await appointmentRequestService.ReplaceRequest(
id, appointmentRequest, ifMatch, cancellationToken);
HttpContext.Response.Headers.Add(HeaderNames.ETag, concurrencyToken);
return NoContent();
}
[HttpPatch("{id:guid}", Name = "UpdateAppointmentRequest")]
[ProducesResponseType(typeof(AppointmentRequestDto), StatusCodes.Status200OK, MediaTypeNames.Application.Json)]
[Consumes("application/json-patch+json")]
public async Task<IActionResult> Update(Guid id, JsonPatchDocument<AppointmentRequestDto> patchDocument,
[FromHeader(Name = "If-Match")] string? ifMatch, CancellationToken cancellationToken = default)
{
var (result, concurrencyToken) = string.IsNullOrWhiteSpace(ifMatch)
? await appointmentRequestService.UpdateRequest(id, patchDocument, cancellationToken)
: await appointmentRequestService.UpdateRequest(id, patchDocument, ifMatch, cancellationToken);
HttpContext.Response.Headers.Add(HeaderNames.ETag, concurrencyToken);
return Ok(result);
}
[HttpDelete("{id}", Name = "RemoveAppointmentRequest")]
[ProducesResponseType(StatusCodes.Status204NoContent)]
public async Task<IActionResult> Remove(Guid id, [FromHeader(Name = "If-Match")] string? ifMatch,
CancellationToken cancellationToken = default)
{
if(string.IsNullOrWhiteSpace(ifMatch))
await appointmentRequestService.RemoveRequest(id, cancellationToken);
else
await appointmentRequestService.RemoveRequest(id, ifMatch, cancellationToken);
return NoContent();
}
}
</code></pre>
<blockquote class="blockquote">
<p>There are inherent complexities in a Web API. It needs to present an interface usable on the Web and utilizes open standards as much as possible. You'll notice that the <code>AppointmentController</code> PATCH implementation uses <code>JsonPatchDocument</code>, an implementation of the <a href="https://jsonpatch.com/">JSON Patch</a> (IETF RFC 6902) standard. This standard is specific to the Web, specific to JSON, and deals with operations intended to be specifically applied to JSON representations equivalent to the model defined in the interface (i.e., the model, not what is represented in the database or an in-memory representation of a domain object.)</p>
</blockquote>
<p>This controller is isolated from the collaboration with the database and delegates that interaction to an Application Service via the <code>appointmentRequestService</code> field (declaration removed for readability). In state-modifying HTTP methods (PUT, DELETE, PATCH), the actions have an <code>ifMatch</code> parameter passed in through the <code>If-Match</code> HTTP request header. When present, it is passed along to the application service for optimistic concurrency. This example shows an <em>optional</em> use of <code>If-Match</code>; it's plausible that another implementation might <em>require</em> the <code>If-Match</code> header and respond with status code 428 Precondition Required.</p>
<p>Of note is that this controller abstracts etag header values as <em>concurrency token</em> text so that nothing else has to deal with HTTP headers.</p>
<p>Let's look at the MVC model (I prefer to refer to it as a Data Transfer Object).</p>
<pre><code class="language-csharp">public class AppointmentRequestDto
{
[Required]
public DateTime? CreationDate { get; set; }
public IEnumerable<string>? Categories { get; set; }
[Required]
public string? Description { get; set; }
public string? Notes { get; set; }
[Required]
public AppointmentRequestStatus? Status { get; set; }
[Required]
public MeetingDuration? Duration { get; set; }
[Required]
public IEnumerable<string>? Participants { get; set; }
[Required]
public IEnumerable<DateTime>? ProposedStartDateTimes { get; set; }
}
</code></pre>
<p>Since we're delegating serialization to ASP.NET (which requires writable properties), the properties are nullable but annotated with <code>RequiredAttribute</code> to signal to the framework what properties are required. There is no identifier in the <code>AppointmentRequestDto</code> class because we don't want to duplicate it there and in the resource's URI.</p>
<p>Azure Cosmos has implemented optimistic concurrency control and stores an ETag per document. I'll use Azure Cosmos for the database implementation to show how this can be re-used in your WebAPI.</p>
<h2 id="azure-cosmos-example">Azure Cosmos Example</h2>
<p>In Azure Cosmos, each document has several mandatory properties: <code>id</code>, <code>_rid</code>, <code>_self</code>, <code>_etag</code>, <code>_attachements</code>, and <code>_ts</code>. These are implementation details of the database that we don't want to leak into our API as body content. When we use the <a href="https://www.nuget.org/packages/Microsoft.Azure.Cosmos">Azure Cosmos SDK</a>, we need serialization classes to serialize the data to and from a container. Let's see an example with a fictitious appointment request resource:</p>
<pre><code class="language-csharp">public class AppointmentRequestEntity : CosmosEntityBase
{
[JsonProperty(PropertyName = "id")]
public Guid Id { get; set; }
[JsonProperty(PropertyName = "_rid")]
public string? ResourceId { get; set; }
[JsonProperty(PropertyName = "_self")]
public Uri? SelfUri { get; set; }
[JsonProperty(PropertyName = "_etag")]
public string? ETag{ get; set; }
[JsonProperty(PropertyName = "_ts")]
public int? TimestampText{ get; set; }
public DateTime? CreationDate { get; set; }
public IEnumerable<string>? Categories { get; set; }
public string? Description { get; set; }
public string? Notes { get; set; }
public AppointmentRequestStatus? Status { get; set; }
public MeetingDuration? Duration { get; set; }
public IEnumerable<string>? Participants { get; set; }
public IEnumerable<DateTime>? ProposedStartDateTimes { get; set; }
}
</code></pre>
<p>Notice the first five properties that are necessary to access the Azure Cosmos implementation details. (in <a href="https://github.com/peteraritchie/Examples.Etag">this repo</a> this is split out into a <code>CosmosEntityBase</code> class.)</p>
<p>For my example, I'm going to draw on Domain-Driven design patterns and use a Repository implementation in the database collaboration. I want to delegate all the logic related to database-specific details to the repository implementation. This includes encapsulating the use of the database entity <em>serialization</em> class (translation to/from the database entity class), associating an identifier and etag with the resource, etc. To separate the existence of the database entity class from clients of the repository, we'll define a generic interface that I'll name <code>IOptimisticallyConcurrentRepository</code> that works with different types of domain entity classes:</p>
<pre><code class="language-csharp">public interface IOptimisticallyConcurrentRepository<TDomainEntity>
{
Task<TDomainEntity> Get(Guid id, CancellationToken cancellationToken = default);
IAsyncEnumerable<TDomainEntity> Get(CancellationToken cancellationToken = default);
Guid GetId(TDomainEntity entity);
bool TryGetIfModified(Guid id, string concurrencyToken, out TDomainEntity? entity);
string GetConcurrencyToken(TDomainEntity entity);
Task<Guid> Add(TDomainEntity entity, CancellationToken cancellationToken = default);
Task Remove(Guid id, CancellationToken cancellationToken = default);
Task Replace(Guid id, TDomainEntity entity, CancellationToken cancellationToken = default);
Task RemoveIfMatch(Guid id, string token, CancellationToken cancellationToken = default);
Task ReplaceIfMatch(Guid id, TDomainEntity entity, string token, CancellationToken cancellationToken = default);
}
</code></pre>
<p>Next is a generic repository class to support Azure Cosmos that deals with arbitrary domain (<code>TDomainEntity</code>) and database serialization classes (<code>TDbEntity</code>):</p>
<pre><code class="language-csharp">public class CosmosOptimisticallyConcurrentRepository<TDomainEntity, TDbEntity>
: IOptimisticallyConcurrentRepository<TDomainEntity>
where TDomainEntity : class
where TDbEntity : CosmosEntityBase
{
private class EntityContext
{
public EntityContext(Guid id, string concurrencyToken)
{
Id = id;
ConcurrencyToken = concurrencyToken;
}
public Guid Id { get; }
public string ConcurrencyToken { get; }
}
private readonly Container container;
private readonly ITranslator<TDomainEntity, TDbEntity> dbEntityTranslator;
private readonly Action<TDbEntity, Guid> setDbEntityId;
protected CosmosOptimisticallyConcurrentRepository(Container container, ITranslator<TDomainEntity, TDbEntity> dbEntityTranslator,
Action<TDbEntity, Guid> setDbEntityId)
{
this.container = container;
this.dbEntityTranslator = dbEntityTranslator;
this.setDbEntityId = setDbEntityId;
}
public async Task<Guid> Add(TDomainEntity entity, CancellationToken cancellationToken = default)
{
var id = Guid.NewGuid();
var dbEntity = dbEntityTranslator.ToData(entity);
setDbEntityId(dbEntity, id);
try
{
var result = await container.CreateItemAsync(dbEntity, new PartitionKey(id.ToString("D")), cancellationToken: cancellationToken);
conditionalWeakTable.Add(entity, new EntityContext(id, result.ETag));
return id;
}
catch (CosmosException ex) when(ex.StatusCode == HttpStatusCode.PreconditionFailed)
{
throw new ConcurrencyException();
}
}
public bool TryGetIfModified(Guid id, string concurrencyToken, out TDomainEntity? entity)
{
var idText = id.ToString("D");
try
{
var result = container.ReadItemAsync<TDbEntity>(
idText,
new PartitionKey(idText),
requestOptions: new ItemRequestOptions() { IfNoneMatchEtag = concurrencyToken })
.Result;
entity = dbEntityTranslator.ToDomain(result.Resource);
conditionalWeakTable.Add(entity, new EntityContext(id, result.ETag));
return true;
}
catch (AggregateException aggregateException) when (aggregateException.InnerExceptions.Count == 1 &&
aggregateException.InnerExceptions.Single() is
CosmosException
{
StatusCode: HttpStatusCode.NotModified
})
{
entity = default;
return false;
}
catch (AggregateException aggregateException) when (aggregateException.InnerExceptions.Count == 1 &&
aggregateException.InnerExceptions.Single() is
CosmosException
{
StatusCode: HttpStatusCode.NotFound
})
{
throw new EntityNotFoundException(id);
}
catch (CosmosException ex) when(ex.StatusCode == HttpStatusCode.NotFound)
{
throw new EntityNotFoundException(id);
}
}
public async IAsyncEnumerable<TDomainEntity> Get([EnumeratorCancellation] CancellationToken cancellationToken = default)
{
var iterator = container.GetItemQueryIterator<TDbEntity>();
while (iterator.HasMoreResults)
{
var set = await iterator.ReadNextAsync(cancellationToken);
foreach (var e in set)
{
var entity = dbEntityTranslator.ToDomain(e);
conditionalWeakTable.Add(entity, new EntityContext(e.Id, e.ETag!));
yield return entity;
}
}
}
public async Task<TDomainEntity> Get(Guid id, CancellationToken cancellationToken = default)
{
var idText = id.ToString("D");
try
{
var result = await container.ReadItemAsync<TDbEntity>(idText, new PartitionKey(idText), cancellationToken: cancellationToken);
var entity = dbEntityTranslator.ToDomain(result.Resource);
conditionalWeakTable.Add(entity, new EntityContext(id, result.ETag));
return entity;
}
catch (CosmosException ex) when(ex.StatusCode == HttpStatusCode.NotFound)
{
throw new EntityNotFoundException(id);
}
}
public async Task Replace(Guid id, TDomainEntity entity, CancellationToken cancellationToken = default)
{
var dbEntity = dbEntityTranslator.ToData(entity);
setDbEntityId(dbEntity, id);
try
{
_ = await container.UpsertItemAsync(dbEntity, cancellationToken: cancellationToken);
}
catch (CosmosException ex) when(ex.StatusCode == HttpStatusCode.NotFound)
{
throw new EntityNotFoundException(id);
}
}
public async Task ReplaceIfMatch(Guid id, TDomainEntity entity, string token, CancellationToken cancellationToken = default)
{
var idText = id.ToString("D");
var dbEntity = dbEntityTranslator.ToData(entity);
setDbEntityId(dbEntity, id);
var requestOptions = new ItemRequestOptions { IfMatchEtag = token };
try
{
_ = await container.ReplaceItemAsync(dbEntity, idText, new PartitionKey(idText), requestOptions: requestOptions, cancellationToken: cancellationToken);
}
catch (CosmosException ex) when (ex.StatusCode == HttpStatusCode.PreconditionFailed)
{
throw new ConcurrencyException();
}
catch (CosmosException ex) when(ex.StatusCode == HttpStatusCode.NotFound)
{
throw new EntityNotFoundException(id);
}
}
public async Task Remove(Guid id, CancellationToken cancellationToken = default)
{
var idText = id.ToString("D");
try
{
_ = await container.DeleteItemAsync<TDbEntity>(idText, new PartitionKey(idText), cancellationToken: cancellationToken);
}
catch (CosmosException ex) when(ex.StatusCode == HttpStatusCode.NotFound)
{
throw new EntityNotFoundException(id);
}
}
public async Task RemoveIfMatch(Guid id, string token, CancellationToken cancellationToken = default)
{
var idText = id.ToString("D");
var requestOptions = new ItemRequestOptions { IfMatchEtag = token };
try
{
_ = await container.DeleteItemAsync<TDbEntity>(idText, new PartitionKey(idText), requestOptions: requestOptions, cancellationToken: cancellationToken);
}
catch (CosmosException ex) when (ex.StatusCode == HttpStatusCode.PreconditionFailed)
{
throw new ConcurrencyException();
}
catch (CosmosException ex) when(ex.StatusCode == HttpStatusCode.NotFound)
{
throw new EntityNotFoundException(id);
}
}
}
</code></pre>
<p>As a reminder, a "concurrency token" is synonymous with an "etag" in the context of the repository.</p>
<p>The persistence needs of an application are independent of a domain entity, so the domain entity is isolated from web/database identifiers, concurrency tokens, HTTP, etags, etc. So, the repository needs to translate from a domain object to the serialization object, which is performed mostly by an <code>ITranslator<TDomain, TData></code> implementation but also with the assignment of the identifier to the serialization object. To keep the non-domain details isolated from the domain object, I've used the <code>ConditionalWeakTable<TKey, TValue></code> type to associate database persistence details (ID and etag/concurrency token, as abstracted by <code>EntityContext</code>) to the object without too much management logic.</p>
<blockquote class="blockquote">
<p><code>ConditionalWeakTable</code> is like a dictionary that associates a value with another object. It differs from a traditional dictionary in that when the key is no longer referenced, the associated value is freed/destroyed. This allows us to get associated data with minimal memory impact easily.</p>
</blockquote>
<p>An implementation of the repository now just requires the type to use for the database serialization class, the domain entity type, and how to assign an identifier to the Azure Cosmos <code>id</code> property:</p>
<pre><code class="language-csharp">public sealed class CosmosAppointmentRequestRepository : CosmosOptimisticallyConcurrentRepository<AppointmentRequest, AppointmentRequestEntity>
{
public CosmosAppointmentRequestRepository(Container container, ITranslator<AppointmentRequest, AppointmentRequestEntity> appointmentRequestEntityTranslator)
: base(container, appointmentRequestEntityTranslator, (entity, guid) => entity.Id = guid)
{
}
}
</code></pre>
<p>The only remaining part is the implementation of the application/database collaboration, the <em>application service</em>:</p>
<pre><code class="language-csharp">public class AppointmentRequestService
{
private readonly AppointmentRequestDtoTranslator appointmentRequestDtoTranslator;
private readonly IOptimisticallyConcurrentRepository<AppointmentRequest> repository;
public AppointmentRequestService(AppointmentRequestDtoTranslator appointmentRequestDtoTranslator, IOptimisticallyConcurrentRepository<AppointmentRequest> repository)
{
this.appointmentRequestDtoTranslator = appointmentRequestDtoTranslator;
this.repository = repository;
}
public async Task<(Guid, string)> CreateRequest(AppointmentRequestDto appointmentRequest, CancellationToken cancellationToken = default)
{
var entity = appointmentRequestDtoTranslator.AppointmentRequestDtoToAppointmentRequest(appointmentRequest);
var guid = await repository.Add(entity, cancellationToken);
return (guid, repository.GetConcurrencyToken(entity));
}
public async Task<(AppointmentRequestDto, string)> GetRequest(Guid id, CancellationToken cancellationToken = default)
{
var appointmentRequest = await repository.Get(id, cancellationToken);
return (appointmentRequestDtoTranslator.AppointmentRequestToAppointmentRequestDto(appointmentRequest), repository.GetConcurrencyToken(appointmentRequest));
}
public Task<(AppointmentRequestDto, string)> GetRequest(Guid id, string etag, CancellationToken _ = default)
{
if(repository.TryGetIfModified(id, etag, out var appointmentRequest))
{
return Task.FromResult((appointmentRequestDtoTranslator.AppointmentRequestToAppointmentRequestDto(appointmentRequest!), repository.GetConcurrencyToken(appointmentRequest!)));
}
throw new ConcurrencyException();
}
public async IAsyncEnumerable<(AppointmentRequestDto, Guid, string)> GetRequests([EnumeratorCancellation] CancellationToken cancellationToken = default)
{
var result = repository.Get(cancellationToken);
await foreach (var item in result.WithCancellation(cancellationToken))
{
yield return (appointmentRequestDtoTranslator.AppointmentRequestToAppointmentRequestDto(item), repository.GetId(item),
repository.GetConcurrencyToken(item));
}
}
public async Task RemoveRequest(Guid id, CancellationToken cancellationToken = default)
{
await repository.Remove(id, cancellationToken);
}
public async Task RemoveRequest(Guid id, string etag, CancellationToken cancellationToken = default)
{
await repository.RemoveIfMatch(id, etag, cancellationToken);
}
internal async Task<string> ReplaceRequest(Guid id, AppointmentRequestDto appointmentRequest,
CancellationToken cancellationToken = default)
{
var entity = appointmentRequestDtoTranslator.AppointmentRequestDtoToAppointmentRequest(appointmentRequest);
await repository.Replace(id, entity, cancellationToken);
return repository.GetConcurrencyToken(entity);
}
internal async Task<string> ReplaceRequest(Guid id, AppointmentRequestDto appointmentRequest, string etag,
CancellationToken cancellationToken = default)
{
var entity = appointmentRequestDtoTranslator.AppointmentRequestDtoToAppointmentRequest(appointmentRequest);
await repository.ReplaceIfMatch(id, entity, etag, cancellationToken);
return repository.GetConcurrencyToken(entity);
}
public async Task<(AppointmentRequestDto, string)> UpdateRequest(Guid id, JsonPatchDocument<AppointmentRequestDto> patchDocument,
CancellationToken cancellationToken = default)
{
var current = await repository.Get(id, cancellationToken);
var currentDto = appointmentRequestDtoTranslator.AppointmentRequestToAppointmentRequestDto(current);
patchDocument.ApplyTo(currentDto);
await repository.Replace(id, appointmentRequestDtoTranslator.AppointmentRequestDtoToAppointmentRequest(currentDto), cancellationToken);
return (currentDto, repository.GetConcurrencyToken(current));
}
public async Task<(AppointmentRequestDto, string)> UpdateRequest(Guid id, JsonPatchDocument<AppointmentRequestDto> patchDocument,
string etag, CancellationToken cancellationToken = default)
{
var current = await repository.Get(id, cancellationToken);
var currentDto = appointmentRequestDtoTranslator.AppointmentRequestToAppointmentRequestDto(current);
patchDocument.ApplyTo(currentDto);
await repository.ReplaceIfMatch(id, appointmentRequestDtoTranslator.AppointmentRequestDtoToAppointmentRequest(currentDto), etag, cancellationToken);
return (currentDto, repository.GetConcurrencyToken(current));
}
}
</code></pre>
<p><code>AppointmentRequestService</code> contains the interaction logic specific to the application and the repository. Since there</p>
<p>Dealing with translation to and from DTO, domain, and serialization classes is made less of a chore with tools like <a href="https://github.com/riok/mapperly">Mapperly</a>. <a href="https://github.com/riok/mapperly">Mapperly</a> will generate translation code based on property names. To create a translator to/from two types is easy as creating a partial class with a <code>MapperAttribute</code> attribute with partial methods that take one type as parameter and the other as a return:</p>
<pre><code class="language-csharp">[Mapper]
public partial class AppointmentRequestDtoTranslator
{
public partial AppointmentRequest AppointmentRequestDtoToAppointmentRequest(AppointmentRequestDto dto);
public partial AppointmentRequestDto AppointmentRequestToAppointmentRequestDto(AppointmentRequest entity);
}
</code></pre>
<p><code>AppointmentRequestDtoTranslator</code> translates <code>AppointmentRequestDto</code> instances to/from <code>AppointmentRequest</code> domain entity instances. And to translate to/from <code>AppointmentRequestEntity</code>:</p>
<pre><code class="language-csharp">[Mapper]
public partial class AppointmentRequestEntityTranslator : ITranslator<AppointmentRequest, AppointmentRequestEntity>
{
[MapperIgnoreSource(nameof(AppointmentRequestEntity.Id))]
[MapperIgnoreSource(nameof(AppointmentRequestEntity.ResourceId))]
[MapperIgnoreSource(nameof(AppointmentRequestEntity.ETag))]
[MapperIgnoreSource(nameof(AppointmentRequestEntity.SelfUri))]
[MapperIgnoreSource(nameof(AppointmentRequestEntity.TimestampText))]
public partial AppointmentRequest ToDomain(AppointmentRequestEntity data);
[MapperIgnoreTarget(nameof(AppointmentRequestEntity.Id))]
[MapperIgnoreTarget(nameof(AppointmentRequestEntity.ResourceId))]
[MapperIgnoreTarget(nameof(AppointmentRequestEntity.ETag))]
[MapperIgnoreTarget(nameof(AppointmentRequestEntity.SelfUri))]
[MapperIgnoreTarget(nameof(AppointmentRequestEntity.TimestampText))]
public partial AppointmentRequestEntity ToData(AppointmentRequest domain);
}
</code></pre>
<p>Since <code>AppointmentRequestEntity</code> has some Azure Cosmos implementation details, we use Mapprerly's <code>MapperIgnoreTargetAttribute</code> and <code>MapperIgnoreSourceAttribute</code> to tell <a href="https://github.com/riok/mapperly">Mapperly</a> that not all properties need translation.</p>
<p>Dealing with concurrency issues and implementing concurrency control can be intimidating. In this post, I make it less intimidating by clarifying some specifics by showing an example implementation with ASP.NET Core and Azure Cosmos DB. Additionally, the Domain-Driven Design patterns Repository and Application Service are used to isolate etag implementation details from the Web API to delegate that to Azure Cosmos.</p>
<blockquote class="blockquote">
<p>There are multiple ways of implementing optimistic concurrency; HTTP ETags are but one way. If you can't abide by the expectations set out by the HTTP standards, don't use Etags. There's nothing that forces you to use HTTP precondition header fields. But, remember, the means exist in HTTP, and embracing it will promote interoperability and reliability (to implement something different than something introduced at least 26 years ago fails to recognize the huge amount of validation and verification that's gone into making it correct.)</p>
</blockquote>
<p>In a future post, I will show an example of a repository implementation that uses Entity Framework and its expectations for concurrency tokens.</p>
<p><img src="../assets/farside-etags-in-asp-dot-net.jpg" class="img-fluid" alt="lots of things going on at the same time, in the style of Farside"></p>http://blog.peterritchie.com/posts/http-and-etag-header-fieldsHTTP and ETag Header Fields2023-06-15T00:00:00Z<p><img src="../assets/accidental-overwrite.jpg" class="img-fluid" alt="A stuffed tiger corrupted appearance due to accidental overwrite" /></p>
<p>Update: corrected mention of <code>412</code> in the context of <code>GET</code> and <code>If-Modified-Since</code> to <code>304</code>.</p>
<p>Over the last four-plus years, I have been almost exclusively working on some sort of *-as-a-Service (*aaS)—for example, Mortgage Origination as a Service, Insurance Claims as a Service. I always see a couple of things when implementing Web (HTTP) services: the <em>reinvention of the wheel</em> and recognizing the problem ETags solves after publishing a specification (sometimes both).</p>
<p>With *aaS as a Web API, the intention is to have multiple API clients providing access to representations of shared resources. Early in projects like this involves an initial (single) client, so the chances of a client having a stale resource representation are slim. When another client starts to use the API and an update gets accidentally overwritten, things get needlessly complicated.</p>
<p>I've seen teams address this problem in a number of ways, often involving a date-time stamp. With multiple clients on an API, scalability is an issue, and a date-time stamp can mean different things to different servers (as we'll see below). You need a single authority for a resource's last modified date-time to avoid exchanging one problem for another. See <a href="https://datatracker.ietf.org/doc/html/rfc7232#section-2.2.2">Last-Modified/Comparison</a> for more details.</p>
<p>The creators of HTTP encountered this issue and added features to HTTP to deal with this (I assume that's why they added these features). I don't know when these features were devised, but they proposed them in 1997. So, they've been in the wild for at least 25 years with the entire web as a test bed. So, many brilliant people either created or scrutinized the solution. i.e., it's a wheel.</p>
<p>The HTTP features are <em>ETags</em> and <em>conditional requests</em> and enable <em>optimistic concurrency</em>.</p>
<h2 id="etags">ETags</h2>
<p>An <a href="https://datatracker.ietf.org/doc/html/rfc7232#section-2.3">ETag</a> (AKA entity-tag) addresses the "lost update" problem where there are two clients of an API that have received the representation of a version of an entity. Still, another client updates the entity before the other: the second update causes the first the be "lost." See the following diagram for a visualization:</p>
<p><img src="./assets/lost-update-sequence.png" class="img-fluid" alt="Lost-update" /></p>
<p>An ETag addresses accidental overwrite by <em>versioning</em> the resource with an entity-tag (a hash of the representation, a version, etc.). When a client requests a resource, the server may include an ETag validator header field with an entity-tag value in the response. The URI of the resource, along with that entity-tag, constitutes an identifier for a particular version of an entity.</p>
<p>When a client requests a change to the entity, it includes the entity-tag as a basis version with a conditional header field (like <code>If-Match</code>.) The server responds with <code>412 (Precondition Failed)</code>, and the client can retrieve the latest version, re-apply their change, and re-send. See the following diagram for a visualization:</p>
<p><img src="./assets/lost-update-solution-sequence.png" class="img-fluid" alt="Lost-update" /></p>
<h2 id="falling-back-to-date-and-time">Falling back to date and time</h2>
<p>Even if you use date and time, <strong>HTTP also covers you with other precondition header fields involving modification date</strong>. The <code>If-Unmodified-Since</code> and <code>If-Modified-Since</code> precondition header fields allow you to pass modification date preconditions to make a request conditional. When the precondition isn't met, a <code>412 (Precondition Failed)</code> status code will be in the response, or for <code>GET</code> or <code>HEAD</code>, a <code>304 (Not Modified)</code> status code will be in the response.</p>
<p>The initial GET of a resource that supports modification dates in conditional requests will include a <code>Last-Modified</code> header field validator. The <code>Last-Modified</code> validator is in the form of an <a href="https://datatracker.ietf.org/doc/html/rfc7231#section-7.1.1.1">HTTP-date</a>.</p>
<h2 id="being-successful">Being Successful</h2>
<p>RFC 7232, the HTTP 1.1 specification, section 2.3 describes the <em>entity-tag</em>:</p>
<blockquote class="blockquote">
<p>An entity-tag is an opaque validator for differentiating between multiple representations of the same resource, regardless of whether those multiple representations are due to resource state changes over time, content negotiation resulting in multiple representations being valid at the same time, or both.</p>
</blockquote>
<p>This means that the ETag value depends on the content-type, so two <strong>different representations of the same resource should have different ETag values</strong> (e.g., one gzip encoded, one not.)</p>
<p>This also means that the ETag value is opaque to requestors but does point out that one of the intents of ETags to be <strong>an alternative to using a date-time stamp due to lack of accuracy</strong>.</p>
<h3 id="patch">PATCH</h3>
<p>Using <code>PATCH</code> with something like <a href="https://jsonpatch.com/">JSONPatch</a> may seem to help alleviate conflicts by providing more granularity in what is changing. Technically true, to implement this would be non-trivial. The ETag specifics a tag of that edition of the entire resource, not any one field. While comparing a change against a delta between two editions of a resource (keeping in mind those editions may not be adjacent) might be one technique for dealing with that, <strong>creating deltas between arbitrary versions of the same resource is non-trivial</strong>. You could introduce that sort of thing. Something like event-sourcing might enable that. But remember that there may be interdependencies between properties of a resource, and just because the current request changes a property that hasn't changed since the resource was retrieved doesn't mean there isn't still a conflict.</p>
<h3 id="last-modified">Last-Modified</h3>
<p>Remember that <code>Last-Modified</code> uses <a href="https://datatracker.ietf.org/doc/html/rfc7231#section-7.1.1.1">HTTP-date</a> format, so <strong><code>Last-Modified</code> only supports second granularity</strong>. With multiple origin servers, more than second granularity may be needed to be accurate 100% of the time.</p>
<h4 id="if-unmodified-since">If-Unmodified-Since</h4>
<p><code>If-Unmodified-Since</code> is used with state-changing methods like PUT, POST, DELETE, and PATCH to avoid accidental overrides (lost updates). <code>If-Unmodified-Since</code> imposes the precondition <em>update this entity only if it hasn't changed since the provided date-time</em>. <strong>Use <code>If-Unmodified-Since</code> to avoid lost update problems when second granularity is not a problem</strong>.</p>
<h4 id="if-modified-since">If-Modified-Since</h4>
<p>When used with <code>GET</code> or <code>HEAD</code>, the <code>If-Modified-Since</code> header field imposes the precondition <em>respond with <code>304 (Not Modified)</code> and not with an entity representation if the modification date of the identified resource is not more recent than the date provided</em>. <strong>Use <code>If-Modified-Since</code> to avoid re-transferring the same data</strong>.</p>
<h3 id="conflict"><code>409 (Conflict)</code></h3>
<p><code>409 (Conflict)</code> may sound like an appropriate response to a conditional PUT/POST/PATCH request, except that <code>412 (Precondition Failed)</code> is expected. <strong>Response status code <a href="https://datatracker.ietf.org/doc/html/rfc7231#section-6.5.8"><code>409</code></a> should be used when something about the current state of the resource means that the server cannot change it</strong>. Also, if you have chosen not to use HTTP precondition features and have included something <em>in the representation of the entity</em> for versioning (like last-modified-date, see above), then <code>409 (Conflict)</code> is appropriate to signify a potential accidental overwrite or lost update.</p>
<h3 id="leveraging-existing-implementations">Leveraging Existing Implementations</h3>
<p>Azure Cosmos DB implements ETags and <code>Last-Modified</code> to be leveraged to support the versioning of resources in your Web API. Technically the ETag is a version of the representation that Cosmos DB provides, so consider generating a new ETag based on what Cosmos DB provides, especially if you support more than one content-type (like XML). Suppose you have the concept of a database DTO or database models different from your MVC models. In that case, you should consider custom entity-tag generation based on the Cosmos-supplied entity-tag.</p>
<p>To leverage the Cosmos-supplied entity-tag, retain it and re-send it in any state-changing requests to Cosmos in the <code>If-Match</code> header field. If the entity-tags do not match, Cosmos DB will respond with <code>412</code>, and the Cosmos DB library will throw a <code>CosmosException</code> with <code>StatusCode</code> == <code>HttpStatusCode.PreconditionFailed</code>.</p>
<!--
title Lost Update Problem
participant "Client 1" as Client1
participant "Client 2" as Client2
participant API
Client1->API:""GET /resource/123""
activate Client1
Client1<--API:""200 OK""\n//resource v1 representation//
create Client2
Client2->API:""GET /resource/123""
activate Client2
Client2<--API:""200 OK""\n//resource v1 representation//
Client1->API:""PUT /resource/123""
Client1<--API:""200 OK""\n//resource v2 representation//
deactivateafter Client1
destroyafter Client1
Client2-#red>API:""PUT /resource/123""
note over Client1,API#pink:Client 2 is updating the resource based from **v1**, not **v2**:\n<align:center>the v2 update is "lost" to //Client 2//</align>
Client2<--API:""200 OK""\n//resource v3 representation//
-->
<!--
title Lost Update Solution
participant "Client 1" as Client1
participant "Client 2" as Client2
participant API
Client1->API:""GET /resource/123""
activate Client1
Client1<--API:""200 OK""\n//resource v1 representation//
create Client2
Client2->API:""GET /resource/123""
activate Client2
Client2<--API:""200 OK""\n//resource v1 representation//
Client1->API:""PUT /resource/123\nIf-Match: v1""
Client1<--API:""200 OK""\n//resource v2 representation//
deactivateafter Client1
destroyafter Client1
Client2->API:""PUT /resource/123\nIf-Match: v1""
Client2<--API:<color:#red>""412 Precondition Failed\nBasis version of resource is out of date""</color>
-->
<h2 id="references">References</h2>
<p><img src="../assets/accidental-overwrite.jpg" class="img-fluid" alt="A stuffed tiger corrupted appearance due to accidental overwrite"></p>http://blog.peterritchie.com/posts/Being-Successful-With-Domain-Driven-Design--Minimal-Complexity-Part-3Being Successful with Domain-Driven Design: Minimal Complexity, Part 32023-06-12T00:00:00Z<p><img src="/assets/complex-relationships.jpg" class="img-fluid" alt="complex-relationships" /></p>
<p>With a name like "Domain-Driven Design," it should be no surprise there is a major focus on the domain and has a huge influence on implementation. We've focused mostly on strategic design patterns and practices like Ubiquitous Language, Bounded Context, etc. But I've also covered a bit of tactical design and implementation. I've transitioned from strategical patterns--that deal with being explicit with domain concepts (Ubiquitous Language, Bounded Contexts)--to tactical patterns that have focused on directly translating domain concepts into code structure or coding patterns (like Services and Aggregates.)</p>
<p>The concepts and their consistency boundaries are only a couple of things that contribute to the complexity of non-trivial domains. For example, the work required to implement a domain is independent of its concepts and consistency boundaries. Additionally, the system's quality attributes and technical constraints are major influencers on the internal structure of that system. The next set of Domain-Driven Design patterns I'll get into (<em>tactical</em> patterns) aid in this respect. As we get closer to implementation, the focus turns more towards isolating domain complexity from implementation complexities.</p>
<p>As with many things in Domain-Driven Design, <em>x</em> for the sake of <em>x</em> is not the intention. Many things in most methodologies can be regurgitated and used by rote, providing little to no value. The principles and practices in Domain-Driven Design are best utilized with purpose and intent. Architectural layering is a good example. Each layer needs a reason for being (a purpose) with unidirectional independence of its concepts from another layer's concepts. Just any two groupings won't do; without the purposeful intent of having two layers with a unidirectional dependency, you'll never gain the benefits of layering. You end up with the added burden of managing a structure that does not give you any layered benefits.</p>
<p>The minimum complexity for layers is that two groups of concepts (contexts) are uni-directionally interdependent. In Domain-Driven Design, those layers focus on isolating the concerns of a User Interface, Application, Domain, and Infrastructure.</p>
<p>The Application layer may seem unique to Domain-Driven Design. There are few patterns/methodologies that isolates the concern that the application layer deals with. Ports and Adapters (Hexagonal) and, by extension, Clean Architecture recognize and isolate high-level <em>use cases</em> from both the domain and the implementation details of a UI. This is the role of the Application layer in Domain-Driven Design to further isolate the domain from how any use case uses the domain (a use case <em>applies</em> or <em>realizes</em> the domain). In Ports and Adapters, uses-cases (or, as Cockburn describes, uses-cases) are sequences of interactions between the system and users/actors. With the recognition of these interactions, they can now be isolated as collaborations within the Application layer.</p>
<p>There are other patterns that isolate interaction behavior structurally within collaborations like the Adapter pattern, but I'll save that for another day.</p>
<p>A UI may have to deal with different form factors, communication protocols, execution contexts, etc. A loosely coupled UI involves designing an interface that takes all of those things into consideration to be successful. A web-based UI requires a backend that supports open protocols and standards. Protocols and standards relating to implementation or delivery are merely constraints on how a system is implemented. At some level, the domain needs to operate correctly regardless of those constraints.</p>
<p>Layers are like different team roles, all working together simultaneously to accomplish specific types of goals. Bounded contexts are also like multiple teams, sometimes like a night shift and a day shift or an on-shore and an off-shore team. These types of teams work with some level of independence: shifts may never work together simultaneously, and on- and off-shore teams only work together for a brief time with much more structured communication.</p>
<p>Recognizing and planning for how teams contribute to the same goals is key for these teams to be effective. It's the recognition that different parts of a larger system need different levels of independence. With teams, this is to utilize resources effectively: like how shifts can use limited resources (human skills) across more of the day (e.g., 24 hours instead of 8.) Conway's law is just an observation (i.e., a reality). A team (or teams) structure imposes a means and cadence to communication. How often and the way inter-team communications occurs has implicit limits on that communication.</p>
<p>Recognizing and working with that communications structure can make teams much more successful. Domain-Driven Design makes domains and sub-domains first-class citizens within the practices. Many aspects of architectural and social boundaries can affect the release of a product. A Bounded Context is more than a consistency boundary or scope of a domain model. A Bounded Context also involves work products (deployments, deliverables) and team organization.</p>
<p>For example, the consistency boundary of a mortgage loan application becoming complete and submitted is a fairly obvious boundary and context. Still, the amount of work involved to support that might be fairly large. The number of people implementing and supporting that context might amount to several teams. The complexity of dealing with several teams of people to deliver parts of the same system can be enormous. Domain-Driven Design also gives us some patterns and practices to address those complexities. You may need to split a domain into more bounded contexts because one context is too complex for a single team to manage. When we start to talk about separating work across teams, we're still talking about bounded contexts. For similar reasons, you may need to split a domain into more bounded contexts (and thus "sub-domains") simply because of an existing team or reporting structure.</p>
<p>For delivery to be more successful, it's important to recognize the different teams, reporting structures, team motivations, and missions within the strategic design of the Bounded Contexts. How two teams and how the work product of those two teams interact is unique. Fortunately, there are some patterns to address the dependencies between two teams and their work products that help us address their inherent complexities. I'm assuming there's always some degree of interdependency and independence, and I'm ignoring mutually independent (Separate Ways) and Big Ball of Mud relationships/structures.</p>
<blockquote class="blockquote">
<p>It's worth noting that as soon as two contexts are recognized the need to translate between the two becomes a reality. As context become complex to the point of being bounded context so too does the need to recognize and isolate translation. Much of what we do in Domain-Driven Design is the isolation of concepts, concerns, responsibilities, etc. The need for a translation layer is no different.</p>
</blockquote>
<p>There is a spectrum to the degree of independence of two teams and or the independence of their work products. The teams are very dependent on one end, while the other is extremely independent. With Domain-Driven Design, very dependent teams exhibit a lot of domain overlap. With a lot of domain overlap, you can have an interdependence where teams work as equals or partners. This partnership can manifest in an early re-org of people working on existing or legacy systems. That partnership may start with different teams working on separate parts of the codebase. This partnership may only be one step in the evolution of the teams; the next step is often to organize teams toward the Shared Kernel model.</p>
<blockquote class="blockquote">
<p>A spectrum of options is a synonym for <em>infinite combinations</em>. It's nice to have flexibility, but an endless set of possibilities is hard to map to a finite set of patterns, and it's hard to use established practices if every situation is novel. There are some ideas and structures that Domain-Driven Design details to add some granularity to the domain we're modeling so that we can more easily map complexity to the patterns and practices that address them.</p>
</blockquote>
<p>In the Shared Kernel model, the team carves off a separate shared codebase or a shared component to contain all the things that two or more teams will always or almost always mutually require. A Shared Kernel model involves organizational behavior, like specific responsibilities, code areas, accountability, etc. But Shared Kernel is a fairly casual relationship. With more formality between two teams or components, you usually see an Upstream/Downstream relationship form. It is easy to view the users of the Shared Kernel downstream dependents and evolve to a more formal Upstream/Downstream relationship. A Customer/Supplier model may emerge in cases like this.</p>
<blockquote class="blockquote">
<p>A shared codebase is more casual than a shared component, but a shared component promotes more independence. At the component level, it's important to ensure that autonomy hasn't allowed the teams to deviate from a shared plot, which Continuous Integration is intended to address. The component should be integrated with client code at every opportunity. The intent of a Published Language is for all contexts to be on the same page in understanding that domain. It's not that all contexts will adopt the published language as their domain, but they know how to translate in and out of their domain.</p>
</blockquote>
<p>In the Customer/Supplier model, one team owns a component the other uses as the consumer of the component's capabilities. The team that owns the component is the Supplier, and the team that uses it is the Customer. With this model comes organizational behavior with more specific responsibilities, more planning, and scheduling. With this increased independence, the supplier team has very specific goals in which the customer team has a stake and influence, represented in a release cadence and a roadmap. The Customer is usually the driver of what capabilities the component provides next. The integration model of Customer/Supplier is usually a web service.</p>
<blockquote class="blockquote">
<p>In a Upstream/Downstream model, there will almost always be some form of Published Language--usually more formal than just a description, often a specification. Translation becomes more formal in a Upstream/Downstream model, often resulting in a translation layer. If the Upstream/Downstream relationship is between two Bounded Contexts with a high degree of independence an Anticorruption Layer is used on one side to manage the differences communicating between the two domains.</p>
</blockquote>
<p>Shared kernel and two-team customer/supplier relationships often exist due to the reporting structure or that reporting structure was created to split work across two teams. In a more product-focused organization, you may have a Customer/Supplier model with more than one customer. Multi-customer relationships can be witnessed in larger organizations with things like shared libraries. The customers are still driving the capabilities that the component and team provide them, but it can become more formal to manage the unique requirements of different customers. The supplier team is often more organized or formal and may have more of a product strategy with a product vision and mission that helps guide their work.</p>
<blockquote class="blockquote">
<p>With more formal relationships come more formal expectations. Those expectations may come in the form of specifications and processes. Continuous Integration is an example of a process that continuously validates integrability. Potentially less formal than a specification may be a Published Language--which in its simplest form is a description of the concepts of a domain. (more complex forms would be varying degrees of specifications.)</p>
</blockquote>
<p>Communications with a customer/supplier model within the same organization can be informal. What the team is working on and how they interact with customers might be more like partnerships; teams may work closely together to implement and integrate components. The number of customers or distance from customers can impact this informalness. The further away a customer is (different division, different organization, different company) may impose more formality to the relationship. The work product of the supplier team may be viewed more like a product. And while customers may drive that product, it may be much more formal to the point where the component is independent of any single customer. This type of relationship may be structured more like a service with a very specific or well-specified interface. Moving towards a well-specified interface is the intent of an Open Host Service where the component is remotely accessed (a service) with a specified protocol and interface.</p>
<p>As a customer has less influence on a service, they may become completely dependent on the supplier team to provide the capabilities they require. They accept the risk that the supplier team may not provide the necessary capabilities in the future. This is extreme, and either no organization would accept this risk, or it is a temporary relationship. In reality, the different models aren't mutually exclusive, but there is a tendency towards one of them. e.g., a relationship tends to be less like a customer/supplier relationship and more like one completely conforming to another. The recognition of this relationship is called the Conformist model.</p>
<p>highlight: Recognize change will happen but don't try to create a design that accommodates all change.</p>
<p><img src="/assets/complex-relationships.jpg" class="img-fluid" alt="complex-relationships"></p>http://blog.peterritchie.com/posts/Being-Successful-With-Domain-Driven-Design--Minimal-Complexity-Part-2Being Successful with Domain-Driven Design: Minimal Complexity, Part 22023-05-29T00:00:00Z<p>In part one, I talked about the complexity in the language used to communicate the domain. Domain-Driven Design (DDD) deals with that complexity by isolating the concepts in a clear language that domain experts understand. Ubiquitous Language helps form the basis of all the other patterns and practices in Domain-Driven Design through the clear isolation of domain <em>concepts</em>. The DDD pattern language context map provides a good example of isolating concepts (in this case, Domain-Driven Design concepts):</p>
<p><img src="/assets/ddd-pattern-language.png" class="img-fluid" alt="ddd-pattern-language" /></p>
<p>Isolating individual concepts, naming them, and detailing how they relate allows each to be thought about independently. We can focus on parts of "Domain-Driven Design" because a Context Map details that isolation.</p>
<p>I'll dig deeper into the Aggregate and Service patterns in this part two. Aggregate and Service enable and embody major domain concepts.</p>
<p>An aggregate is the realization of a logical consistency boundary and the operations contributing to that consistency. An aggregate is a composition of several domain objects: At least one entity object and usually several value objects. Each domain object maintains its own consistency (it has invariants.) A date is a composition of a day, month, and year, but February 31, 1981 is not a valid date. We differentiate an aggregate from any grouping of domain objects because of the invariants and rules beyond that of simply a collection of consistent domain objects. An aggregate models that cross-object consistency requirement.</p>
<p>Aggregates map to major domain concepts and significant domain behavior associated with a particular domain object (the root). The root is the object of behavior and acts as a gateway to the other objects the aggregate comprises. The root takes on the responsibility of maintaining the consistency of the entire aggregate. The complexity of that composition and the consistency are separated from complexities outside the natural boundary of the aggregate.</p>
<p>An aggregate is not a design choice; it is a natural role that a major domain entity plays in the domain that requires recognition in a domain model. Some domain entities naturally take on certain behavior that affects other domain objects. All the behavior must happen in certain ways and have expectedly consistent results. Those consistent results (or state) abide by rules and invariants--it's consistent <em>because</em>... With a loan application, for example, there's no such thing as having negative assets (particularly: it's not a consistent loan application when it contains assets with negative value.) If you <em>owe</em> money, that's a liability.</p>
<p>When understanding and modeling a domain, I like to accurately map behavior to domain concepts that logically have that (or any) behavior. Sometimes it's easy to mis-associate behavior with static concepts when those are the major concepts in the domain. A loan, for example, is a major concept in many financial domains, but it does not exhibit behavior; it's static. A loan is the subject of many behaviors in a financial domain but is just a contract (or a specification.) We know we have complexity to deal with. Mis-associating behavior reduces clarity, making things needlessly more complex.</p>
<p>Starting out understanding a domain, I find focusing primarily on behavior and activities useful. Everything else in a domain is ultimately the subject of a behavior or activity, so I don't model them explicitly. For example, the role of underwriter <em>approves</em> a loan application. A loan application can be approved when the following rules are satisfied: a) the Debt-To-Income Ratio is below 43%, and b) etc... Debt-To-Income Ratio is modeled as an attribute of a loan application and covered in the activity of Approving a Loan. Information that isn't the object of a behavior also adds needless complexity.</p>
<p>When certain logic doesn't require a single object or doesn't require a <em>particular state</em> and requires <em>particular objects</em>, it might not be accurate to model the logic as part of an <em>aggregate</em>.</p>
<p>A Service is the realization of a collaboration between several objects. The concept of that service exists because it is not the natural behavior of a domain <em>entity</em>. A Domain Service is the realization of a collaboration between one or more domain entities. It involves business logic and or business rules that may affect the state of those domain entities. An application service is the realization of a collaboration between one or more domain services, domain objects, and an infrastructure service. And an infrastructure service is a collaboration with a framework or the "outside." The Service pattern recognizes complexity by isolating logic that is otherwise unrelated.</p>
<p>Look again at the Domain-Driven Design pattern language. If we always had to deal with all the complexities detailed in that diagram, it would be much harder to get things done promptly. The fact that the concepts are delineated and we can deal with them in isolation simplifies working with them. Separating interaction-only logic into a separate service from the object behavior affecting their state means we can think about and work with those concepts in isolation. Those concepts are now more loosely-coupled, and we obtain the benefits of loose coupling.</p>
<p>These definitions are easy enough to understand but can be confusing when it comes time to put them into practice. Sometimes the confusion stems from a backward approach to implementing software systems. Teams work backward from patterns when looking for the opportunity to use patterns. It's more successful in understanding the domain and then matching domain concepts to patterns and practices. For example, I've seen people approach a domain with questions like "What are all the entities?" or "What are all the services?" While we might be able to answer those questions after we've understood the domain and started to design solutions, approaching it from that perspective at that stage of understanding can pervert the interpretation of the domain. Services can be hard to recognize and implement when the domain concepts are not yet clearly isolated.</p>
<p>Sometimes it can be easy to delineate interaction logic in a collaboration from the business logic; often, it is not. In a financial domain, <em>transferring funds</em> as a capability can be easily viewed as the behavior of an account, for example. It involves accounts, obviously, so why would it not be an <em>account behavior</em>? But what happens when transferring funds? The situation's complexity comes from the fact that more than one account is involved, and the consistency of each account needs to be managed independently of the other(s). A funds transfer succeeds or fails, but has no state of its own--any change in state is encapsulated in the objects participating in the collaboration.</p>
<p>A clear understanding of the domain concepts is vital to project the isolation of those concepts into design elements more accurately. Modeling domain knowledge requires the delineation and understanding of the concepts as well as correctly associating all the behavior and attributes of those concepts. Maintaining this model is a process of managing complexity, but it only happens in stages. Managing these complexities is an ongoing and iterative process. The more complex a domain is, knowledge fragmentation amongst domain experts is more likely. It's highly unlikely that a single person completely understands the domain. This knowledge fragmentation is one reason we model the domain iteratively, recognizing that understanding evolves over time.</p>
<p>The Aggregate and Service patterns model parts of the domain similarly but provide a means to recognize separate parts of the domain: independent of the level at which they apply as well as how they affect state. Service operates at a higher level to model an activity, a collaboration of objects. An aggregate is the composition of several domain objects that must abide by the same invariants and be consistent in the presence of each other.</p>
<p>In part one, I talked about the complexity in the language used to communicate the domain. Domain-Driven Design (DDD) deals with that complexity by isolating the concepts in a clear language that domain experts understand. Ubiquitous Language helps form the basis of all the other patterns and practices in Domain-Driven Design through the clear isolation of domain <em>concepts</em>. The DDD pattern language context map provides a good example of isolating concepts (in this case, Domain-Driven Design concepts):</p>http://blog.peterritchie.com/posts/Being-Successful-With-Domain-Driven-Design--Minimal-Complexity-Part-1Being Successful With Domain-Driven Design: Minimal Complexity, Part 12023-05-19T00:00:00Z<p><img src="/assets/concepts-contexts-and-boundaries.jpg" class="img-fluid" alt="Concepts, Context, and Boundaries. Abstract Thought" />
The Domain-Driven Design book (the "Blue Book") includes "Tackling complexity at the heart of software" in the title. While "complexity" can be subjective, the takeaway is that Domain-Driven Design intends to address complex software systems. The principles and practices in Domain-Driven Design have their complexities, so for Domain-Driven Design to add value, it needs to address existing/expected complexity and attempt to be net-positive for simplicity.</p>
<p>Interestingly, the title of the Blue Book alludes to questions about <em>what</em> complexity is at the heart of software. Subjectively subjective, but we can look to the intent of some of the patterns and practices to deduce some types of complexity that Domain-Driven Design adequately addresses in the design of software systems.</p>
<p>For this series, I'll tranche away some of the patterns as essential complexity (essential complexity of <em>both</em> Domain-Driven design and the design of almost all software design): Entities, Value Objects, Modules, Layered Architecture, Factories, and Repositories. Any modular software system must deal with identity, value, creation, and storage. There's nothing new about layered architecture, but Domain-Driven Design does detail the isolation of specific responsibilities (like Domain and Infrastructure) that I'll cover. Practices like Side-Effect Free Functions, Standalone Classes, Intention-Revealing Interfaces, Continuous Integration, Assertions, and Declarative Style are aspects of long-championed techniques like cohesion, loose-coupling, naming standards, or functional programming (in my opinion).</p>
<p>None of what I've tranched off are unimportant, but they have been tried and true before Domain-Driven Design, and I want to focus on added value in Domain-Driven Design. To that end, I'll focus on Ubiquitous Language, Bounded Context, Context Map, Aggregates, Services, Domain Layer, Generic Subdomains, Segregated Core, Anti-Corruption Layer, and Core Domain; and touch on Evolving Order.</p>
<h3 id="clean-concepts-contexts-and-boundaries">Clean Concepts, Contexts, and Boundaries</h3>
<p>If I had to distill the intent of Domain-Driven Design to a single statement, it might be "be explicit." Or, more explicitly: "Be explicit with boundaries." Software systems are not the only source (or victim) of complexity. There's a whole science devoted to it: Complex Adaptive Systems. Not to oversimplify Complex Adaptive Systems, but systems with sufficient complexity are inherently unpredictable and exhibit <em>emergent behavior</em>, among other things. Meaning that complex systems will do what they're going to do, and we can only sometimes predict what they will do. Sometimes that emergent behavior is beneficial; sometimes, it isn't. To make systems more predictable (and get the benefits that provides), we have to reduce complexity. The complexity of complex systems arises from the number of dependencies, relationships, and interactions. Each unbounded interconnection increases complexity exponentially.</p>
<p>Complex adaptive systems theory is why explicit boundaries are a major aspect of how Domain-Driven Design combats complexity in software to produce more reliable and robust systems. Explicitness is important here; we're not looking for any-old boundaries. We're looking to constrain and isolate areas of the system based on purpose, meaning, and intent. We could chalk this up as simply an exercise in cohesion, but Domain-Driven Design focuses on getting to and clarifying that purpose, meaning, and intent.</p>
<p>Explicitness starts with unambiguous concepts, descriptions, and terms. If people aren't communicating the same concept things aren't going to get simpler. I speak about <em>Naming Things</em>, and part of what makes that difficult, I've decided, is language (or English). Stemming from human nature, we try to classify an ever-increasing set of concepts with a finite set of words, syntax, and semantics. The first step to explicit boundaries is the agreement on what they are: agreement on the concepts and to which explicit context they apply—the Ubiquitous Language.</p>
<p>The value of Ubiquitous Language isn't just that there is an agreed-upon vocabulary. The added value to a Ubiquitous Language is what it accounts for. The Ubiquitous Language recognizes classifications of concepts, classifications common to most software systems. Classifications that the Ubiquitous Language fosters and isolates: individuals, invariants and consistency rules, operations, processes, collaborations, commands, events, views, and values/properties. By "individuals," I don't just mean people, but anything that exhibits individuality (aka "entity.")</p>
<p>Imagine an amorphous "loan" concept in the financial industry. People apply for loans, obtain loans, and pay back loans. Getting a loan involves evaluating personal information (credit rating, assets, liabilities, etc.). Paying back a loan consists of a term, an interest rate, a payment schedule, etc. Credit rating, assets, liabilities, term, interest rate, and payment schedule are six interconnected concepts. With six concepts (with each having five interconnections to the others), there are 30 interconnections. Or 30 complexity points. But, if we think of these six concepts as two different semi-independent contexts: "loan application" and "loan servicing," we end up with two contexts (and one interconnection) with three concepts (or three interconnections) totaling six interconnections. We've gone from 30 complexity points to 6 simply by defining the actual contexts better. In other words, we're being more explicit.</p>
<p>Explicitness like this--delineating elements of a set into two groups connected by a specific relationship--is creating a boundary between two contexts. This is a very simple example of Bounded Context. As the name implies, Bounded Context is an explicit contextual boundary: where one context ends and another begins. This is a domain's macro level, recognizing that Loan Servicing can only happen after Loan Application is successful. This particular boundary is based on a temporal or procedural boundary. Phases or steps are a good way of organizing domains into bounded contexts.</p>
<p>In these two contexts, the word "loan" exists in both. The word "loan" is used in the application context as well as in the servicing context. But, in the application context, the meaning is really "loan application," and in the servicing context, it really means "serviced loan". Understanding that servicing depends on an approval event in a loan application phase (or activity) allows us to realize an explicit boundary. Sometimes it's as easy as this, but often it's not. There are other ways of teasing out boundaries (or contexts), almost always involving vocabulary elements.</p>
<p>Sometimes you've got overloaded terms like "loan"; sometimes, you have different terms like different <em>rules</em>. Different rules are typically applied in different scenarios or involve different parameters. Different rules offer a window into recognizing different contexts with a boundary in-between. You may recognize concepts like this (events, operations, rules/invariants) from the larger list I mentioned above. A Ubiquitous Language can also account for individuals and entities, commands (often related to an activity), views (reports, screens, results), collaborations, and attributes or properties attributed to individuals and entities. Additionally, attributes or properties can be involved in criteria, and categories or subtypes may group individuals and entities.</p>
<p>Working towards a Ubiquitous Language is working towards concepts more independent from each other. Independent concepts are themselves individual contexts. Any defined concept has a defined context with understandable boundaries. Keeping the complexity of one context bound from others keeps the essential complexity within that context and reduces the accidental complexity that arises from blended contexts.</p>
<p>Domain-Driven Design adds value when you have a minimal complexity, when a subject matter has multiple terms per classification. Terms can be classified as entities, processes, phases, events, rules, views, etc. The focus of this post was a level of complexity where boundaries are recognizable in the nuances of the vocabulary. In future posts, I'll dig deeper into different the subject matter (or domain) classifications, how you can isolate the complexities of each, and the parts of Domain-Driven Design that apply.</p>
<p><img src="/assets/concepts-contexts-and-boundaries.jpg" class="img-fluid" alt="Concepts, Context, and Boundaries. Abstract Thought">
The Domain-Driven Design book (the "Blue Book") includes "Tackling complexity at the heart of software" in the title. While "complexity" can be subjective, the takeaway is that Domain-Driven Design intends to address complex software systems. The principles and practices in Domain-Driven Design have their complexities, so for Domain-Driven Design to add value, it needs to address existing/expected complexity and attempt to be net-positive for simplicity.</p>http://blog.peterritchie.com/posts/installing-dotnet-framework-4-5-targeting-packInstalling .NET Framework 4.5 Targeting Pack2023-02-05T00:00:00Z<p><img src="/assets/DALL%C2%B7E-2023-02-05-13.14.52--ludites-frustration-with-errors-in-integrated-development-environments-(IDE)-pencil-and-watercolor.png" class="img-fluid" alt="When working in an IDE seems like working with crayons" /></p>
<p>Something came up with a client around Live Dependency Validation in Visual Studio recently. Digging into it I ran into several issues, one of which was the error:</p>
<pre><code>Severity Code Description Project File Line Suppression State
Error The reference assemblies for .NETFramework,Version=v4.5 were not found. To resolve this, install the Developer Pack (SDK/Targeting Pack) for this framework version or retarget your application. You can download .NET Framework Developer Packs at https://aka.ms/msbuild/developerpacks DependencyValidation C:\Program Files\Microsoft Visual Studio\2022\Enterprise\MSBuild\Current\Bin\amd64\Microsoft.Common.CurrentVersion.targets 1229
</code></pre>
<p>.NET Framework 4.5 has been out of support since 2016, so its targetting pack isn't available for download. I found a couple blogs posts about editing the modelproj file to add things like <code>ResolveAssemblyReferenceIgnoreTargetFrameworkAttributeVersionMismatch</code> or a <code>PackageReference</code> to <code>microsoft.netframework.referenceassemblies.net45</code> but neither worked.</p>
<p>One of the features of Visual Studio Installers is that they can be a one-stop-shop for all the things you're going to need to develop software (with or without Visual Studio). One of these features is to install .NET targeting packs! Although the latest version of Visual Studio doesn't include out-of-support components, prior versions of Visual Studio are available. Visual Studio 2019 came out before .NET Framework 4.5 was completely unsupported (i.e. still had the option of paid support) so it offers the ability to install some targetting packs that are currently out of support.</p>
<p>You can download older versions of Visual Studio via <a href="https://bit.ly/vs-old">https://visualstudio.microsoft.com/vs/older-downloads/</a>, which seems to redirect you eventually to Visual Studio Subscriptions downloads. For our purposes, Visual Studio Community Editions works fine.</p>
<p>To install, run the Visual Studio installer that you've downloaded (if you already have 2019 installed, run the already installed Visual Studio Installer and click <strong>Modify</strong>) then click <strong>Continue</strong> to go past the <em>set up a view things</em> dialog. (if you have VS 2022 installed, this seems to do nothing.)</p>
<table class="table">
<thead>
<tr>
<th><strong>Note</strong></th>
</tr>
</thead>
<tbody>
<tr>
<td>Make sure you don't change any of the <em>workloads</em> (if you have Visual Studio 2019 install already, some may be checked--don't uncheck them, that will uninstall them).</td>
</tr>
</tbody>
</table>
<p><img src="/assets/vs2019-installer-45-targetting-pack.png" class="img-fluid" alt="Visual Studio Installer" /></p>
<p>Click on the <strong>Individual Components</strong> tab at the top (to the right of <em>Workloads</em> and to the left of <em>Language packs</em>.) In the .NET section, find an check <strong>.NET Framework 4.5 targetting pack</strong>.</p>
<p>Click <strong>Install</strong> or <strong>Install while downloading</strong>.</p>
<p>Once completed, you now have the .NET Framework 4.5 targetting pack. If you're doing this in response to a .NET Framework 4.5 targetting pack error message in Visual Studio, exit and re-start Visual Studio--the error should go away (it does with the modelproj error.)</p>
<hr />
<p>Incidentally, the other issues I encountered are:</p>
<pre><code>Full solution analysis for C# is currently disabled. You may not be seeing all possible dependency validation issues in C# projects. Options... Don't show again
</code></pre>
<p>... with no way to enable full solution analysis in a way that this notice recognizes and goes away.</p>
<p>I'd appreciate any advice to resolve that that doesn't involve clicking <strong>Don't show again</strong>.</p>
<p><img src="/assets/DALL%C2%B7E-2023-02-05-13.14.52--ludites-frustration-with-errors-in-integrated-development-environments-(IDE)-pencil-and-watercolor.png" class="img-fluid" alt="When working in an IDE seems like working with crayons"></p>http://blog.peterritchie.com/posts/things-i-learned-attempting-azure-administrator-associate-part-2Things I Learned Attempting Azure Administrator Associate - Part 2 - Storage2022-12-22T00:00:00Z<p><img src="/assets/DALL%C2%B7E-2022-12-22-17.03.40--distributed-cloud-data-storage-in-the-style-of-salvator-dali.png" class="img-fluid" alt="distributed cloud data storage" /></p>
<p>Azure Administrator Associate certification is about the skills required to be an Azure account, subscription, tenant, etc., administrator. If your end goal is to develop applications on Azure, that involves a lot of <em>administration</em> of Azure resources. Regardless of your plan, storage administration is nuanced. This post focuses on some of those nuances, nuances that may not be apparent in the documentation.</p>
<h3 id="overview">Overview</h3>
<!--capabilities-->
<p>Azure provides storage services for Files, Blobs, Queues, and Tables. Files are blobs that support access via SMB protocol, AKA File Shares. Blobs are web resources that support access via a URI (HTTP). Blob Storage supports two types of blobs: block blobs and page blobs. Table Storage supports key-based relational and document access. Queues support access to ephemeral messages.</p>
<p>Azure Files has a File Sync feature that supports file-level replication across Windows Servers. The Azure File endpoint is also called the Cloud Endpoint and is part of a Sync Group that includes one or more Windows Server file shares.</p>
<p>There are two performance options for Storage Accounts: Standard (general purpose v2) and Premium (for low latency.)</p>
<!--tiers/skus-->
<p>Storage has a couple of storage tiers: Standard and Premium. Storage tiers provide different functionality at different costs. Blob Storage has several access tiers: Hot, Cool, and Archive. Access tiers offer a way to communicate the frequency and type of data access to reduce storage costs. Access tiers can be used to implement a lifecycle for data, moving to lower-cost tiers over time to reduce cost.</p>
<!--durability/high-availability-->
<p>Storage supports data redundancy that makes copies of data to avoid loss due to infrastructure failure. There are several options: Locally-Redundant Storage (LRS), Zone-Redundant storage (ZRS), Geo-Redundant Storage (GRS), and Geo-Zone-Redundant Storage (GZRS). LRS stores three copies of the data asynchronously within a single data center. ZRS duplicates those 3 LRS copies across three availability zones (clusters) in a region. GRS duplicates those 3 LRS asynchronously to a single zone in a secondary region. GZRS duplicates the ZRS data across zones within the secondary region.</p>
<!--data protection-->
<p>Recovery Services is the service responsible for storing backups and recovery points. Recovery Services stores data within Recovery Services Vaults.</p>
<p>Encryption scopes logically group blobs or containers and assign an encryption key specific to that scope.</p>
<!--access control-->
<p>There are a couple options for controlling access to data: Azure AD accounts/groups or Shared Access Signatures (SAS). Azure AD Groups provide a more manageable way to control Azure AD account access to data (than simply Azure AD accounts). SAS provides a granular means to provide delegate access to external entities.</p>
<h3 id="notable-information">Notable information <!--TIL--></h3>
<ul>
<li>LRS protects against rack-level hardware failure (so, if you want data-center-wide failure protection, LRS is insufficient).</li>
<li>LRS is supported for Standard File and Standard Block Blob account types (otherwise, GRS is the default.)</li>
<li>ZRS protects against data loss due to data center failure (so, if you want region-wide failure protection, ZRS is insufficient)</li>
<li>GRS protects against region failure.</li>
<li>GRS and GZRS secondary regions are pre-defined, forcing your data into a specific region.</li>
<li>GZRS protects against region failure and simultaneous data center failure in the secondary region.</li>
<li>When defining a data lifecycle, in the case of a tie, the option that results in the least cost will be chosen.</li>
<li>Migrating from LRS to GRS is supported with a feature called "Live Migration." Migration from LRS in other scenarios (e.g. to ZRS) must be done manually. Since Premium Storage accounts do not support LRS, Live Migration does not support Premium Storage accounts.</li>
<li>Live migration supports the use case of a storage account failure of GRS-replicated data. GRS is a second LRS in a secondary region. If a region fails, GRS reduces to LRS, so recovering means using Live Migration.</li>
<li>Durability is not backup; it provides access to data when infrastructure recoverability isn't an option. Apart from Live Migration, <em>restoration</em> is limited to manually copying live data when needed.</li>
<li>Durability does not protect against application-level failure; use backups or custom (application-level) durability in those scenarios.</li>
<li>Encryption scopes are useful for providing logical data tenancy.</li>
<li>Files added/modified in a File Share are only detected and replicated to the Windows Server file shares once every 24 hours (i.e., only visible after 24 hours).</li>
<li>Adding a file share to a Sync Group acts like all the files and folders within the file share were just added, replicating to the cloud endpoint and any other file shares.</li>
<li>When applying <em>least privilege</em> to storage accounts, the <strong>Reader</strong> role is also required on the Azure AD account if the Azure AD account needs to navigate storage resources in the Azure Portal.</li>
<li>Asynchronous data redundancy options introduce the possibility of data loss. If the asynchronous duplication to the secondary region did not complete, it is out of sync with the last state of the primary region. Application-level logic is required to prevent loss of data in this scenario.</li>
<li>The Archive tier does not have immediate access; it must be <em>rehydrated</em> to a cool/hot tier first (usually with a Copy Blob operation of up to 15 hours completion time).</li>
<li>File Share storage may be backed up to Recover Service vaults, but Blob Storage may not.</li>
</ul>
<p>This table summarizes the types of storage accounts and the features/redundancy that each support.</p>
<table class="table">
<thead>
<tr>
<th>Account</th>
<th>Redundancy</th>
<th style="text-align: center;">block blob</th>
<th style="text-align: center;">page blob</th>
<th style="text-align: center;">append blob</th>
<th style="text-align: center;">file share</th>
<th style="text-align: center;">queue</th>
<th style="text-align: center;">table</th>
</tr>
</thead>
<tbody>
<tr>
<td>Standard account</td>
<td>LRS</td>
<td style="text-align: center;">☑</td>
<td style="text-align: center;">☑</td>
<td style="text-align: center;">☑</td>
<td style="text-align: center;">☑</td>
<td style="text-align: center;">☑</td>
<td style="text-align: center;">☑</td>
</tr>
<tr>
<td>Standard account</td>
<td>GRS</td>
<td style="text-align: center;">☑</td>
<td style="text-align: center;">☑</td>
<td style="text-align: center;">☑</td>
<td style="text-align: center;">☑</td>
<td style="text-align: center;">☑</td>
<td style="text-align: center;">☑</td>
</tr>
<tr>
<td>Standard account</td>
<td>GAZRS</td>
<td style="text-align: center;">☑</td>
<td style="text-align: center;">☑</td>
<td style="text-align: center;">☑</td>
<td style="text-align: center;">☑</td>
<td style="text-align: center;">☑</td>
<td style="text-align: center;">☑</td>
</tr>
<tr>
<td>Standard account</td>
<td>RA-GZRS</td>
<td style="text-align: center;">☑</td>
<td style="text-align: center;">☑</td>
<td style="text-align: center;">☑</td>
<td style="text-align: center;">☑</td>
<td style="text-align: center;">☑</td>
<td style="text-align: center;">☑</td>
</tr>
<tr>
<td>Premium Block blobs account</td>
<td>LRS</td>
<td style="text-align: center;">☑</td>
<td style="text-align: center;">☐</td>
<td style="text-align: center;">☑</td>
<td style="text-align: center;">☐</td>
<td style="text-align: center;">☐</td>
<td style="text-align: center;">☐</td>
</tr>
<tr>
<td>Premium Block blobs account</td>
<td>ZRS</td>
<td style="text-align: center;">☑</td>
<td style="text-align: center;">☐</td>
<td style="text-align: center;">☑</td>
<td style="text-align: center;">☐</td>
<td style="text-align: center;">☐</td>
<td style="text-align: center;">☐</td>
</tr>
<tr>
<td>Premium File shares account</td>
<td>LRS</td>
<td style="text-align: center;">☐</td>
<td style="text-align: center;">☐</td>
<td style="text-align: center;">☐</td>
<td style="text-align: center;">☑</td>
<td style="text-align: center;">☐</td>
<td style="text-align: center;">☐</td>
</tr>
<tr>
<td>Premium File shares account</td>
<td>ZRS</td>
<td style="text-align: center;">☐</td>
<td style="text-align: center;">☐</td>
<td style="text-align: center;">☐</td>
<td style="text-align: center;">☑</td>
<td style="text-align: center;">☐</td>
<td style="text-align: center;">☐</td>
</tr>
<tr>
<td>Premium Page blobs account</td>
<td>LRS</td>
<td style="text-align: center;">☐</td>
<td style="text-align: center;">☑</td>
<td style="text-align: center;">☐</td>
<td style="text-align: center;">☐</td>
<td style="text-align: center;">☐</td>
<td style="text-align: center;">☐</td>
</tr>
<tr>
<td>Premium Page blobs account</td>
<td>ZRS</td>
<td style="text-align: center;">☐</td>
<td style="text-align: center;">☑</td>
<td style="text-align: center;">☐</td>
<td style="text-align: center;">☐</td>
<td style="text-align: center;">☐</td>
<td style="text-align: center;">☐</td>
</tr>
</tbody>
</table>
<p><img src="/assets/DALL%C2%B7E-2022-12-22-17.03.40--distributed-cloud-data-storage-in-the-style-of-salvator-dali.png" class="img-fluid" alt="distributed cloud data storage"></p>http://blog.peterritchie.com/posts/things-i-learned-attempting-azure-administrator-associateThings I Learned Attempting Azure Administrator Associate - Part 12022-12-20T00:00:00Z<p><img src="/assets/DALL%C2%B7E-2022-12-20-15.03.33---A-woman-going-through-the-process-of-certifying-knowledge.png" class="img-fluid" alt="person going through the process of certifying knowledge" /></p>
<p>I recently earned certification for Azure Administrator Associate. My goal is to make my experience and skills more verifiable in areas like application solution architecture. Azure Administrator Associate is a prerequisite for Azure Solutions Architect Expert and DevOps Engineer Expert (I imagine it's a prerequisite for all Azure * {Expert|Associate} certs.)</p>
<p>Certifications aren't perfect, "certification" has different meanings to the observer and the certification itself. Most certifications bring with them an expected minimum understanding of the subject. Does it mean the earner will do everything perfectly with the subject? Of course not, but it gives the person a certain vocabulary to communicate more efficiently on the subject.</p>
<p>The road to Azure Administrator Associate was interesting, and sharing some notable information would be helpful for others.</p>
<h2 id="making-the-implicit-explicit">Making The Implicit Explicit</h2>
<p>The key to good communication is clearly understanding a subject and eliminating assumptions and misunderstandings. While understanding what is expected of a certified Azure Administrator Associate, I noticed some knowledge that I realized is typically implicit. Another way of looking at the following is that each starts with "It may seem obvious, but...".</p>
<p>Implicit knowledge is knowledge obtained through incidental activities; knowledge gained without awareness of learning is occurring.</p>
<h3 id="line-of-business-lob-applications">Line of Business (LoB) Applications</h3>
<p>Line of Business (LoB) applications is ubiquitous in the computing industry. Everyone knows what it <em>means</em>, but if you ask two people to define it, you'll get more than one answer. While agreement/standardization on what a LoB application is isn't going to happen any time soon, there are certain truths about LoB applications:</p>
<ul>
<li>An in-house, custom web application</li>
<li>Not accessible via the Internet, either behind a firewall or strict access control (authentication and authorization)</li>
<li>Access <em>may</em> occur via an application gateway or load balancer</li>
<li>Specific to the company, business area, or industry</li>
</ul>
<h3 id="azcopy">AzCopy</h3>
<p>AzCopy works with Azure storage but only Azure Blob Storage and Azure Files.</p>
<h2 id="conclusion">Conclusion</h2>
<p>There are many areas of clarification with Azure Administrator Associate. Future posts on the subject will address clarifications involving important explicit limits, restrictions, constraints, rules, etc.</p>
<p>Are there other implicit aspects of Azure administration that can be made explicit?</p>
<p><img src="/assets/DALL%C2%B7E-2022-12-20-15.03.33---A-woman-going-through-the-process-of-certifying-knowledge.png" class="img-fluid" alt="person going through the process of certifying knowledge"></p>http://blog.peterritchie.com/posts/fundamental-webapi-integration-testsFundamental ASP.Net Minimal API Integration Tests2022-11-03T00:00:00Z<p><img src="/assets/robotic%20computing%20testing.png" class="img-fluid" alt="Robotic computing testing" /></p>
<p>I've been involved with some fairly large projects that involved RESTful APIs. When dealing with multiple team members, multiple teams, and OpenAPI specs, there can be many risks. Even when an OpenAPI specification is generated from source code, what the code does can easily be unaligned with the spec. Luckily the spec is a machine-readable contract of the <em>intent and purpose</em> of the API.</p>
<p>Automated testing to the rescue! With ASP.NET, you can inject into and observe the middleware pipeline. ASP.NET integration tests are a common way of verifying the pipeline and how it is used. We can create integration tests that process the OpenAPI spec and verify operations are working as expected in various ways. This article dives into a couple of these ways.</p>
<h2 id="fundamental-api-integration-tests">Fundamental API Integration Tests</h2>
<p>With a functioning Web API and an OpenAPI specification that describes it there are some fundamental things we can verify:</p>
<ul>
<li>The generated OpenAPI document is valid</li>
<li>The paths have endpoints implemented</li>
<li>The operations respond with the correct type of response</li>
</ul>
<p>First, let's set up our solution, projects, and integration testing scaffolding.</p>
<h2 id="setting-up-the-solution-and-projects">Setting Up the Solution and Projects</h2>
<p>We're dealing with a Web API and integration tests, so let's create a Web API project and make the <code>Program</code> class <code>public</code>. You can do that manually in Visual Studio; but for consistency, the CLI is powerful (I'm being intentional with framework versions and some configuration options--appending <code>public partial class Program { }</code> to Program.cs to make the class public):</p>
<pre><code class="language-PowerShell">dotnet new solution
dotnet new webapi -o WebApi --use-minimal-apis true --framework net6.0 --use-program-main false
echo public partial class Program { } >> WebApi\Program.cs
dotnet sln add WebApi\WebApi.csproj
</code></pre>
<p>Next, we want to add a test project. xUnit is my go-to, so we'll use that and add a reference to the Web API project. Again, in the CLI:</p>
<pre><code class="language-PowerShell">dotnet new xunit -o IntegrationTests --framework net6.0
del IntegrationTests\UnitTest1.cs
dotnet add IntegrationTests\IntegrationTests.csproj reference WebApi\WebApi.csproj
dotnet sln add IntegrationTests\IntegrationTests.csproj
</code></pre>
<p>For ASP.Net integration tests, we will use <code>WebApplicationFactory<T></code>, which requires a reference to <code>Microsoft.AspNetCore.Mvc.Testing</code>. In addition, to process OpenAPI documents, we'll need the <code>Microsoft.OpenApi.Readers</code> package. Again, via the CLI:</p>
<pre><code class="language-PowerShell">dotnet add IntegrationTests\IntegrationTests.csproj package Microsoft.OpenApi.Readers
dotnet add IntegrationTests\IntegrationTests.csproj package Microsoft.AspNetCore.Mvc.Testing
</code></pre>
<h2 id="integration-test-scaffolding">Integration Test Scaffolding</h2>
<p>I got into some of the scaffolding of ASP.NET 6 integration tests in <a href="#setting-up-the-solution-and-projects">Setting Up the Solution and Projects</a> concerning the required package references. the <code>Microsoft.AspNetCore.Mvc.Testing</code> package is required so that we may use the <a href="https://learn.microsoft.com/en-us/dotnet/api/microsoft.aspnetcore.mvc.testing.webapplicationfactory-1?view=aspnetcore-6.0"><code>WebApplicationFactory<TEntryPoint></code></a> class--which allows us to bootstrap a web application in memory, specifically for testing.</p>
<p>We'll use <code>WebApplicationFactory</code> to create an instance of an <code>HttpClient</code> test fake that works with our in-memory host. In addition, we'll override <code>WebApplicationFactory</code> to get at some of the Swashbuckle details from the pipeline. We're interested in the generated OpenAPI document for processing and the name of that document to generate the OpenAPI specification URI for verification. Here's an example of a <code>WebApplicationFactory</code> implementation that does what we need:</p>
<pre><code class="language-csharp">public class MyWebApplicationFactory : WebApplicationFactory<Program>
{
public OpenApiDocument? OpenApiDocument { get; private set; }
public string OpenApiDocumentName { get; private set; } = string.Empty;
protected override IHost CreateHost(IHostBuilder builder)
{
var host = base.CreateHost(builder);
using var scope = host.Services.CreateScope();
var sp = scope.ServiceProvider;
var swaggerGeneratorOptions = sp.GetRequiredService<IOptions<SwaggerGeneratorOptions>>().Value;
OpenApiDocumentName = swaggerGeneratorOptions.SwaggerDocs.First().Key ?? string.Empty;
var swaggerProvider = sp.GetRequiredService<ISwaggerProvider>();
OpenApiDocument = swaggerProvider.GetSwagger(OpenApiDocumentName);
return host;
}
}
</code></pre>
<p>The important parts are the <code>OpenApiDocument</code> and <code>OpenApiDocumentName</code> properties.</p>
<p>Now that we've got integration testing scaffolded let's create a test base class to make creating multiple integration tests clean and tidy.</p>
<h2 id="some-test-conventions">Some Test Conventions</h2>
<p>Automated testing classes and methods offer an opportunity to isolate and categorize tests to reduce work and clarify what is being tested (more importantly, what isn't passing). I tend towards a given/when/then structure when designing tests. The test class encapsulates the given/when (as well as the <em>arrange</em> from arrange/act/assert) whose name is suffixed with "Should." Each test method in the class is then given a name that describes the <em>then</em> condition. I try to ensure that there is one condition and thus one assert per method. YMMV.</p>
<p>For the tests I want to describe in this article, I've created a base class to encapsulate related given/when scenarios (or <em>shoulds</em>) that require the details we're accessing with the <code>WebApplicationFactory<Program></code> implementation. Naming is hard, so I'm starting simple with a <code>WebApiShouldBase</code> class that encapsulates the parts we're getting with <code>MyWebApplicationFactory</code> and an ability to get a stream to the "live" OpenAPI spec document (JSON). It also deals with the responsibility of owning those things (e.g., disposal):</p>
<pre><code class="language-csharp">public class WebApiShouldBase : IDisposable
{
private readonly string openApiSpecUriText;
protected readonly HttpClient WebApiClient;
protected OpenApiDocument? OpenApiDocument { get; }
protected Task<Stream> GetOpenApiDocumentStreamAsync() => WebApiClient.GetStreamAsync(openApiSpecUriText);
protected WebApiShouldBase()
{
var factory = new MyWebApplicationFactory();
WebApiClient = factory.CreateClient();
OpenApiDocument = factory.OpenApiDocument;
this.openApiSpecUriText = $"/swagger/{factory.OpenApiDocumentName}/swagger.json";
}
protected virtual void Dispose(bool isDisposing)
{
if (isDisposing)
{
Dispose();
}
}
public void Dispose() => WebApiClient.Dispose();
}
</code></pre>
<p>The important parts are the <code>OpenApiDocument</code> property which re-surfaces the <code>MyWebApplicationFactory.OpenApiDocument</code> to implementors, the <code>WebApiClient</code> property to access the API, and the <code>GetOpenApiDocumentStreamAsync</code> method that holds the OpenAPI spec document that the API provides. This class hides things like the URI to the swagger.json, the use of <code>MyWebApplicationFactory</code>, disposal, etc.</p>
<p>With that, let's start doing some tests!</p>
<h2 id="verifying-the-generated-openapi-is-valid">Verifying The Generated OpenAPI Is Valid</h2>
<p>"Valid" is subjective with OpenAPI. An OpenAPI spec is very <em>forgiving</em> in allowing for many opinions on what a <em>good</em> API looks like. I'm not going to go deep on what <em>good</em> might mean; just dive into facilitating validation of that generated document. The fact that there is an OpenApiDocument instance, and a raw OpenAPI specification, is an implementation detail. We'll use that OpenApiDocument instance shortly, but I want to ensure that the raw document meets some minimum requirements. For this example, the OpenAPI document is processed, not errors we detected, and there <em>are</em> paths. Very simple:</p>
<pre><code class="language-csharp"> [Fact]
public async Task ProduceValidOpenApi()
{
var readerResult = await new OpenApiStreamReader()
.ReadAsync(await GetOpenApiDocumentStreamAsync().ConfigureAwait(false)).ConfigureAwait(false);
Assert.NotNull(OpenApiDocument);
Assert.NotEmpty(readerResult.OpenApiDocument.Paths);
Assert.Empty(readerResult.OpenApiDiagnostic.Errors);
}
</code></pre>
<p>Client requirements can be less strict than development requirements (development objectives), and there may be different subsets of requirements in the case of multiple clients. This example doesn't implement that specifically but does provide the means to do it (by adding distinct test methods.)</p>
<p>OpenAPI.Net has can do very complex verification and validation, but I expect that sort of testing to be performed at a different level--I want to make sure client-oriented tests are handled here.</p>
<h2 id="verifying-the-paths-have-endpoints-implemented">Verifying The Paths Have Endpoints Implemented</h2>
<p>Publishing an API with paths and operations, and hosting an API that hasn't implemented those operations is silly. So the next test verifies they are implemented (at least the GET operations) as specified:</p>
<pre><code class="language-csharp"> [Fact]
public async Task EndpointsRespondOkToGet()
{
Assert.NotNull(OpenApiDocument);
var pathsWithGetOperations = OpenApiDocument.Paths.Where(w => w.Value.Operations.ContainsKey(OperationType.Get));
foreach (var (requestUriText, _) in pathsWithGetOperations)
{
var response = await WebApiClient.GetAsync(requestUriText).ConfigureAwait(false);
Assert.True(response.IsSuccessStatusCode);
}
}
</code></pre>
<p>GET operations are <em>easy</em>; they shouldn't have a request body and almost always have a success response specified. In the future, I can dive into other types of operations like POST, how to extract samples from the OpenAPI specification, and how to verify operations with request data and or error responses.</p>
<h2 id="verifying-the-operations-respond-with-the-correct-type-of-response">Verifying The Operations Respond With The Correct Type Of Response</h2>
<p>HTTP, and thus OpenAPI, don't enforce that any operation responds with anything in particular. But, if you're reading <em>this</em> blog, you are probably of the opinion that given the opportunity to specify behavior, you should be at least as detailed in specifying the type and schema of the responses. I'll leave out validating response schema in this article, but I will show verifying that each request responds with the correct media type. For example:</p>
<pre><code class="language-csharp"> [Fact]
public async Task EndpointsRespondWithCorrectMediaTypeToGet()
{
Assert.NotNull(OpenApiDocument);
var pathsWithGetOperations = OpenApiDocument.Paths.Where(w => w.Value.Operations.ContainsKey(OperationType.Get));
foreach (var (requestUriText, pathItem) in pathsWithGetOperations)
{
var responseContentType = pathItem.Operations[OperationType.Get]
.Responses[OkResponseCodeText]
.Content
.Single().Key;
var request = new HttpRequestMessage
{
Method = HttpMethod.Get,
RequestUri = new Uri(requestUriText, UriKind.Relative),
Headers =
{
{
HttpRequestHeader.Accept.ToString(),
responseContentType
}
}
};
var response = await WebApiClient.SendAsync(request).ConfigureAwait(false);
Assert.True(response.Content.Headers.ContentType?.MediaType ==
responseContentType);
}
}
</code></pre>
<h2 id="caveats">Caveats</h2>
<p>Of course, you can have or create an OpenAPI that does little more than document an endpoint and ignore that there are operations and those operations do specific things.</p>
<p>This article is an overview. I recognize that Swashbuckle and <del>Swagger</del>OpenAPI support in ASP.NET is powerful, but this article doesn't take into account many things you can do with it (like multiple OpenAPI documents.)</p>
<p>I also recognize that operations that take no parameters are rare, but I trust that my readers are good with taking on that as an exercise. Or, at least let me know if that's detail I should post in the future.</p>
<h2 id="summary">Summary</h2>
<p>This article provides a very high-level overview of integration testing ASP.NET minimal APIs. We then got into some details of general Web API integration tests that focus on OpenAPI specification aspects of the Web API middleware.</p>
<p>What sort of automated testing of an API specification do you see as beneficial to your projects?</p>
<h2 id="references">References</h2>
<ul>
<li><a href="https://learn.microsoft.com/en-us/aspnet/core/test/integration-tests?view=aspnetcore-6.0">Integration tests in ASP.NET Core</a></li>
<li><a href="https://learn.microsoft.com/en-us/dotnet/api/microsoft.aspnetcore.mvc.testing.webapplicationfactory-1?view=aspnetcore-6.0"><code>WebApplicationFactory<TEntryPoint></code></a></li>
</ul>
<p>The source for the examples, including the creation scripts can be found at <a href="https://github.com/peteraritchie/fundamental-webapi-integration-testing">https://github.com/peteraritchie/fundamental-webapi-integration-testing</a></p>
<p><img src="/assets/robotic%20computing%20testing.png" class="img-fluid" alt="Robotic computing testing"></p>http://blog.peterritchie.com/posts/visual-studio-defender-performanceVisual Studio Performance with Microsoft Defender2022-10-27T00:00:00Z<p><img src="/assets/antivirus%20exclusion%20developer.png" class="img-fluid" alt="Antivirus Exclusion Developer Dall-E image" /></p>
<p>Steve Smith posted about <a href="https://ardalis.com/speed-up-visual-studio-build-times/?utm_sq=h3m43zzlkm">speeding up built times in Visual Studio</a> by configuring Windows Defender. That was in 2016 and to say things have changed a bit is probably an understatement. Configuring a new laptop, I thought I'd revisit this briefly.</p>
<p>Before changing anything in Windows Virus & Threat Protection, go ahead and run a scan to make sure we're starting with a clean slate. <del>Go to <a href="ms-settings:windowsdefender">Windows Security</a> and click <strong>Virus & threat protection</strong> then click the <strong>Quick scan</strong> button.</del> I've been advocating scripting all-the-things, to run a quick scan in an administrator Powershell terminal run <code>Start-MpScan -ScanType QuickScan</code>. You can also run a full-scan, if that makes you more comfortable: <code>Start-MpScan -ScanType FullScan</code>.</p>
<p>Once that's complete we can configure Windows Virus and Threat Protection to "trust" (exclude) Visual Studio. To do that in PowerShell you can use the <a href="https://learn.microsoft.com/en-us/powershell/module/defender/add-mppreference?view=windowsserver2022-ps"><code>App-MpPreference</code> cmdlet</a> (as well as see what's already configured with the <a href="https://learn.microsoft.com/en-us/powershell/module/defender/get-mppreference?view=windowsserver2022-ps"><code>Get-MpPreference</code> cmdlet</a>). Some examples:</p>
<p>With Visual Studio 2022 Enterprise:</p>
<pre><code class="language-PowerShell">Add-MpPreference -ExclusionProcess "$Env:ProgramFiles\Microsoft Visual Studio\2022\Enterprise\Common7\IDE\devenv.exe"
</code></pre>
<p>With Visual Studio 2022 Professional:</p>
<pre><code class="language-PowerShell">Add-MpPreference -ExclusionProcess "$Env:ProgramFiles\Microsoft Visual Studio\2022\Professional\Common7\IDE\devenv.exe"
</code></pre>
<p>With Visual Studio 2022 Community:</p>
<pre><code class="language-PowerShell">Add-MpPreference -ExclusionProcess "$Env:ProgramFiles\Microsoft Visual Studio\2022\Community\Common7\IDE\devenv.exe"
</code></pre>
<p>And, with Visual Studio 2022 Preview:</p>
<pre><code class="language-PowerShell">Add-MpPreference -ExclusionProcess "$Env:ProgramFiles\Microsoft Visual Studio\2022\Preview\Common7\IDE\devenv.exe"
</code></pre>
<p>You can also do that for Visual Studio Code:</p>
<pre><code class="language-PowerShell">Add-MpPreference -ExclusionProcess "$Env:LocalAppData\Programs\Microsoft VS Code\code.exe"
</code></pre>
<p>You can also exclude the location of where you store your source code. The default location is <code>C:\Users\<user-name>\source\repos</code> for Visual Studio. So, in PowerShell, you can add a path exclusion:</p>
<pre><code class="language-PowerShell">Add-MpPreference -ExclusionPath "$Env:USERPROFILE\source\repos"
</code></pre>
<table class="table">
<thead>
<tr>
<th>Note:</th>
</tr>
</thead>
<tbody>
<tr>
<td>If you're working with Git repositories that you're unsure of what they contain, you may want to separate where you clone those repos from where you exclude.</td>
</tr>
</tbody>
</table>
<p>Or, if you want a PowerShell script to just do all thee things, see <a href="https://gist.githubusercontent.com/peteraritchie/d6025591566821b4ae5995eb831b6e8d/raw/912b5b20b749d506562437f40e169e6a3e24d279/optimize-defender.ps1">optimize-defender.ps1</a></p>
<p>Other processes to consider:</p>
<pre><code class="language-PowerShell">Add-MpPreference -ExclusionProcess "$Env:ProgramFiles\dotnet\dotnet.exe"
</code></pre>
<p>Any other processes or paths that you'd consider for exclusion?</p>
<p><img src="/assets/antivirus%20exclusion%20developer.png" class="img-fluid" alt="Antivirus Exclusion Developer Dall-E image"></p>http://blog.peterritchie.com/posts/By-Reference-in-csharpBy Reference in C#2022-09-28T00:00:00Z<p>I became aware recently that there were many C# compiler errors that do not have a corresponding documentation page. That documentation is open-source and I chose to spend some time contributing some pages for the community. Looking at a language feature from the perspective of its compile-time errors is rather enlightening, so I'd though I'd write a bit about these features in hopes of offering a better understanding for my readers.</p>
<p>C# compiler errors can be categorized (arbitrarily) by different areas of C# syntax, and I started to focus on one category at a time. One of those areas involves referenced variables. C# has always has <code>ref</code> arguments, but <code>ref</code> return, <code>ref</code> locals, <code>ref</code> structs, and <code>ref</code> fields have been additions to the syntax.</p>
<p>The declaration of a variable in C# influences its syntax in a couple ways: binding and accessibility. Accessibility is whether an identifier is <em>visible</em> at compile-time in a given context. Binding is how an identifier or name is bound at run-time to resources like data and code. Binding uniquely affect the compile-time correctness of any particular usage of a <code>ref</code> variable.</p>
<p>Binding affects the compile-time usage of an identifier because of the run-time lifetime of the resources it is bound to. You're probably familiar with a static method accessing instance data and the errors caused in this context. <code>ref</code> variables have a similar context when they are stack allocated. Heap-allocated objects (objects bound to the heap) can have their lifetime extended to be long-lived because the heap shares the same lifetime as the application. Variables bound to stack-allocated resources cannot have their lifetime extended beyond a specific scope. The stack is a sequential collection of elements with elements implicitly partitioned by a shared scope. The most recognized scope is probably a method call or method/lambda body. Local variables bound to stack elements do not have a lifetime beyond the method call. A reference to a stack object cannot be assigned to a variable or expression with a broader scope.</p>
<p>How far the value of an expression can leave the confines of its declaration scope is called "escape scope". Sometimes the escape scope is the same as the declaration scope. The compiler verifies compatible escape scopes during assignment. For example:</p>
<pre><code class="language-csharp"> void M(ref int ra)
{
int number = 0;
ref int rl = ref number;
if (ra == 0)
{
int x = number;
rl = ref x;
}
}
</code></pre>
<p><code>x</code> is local to the <code>if</code> body, it is bound to the stack, and its escape scope is narrower than <code>ref rl</code> because <code>ref rl</code> is declared in the outer scope. Since <code>ref rl</code> is an alias to another variable, it cannot reference a variable bound to a resource that will go out of scope before it does. <code>rl = ref x</code> results in a compiler error. If <code>rl</code> were not a reference to a value type, the assignment would be okay because <code>x</code> would be bound to heap and have a broader escape scope.</p>
<p>The compiler also verifies compatible escape scopes when returning values. For example:</p>
<pre><code class="language-csharp"> ref int M(ref int ra)
{
int number = 0;
ref int rl = ref number;
if (ra == 0)
{
ref int x = ref number;
return ref x;
}
return ref ra;
}
</code></pre>
<p><code>return ref x</code> results in a compiler error because the escape scope of <code>x</code> is local to the method. The error message here may not be as clear as the first because it doesn't mention the narrower escape scope.</p>
<p>There are the basic escape scopes. A calling method scope, a current method scope, and a return-only scope.</p>
<p>The calling method scope is a scope outside of the containing method/lambda. References can reach this scope via either a <code>ref</code> parameter or a return.</p>
<p>The current method scope is a scope within a containing method/lambda.</p>
<p>The return-only scope is a special case for <code>ref struct</code> types that can only leave the method scope via a return and not through a <code>ref</code> or <code>out</code> parameter.</p>
<p>I became aware recently that there were many C# compiler errors that do not have a corresponding documentation page. That documentation is open-source and I chose to spend some time contributing some pages for the community. Looking at a language feature from the perspective of its compile-time errors is rather enlightening, so I'd though I'd write a bit about these features in hopes of offering a better understanding for my readers.</p>http://blog.peterritchie.com/posts/the-fundamental-quality-attributes-of-technology-systemsFundamental Quality Attributes of Technology Systems2022-09-16T00:00:00Z<p>What are quality attributes? The term "non-functional requirements" has been more prevalent, but that is a technologist's term. The first time you bring up "non-functional requirements" with a customer, there's always confusion, then concern. I've heard more than once, "we want something functional."</p>
<p>The most important attribute of a system is its functionality. Functionality is whether a system is <em>fit for purpose</em>. A system's functionality is what makes it unique, so I'll defer detail on functional attributes for another time. In this post I'll focus this post on cross-cutting quality attributes that permeate all aspects of a solution.</p>
<p>Quality attributes are the characteristics a system needs to exhibit--qualifications of the system's desired functionality. Quality attributes address customer concerns regarding the degree of success of a system. A customer's concerns of a system are unique and thus precludes having a universal prioritized list of quality attributes.</p>
<p>It is simple to describe the characteristics a customer expects of a system that provides the features they need: Customers expect features that operate without fault or error, operate consistently within expectations, operate within resource constraints, and protect from unauthorized access. Translating that into a collection of system-specific measures is an enormous undertaking that cannot be taken lightly.</p>
<p>Many philosophies about quality attributes (usually termed "quality models") exist, like FURPS, ISO/IEC 9126/25010, McCall, etc. These models detail several categories that organize the many adjectives that can apply to software systems. Some common categories may include Reliable, Efficient, Maintainable, Secure, etc. I view quality attributes as a palette of possible adjectives; no one list is perfect for every situation. There are top-/high-level categorizations that can apply more broadly. When discussing quality attributes, we use the noun form (Reliability vs. Reliable.) I've landed on the following top-level categories (in no particular order): Performance, Operability, Security, and Dependability.</p>
<h2 id="dependability-of-features">Dependability (of features)</h2>
<ul>
<li>to function, without fault or error</li>
</ul>
<p>Dependability involves concerns such as:</p>
<ul>
<li>availability — readiness for usage</li>
<li>reliability — continuity of service</li>
<li>safety — non-occurrence of catastrophic consequences on the environment</li>
<li>confidentiality — non-occurrence of unauthorized disclosure of information</li>
<li>integrity — non-occurrence of improper alterations of information</li>
<li>maintainability — aptitude to undergo repairs and evolution</li>
</ul>
<h2 id="performance-of-execution">Performance (of execution)</h2>
<ul>
<li>To function within resource constraints (time, compute, storage, memory, network) constraints</li>
</ul>
<p>Performance involves concerns such as:</p>
<ul>
<li>latency - the degree of responsiveness.</li>
<li>throughput - the rate at which work can be performed</li>
<li>capacity - the amount of work that can be performed</li>
</ul>
<h2 id="operability-of-function">Operability (of function)</h2>
<ul>
<li>to become and remain operable.</li>
</ul>
<p>Operability involves concerns such as:</p>
<ul>
<li>deployability - the ability of a system to be put into production</li>
<li>monitorability - the ability of a system's health and operation to be monitored</li>
<li>configurability - the ability of a system's behavior to be customized</li>
</ul>
<h2 id="security">Security</h2>
<ul>
<li>To ensure authorized usage.</li>
</ul>
<p>Security involves concerns such as:</p>
<ul>
<li>Confidentiality - the quality of a system restrict access to information</li>
<li>Integrity - the quality of a system adhere to accuracy and consistency of information and behavior</li>
<li>Availability - the quality of a system to provide access to information to those authorized</li>
<li>accountability - the quality of a system to account for its actions when fulfilling its responsibilities</li>
</ul>
<p>Because quality attributes address customer concerns, there are overlaps between categories. For example, there's a dependability concern that data integrity does not impact functionality; there's also a security concern that data integrity doesn't result in data loss. Don't let the overlap distract you from what's best for your solution. It will change even if you could pre-define a complex structure of quality attributes most suitable for your solution. Needs change, priorities change, and quality attributes are philosophic influencers to a solution that requires nurturing.</p>
<p>What are quality attributes? The term "non-functional requirements" has been more prevalent, but that is a technologist's term. The first time you bring up "non-functional requirements" with a customer, there's always confusion, then concern. I've heard more than once, "we want something functional."</p>http://blog.peterritchie.com/posts/naming-things-events-and-actionNaming Things - Common Actions and Events2022-09-01T00:00:00Z<p>In this multi-part series on Naming Things, I dig into the benefits of having a clear understanding of common terms and concepts--in this case, common actions and events.</p>
<p>What does Deleted mean? Is it the same as Removed or Destroyed? What if you want to support soft delete as well as hard delete?</p>
<p>I want to be clear; these aren't developer decisions. They're developer problems based on a lack of clarity in the customer's domain. A customer likely won't use terms like "soft delete" and "hard delete ."The customer will probably refer to the most common form of delete as "delete." An architect role is responsible for teasing out the nuances of meaning into terms that the subject matter experts agree upon and getting consensus on usage with the development team.</p>
<p>Every system involves mutating data and information, yet it can be a common source of confusion regarding naming things. There are multiple types of data changes. Systems can create new data and may add that new data to a collection--physical or logical. Systems may change data, or designers may change the structure of data. Data is often removed--from a particular view or existence.</p>
<p>English allows us to reuse terms to mean many things. "Delete" and "changed," for example. There are well-known terms that enable us to communicate intent and consequences easily. But, when we reuse these terms across different contexts with different intents and consequences, we introduce the possibility of confusion, making naming things seem difficult.</p>
<p>It's important to understand the different intents and resulting consequences to data and attempt to get consensus on names and terms that adequately and uniquely represent these situations.</p>
<p>"Delete" is a common point of confusion. There may be a need for <em>soft deletes</em> and <em>hard deletes</em>; both of WHICH make data inaccessible in a context. But, data may also be moved from one context to another, changing its accessibility but not making it <em>inaccessible</em>. To use a single term like "delete" for all of these situations leads to confusion and issues in naming things.</p>
<!--
An important aspect of naming: name the consistency boundary. What is a date? A year, month, and day. If we included time, is it still a date? Typically that would be called date/time.
-->
<p>Each domain can be different, but the situations I've just described are very common. For those, I start with unique terms for each and work with the subject matter experts to refine them (if needed):</p>
<ul>
<li><strong>Deleted</strong> means something is no longer accessible in some context.</li>
<li><strong>Destroyed</strong> means removed from existence; no possible way to ever get it back.</li>
<li><strong>Removed</strong> means a thing has been moved out of or removed from a container/collection.</li>
</ul>
<p>Inverse terms:</p>
<ul>
<li><strong>Created</strong> signifies something new has come into existence (rather than Added).</li>
<li><strong>Added</strong>* signifies something has been added to a container or collection.</li>
</ul>
<p>Mutating information seems like such a simple concept. But, we often need to know if data changes within the context of other data. Unique data mutation terms I start out with when working with subject matter experts:</p>
<ul>
<li><strong>Updated</strong> signifies the value properties or attributes of an existing thing (entity) have been changed.</li>
<li><strong>Changed</strong> signifies an entire thing (entity) has been replaced with another.</li>
</ul>
<h2 id="word-form">Word Form</h2>
<p>Actions are verbs, and events are past participles constructed from verbs. Most event names are constructed from regular verbs by adding the prefix -ed. Delete + ed: deleted; create + ed: created. Sometimes events are constructed from irregular verbs and don't end in ed. Bind: bound; drive: driven; sleep: slept.</p>
<p>Events are not simply verbs in past tense form. An event's context is that it is related to a subject. For example, the event "deleted" involves a subject and is used to describe the current state as a result of a past action. In grammar, this is <em>present perfect tense</em> and implies an auxiliary verb of "has been ."e.g., <em>The customer record has been deleted</em> or <em>The customer record is deleted</em>. Since this details the subject somehow, it is also in a <em>present indicative</em> form. This detail is important but normally for edge cases. Normal domain narratives should align with this because that's normal language in these scenarios.</p>
<ul>
<li>Match events to actions; don't define events when no action would result in that event.</li>
<li>Don't assume events always end in "ed."</li>
<li>Events are present indicative, past participles, and in the present perfect tense.</li>
</ul>
<p>Do you have other actions and events that you commonly encounter?</p>
<p>In this multi-part series on Naming Things, I dig into the benefits of having a clear understanding of common terms and concepts--in this case, common actions and events.</p>http://blog.peterritchie.com/posts/Message-oriented-Minimal-APIs-in-ASP.NETMessage-oriented Minimal APIs in ASP.NET Core2022-08-28T00:00:00Z<p>TL;DR - <a href="#implementation-details">go to the implementation details</a></p>
<p>First of all, what is message-oriented? Like many things in technology and life, terms like "oriented" are frequently used whose meaning may not be immediately connected to its usage. Object-oriented, message-oriented, aspect-oriented, etc. have a vague meaning when used, which can sometimes introduce a lack of clarity.</p>
<p>*-Oriented means that a particular concept always taken into consideration and utilized in the specified circumstances. Message-oriented means that interactions between concepts <em>at a certain layer</em> involve messages. This is sometimes referred to as Message Passing; or that communication between layer-specific concepts it done by passing messages to each other.</p>
<p>Things can be oriented in many ways at the same time. A system may be simultaneously message-oriented and object-oriented, for example--which usually means that what produces and consumes messages are implemented as <em>objects</em>.</p>
<!--Some may say that the Command Pattern is message-oriented. For the purposes of this article, that's not the complete story. A command can (and should) be a message; but a message on it's own does not constitute message-orientation. Falling back to Message Passing, it's how the command is communicated is what makes a component message-oriented.
## Is HTTP Message-Oriented?
Yes, HTTP is message-oriented. Underlying layers (like TCP) implement HTTP via frames and streams (and packets, datagrams, etc.) but at the HTTP layer, communication between major concepts is done with what is called "messages".
-->
<h2 id="being-message-oriented">Being Message-Oriented</h2>
<p>Message-orientation demands a level of loose coupling. In object-oriented message-orientation objects donn't communicate with each other directly, a third-party transports message from the sender (or producer) to the receiver (or the consumer). There an many ways that can happen: queues (or simply collections), mediators, buses, etc. The type of the third-party component depends on the degree of loose coupling and how much work the third party takes on to transport those messages. For the purposes of this article I'll focus on <em>Bus Architecture</em>. Bus Architecture is a combination of a common data model, a common command set, and an infrastructure that provides a shared set of interfaces to transport messages.</p>
<h2 id="being-successfully-message-oriented">Being Successfully Message-Oriented</h2>
<p>As with any layered architecture, implementation details at different layers allow us to compartmentalize different concerns to ease understanding and simplify implementation. Message-oriented systems often classify messages to better implement and support a subdomain. Messages are often classified as commands, events, or documents to better support common subdomain and communication scenarios. Commands are messages that communicate a request or imperative intent. Events are messages that communicate or encapsulate a change in state. And documents are messages that contain data independent of a command or an event.</p>
<h2 id="why-minimal-apis">Why Minimal APIs?</h2>
<p>You may have read articles like "<a href="https://ardalis.com/mvc-controllers-are-dinosaurs-embrace-api-endpoints/">MVC Controllers are Dinosaurs - Embrace API Endpoints</a>" that suggest that modern MVC implementation have controllers that don't actually "control" anything. MVC details that Models, Views, and Controllers are decoupled from one another and cohesive in and of themselves. Controllers should interpret input and convert it into invocations upon the model and the view. Models are a dynamic data structure that directly manages data, logic, and rules for a given context. When MVC was devised, the controller was far closer to the user and took on more responsibility to imperatively translate and route data. With modern systems and technologies like JSON and programming language syntax, much of that translation and routing can be declarative--wiring up a request, its route, and the receiver of the command directly.</p>
<table class="table">
<thead>
<tr>
<th style="text-align: center;">As an aside, many have argued that <em>model</em> and <em>view</em> have been muddied and that <em>view</em> doesn't exist with RESTful APIs, questioning "MVC" implementations altogether.</th>
</tr>
</thead>
</table>
<p>When you don't really have a controller and data translation occurs under the hood, going through the motions of controllers and models with core subdomain objects is viewed as needless ceremony.</p>
<h2 id="message-oriented-frameworks">Message-Oriented Frameworks</h2>
<p>I've been working with a simple-but-no-simpler messaging library for several years. It's a set of libraries that I maintain called <code>[PRI.Messaging](https://github.com/peteraritchie/Messaging)</code>. It consists of primitive types (abstractions) (<code>PRI.Messaging.Primitives</code>) and pattern implementations (<code>PRI.Messaging.Patterns</code>). It makes ideas like consumers, producers, and buses first-class concepts. PRI.Messaging.Patterns includes a bus implementation that assumes the role of dependency injection and message routing, allowing you to simply create message producers, message consumers, and have them automatically wired-up and messages routed appropriately.</p>
<p>I'll be using these libraries to implement message-oriented minimal APIs in ASP.NET Core 6+.</p>
<h2 id="implementation-details">Implementation Details</h2>
<p>For simplicity, I'll show making the default project created for minimal APIs message-oriented; get ready for some weather forecasting.</p>
<p>Starting with creating the default project:</p>
<pre><code class="language-powershell">dotnet new webapi -minimal -o WebApi
dotnet new sln -n example
dotnet sln example.sln add WebApi
</code></pre>
<p>This gives us OpenAPI (Swagger) and HTTPs support along with a single <code>weatherforecast</code> endpoint and a <code>WeatherForecast</code> response model (message).</p>
<p>The important code from Program.cs:</p>
<pre><code class="language-csharp">var summaries = new[]
{
"Freezing", "Bracing", "Chilly", "Cool", "Mild", "Warm", "Balmy", "Hot", "Sweltering", "Scorching"
};
app.MapGet("/weatherforecast", () =>
{
var forecast = Enumerable.Range(1, 5).Select(index =>
new WeatherForecast
(
DateOnly.FromDateTime(DateTime.Now.AddDays(index)),
Random.Shared.Next(-20, 55),
summaries[Random.Shared.Next(summaries.Length)]
))
.ToArray();
return forecast;
})
.WithName("GetWeatherForecast")
.WithOpenApi();
app.Run();
record WeatherForecast(DateOnly Date, int TemperatureC, string? Summary)
{
public int TemperatureF => 32 + (int)(TemperatureC / 0.5556);
}
</code></pre>
<p>To become message-oriented we need to first create explicit messages to represent the interactions. For this I've created a <code>GetWeatherForecastCommand</code> command and a <code>WeatherForecastedEvent</code> event:</p>
<pre><code class="language-csharp">public class GetWeatherForecastCommand : ICommand
{
public string CorrelationId { get; set; } = Guid.NewGuid().ToString();
}
</code></pre>
<pre><code class="language-csharp">public class WeatherForecastedEvent : IEvent
{
public WeatherForecast[] Forecasts { get; }
public WeatherForecastedEvent(WeatherForecast[] forecasts)
{
Forecasts = forecasts;
}
public DateTime OccurredDateTime { get; set; } = DateTime.UtcNow;
public string CorrelationId { get; set; } = Guid.NewGuid().ToString();
}
</code></pre>
<p>Now we need something that explicitly handles (consumes) the <code>GetWeatherForecastCommand</code> command produces the <code>WeatherForecastedEvent</code> event. I prefer to call these types of things "Command Handlers". So,:</p>
<pre><code class="language-csharp">public class GetWeatherForecastCommandHandler : IConsumer<GetWeatherForecastCommand>, IProducer<WeatherForecastedEvent>
{
private IConsumer<WeatherForecastedEvent> consumer = new ActionConsumer<WeatherForecastedEvent>((_) => { });
public void AttachConsumer(IConsumer<WeatherForecastedEvent> consumer)
{
this.consumer = consumer;
}
private readonly string[] summaries = new[]
{
"Freezing", "Bracing", "Chilly", "Cool", "Mild", "Warm", "Balmy", "Hot", "Sweltering", "Scorching"
};
public void Handle(GetWeatherForecastCommand message)
{
var forecasts = Enumerable.Range(1, 5).Select(index =>
new WeatherForecast
(
DateTime.Now.AddDays(index),
Random.Shared.Next(-20, 55),
summaries[Random.Shared.Next(summaries.Length)]
))
.ToArray();
consumer.Handle(new WeatherForecastedEvent(forecasts));
}
}
</code></pre>
<p>As you'll see in this command handler, I've moved the implementation details from Program.cs and encapsulated them into this class (i.e. <code>summaries</code> and the creation of the <code>WeatherForcast</code> array.)</p>
<p>Returning to Program.cs, we now need to inject a <code>Bus</code> service and update the route endpoint to accept a bus instance, translate to our command, and send it to the bus.</p>
<pre><code class="language-csharp">app.MapGet("/weatherforecast", async (IBus bus) =>
{
WeatherForecastedEvent result =
await bus.RequestAsync<GetWeatherForecastCommand, WeatherForecastedEvent>(
new GetWeatherForecastCommand());
return Results.Ok(result.Forecasts);
})
.WithName("GetWeatherForecast")
.WithOpenApi();
</code></pre>
<p>The <code>IBus</code> <code>RequestAsync</code> extension method implements the asynchronous request-reply pattern.</p>
<p>With our messages, consumers, and producers we can now create a message bus singleton and have it wire-up the producers and the consumers. This is simply done by invoking the <code>IBus.AddHandlersAndTranslators</code> method in addition to registering a singleton bus:</p>
<pre><code class="language-csharp">IBus bus = new Bus();
bus.AddHandlersAndTranslators(
Path.GetDirectoryName(typeof(Program).Assembly.Location)!,
Path.GetFileName(typeof(Program).Assembly.Location), "");
builder.Services.AddSingleton(bus);
</code></pre>
<h2 id="summary">Summary</h2>
<p>As you can see, the route endpoint has an <code>IBus</code> injected into it (i.e. Dependency Injection) and is only concerned with sending a <code>GetWeatherForecastCommand</code> message and receiving a <code>WeatherForecastedEvent</code> message. Where that command goes and where the event comes from (and how it gets created) are irrelevant (i.e. neither knows nor cares about <code>GetWeatherForecastCommandHandler</code>). With the implementation details of weather forecasting moved out into <code>GetWeatherForecastCommandHandler</code> those details are now longer directly coupled to a web API. <code>GetWeatherForecastCommandHandler</code> exists in its own library and can be used by several types of applications. <code>GetWeatherForecastCommandHandler</code> could be used, as is, within a console application, a PowerShell CmdLet, etc. As with any well designed loosely coupled system, it's just a matter of correctly setting up a service container for the specific circumstances.</p>
<p>How will you use event-orientation?</p>
<h2 id="references">References</h2>
<ul>
<li><a href="https://ardalis.com/mvc-controllers-are-dinosaurs-embrace-api-endpoints/">MVC Controllers are Dinosaurs - Embrace API Endpoints | Ardalis Blog</a></li>
<li><a href="https://github.com/peteraritchie/Messaging">PRI.Messaging - GitHub</a></li>
<li><a href="https://www.nuget.org/packages/PRI.Messaging.Primitives/3.0.0-beta">PRI.Messaging.Primitives - Nuget</a></li>
<li><a href="https://www.nuget.org/packages/PRI.Messaging.Patterns/3.0.0-beta">PRI.Messaging.Patterns - Nuget</a></li>
<li><a href="https://docs.microsoft.com/en-us/azure/architecture/patterns/async-request-reply">Asynchronous Request-Reply Pattern - Azure Architecture Center</a></li>
<li><a href="https://www.enterpriseintegrationpatterns.com/RequestReply.html">Request-Reply Messaging Pattern - Enterprise Integration Patterns</a></li>
</ul>
<p>TL;DR - <a href="#implementation-details">go to the implementation details</a></p>http://blog.peterritchie.com/posts/data-urls-in-markdown Data URLs in Markdown2022-06-27T00:00:00Z<ul>
<li>Data URLs embed data within the URI instead of being a link</li>
<li>Data URLs can be used to embed images into a web page</li>
<li>Data URLs can be used for images in markdown</li>
</ul>
<p><a href="#data-urls">TL;DR</a>⮷</p>
<h2 id="background">Background</h2>
<h3 id="links-in-markdown">Links in Markdown</h3>
<p>URLs can be used in markdown as a hyper-link to another location (another page, or an anchor in the current page, or both.) These links in markdown come in two varieties: "conventional" and reference-style. Conventional links have the format <code>[text](url)</code>.</p>
<p>Reference-style links have a two-part format. The first is the reference declaration which consists of a name and the URL. For example:</p>
<pre><code class="language-markdown">[twitter]: https://twitter.com/peterritchie
</code></pre>
<p>Typically the reference declarations appear at the end of the markdown file.</p>
<p>That second-part of the format is slightly different than a conventional link: <code>[Visible Text][reference-name]</code> (notice that both parts are enclosed in square brackets)</p>
<p>For example: <code>[Me@Twitter][twitter]</code>. Which would result in <a href="https://twitter.com/peterritchie">Me@Twitter</a>.</p>
<p>That reference can be used any number of times within markdown by referencing the reference name.</p>
<h3 id="images-in-markdown">Images in Markdown</h3>
<p>An image in markdown is a special kind of link; it follows the same format as other links <em>except</em> that it starts with an exclamation mark (!) and has an optional title-text enclosed in quotes. For example <code>![alt-text](url "title-text")</code>. Since the image is what is visible, the title text portion of the link is text shown when hovering over the image and the alternative text is used by accessibility features.</p>
<h2 id="data-urls">Data URLs</h2>
<p>There exists the ability to encode content within a URL (so the URL is actually the "response" in the conventional URI request scenario). <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/Data_URLs">Data URLs</a> (Formally known as Data <em>URIs</em>) use the <code>data:</code> URI schema followed by an optional media type, an optional base64 extension (<code>;base64</code>) followed by data (if the <code>base64</code> extension is used the data is binary and is base-64 encoded).</p>
<p>For example, with binary data for a red dot png a data URL may look like this:</p>
<pre><code>data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAUAAAAFCAYAAACNbyblAAAAHElEQVQI12P4//8/w38GIAXDIBKE0DHxgljNBAAO9TXL0Y4OHwAAAABJRU5ErkJggg==
</code></pre>
<h3 id="images-data-urls-and-reference-style-markdown-links">Images, Data Urls, and Reference-Style Markdown Links</h3>
<p>Data URLs may be used in markdown image links. With image markdown format and a data URL:</p>
<pre><code class="language-text">![a red dot](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAUAAAAFCAYAAACNbyblAAAAHElEQVQI12P4//8/w38GIAXDIBKE0DHxgljNBAAO9TXL0Y4OHwAAAABJRU5ErkJggg==)
</code></pre>
<p>... and result in: <img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAUAAAAFCAYAAACNbyblAAAAHElEQVQI12P4//8/w38GIAXDIBKE0DHxgljNBAAO9TXL0Y4OHwAAAABJRU5ErkJggg==" class="img-fluid" alt="a red dot" title="The Image" /></p>
<p>...or with title text:</p>
<pre><code class="language-text">![a red dot](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAUAAAAFCAYAAACNbyblAAAAHElEQVQI12P4//8/w38GIAXDIBKE0DHxgljNBAAO9TXL0Y4OHwAAAABJRU5ErkJggg== "The Image")
</code></pre>
<p>... and result in: <img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAUAAAAFCAYAAACNbyblAAAAHElEQVQI12P4//8/w38GIAXDIBKE0DHxgljNBAAO9TXL0Y4OHwAAAABJRU5ErkJggg==" class="img-fluid" alt="a red dot" title="The Image" /></p>
<p>Or, with a reference-style image link:</p>
<pre><code class="language-markdown">![a red dot][red-dot]
[red-dot]: data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAUAAAAFCAYAAACNbyblAAAAHElEQVQI12P4//8/w38GIAXDIBKE0DHxgljNBAAO9TXL0Y4OHwAAAABJRU5ErkJggg==
</code></pre>
<p>... and result in: <img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAUAAAAFCAYAAACNbyblAAAAHElEQVQI12P4//8/w38GIAXDIBKE0DHxgljNBAAO9TXL0Y4OHwAAAABJRU5ErkJggg==" class="img-fluid" alt="a red dot" /></p>
<p>Data URLs are a handy way to reduce the number of files involved in a page. Like any feature, that can get ridiculous so the value comes when working with small chunks of data (like a red dot image.)</p>
<h2 id="references">References</h2>
<p><a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/Data_URLs">Data URLs</a></p>
<p><a href="https://en.wikipedia.org/wiki/Data_URI_scheme">Data URI Scheme (wikipedia)</a></p>
<p><a href="#data-urls">TL;DR</a>⮷</p>http://blog.peterritchie.com/posts/agile-off-the-railsAs a Delivery Team Member, I Want To Know if My Organization's Agile Initiative Is off the Rails2022-06-19T00:00:00Z<p><img src="/assets/adult-gd72730acb_1920.jpg" class="img-fluid" alt="concentration" /></p>
<p>Image by <a href="https://pixabay.com/users/pexels-2286921/?utm_source=link-attribution&utm_medium=referral&utm_campaign=image&utm_content=1850268">Pexels</a> from <a href="https://pixabay.com/?utm_source=link-attribution&utm_medium=referral&utm_campaign=image&utm_content=1850268">Pixabay</a></p>
<p>Or, <em>as a delivery team member, I want to know if my organization's agile initiative is off the rails, so that I may compensate for it</em>.</p>
<p>I have been an agile team member (delivery, engineering) in many organizations. There is a spirit to any defined agile process, a spirit that addresses known time-to-market and quality fallacies. Agile processes are like any good guidance; they are based on experience and techniques proven to address known problems.</p>
<p>First, some context:</p>
<p><strong>Excellence</strong> is not about being perfect but about recognizing and exploiting opportunity.</p>
<p>A <strong>Project</strong> is a temporary, planned effort to achieve a particular aim.</p>
<p>A <strong>Sprint</strong> is a time-boxed period where a team works to deliver usable functionality to stakeholders.</p>
<p>Goals and objectives are often <strong>Means Goals</strong> and <strong>Means Objectives</strong>, signifying they are a catalytic end to realize another end.</p>
<p>Objectives that are meant to accomplish other objectives exist in a continuum of objectives called <strong>Cascading Objectives</strong>.</p>
<p>A <strong>Key Result</strong> is not qualitatively measured ("done," "improved," etc.); they are measured quantitatively ("improved by 25%," "decreased by a factor of 2," etc.)</p>
<p>Agile methodologies embody a continuum of purpose, motivation, and improvement. Purpose, motivation, and improvement are not team-specific concepts but are organization-wide constructs. This continuum is the result of the act of leadership.</p>
<p>Over the years, I have witnessed many patterns of behavior that have resulted in failed agile delivery. Following are some common of those practices,</p>
<h2 id="agile-is-the-only-process">Agile Is the Only Process</h2>
<p>Agile's raison d'être is to deliver value to the stakeholders. Agile is a project management technique; no enterprise devotes 100% of its resources to projects. An enterprise has a purpose for existing (their <em>why</em>, the vision) and has a current means to achieve that purpose (their mission). Any effort not gauged by whether it satisfies the overall mission (and thus aligns with the purpose) can only succeed accidentally. Planning to succeed accidentally is not planning, and that sort of "planning" is a waste of time. Agile organizations put effort into addressing assumptions before planning value delivery.</p>
<p>You know an organization is working against itself and pretending to be agile when:</p>
<ul>
<li>100% of engineering time is devoted to "sprints."</li>
<li>Sprints are planned correctly 100% of the time and are never canceled due to change.</li>
<li>A project plan does not focus on an operational outcome.</li>
<li>Spikes are exceedingly rare.</li>
<li>No stakeholder has communicated what they value.</li>
<li>Stakeholders do not declare a spike's usable functionality; the delivery team declares it.</li>
</ul>
<h2 id="no-one-has-okr-training-or-expertise">No One Has OKR Training or Expertise</h2>
<p>Goals and objectives are easily understood conceptually but are hard to implement in reality. One of the impetus' of OKRs is to recognize and address that. Objectives and deliverables are consequents of goals; they are the means to a larger end while still being an end in and of themselves. In isolation, objectives and deliverables are meaningless and, like any other misguided activities, detract from the purpose of accomplishing them. OKRs attempt to associate goal-oriented <em>key results</em> with individual objectives, objectives that cascade from higher-level objectives.</p>
<p>You know OKRs are in name only when:</p>
<ul>
<li>Objectives do not cascade from higher-level objectives</li>
<li>Key results are an action, not an event</li>
<li>Key results are not measurable</li>
</ul>
<p>To be honest, I've only ever seen failed implementations of OKRs--OKRs more often address perceived delivery failures rather than leadership failures. i.e., they are a management technique rather than a result of leadership.</p>
<h2 id="leadership-in-name-only">Leadership in Name Only</h2>
<p>A manager creates and judges the attainment of goals (doing things right). A leader communicates and cultivates purpose and vision (doing the right things).</p>
<ul>
<li>There are only "leaders" and no "managers."</li>
<li>"Leaders" that are late to every meeting.</li>
<li>Activities are judged, not outcomes.</li>
<li>Goals and objectives are only ever qualitative, not quantitative.</li>
<li>Personal improvement is not a measured performance metric.</li>
</ul>
<p>Of course, I could go on. There are many more examples and many bad practices. I'd love to hear about what you've witnessed and your thoughts on these and other bad practices.</p>
<!--
## References:
https://www.forbes.com/sites/williamarruda/2016/11/15/9-differences-between-being-a-leader-and-a-manager/?sh=2432b1424609
-->
<p><img src="/assets/adult-gd72730acb_1920.jpg" class="img-fluid" alt="concentration"></p>http://blog.peterritchie.com/posts/Environment-Variables-with-CSharp-Conditional-Compilation-SymbolsEnvironment Variables with C# Conditional Compilation Symbols2019-12-12T00:00:00Z<p>Have you ever thought, it would be nice to have a symbol like <code>PETERRIT</code> that is unique to your domain account that you could use for code that YOU maybe working on but don't want to break the build?</p>
<p>I occaisionally think I would like to do this:</p>
<pre><code class="language-csharp">#if PETERRIT
public class ViolatileExperiment()
{
//...
}
#endif
</code></pre>
<p>When I think of this I go look at the docs or on Stackoverflow, but I never find anything that allows me to do that.</p>
<p>I had that thought recently and poked around in the Project Settings for a few minutes to see what's going on. Interestingly <code>"%USERNAME%"</code> causes and error, but doesn't break the build.</p>
<p>Damn, I thought. But % is so... DOS, maybe they use a different delimiter. So, I stuck in <code>${USERNAME}</code>. Nope. Then I thought, wait, macros in build events have a specific format! I entered <code>$(USERNAME)</code> and low-and-behold <strong>it worked!</strong></p>
<p>It's a little wonky though, in the Project Settings it shows the expanded variable (<code>PETERRIT</code>), but in the project file it shows the macro reference. (<code>$(USERNAME)</code>). I can see the macro reference getting overwritten from time to time.</p>
<p>Enjoy!</p>
<p>Have you ever thought, it would be nice to have a symbol like <code>PETERRIT</code> that is unique to your domain account that you could use for code that YOU maybe working on but don't want to break the build?</p>http://blog.peterritchie.com/posts/RESTful-VersioningRESTful Versioning2019-11-12T00:00:00Z<p>Versioning is not new. Versioning seems to be one of those things that people find hard to do or difficult to fully understand, especially with services and APIs. RESTful versioning seems to be in the realm of Tabs v Spaces, but I want to detail my related observations (mostly of other's writings, but with some added color).</p>
<h2 id="what-is-a-version">What is a Version?</h2>
<p>Before going further, I find defining terms so their meaning is explicit and understood. <em>Version</em> is no exception.</p>
<p>A <em>version</em> recognizes a change to <em>something</em> already established and assigns it an unique identity. That identity serves as a moniker for <em>what</em> changed so that when the <em>something</em> that changed is processed, it can be differentiated from other <em>somethings</em> of different versions.</p>
<h2 id="why-do-we-need-a-version">Why Do we Need a Version?</h2>
<p>Based on what a version <em>is</em>, it may seem easy to at least deduce <em>why</em>. That deduction usually <em>to differentiate different versions of things</em>. This is <em>what</em> not <em>why</em>. This is the part that many people seem to dismiss or let slip by. In the context the knee-jerk response is "different version of the API" (API Versioning). But, this simply restating what versioning is. It's similar to defining "version" as</p>
<blockquote class="blockquote">
<p>A version is the version of something in relation to other versions of the same thing.</p>
</blockquote>
<p>Yeah, using the word you're defining in the definition is <em>helpful</em>. "Version" must provide:</p>
<h3 id="support-for-past-representations">Support For Past <em><strong>Representations</strong></em></h3>
<p>The major reason versioning comes into play is because any one <em>representation</em> of something evolves over time. Needs change, understanding improves, technology evolves, imperfections are found, etc. and how something is stored or communicated needs to change to accommodate that evolution.</p>
<p>"Requirements" are an obvious agent of change, and it would be easy to provide a trivial requirements example but a <em>fixing imperfection</em> example would be more persuasive. Humans like to be open-minded but inherently we live in our own worlds (our own mental model of the world). Some of us are empathic and recognize parts of other worlds of the people around us. Or we know about a set of archetypes for which we can optimized interaction. But in reality there are really 8+ billion other worlds out there and it's simply not humanly possible to know the intricacies of each. Which means we make assumptions and trade-offs of what is acceptable to all of those other worlds. Usually our audience isn't all 8+ billion people, so we're generally more correct than incorrect in our assumptions. But, being incorrect is inevitable and expected. Much like we need to support many personalities, preferences, and needs; we also need to:</p>
<h3 id="support-multiple-representations-of-concepts">Support Multiple <em>Representations</em> of Concepts</h3>
<p>An example of this type of imperfection are date/time representations. We live in our own <em>locus</em> (which is like a personal <em>locale</em>) and take for granted things we use or do in our locus from day-to-day, like Date/Time representations. Local time has been working for each of us for all our lives, we take that for granted and use it in a representation without thinking.</p>
<p>There are many things that make this problematic and error-prone. I won't get into detail what <em>all of those</em> may be (a blog isn't the place for a tome like that). The <em>fix</em> is, of course, to use a <em>different representation</em>. Implementing that fix and supporting existing representations of complex data means we need to be able to tell different representations of the same data from one another.</p>
<p>This allows us to know how to translate each representation to the same in-memory structure when we (i.e. our code) encounter these representations; reinforcing that <em>representations</em> differ from the <em>conceptual</em> resource <em>and</em> from the <em>implementation</em> translation of the representation</p>
<p>We need the ability to translate <em>multiple representations</em> because there will be instances of differing representations <em>in the wild</em> at a time.</p>
<h3 id="support-multiple-active-representation-versions">Support Multiple <em>Active</em> Representation Versions</h3>
<p>Increasingly we work in asynchronous environments where communication of data is off-loaded to asynchronous communication technology (like queues, topics, and even threads). And with the drive to 100% availability this means that we perform software updates <em>in situ</em> or without taking our systems completely off-line (or unavailable). e.g. <em>indirect</em> communication through a queue makes communication of the message independent of the processes. Until it reaches and is consumed by the destination, the message can be in a queue with neither process executing. This requires components to support <strong>at least</strong> two versions of a representation <em>at the same time</em>.</p>
<h2 id="attributes-of-rest">Attributes of REST</h2>
<p>Since we're in the context of <strong>Re</strong>presentational <strong>S</strong>tate <strong>T</strong>ransfer where resource <em>representation</em> is front-and-center as well as <em>state</em> of that resource, the following is a review of main features of RESTful services:</p>
<ul>
<li>REST is not a distributed object style <a href="https://www.ics.uci.edu/%7Efielding/pubs/dissertation/rest_arch_style.htm#sec_5_3_1">3</a> <!--(5.2.1-1)--></li>
<li>A resource identifier (URI/URN) is a reference to a particular conceptual resource, not to a particular representation of it. <a href="https://www.ics.uci.edu/%7Efielding/pubs/dissertation/rest_arch_style.htm#sec_5_2_1_1">1</a> <!--(5.2.1.1-4)--></li>
<li>A representation of a resource is transfered between REST components. <a href="https://www.ics.uci.edu/%7Efielding/pubs/dissertation/rest_arch_style.htm#sec_5_2_1_1">1</a> <!--(5.2.1.1-4)--></li>
<li>A resource maps to a set of entities that varies over time, not just the representation at the moment <a href="https://www.ics.uci.edu/%7Efielding/pubs/dissertation/rest_arch_style.htm#sec_5_2_1_1">2</a> <!--(5.2.1.1-2)--></li>
<li>The set of entities that are mapped to a resource are considered equal (by resource identifier and/or representation). <a href="https://www.ics.uci.edu/%7Efielding/pubs/dissertation/rest_arch_style.htm#sec_5_2_1_1">2</a> <!--(5.2.1.1-2)--></li>
<li>The semantics of mapping a resource to an entity distinguishes one resource from another and is constant. <a href="https://www.ics.uci.edu/%7Efielding/pubs/dissertation/rest_arch_style.htm#sec_5_2_1_1">2</a> <!--(5.2.1.1-2)--></li>
<li>Representations are late-bound and based on characteristics of the request. <a href="https://www.ics.uci.edu/%7Efielding/pubs/dissertation/rest_arch_style.htm#sec_5_2_1_1">2</a> <!--(5.2.1.1-4)--></li>
<li>An identifier may exist without, or before, any realized representations. <a href="https://www.ics.uci.edu/%7Efielding/pubs/dissertation/rest_arch_style.htm#sec_5_2_1_1">2</a> <!--(5.2.1.1-2)--></li>
</ul>
<h2 id="restful-versioning-options"><em>RESTful</em> Versioning Options</h2>
<p>A quick review of objectives, for any given resource representation:</p>
<ul>
<li>we need to differentiate change independently from unrelated representations</li>
<li>we need to differentiate different changes to related representations at the same time</li>
</ul>
<p>There are really only two fundamental options for <em>API Versioning</em> (I didn't use <em>RESTful Versioning</em> for reasons I hope will become clear):</p>
<ul>
<li>Version moniker in the URI/URN, or</li>
<li>Version moniker in the headers (or media-type)</li>
</ul>
<h3 id="uriurn">URI/URN</h3>
<p>In REST, a URL/URI <strong>only</strong> identifies a resource, it is not a content-type identifier. One reason for this is <em>Content Negotiation</em>. Content negotiation details that in the <strong>request for any particular resource</strong>, the <em>representation</em> of the resource (the response) can be negotiated through headers and responses. That negotiation occurs through the single URL/URI.</p>
<p>I.e. the response format does not need to be consistent per URL/URI. <a href="https://tools.ietf.org/html/rfc7231#section-3.4">4</a> If it's not clear, supporting multiple representations means there can be <em>many</em> response formats for any single URI/URN. Since we've already shown that each representation is independent of other, <strong>multiple versions need to be represented per URL</strong>. Making something like <code>example.com/picture/v2</code> for the SVG format meaningless. Therefore URI/URN versioning doesn't support some fundamental REST features.</p>
<h3 id="media-types-headers">Media Types (Headers)</h3>
<p>Media types are monikers for a particular re presentation. XML and Json or JPEG and PNG are examples of particular representations of the same resource. But media types are more complex than that.
Media types can consist of a registered type (<code>application</code>, <code>audio</code>, <code>example</code>, <code>font</code>, <code>image</code>, <code>message</code>, <code>model</code>, <code>multipart</code>, <code>text</code>, and <code>video</code>), a subtype (the registered format in the standards tree, or a dot-delimited subtype tree), a suffix (prefixed with <code>+</code>), and optional parameters (key/optional-value pairs prefixed with <code>;</code>). Suffixes can be used to specify the underlying <em>structure</em> of a type/subtype, e.g. JSON and XML. Formats like SVG can be either textual or binary, so although being a image and SVG, simply specifying <code>image/svg</code> is not enough to cover both of those structures. The media type for the XML format of SVG ends up being <code>image/svg+xml</code>. Application-specific types use the <code>application</code> type and a subtype in the vendor tree (<code>vnd</code>). If a custom application format for a <em>person</em> resource that uses XML format would have a media type of <code>application/vnd.person+xml</code>. If the service also supports JSON it would have another media type <code>application/vnd.person+json</code>. <code>charset</code> is a reserved parameter, other parameters have unique meaning defined within the type/subtype. for example <code>text/html; charset=UTF-8</code> and <code>application/vnd.person+json; version=2.0.0</code>.</p>
<p>The take away is that <strong>media-types are independent from the endpoint</strong>.</p>
<h1 id="wrapping-up">Wrapping Up</h1>
<p>It may make sense to think that the resource is changing, but in reality it is the representation that changes. The resource is abstract, like <em>client</em>. Changing Birth Date from local to UTC doesn't change the fact that the <strong>resource is still a <em>client</em></strong>. If the resource fundamentally changes, that's when you change the URL/URI. But not with a version identifier, but a new <em>resource</em>. If something previously considered a "client" changes so is conceptually no longer a "client", then a new URI/URN should be used (like "client" to "lead"). <em>We wouldn't have different versions of clients, merely different representations of client information</em>.</p>
<h2 id="tldr">TL;DR</h2>
<p>A URI/URN is a reference to the single conceptual resource and not to a particular representation. Since media types <em>are</em> the data format of the representation and the same conceptual resource has representations that can change, <strong>media type must be used to specify different representations that a single URI/URNs supports</strong>.</p>
<p>Versioning is not new. Versioning seems to be one of those things that people find hard to do or difficult to fully understand, especially with services and APIs. RESTful versioning seems to be in the realm of Tabs v Spaces, but I want to detail my related observations (mostly of other's writings, but with some added color).</p>