At Populate, building software follows a “north star” principle: writing minimal code that is hard to break yet flexible to adaptation.
In this article, I will share some of the ideas in software engineering we are experimenting with to build a quality system. A system is beautiful when it is robust, works flawlessly, is easy to interpret, and remains solid over time as change is continuously introduced.
Model as the source of Truth
Models are the source of truth in the Populate system. As such, great care has been taken to represent the business world in the Model in an accurate way, and great care continually needs to be taken to ensure that the validity of the domain remains intact.
Populate applies Domain Driven Design to achieve that goal, where the Model entities represent the business domain the software is built for. We pursue the following objectives when thinking about Modeling the business world:
- Accurate representation of the business domain with no ambiguity
- Design should break if the business domain changes ensuring the original representation was tight
- Adaptation to change should trigger minimal changes ensuring loose coupling in the first place
We can maintain the integrity of the domain modeling by carefully defining the data structures and the functions that operate on them. The model must have all the constraints and checks in place to uphold the system invariants. Any violation of the business invariants in the domain model will result in an unstable system at best and unchecked corrupted data at worst.
Validity checking on every data impacting the model is done on the edge of the context boundary. Models are kept pristine by gatekeepers that do not allow anything invalid to reach the models.
Mechanism for Model purity
Models are kept pristine in two ways:
DTOs as gatekeepers
Use of the Type system
Section: DTO objects as gatekeepers
DTO objects are the vehicle that carries the model data outside of the bounded context of the Populate system and receives information into Populate from the external world. As such, DTO objects are most flexible in terms of type restrictions. IsValidDTO() method, implemented by all IInteractWithModel DTO objects, must work as a gatekeeper to stop ANY invalid data from filtering into the Model space. Data should never get to models that may result in an invalid state, and all data must be validated and prevented at the DTO level through the IsValidDTO level. The check for validity takes the form of the following:
if (!IsValidDTO)
return None;
Model = DTO.ToModel();
DTOs either read from the Model or Interact with a model (both read/write) – Interfaces are in place to guarantee that the DTO objects employ necessary transformers – ToModel and FromModel for any such conversion to and from models. The IReadFromModel DTOs do not need validity checking as the outbound Model to DTO is guaranteed to be pure. The IInteractWithModel DTOs need the IsValidDTO() implementation to check DTO validity before any inbound DTO to Model conversion. A properly constructed IsValidDTO is the single most important thing we can do to continue to keep the system stable as it grows in size.
Section: Use of the Type system
There are few places where the invariants of the system can be maintained. The database layer is one, using Database Foreign Keys, Constraints, and Checks. However, in the Code-First-Database-Design approach, we use the Type system to hold and maintain the invariants of the system.
Example: If the Firstname of a patient is constrained to be of 64 letters, instead of using
string Firstname;
and applying checks in the form of
if (String.IsNullOrEmpty(fname) || fname.Length > 64),
define a type NonemptyString64 that guarantees the invariant. The approach guarantees no illegal state will ever be created in the system as long as the Type definitions and usage is correct. It also advances the idea of “self-documenting code.”
NonemptyString64 Firstname {get; set;}
is fully descriptive of the nature and constraints of the property Firstname. A well-designed type system also eliminates the need for unnecessary commenting. There’s no need to write “The property first name must be of 64 characters maximum” anywhere in the code. Strive for self-documenting code.
Type systems allow us to embrace the idea of “Making illegal states unrepresentable.” HOWEVER, humans are not perfect, and invalid data or transformation may creep into the Model layer. If this happens, Model Type objects throw exceptions as a last resort – rather blow up, then allow the system to get into an invalid state. It will only happen if the DTO gatekeepers fail them. And any such exception should light a fire for us to scramble and close the validation gap – meaning, it’s time to update the respective IsValidDTO so the exception must not happen ever again.
Relationship between the Model and the external world
The models need to be persistent for Populate to become a stateful system. Models are EF Entities that remain at the core of the system. They are not aware of any other layers. Only Entity Framework Core knows about the models, which is essential to map the models to the persistence storage and back.
Build a predictable system
In a deterministic system, there is no need for exception handling. Because nothing is an exception and everything is part of the design. Strive to build a deterministic system. Anticipate all possible outcomes of an operation and deal with all of them in the design.
A few common patterns of errors in a C# system involve the appearance of NULLs. Compiler static analysis will give you warnings where it thinks a possible dereferencing of null value may occur. This is a hint to the developer to properly analyze the flow and ensure that the value in question can indeed be not null. In that case, use the null forgiving (postfix !) operation to inform the compiler that you have ensured that will not be the case. But only use the null-forgiving when you absolutely know the value cannot be null; otherwise, deal with the condition if the value is indeed null.
Another area of possible unpredictability happens when the service layers deal with the potentially messy DTO objects. DTO objects carry information from the external world outside of the bounded context that Populate has no control over. They are constructed from primitive types, and therefore the concept of type safety cannot be applied. Service layers must anticipate all possible error conditions and communicate the appropriate error type back to the API so the information can be transferred to the client. Populate uses the Option<T> type for such communication, where T is the expected return type. In the event of an error, Option sets the Error attribute, which describes the nature of the error. API layer can interpret the error appropriately and return a meaningful HTTP Status code back to the caller.
Finally, use the Type system to build a deterministic system. Well-designed Types have built-in checks and constraints and will not allow an invalid operation. By stopping any error state from being created and dealing with all possible error conditions, we can build a deterministic system.
Having no exception handling is a beautiful side-effect of building a predictable system. If the behavior of a system is fully deterministic, then nothing is an exception. The system has zero try-catch blocks at the moment. The only place for exception handling is when interacting outside the bounded context where the external world can throw wrenches we cannot control and did not anticipate yet have to deal with.
Code hygiene best practices
Don’t hack a solution. Create a framework for the solution so all similar changes can be systematically absorbed in the design.
Write less code. The volume of code does not correlate to quality. The probability of errors increases with more code.
Extract every repeatable pattern into a common utility so the utility can be shared
Before introducing a new utility, check if any existing solution exists that matches the pattern of the problem you are trying to solve
Relentlessly reuse wherever possible instead of writing new code.
Use interfaces to conform to common properties and behaviors. If there exists a common connection between entities – consider applying an interface to guarantee the connection remains intact in the future. And don’t forget to apply the Interface Segregation Principle to keep the interfaces clean and maintainable.
Don’t create unnecessary references all over the place. You will be creating a rigid model where changes in one place result in a cascade of changes all over the code because of the overly coupled system.
Develop a deep understanding of the SOLID principles. Apply SOLID principles rigorously. The principles exist for a reason.
Simple Mental Model
A concept often overlooked in modeling a system is the idea of keeping the mental model simple. The Model should be no harder to follow than the complexity of the business domain it represents. The complexity of operations and model representation in views can easily transfer into overly complex models without awareness. Be careful about keeping the model simple. It should be easy to follow, must accurately paint a picture of the business domain, and present a coherent view of the system.
Changing the Model
Domain modeling is not a one-time process. The domain will continue to evolve as the business requirements change and the nature of the invariants evolve. The change needed to be handled with care. Everything in the system is designed the way it is on purpose. The Model is protected through layers of constraints, interfaces, and transformers, so the state of the domain model can never be invalid. When analyzing a change to the model, keep the following in mind. Models can be either:
Accurately and correctly reflects our current understanding of the domain.
Accurately reflected our understanding of the domain but has become outdated and needs to be updated.
Incorrectly reflects our current understanding of the domain.
As such, when faced with an issue or a change request, IT IS IMPORTANT first to assess which of the above three it falls under. Putting a hack to address a change request/bug will compromise the integrity of the entire model and will break many invariants in subtle ways. This will be bad as these violations will manifest in the production system, which will not be a good experience for our customers and the engineering team, as we are the ones who have to deal with the mess. Therefore, always address the change with a proper model upgrade that naturally solves the bug or adopts the change instead of wantonly changing a private attribute to a public attribute or adjusting the Interface type, or removing a model constraint.
Other elements
Onion Architecture
Populate architecture falls under the category of “Onion architecture.” Still, instead of approaching “Onion Architecture” as a strict goal, this particular architecture design naturally surfaced, having loose coupling and separation of concerns as a target. Populate architecture layers look like this, from the innermost to the outermost layer within the bounded context:
Models -> DTOs -> Services -> WebAPI
At every stage, the outer layer is aware of the inner layer, but the opposite is not true. I.e., Model is not aware of any other layers.
Presence of ViewModels
ViewModels exist to provide support to a View page when the page requires a data model that is a combination of multiple DTOs. They are best seen as an aggregator to serve the specific need of a UI page more than anything else.
Use of Option<T> types
When a WebAPI requests the Service layer, the Service layer may be able to serve the request or encounter different kinds of errors in doing so. The error conditions must be appropriately propagated back to the caller, so the caller (Web APIs) can respond to the client with proper Http Error codes. This is achieved by using Option<T> types where the type T in the Option represents the expected response type. Option<T> type provides an Error property that the Service layer can appropriately set to propagate the error. Web API must check the IsValidResponse value to know if any error has occurred. If true, then return the response of type T to the caller; if false, then retrieve the error and formulate a proper HTTP Status Code to send to the caller.
Role of Unit Testing
Much of the need for unit testing is eliminated by the thoughtful and thorough use of the Type system. Any reporting bugs must first be considered for fixing by updating the design and the Type system.
Unit tests have a role to play in the Populate system. Consider the following areas where Unit Testing is important to maintain the integrity of the system:
Interacting with third-party integrations where we do not control the types and the behavior of the external API calls.
Ensuring any behavioral change within Populate remains constant.
Resources
Domain Driven Design There are many online resources out there.
Domain Driven Design is an excellent book: https://www.amazon.com/Domain-Driven-Design-Tackling-Complexity-Software/dp/0321125215
I would also recommend watching this video: https://www.youtube.com/watch?v=PLFl95c-IiU
Using Type system to “make illegal states unrepresentable”
Please read the following series to develop an understanding of using the Type system to your advantage in making illegal states unrepresentable. The programming language used in the posts is F# (a language with fantastic support for Type systems by the way), but it can be easily followed even if you don’t know F#: https://fsharpforfunandprofit.com/posts/designing-with-types-intro/