Dependency Indirection with Injectable Factory

Overview

Dependency indirection (DID) is a way to avoid one software component’s direct dependencies on another component’s implementation. Most of the component’s design issues related to testability and portability are caused by rigid direct dependencies. There are many ways to avoid direct dependencies. DI (Dependency Injection) which is so popular that it has almost become a religious belief, is just one of them. But DI comes with costs. With DI frameworks, people tend to abuse DI, DI everything approach dramatically increases the accidental complexity¹. In this article, I will introduce the injectable factory pattern which uses a dependent intermediation way to solve direct dependency issues. It has been intensively used for several mid-sized projects successfully deployed to production serving millions of users. We enjoyed the readable, flexible, testable and portable components without increasing the complexity. The injectable factory is not something new from scratch, basically, it’s a combination of service locator and factory patterns without their caveat. These two patterns are considered anti-patterns according to most DI articles. You may wonder why it’s a combination of two anti-patterns. But hold on, I’ll explain why you don’t need DI in most situations. (I’ll talk about the philosophy behind before I delve into the details, if you are impatient, you can jump to The Solution section directly)

Dependency Problems, Practices, Misconceptions

Dependencies are important parts of every system. Most systems are organized as layers, the upper layer has natural dependencies on the lower layers. Most systems also have dependencies on external downstream systems and runtime environments such as operating systems and application frameworks. Following diagram of an order component shows the typical dependencies:

The problem is Rigid Dependencies instead of Prewired Dependencies

To make components testable and portable, we have to make those dependencies loosely coupled so that most dependencies can be easily replaced. But does loose coupling mean disconnected? Is prewired dependencies bad? The answer is NO. The real problem is not about prewired dependencies, but rigid dependencies, those dependencies that you have no way to change without modifying the original code, thus it violates OCP. That will limit the reusability of components in a different context and make it hard to test. Many people with DI framework experience, have the misconception that prewired dependencies are bad and try to remove them which dramatically increases the accidental complexity. They are solving the problem with the wrong tool.

The component should be Self-Contained

As a user, we all like “battery included but swappable” experience. When we construct a higher-level component, we don’t want to deal with the details of lower-level components’ internal wiring.

For example, if we want to assemble a car, we need to install an engine into it, we prefer a pre-assembled engine instead of a pile of engine parts, so we don’t have to deal with the details on how the engine is assembled internally. We only care about the usage boundary of the engine such as connecting with gas pipe and exhaust pipe, these are legitimate and mandatory DI points, without them, the engine won’t work. But the majority of internal wiring such as piston assembly, spark plugs are implementation details, they are prewired and hidden from us. Even pre-wired, they are still loosely coupled with screws and plugs instead of glue and solder, so most of the sub-parts are still easily replaceable. Pre-wiring brings user benefits, but it doesn’t necessarily introduce tight coupling. Here DI points are only for those dependencies that normal users (80% users) are interested in, they are part of the usage boundary. A well-designed component should minimize usage boundaries to reduce the burden of normal users. That also applies to DI points, they should be kept to a minimum. This is how we manage complexity to build complex machines in a composition manner by hiding lower-level details. We try to deal with one level of abstraction at one time by using ready-to-use, self-contained sub-components.

For software components, the same principle applies. It should be self-contained. When we deliver a component, the lower-level internal sub-components should be pre-wired with sensible defaults (The OEM parts) and ready to use. They are the implementation details the are not of the concern of the majority of the component users. We should apply the dependency hiding rule to hide them from the usage boundary. Only the minimum number of dependencies that users are interested in should be explicitly exposed as part of the usage boundary, they are the legitimate DI points. For internal dependencies, even prewired, they should still be loosely coupled without rigid dependencies (such as avoiding using new keyword) so that they can be easily replaced with aftermarket parts for whatever reasons.

However, with DI frameworks, it encourages the DI everything approach which increases the components' usage boundary which in turn causes complexity and deep coupling around usage boundary.

People may argue with DI frameworks, the wiring is handled by frameworks, so clients only depend on interfaces. However, you still expose the internal dependencies to the framework and you still need to take care of the wiring somewhere which essentially creates implicit coupling which is much worse than explicit coupling. The code is much less readable and less robust because of the fragmented logic. Many wiring issues can only be discovered during runtime, reducing the effectiveness of compiler checks and static analyses. It also forces users of the component to depend on the same DI framework the component internally uses. This is like to force the engine users to depend on the engine assembly line, but what they need is just an assembled engine! You can no longer claim your components are self-contained and portable.

Separations of Usage Boundary and Testing Boundary

Most home appliances, come with user manuals and repair manuals, they are for different audiences. For well-designed appliances, the user manual should be as simple as possible, so that reduce the burden of the user, remember an appliance is mainly to serve users with values instead of adding their burdens. In contrast, the repair manual provides many more details on how internal components are wired, and how those components can be tested and replaced. These are for advanced users who are capable of repairing or customizing their appliances. With the separation of different manuals, we avoid overwhelming normal users with too many implementation details. That achieves the principle of “Simple things should be simple, complex things should be possible.”

For software components, the same principle applies, we should have a clear separation between usage boundary and testing boundary. With DI frameworks, people tend to mix these two kinds of audiences: normal component users and unit testers. It encourages to use DI everything approach which is mainly for testing purposes (Actually for most of the developers, that could be the main if not the only reason to use DI). That pollutes the usage boundary, adds an unnecessary burden to the normal users. For the majority of the testing and customization problems, DI is not the right tool due to the accidental complexities it brings.

Hide Most Dependencies as They are Implementation Details

Based on guidance from most DI books, people tend to declare every dependency explicitly either through constructors or setter methods. This is another big misleading, as I mentioned earlier, most of the internal dependencies are implementation details, we should hide them as deep as possible just as how we hide other implementation details. If a dependency is only used by a particular method, we should hide it deep inside the method’s local scope. That’s how we manage complexity, always hide information inside the smallest scope, which will make future refactoring much easier as changes are isolated inside the smallest scope without worrying about breaking other parts of the system.

With explicit dependency declarations, we pollute the usage boundary, pollute the component state space. As a component user, how painful to use such a component, that you just ask it for one function, but it’ll ask you back for 5 direct dependencies and dozens of indirect dependencies recursively before it can serve you! As a code reviewer, how enjoyable when you read a class with a constructor with 5 parameters and 5 lines of its body just to copy the 5 parameters into 5 member variables. And these 5 member variables almost hide the other 1 or 2 legitimate member variables represent the object’s state (Member variables are mainly for representing the object’s state instead of its dependencies). What amount of noise and confusion it introduced. Following is a typical example of constructor over injection.

Noises in A typical example of DI everything component

In [DI Principle book]², it mentioned [service facade Pattern]⁴ to solve the constructor over-injection issue. That seems like a solution, but it again could introduce more accidental complexity, the intermediate service facade components are artificially created just to solve one accidental complexity, but it adds more indirections in the dependency graph, thus more accidental complexities.

Compile Time Dependency doesn’t Enforce Runtime Dependency

In the [DI Principle book]², it mentions [Local Default and Foreign Default]³. People may worry about the pre-wiring of default implementation in an external library. For example, if your business logic(OrderService.jar) and the pre-wired foreign default data access layer (OrderRepo_SQLDB.jar) are in different libraries as shown in the following diagram:

The pre-wired default is suitable for most users. But as a portable component, if a particular application wants to reuse OrderService with NoSQL storage. We can switch the dependency with another NoSQL library as long as it implements the same OrderRepository interface. But how about the original SQLDB Library dependencies? Will it also be unnecessarily loaded during runtime? This is not a problem. Even the Foreign Default does introduce compile-time dependency, most of the modern languages have a clear separation between compile and runtime dependencies, they load classes lazily, so until a class is used, the class won’t be loaded. And we can even exclude them from the runtime bundle, for example: In the Java maven project, we can use “provided” dependency scope, or optional dependency to achieve it. So the foreign default won’t necessarily cause runtime dependencies.

The Solution

Based on previous discussions, we understand DI is not a viable tool for unit testing and customization. There are many better alternatives ⁸. In this section, I will introduce one of the alternative InjectableFactory / InjectableInstance for static-typed languages such as Java or C#. We will see how it solves implicit dependency⁹ issues without introducing complexity.

InjectableFactory Explained

The pattern is pretty simple. It’s implemented as a mutable value holder. This pattern is related to the factory, service locator, singleton pattern, and ambient context pattern. All these patterns are considered anti-patterns according to most DI articles ⁵. But in practice, those patterns, especially singleton, are very handy and useful. They are prevalent in almost every non-trivial project. But they come with caveats. InjectableFactory tries to prevent those caveats at the same time keep the convenience and simplicity of those patterns.

There’s a reference implementation⁶ which I have put in our open source project: jsonex, the whole implementation consists of just two simple classes without any additional runtime dependencies:

1. InjectableFactory: An injectable factory with default object creator
2. InjectableInstance: A special InjectableFactory represents a Singleton instance with default object creator.

Following UML diagram shows the pattern structure:

The ComponentInterface contains a static variable factory (or instance) which references an InjectableFactory (or InjectableInstance) instance and provides a default implementation using a declarative way. At runtime, The client will call its get() method to get its implementation. get() the method will delegate to the factory.get()(or instance.get()). In unit testing or customization, setInstance() (or setObjectCreator()) method will be called to replace the default implementation with alternative implementation such as mocks.

Following is an example of the TimeProvider interface which is an abstraction of system time to avoid application’s direct dependency on system time, so that we can simulate different moments in unit testing. It’s a Singleton. (For clarity, I’ll use my favorite Lombok⁷ annotations in the code examples):

TimeProvider example that uses InjectableInstance

1. TimeProvider interface contains a static reference of InjectableInstance, it will pass the default implementation class: Impl as a parameter. The of() method also accepts a lambda function (Supplier) as a parameter.
2. Static get() method returns the singleton instance by delegating it to instance.get();
3. Inner class Impl is the default implementation. For this simple class, an inner class is appropriate. Usually, it could be a separate top-level class either in the same library or external library (Foreign Default²)
4. An inner class of Mock implementation for unit testing. For this specific class, I prefer to put mock in production code for better reusability.
5. OrderService will depend on TimeProvider instead of new Date() to get system time. So that it won’t be deeply coupled with system Date.
6. In the unit test, we will create a stateful mock instance of TimeProvider.Mock.
7. We injected the TimeProvider with a Mock implementation
8. We call the Mock.add() method to simulate time passed 1 hour, and 48 hours to test different test outcomes.

Here is another example of the DataStore interface demonstrates how to use InjectableFactory and how a client can customize certain dependencies:

1. DataStore interface contains a static reference to InjectableFactory, We provide the generic types for the key <String> to identify a particular instance of type <DataStore>, For the of() method, we provide the default object creator using constructor reference DataStoreImpl::new. We also specify the cache scope as Global, so that the created Object will be cached globally. The available options are NO_CACHE, THREAD_LOCAL, GLOBAL.
2. The commented-out statement is another way to pass the creator as a lambda function. This is a more flexible way for custom creation logic.
3. In the DataStoreImpl class, we will have a constructor which takes a String as a parameter, so that the constructor reference ::new complies to Function interface as required by InjectableFactory.of() method.
4. For the client of the DataStore, it will call DataStore.get(DS_NAME) to retrieve the DataStore implementation. The client is decoupled from a particular implementation. In the unit test, it will be very easy to replace it with a mock implementation by calling DataStore.factory.setObjectCreator(mockCreator)
5. An alternative client which wants to reuse the OrderRepo component but with a different implementation of DataStore can easily call the DataStore.factory.setObjectCreator() to inject a different implementation in its initialization logic.

Following is the complete OrderService class which shows how we achieve testability without introducing a lot of complexities and noises with the InjectableFactory pattern.

In this class, we can see the static method get() in interfaces are replacing the new operator of a concrete class, it’s a new new operator. With that, all the dependencies such as Validator, UserSession, InventoryService, ShoppingCartService, etc, all can be easily replaced at either compile-time or runtime which makes the business logic extremely flexible, portable and testable. But the logic is still very expressive without any noise such as a long list of dependency parameters in constructor and class members of dependencies.

For the complete sample code, please refer to the GitHub repo.

Advantages

Compare with DI framework, factory, service locator, and singleton patterns, InjectableFactory has the following advantages:

1. No framework dependencies, the pattern is very simple, you can either depend on a very lightweight library or you can easily implement your own.
2. Avoid boilerplate code for factory and singleton as most of the logic is provided by those generic value holder library classes.
3. The factory is self-contained within the interface for easy discovery and code navigation.
4. There are no global registries as in Service Locator or DI framework.
5. Very easy to replace certain dependencies with different implementations both at compile time or runtime (It’s impossible for most DI frameworks to swap dependencies at runtime)
6. Dependent components are created lazily. So the application can start up faster without unnecessarily initializing components if they are not used.

Caveats

The singleton or factory injection is changing global state, so it could cause interferences between different unit tests if different tests need different mocks. However, in most unit test frameworks, this may not be a problem as each test is run in an isolated process or class loader. I do encounter problems when running in IDEs that may reuse the same class loader for different tests. In that case, it can be solved by calling the reset() method to make proper teardown after each test (Assume there’s no parallel testing).

Future enhancement

The reference implementation is a basic implementation that meets our needs. Few features could potentially be added to support more advanced use cases:

1. More sophisticated scope management. Currently, CacheScope only supports NO_CACHE, THREAD_LOCAL, GLOBAL. For applications with async worker threads, thread-local may not be appropriate as multiple threads will be used to handle a single request.
2. Crosscut function injection such as logging and tracing. As all the object creation is going through common classes, potentially, we could easily add proxy decorators of implementation to intercept method calls.

Summary

In this article, I highlighted some common misconceptions about dependency problems. I explained why DI is not an appropriate tool for unit testing. In the end, I introduced the alternative solution InjectableFactory for component testing and customizations. Following are the main principles behind this pattern:

1. Self-contained Components. Components should be self-contained without framework dependencies. Internal dependency should be loosely prewired.
2. Dependency hiding. Most internal dependencies are implementation details. We should hide them from the usage boundary. Never pollute the usage boundary just because of testing and customization. Remember design mainly for users, not for testings.
3. Code navigability. Pre-wiring logic should be colocated with the component interface itself for easy discovery and code navigation.

Next time, whenever you plan to depend on a DI framework, think again.

Related readings

1. Dependency Injection Tradeoff

2. Dependency Injection Inversion

3 Design for test vs design for API

[1] Accidental complexity refers to challenges that developers unintentionally make for themselves as a result of trying to solve a problem.

[2]DI Principle book: by Steven van Deursen and Mark Seemann: https://www.manning.com/books/dependency-injection-principles-practices-patterns

[3] According to DI principle book², local defaults are the default dependencies defined in the same library, (jar file as in Java, assembly as in .NET). Foreign defaults are the default dependencies defined in a separate library.

[4] According to DI principle book², the service facade pattern is to create an intermediate component to group related downstream dependencies, such that we can avoid Constructor Over Injection.

[5] [DI principle book]² has very good elaborations on it’s DI anti-pattern chapter

[6] This is just a reference implementation that is suitable for our internal projects. You can either reuse it or easily create your implementation.

[7] With Lombok annotation, lots of noise such as setter/getter in example code has been removed, please refer to the Lombok project site if the annotation is unclear to you.

[8] Frontend javascript community usually takes a more pragmatic and creative approach on solving unit testing problems without introducing lots of complexities for the component user, most of the popular frameworks such as VueJS and ReactJS give up the DI concept in the core framework, but provide much better alternative such as jest mock for unit testing. In contrast, Angular stuck with DI in its core which becomes a major blocker for adoption because of the unnecessary complexity.

[9] Explicit dependencies are those dependencies that the majority of the users are interested in which should be exposed as component boundaries. Implicit Dependencies are internal dependencies that are not about normal users’ concerns. They are implementation details that should be hidden from the component boundary.