Take a look at this picture. What do you see?
If you haven’t seen it before it’s an example of an epic online shopping experience fail. I’m sure that the person who bought these didn’t expect to get a pair of flappy flip-flops…
When placing a purchase order they probably read the item description, viewed the photographs and maybe even glanced over opinions about the seller.
The act of purchasing was technically part of some "contract". The seller promised to deliver fancy shoes, and the buyer paid for them.
It is merely a silly life analogy but the same applies to software development. Broken production caused by an unexpected change in a REST API path. False-positive unit tests caused by incorrectly mocked data. We all know it!
Among many others, there are several reasons why this stuff keeps happening:
- clients are not explicit enough about their requirements
- producers don’t communicate the deliverable well
- we don’t check whether the requirements or actual output didn’t change over time
Consumer-Driven Contract Testing
Luckily, software developers have several tricks up their sleeves to ensure software quality: unit/integration/contract/E2E tests, reviews, CI. The tech-deck is full of aces.
Contract Testing, in particular, is a technique that keeps engineers certain that the contract they "signed" doesn’t unexpectedly change over time.
There are plenty of great articles that deep into the subject of Consumer-Driver Contract Testing and many great tools supporting it (like Dredd) were created.
Without going into details keep in mind the following quote from Martin Fowler:
Contract tests check the contract of external service calls, but not necessarily the exact data.
In other words, we perform Contract Tests to verify that the APIs work the way we agreed they would.
For example, if I tell my teammate that the /store/shoes path requires Content-Type: application/json header and returns { "brand": "Dasidasi" } object then it MUST do that.
Shine On You Crazy Diamond
Existing tools, like Pact or Dredd, do the job quite well. Two caveats, though, are that they either require some boilerplate to get started or they usually support only programmatic testing.
I borrowed that image (with a permission) from Liran’s great article
Consider Pact for example. Defining a contract in code is certainly one way to do it. There is nothing wrong with it.
But what if there was a tool that acts like a proxy server enriched with automatic contract validation so that you validate your contracts as you write your app?
What if the same tool could be reused for automated, zero-code contract testing?
What if contract testing could be a task for a Project Manager and not a software engineer?
And what if you could reuse the same API contract files for contract testing, generating documentation, running mocked HTTP servers and style checking it?
I’m talking about Prism!
Prism is an OpenAPI Driven tool for mocking and validating HTTP servers. You can read more about basics of using Prism in my previous article. However, if you haven’t read it keep in mind that Prism consumes an OpenAPI description file and automatically creates an HTTP server from it.
Contract Validation in Prism
My friends from Stoplight (mainly Vincenzo Chianese & Karol Maciaszek) have recently released a powerful feature called Contract Validation. It enables you to instantly turn your mocking server into a proxy, validating everything that goes in and out your upstream!
This is a game-changer and the possibilities it creates are truly exceptional.
For example, if you run Prism in a proxy mode with an --errors flags it will reply with an error response each time the request or response invalidates the contract! Otherwise, it will add validation errors in a response header.
This means that you can, for instance, run Prism proxy while developing your front-end and see a contract failure instantly!
Quick demo
Let’s run two instances of Prism.
- One, on port
4011, will be a mock server imitating an actual API. Imagine that, for example, this is your development environment API server. - Another one, on port
4010, will be a proxy that your client (e.g. your mobile app or front-end web app) would connect to when developing.
The second instance will proxy to 4011 and will automatically verify that each HTTP request complies with the contract.
prism mock -p 4011 petstore-actual.oas3.yamlLink to petstore-actual.oas3.yaml.
/curl localhost:4011/pets/1 will return something like:
{
"status" : "available",
"photoUrls" : [
"string"
]
}Second, run Prism in proxy mode with the "expected" API.
prism proxy -p 4010 --errors petstore-expected.oas3.yaml http://localhost:4011Link to petstore-expected.oas3.yaml
curl http://localhost:4010/pets/1 returns a validation error!
{
"detail" : "Your request/response is not valid and the --errors flag is set, so Prism is generating this error for you.",
"type" : "https://stoplight.io/prism/errors#VIOLATIONS",
"status" : 500,
"title" : "Request/Response not valid",
"validation" : [
{
"location" : [
"response",
"body"
],
"message" : "should have required property 'name'",
"severity" : "Error",
"code" : "required"
}
]
}It happens because petstore-expected.oas3.yaml contract expects name property in the response object. However, the mocking server doesn’t include it!
"Silent" mode
If being yelled at with errors isn’t quite your thing then skip the --errors flag and Prism will dump its validations into sl-validations header.
HTTP/1.1 200 OK
access-control-allow-origin:
vary: Origin
access-control-allow-credentials: true
sl-violations: [{"location":["response","body"],"severity":"Error","code":"required","message":"should have required property 'name'"}]
content-type: application/json
content-length: 45
date: Wed, 13 Nov 2019 14:25:07 GMT
connection: close
{"photoUrls":["string"],"status":"available"}That option is very useful too, especially if you want to automate your contract testing.
Contract Testing with Prism
Not many people know but Prism has an experimental programmatic API (bear in mind it’s still unstable if you decide to use it). It’s a topic for an entire book but let me show you something cool you could do with very little effort.
I’ve put together a sample repo illustrating how to use Prism & jest framework to automate contract testing.
Disclaimer: if you start wondering how I got to know that programmatic API bear in mind I am one of the early co-implementers of Prism.
Anyway, feel free to go ahead and take a look at the source code. To keep this article relatively light I will only highlight the relevant parts.
/src/contract.test.ts
I first use the programmatic API to set up the config
Most of this is an obscure, secret, CSI stuff.
The important bit is the upstream. It turns Prism into a proxy mode.
const config = {
cors: false,
config: {
mock: false,
checkSecurity: true,
validateRequest: true,
validateResponse: true,
upstream: new URL('http://httpbin'),
} as IHttpConfig,
components: { logger: console },
errors: false,
};In beforeAll I:
- read the
api.oas2.yamlfile and convert it into an array of spec-agnostic HTTP Operations - create and start prism server in "proxy" mode
beforeAll(async () => {
operations = await getHttpOperationsFromResource(resolve(__dirname, 'api.oas2.yaml'));
server = createHttpServer(operations, config);
const address = await server.listen(4011, 'localhost');
});Now the grand finale! Because I have all API operations in a collection I can iterate through them, convert them to Axios request object, make a request and then check if the sl-violations is defined.
If it is, contract validation failed!
test('test operations', () => {
return Promise.all(operations.map(async operation => {
const request = operation2Request(operation);
const response = await axios(request);
expect(response.headers['sl-violations']).toBeUndefined();
}))
});What that means is that I can update my OpenAPI description file and all my operations could be tested automatically!
Summary
Reusing your OpenAPI description files for contract testing is a powerful concept. Instead of writing test cases based on yet another syntax you can use the same file that you already use to design, document, and style check your APIs.
As a company that helps engineers and businesses build their APIs using the best possible tools and techniques, we will definitely add this to our toolbelt.
And, if you value your time, we recommend you give a try to.