The Rest.li Cross-Language Test Suite is a framework for testing and comparing Rest.li implementations. The framework contains the Rest.li Test Suite Specification, which provides test data and testing guidelines that specify which features and scenarios to test, and how to test them. These test guidelines should be followed when testing a particular language implementation of Rest.li. Currently, the framework only supports client testing, though we hope to add server testing support in the future.
The test suite also includes Java tests that follow the Rest.li Test Suite Specification, demonstrating how the spec can be used to test the Java implementation of Rest.li client.
The Rest.li Test Suite Specification will help us achieve consistency and quality across multiple Rest.li client language bindings by:
We have leveraged this Rest.li Test Suite Specification to implement tests for Java and Python Rest.li client bindings, respectively. With this sharable test suite spec, we have identified their coverage and feature parity in this Compatibility Matrix.
Make sure you have installed gradle, which is used for building the test suite.
Download the test suite from its git repository:
git clone <coming soon>
cd restli-testsuite
For new test suite users, take a look at How to Run the Java TestNG Tests and Expand the Test Suite Specification. This will walk you through running the provided Java TestNG tests. It will also explain how to expand the Rest.li Test Suite Specification.
If you would like to follow the spec and add tests for a new language binding, refer to How to Add Tests to a New Client Language Binding. This is a more involved process that requires understanding the spec. It will be helpful to go over the Design and Code Structure section of this document first.
This documentation assumes a background in Rest.li. Specifically, test developers should be familar with Rest.li data schema, IDL, and wire protocol format as demonstrated by the following links:
.pdl) - https://linkedin.github.io/rest.li/pdl_schema.restspec.json) - https://linkedin.github.io/rest.li/spec/restspec_formatImplementers are encouraged to explore the test data and reference documentation in tandem to learn the details of Rest.li.
There are two main ways of interacting with the Rest.li Test Suite Specification:
These distinct ways of using the Rest.li Test Suite Specification point to an important distinction between language-independent and language-specific components within this test suite. The Rest.li Test Suite Specification consists of language-independent data and test guidelines, meant to be used for standardizing cross-platform testing. It cannot immediately be used to test a specific language implementation of Rest.li client. As a test developer, you follow the Rest.li Test Suite Specification to write tests in your desired language.
Throughout the documentation, we use “Rest.li Test Suite Specification” to refer to the language-independent components of the project. We use “Java TestNG Tests” or “Python pytest Tests” for its language-dependent components, which follow the Test Suite Specification.
This test suite is composed of:
.json, .pdsc,.restspec.json.manifest.json file containing all the test data provided in this suite to help drive automated test execution.The test suite is structured as follows:
.
|—— client-testsuite
| |—— manifest.json
| |—— data/
| |—— requests/
| |—— requests-v2/
| |—— responses/
| |—— responses-v2/
| |—— restspecs/
| |—— schemas/
| |—— snapshots/
| |—— src/
| |—— ...
|—— restli-testsuite-server
| |—— src
| |—— ...
|—— build.gradle
|...
This manifest file provides machine readable information about all the automated tests included in this spec. It is shared across languages, and does not need to be changed when adding a test suite for a new language. It should be changed when expanding the Rest.li Test Suite Specification.
This file is intended to help drive portions of the test suite execution for each language. Although it does help, writing additional assertions by hand will still be needed in most languages to validate correctness of in-memory representations and language bindings.
The file is broken down into a few main sections:
.pdsc) as well as JSON data files matching the schemas..restspec.json) as well as test HTTP requests and responses,
in the form of files, for operations supported by the interface definition.These folders contain testing data used by the automated tests in the spec. Most of them were generated by Java Rest.li, and should be used when following the spec to test your Rest.li client implementation.
The following folders are included:
In src/test/java, you can find the Java test suite, which uses the TestNG testing framework.
You can add test suites for new languages in src/test/. The Java suite is separated into two folders:
test: Java files containing the tests (TestRestClientAgainstStandardTestSuite and TestRestClientWithManualAssertions), and utility methods (StandardTestSuiteBase) for running tests.
testsuite: Java files for building and loading requests and responses.
The restli-testsuite-server directory contains code for a Rest.li Java server. Based on java resources defined there,
Java Rest.li will generate language-independent restspecs in the client-testsuite/restspecs folder. Using these restspecs,
other Rest.li implementations can generate request builders without modifying server code.
When adding a new test, you may want to update or add a resource to restli-testsuite-server/src/main/java/testsuite.
See the section on wire protocol tests in How To Run the Java TestNG Tests and Expand the Test Suite Specification.
The test suite spec is intended to cover three categories of Rest.li client behavior: JSON serialization, data template generation, and wire protocol. Each test category is described in more detail below.
These are tests for serialization and deserialization. Tests cover basic JSON and JSON corner cases such as large numbers, special characters and encodings.
| JSON feature | Details |
|---|---|
| Basic Types | string, number, boolean, null, object, array |
| Empty Collections | empty object, empty array, object with empty arrays, object with empty objects, array of empty object, array of empty arrays |
| Large Numbers | int32 and int64 |
| Special Characters | periods in key |
| Unicode | Chinese character and e with an accent mark |
These are tests for data template generation from schema. Tests cover schema types (records, unions, enums, typerefs, …), primitive types, optionals and defaults. Backward compatibility rules are also covered.
| Schema feature | Details |
|---|---|
| Primitive Types | int, long, float, double, bytes, string |
| Complex Types | array of maps, map of ints, record |
| Complex Type: Unions | union of complex types, union of primitives, union of same types |
| Enums | with properties and with alias |
| Fixed Type | |
| Typerefs | for string, array, chained typeref, array, map, field, union |
| Include | include, and include with include |
| Default Fixup | to see if required fields are “fixed up” with defaults |
| Optional Fields |
These are tests for building requests and decoding responses. Tests cover serializing/deserializing of URLs, headers and bodies, escaping/encoding, batch formats, projections, and partial updates.
We test for well-formed requests by comparing the built HTTP request
with the expected HTTP request in the requests/ or requests-v2/ folder. We compare url, method, headers, and body.
We test that Rest.li can decode a Rest.li response from an HTTP response by checking the decoded Rest.li response for
the correct values. We compare the response’s status and error message with the status and error message specified by
manifest.json. The body of the response is tested through manual assertions that check for the correct values.
| Rest.li Method | Collection | Simple | Association | Action Set |
|---|---|---|---|---|
| get | x | x | x | |
| batch-get | x | x | ||
| finder | x | x | ||
| batch-finder | o | o | ||
| create | x | |||
| create with returned entity | x | |||
| batch-create | x | |||
| update | x | x | x | |
| partial update | x | o | ||
| batch-update | x | x | ||
| batch-partial-update | x | o | ||
| delete | x | x | x | |
| batch-delete | x | x | ||
| action | o | o | o | x |
“x” - test is included in Rest.li Test Suite Specification
“o” - test is not included but method should be supported by the resource
“ “ - test is not included and method should NOT be supported by resource
| Key Feature | Rest.li Method used |
|---|---|
| Key with Union of Simple and Complex Members | get |
| Query Params (int, string, long, string array, message array, string map, primitive union, complex types union, optional string, url typeref) | get |
| Complex Key (simple record) | get, create, delete, update batch-create, batch-delete, batch-get, batch-update, partial-update |
| Special Chars in ComplexKey Strings | get, batch-get |
| Error | Resource used | Rest.li Method | Details |
|---|---|---|---|
| 404 | Collection | get | Send empty get request |
| 400 | Collection | update | Request has a missing required field |
| 500 | Collection | create | Create request with id field |
| Error Details | Collection | create | CreateResponse with Error Details |
| Batch Results with Errors | Collection | batch_update | Batch update with one good and two bad requests |
| Feature | Resource tested | Method used |
|---|---|---|
| Typeref | Collection, Association | get |
| Subresource | Collection, Association | get |
| Projection | Collection, Association, ComplexKeyResource | get |
For questions or troubleshooting, refer to Test Suite Troubleshooting.