Cross-Language Test Suite Framework



The Cross-Language Test Suite is a framework for testing and comparing implementations. The framework contains the Test Suite Specification, which provides test data and testing guidelines that specify which features and scenarios to test, and how to test them. These test guidelines should be followed when testing a particular language implementation of Currently, the framework only supports client testing, though we hope to add server testing support in the future.

The test suite also includes Java tests that follow the Test Suite Specification, demonstrating how the spec can be used to test the Java implementation of client.


The Test Suite Specification will help us achieve consistency and quality across multiple client language bindings by:

  • Reducing testing effort required to add a new client implementation
  • Standardizing test plans that can be shared among multiple client implementations
  • “Certifying” client implementations with a quantifiable measure of quality
  • Making it easier to develop compatible implementations, so that a client in one language can be used with a server in another language.

We have leveraged this Test Suite Specification to implement tests for Java and Python client bindings, respectively. With this sharable test suite spec, we have identified their coverage and feature parity in this Compatibility Matrix.

Getting Started

How to Download Project

Make sure you have installed gradle, which is used for building the test suite.

Download the test suite from its git repository:

git clone <coming soon>
cd restli-testsuite

How to Run Java Tests and Expand Test Spec

For new test suite users, take a look at How to Run the Java TestNG Tests and Expand the Test Suite Specification. This will walk you through running the provided Java TestNG tests. It will also explain how to expand the Test Suite Specification.

How to Add Tests to a New Client Language Binding

If you would like to follow the spec and add tests for a new language binding, refer to How to Add Tests to a New Client Language Binding. This is a more involved process that requires understanding the spec. It will be helpful to go over the Design and Code Structure section of this document first.

Design and Code Structure


This documentation assumes a background in Specifically, test developers should be familar with data schema, IDL, and wire protocol format as demonstrated by the following links:

  • Data schema (.pdsc) -
  • IDL (.restspec.json) -
  • Wire Protocol -

Implementers are encouraged to explore the test data and reference documentation in tandem to learn the details of

Design Principles

There are two main ways of interacting with the Test Suite Specification:

  1. Expanding or updating the Test Suite Specification for new features or updated behavior.
  2. Following the spec to add tests for a client binding implemented in a new language.

These distinct ways of using the Test Suite Specification point to an important distinction between language-independent and language-specific components within this test suite. The Test Suite Specification consists of language-independent data and test guidelines, meant to be used for standardizing cross-platform testing. It cannot immediately be used to test a specific language implementation of client. As a test developer, you follow the Test Suite Specification to write tests in your desired language.

Throughout the documentation, we use “ Test Suite Specification” to refer to the language-independent components of the project. We use “Java TestNG Tests” or “Python pytest Tests” for its language-dependent components, which follow the Test Suite Specification.


This test suite is composed of:

  • Test Suite Specification
    • Data files in language-neutral formats such as .json, .pdsc,.restspec.json.
    • A manifest.json file containing all the test data provided in this suite to help drive automated test execution.
    • Guidelines on how to use the spec to validate a client implementation.
  • A Java TestNG suite that uses the spec’s test data to validate the Java client implementation.
  • A Java server for wire protocol data generation and potential integration testing

The test suite is structured as follows:

|—— client-testsuite
|   |—— manifest.json
|   |—— data/
|   |—— requests/
|   |—— requests-v2/
|   |—— responses/
|   |—— responses-v2/
|   |—— restspecs/
|   |—— schemas/
|   |—— snapshots/
|   |—— src/
|   |—— ...
|—— restli-testsuite-server
|   |—— src
|   |—— ...
|—— build.gradle
|... Test Suite Specification


This manifest file provides machine readable information about all the automated tests included in this spec. It is shared across languages, and does not need to be changed when adding a test suite for a new language. It should be changed when expanding the Test Suite Specification.

This file is intended to help drive portions of the test suite execution for each language. Although it does help, writing additional assertions by hand will still be needed in most languages to validate correctness of in-memory representations and language bindings.

The file is broken down into a few main sections:

  • jsonTestData - list of JSON data files.
  • schemaTestData - list of data schema files (.pdsc) as well as JSON data files matching the schemas.
  • wireProtocolTestData - list of interface definition files (.restspec.json) as well as test HTTP requests and responses, in the form of files, for operations supported by the interface definition.
Test data folders

These folders contain testing data used by the automated tests in the spec. Most of them were generated by Java, and should be used when following the spec to test your client implementation.

The following folders are included:

  • data: input JSON data for testing schemas and json serialization
  • requests: incoming HTTP requests for wire protocol tests ( protocol 1.0)
  • requests-v2: incoming HTTP requests for wire protocol tests ( protocol 2.0)
  • responses: expected HTTP responses for wire protocol tests ( protocol 1.0)
  • responses-v2: expected HTTP responses for wire protocol tests ( protocol 2.0)
  • restspecs: IDL (Interface Description Language) generated from Java resources. These should be used by implementations to make request builders.
  • snapshots: snapshots for resource compatibility checker

Java TestNG Tests

In src/test/java, you can find the Java test suite, which uses the TestNG testing framework. You can add test suites for new languages in src/test/. The Java suite is separated into two folders:

  • test: Java files containing the tests (TestRestClientAgainstStandardTestSuite and TestRestClientWithManualAssertions), and utility methods (StandardTestSuiteBase) for running tests.

  • testsuite: Java files for building and loading requests and responses.

Sample server

The restli-testsuite-server directory contains code for a Java server. Based on java resources defined there, Java will generate language-independent restspecs in the client-testsuite/restspecs folder. Using these restspecs, other implementations can generate request builders without modifying server code. When adding a new test, you may want to update or add a resource to restli-testsuite-server/src/main/java/testsuite. See the section on wire protocol tests in How To Run the Java TestNG Tests and Expand the Test Suite Specification. Test Suite Specification Coverage

The test suite spec is intended to cover three categories of client behavior: JSON serialization, data template generation, and wire protocol. Each test category is described in more detail below.

JSON Tests

These are tests for serialization and deserialization. Tests cover basic JSON and JSON corner cases such as large numbers, special characters and encodings.

JSON feature Details
Basic Types string, number, boolean, null, object, array
Empty Collections empty object, empty array, object with empty arrays, object with empty objects, array of empty object, array of empty arrays
Large Numbers int32 and int64
Special Characters periods in key
Unicode Chinese character and e with an accent mark

Data Schema Tests

These are tests for data template generation from schema. Tests cover schema types (records, unions, enums, typerefs, …), primitive types, optionals and defaults. Backward compatibility rules are also covered.

Schema feature Details
Primitive Types int, long, float, double, bytes, string
Complex Types array of maps, map of ints, record
Complex Type: Unions union of complex types, union of primitives, union of same types
Enums with properties and with alias
Fixed Type  
Typerefs for string, array, chained typeref, array, map, field, union
Include include, and include with include
Default Fixup to see if required fields are “fixed up” with defaults
Optional Fields  

Wire Protocol Tests

These are tests for building requests and decoding responses. Tests cover serializing/deserializing of URLs, headers and bodies, escaping/encoding, batch formats, projections, and partial updates.

We test for well-formed requests by comparing the built HTTP request with the expected HTTP request in the requests/ or requests-v2/ folder. We compare url, method, headers, and body. We test that can decode a response from an HTTP response by checking the decoded response for the correct values. We compare the response’s status and error message with the status and error message specified by manifest.json. The body of the response is tested through manual assertions that check for the correct values.

Basic Resource Method Tests for Requests/Responses Method Collection Simple Association Action Set
get x x x  
batch-get x   x  
finder x   x  
create x      
create with returned entity x      
batch-create x      
update x x x  
partial update x   o  
batch-update x   x  
batch-partial-update x   o  
delete x x x  
batch-delete x   x  
action o o o x

“x” - test is included in Test Suite Specification
“o” - test is not included but method should be supported by the resource
“ “ - test is not included and method should NOT be supported by resource

Resource Key and Parameter Tests

Key Feature Method used
Key with Union of Simple and Complex Members get
Query Params (int, string, long, string array, message array, string map, primitive union, complex types union, optional string, url typeref) get
Complex Key (simple record) get, create, delete, update batch-create, batch-delete, batch-get, batch-update, partial-update
Special Chars in ComplexKey Strings get, batch-get

Error tests

Error Resource used Method Details
404 Collection get Send empty get request
400 Collection update Request has a missing required field
500 Collection create Create request with id field
Error Details Collection create CreateResponse with Error Details
Batch Results with Errors Collection batch_update Batch update with one good and two bad requests

Misc. Tests

Feature Resource tested Method used
Typeref Collection, Association get
Subresource Collection, Association get
Projection Collection, Association, ComplexKeyResource get

Next Steps

Improvements to Test Suite Specification

  • Add test for record with lowercase name
  • Add test for tunneled query params
  • Add test that includes unicode (non-ascii) characters on wire
  • Extend spec to include cross-language server testing

Gaps in example Java Tests

  • Enrich manual assertions for wire protocol and schema tests
  • Fill in the gaps listed in the Base Resource Method Tests table

Future Work

  • Expand reference implementations of spec to different languages, such as mobile.


For questions or troubleshooting, refer to Test Suite Troubleshooting.