Open Source at LinkedIn

The open source projects behind the world's largest professional network. These are the technologies we created to connect the world's professionals to make them more productive and successful.

rest.li

Frameworks

Rest.li is a REST+JSON framework for building robust, scalable service architectures using dynamic discovery and simple asynchronous APIs.

dynamometer

Other

A tool for scale and performance testing of HDFS with a specific focus on the NameNode.

gdmix

Other

A deep ranking personalization framework

li-apache-kafka-clients

Other

li-apache-kafka-clients is a wrapper library for the Apache Kafka vanilla clients. It provides additional features such as large message support and auditing to the Java producer and consumer in the open source Apache Kafka.

linkedin-orc

Other

LinkedIn's version of Apache ORC

LayoutTest-iOS

Other

Write unit tests which test the layout of a view in multiple configurations

QueryAnalyzerAgent

Other

Analyze MySQL queries with negligible overhead

dagli

Other

Framework for defining machine learning models, including feature generation and transformations, as directed acyclic graphs (DAGs).

apk-bitminer

Other

null

pyexchange

Other

Python wrapper for Microsoft Exchange

Avro2TF

Other

Avro2TF is designed to fill the gap of making users' training data ready to be consumed by deep learning training frameworks.

databus

Data

Source-agnostic distributed change data capture system

PalDB

Other

An embeddable write-once key-value store written in Java

dexmaker

Other

A utility for doing compile or runtime code generation targeting Android's Dalvik VM

isolation-forest

Other

A distributed Spark/Scala implementation of the isolation forest algorithm for unsupervised outlier detection, featuring support for scalable training and ONNX export for easy cross-platform inference.

eyeglass

Other

NPM Modules for Sass

icon-magic

Other

Automated icon build system for iOS, Android and Web

smart-arg

Other

Smart Arguments Suite (smart-arg) is a slim and handy python lib that helps one work safely and conveniently with command line arguments.

instantsearch-tutorial

Other

Sample code for building an end-to-end instant search solution

o19-bmc-firmware

Other

OpenBMC is an open software framework to build a complete Linux image for a Board Management Controller (BMC)

venice

Other

Venice, Derived Data Platform for Planet-Scale Workloads.

greykite

Other

A flexible, intuitive and fast forecasting library

tracked-queue

Other

An autotracked implementation of a ring-buffer-backed double-ended queue

play-restli

Other

A library that simplifies building restli services on top of the play server.

ml-ease

Data

ADMM based large scale logistic regression

atscppapi

SysOps

This library provides wrappers around the existing Apache Traffic Server API which will vastly simplify the process of writing Apache Traffic Server plugins.

high-school-trainee

Other

LinkedIn Women in Tech High School Trainee Program

play-parseq

Other

Play-ParSeq is a Play module which seamlessly integrates ParSeq with Play Framework

forthic

Other

null

Cytodynamics

Other

Classloader isolation library.

cassette

Other

An efficient, file-based FIFO Queue for iOS and macOS.

thirdpartysources

Other

Third-Party Source Code made available to comply with license agreements.

LiFT

Other

The LinkedIn Fairness Toolkit (LiFT) is a Scala/Spark library that enables the measurement of fairness in large scale machine learning workflows.

school-of-sre

Other

At LinkedIn, we are using this curriculum for onboarding our entry-level talents into the SRE role.

linkedin.github.com

Other

Listing of all our public GitHub projects.

kafka-tools

Other

A collection of tools for working with Apache Kafka.

shiv

Other

shiv is a command line utility for building fully self contained Python zipapps as outlined in PEP 441, but with all their dependencies included.

css-blocks

Other

High performance, maintainable stylesheets.

linkedin-calcite

Other

LinkedIn's version of Apache Calcite

asciietch

Other

A graphing library with the goal of making it simple to graphs using ascii characters.

iris

Other

Iris is a highly configurable and flexible service for paging and messaging.

Spyglass

Mobile

A library for mentions on Android

dustjs

Frameworks

Asynchronous Javascript templating for the browser and server

datahub-gma

Other

General Metadata Architecture

kube2hadoop

Other

Secure HDFS Access from Kubernetes

opticss

Other

A CSS Optimizer

dr-elephant

Other

Dr. Elephant is a job and flow-level performance monitoring and tuning tool for Apache Hadoop and Apache Spark

oncall

Other

Oncall is a calendar tool designed for scheduling and managing on-call shifts. It can be used as source of dynamic ownership info for paging systems like http://iris.claims.

brooklin

Other

An extensible distributed system for reliable nearline data streaming at scale

python-avro-json-serializer

Other

Serializes data into a JSON format using AVRO schema.

iceberg

Other

A temporary home for LinkedIn's changes to Apache Iceberg (incubating)

dex-test-parser

Other

Find all test methods in an Android instrumentation APK

spark-inequality-impact

Other

null

gobblin-elr

Other

This is a read-only mirror of apache/gobblin

rest.li-test-suite

Other

A language-independent Rest.li test suite.

ambry

Other

Distributed object store

parseq

Frameworks

Asynchronous Java made easier

kafka-monitor

Other

Xinfra Monitor monitors the availability of Kafka clusters by producing synthetic workloads using end-to-end pipelines to obtain derived vital statistics - E2E latency, service produce/consume availability, offsets commit availability & latency, message loss rate and more.

Hakawai

Mobile

A powerful, extensible UITextView.

detext

Other

DeText: A Deep Neural Text Understanding Framework for Ranking and Classification Tasks

spaniel

Other

LinkedIn's JavaScript viewport tracking library and IntersectionObserver polyfill

cruise-control-ui

Other

Cruise Control Frontend (CCFE): Single Page Web Application to Manage Large Scale of Kafka Clusters

Tachyon

Other

An Android library that provides a customizable calendar day view UI widget.

FeatureFu

Data

Library and tools for advanced feature engineering

test-butler

Other

Reliable Android Testing, at your service

goavro

Other

Goavro is a library that encodes and decodes Avro data.

spark-tfrecord

Other

Read and write Tensorflow TFRecord data from Apache Spark.

data-integration-library

Other

The Data Integration Library project provides a library of generic components based on a multi-stage architecture for data ingress and egress.

linkedin-gradle-plugin-for-apache-hadoop

Other

null

coral

Other

Coral is a translation, analysis, and query rewrite engine for SQL and other relational languages.

Burrow

Other

Kafka Consumer Lag Checking

sysops-api

SysOps

sysops-api is a framework designed to provide visability from tens of thousands of machines in seconds.

transport

Other

A framework for writing performant user-defined functions (UDFs) that are portable across a variety of engines including Apache Spark, Apache Hive, and Presto.

self-focused

Other

Helps make a single page application more friendly to screen readers.

LiTr

Other

Lightweight hardware accelerated video/audio transcoder for Android.

qark

Other

Tool to look for several security related Android application vulnerabilities

URL-Detector

Other

A Java library to detect and normalize URLs in text

migz

Other

Multithreaded, gzip-compatible compression and decompression, available as a platform-independent Java library and command-line utilities.

performance-quality-models

Other

Personalizing Performance model repository

WomenConnect

Other

null

avro-util

Other

Collection of utilities to allow writing java code that operates across a wide range of avro versions.

flashback

Other

mock the internet

iris-mobile

Other

A mobile interface for linkedin/iris, built for iOS and Android on the Ionic platform

photon-ml

Other

A scalable machine learning library on Apache Spark

swift-style-guide

Other

LinkedIn's Official Swift Style Guide

lambda-learner

Other

Lambda Learner is a library for iterative incremental training of a class of supervised machine learning models.

pygradle

Other

Using Gradle to build Python projects

luminol

Other

Anomaly Detection and Correlation library

concurrentli

Other

Classes for multithreading that expand on java.util.concurrent, adding convenience, efficiency and new tools to multithreaded Java programs

iris-relay

Other

Stateless reverse proxy for thirdparty service integration with Iris API.

cruise-control

Other

Cruise-control is the first of its kind to fully automate the dynamic workload rebalance and self-healing of a Kafka cluster. It provides great value to Kafka users by simplifying the operation of Kafka clusters.

shaky-android

Other

Shake to send feedback for Android.

DuaLip

Other

DuaLip: Dual Decomposition based Linear Program Solver