DuaLip: Dual Decomposition based Linear Program Solver
DuaLip is an extreme-scale Linear Program (LP) solver based on Apache Spark. It solves structured LP problems of the following form arising from web-applications:
where \(x = (x_1, ..., x_I)\) is the full vector of optimization variables, \(x_i\) is the vector of optimization variables associated with one \(i\), and \(A,b,c\) and \(C_i\) are user-supplied data.
It is a distributed solver that solves a perturbation of the LP problem at scale via gradient-based algorithms on the smooth dual of the perturbed LP with computational guarantees. DuaLip can easily scale to problems in trillions of variables.
This library was created by Yao Pan, Kinjal Basu, Rohan Ramanath, Konstantin Salomatin, Amol Ghoting, Sathiya Keerthi, and Miao Cheng from LinkedIn.
Code available on GitHub.
Citing DuaLip
If you are using DuaLip for your work, we encourage you to
We are looking forward to hearing your success stories with DuaLip! Please share them with us.
Features
- Extreme Scale
DuaLip is specifically developed to tackle problems arising in web-applications that usually have hundreds of millions of users and millions of items, pushing the number of optimization variables in the trillions range (if not more). It uses a dual decomposition technique to be able to scale to such large problems. See the Problem Solution section for details. For a wide range of applications, see Ramanath et. al. (2021) and Basu et. al (2020).
- Parallelism
We support extreme-scale parallelism in our DuaLip solver, which can solve tens of millions of small separate LPs simultaneously. Such problems arise in applications like personalized constrained optimization in recommender systems which have personalized constraints for each unique user. In this case, each LP is dedicated to one user and only contains a small number of decision variables. Our parallel version of DuaLip can solve tens of millions of separate small LPs for all different users in parallel.
- Efficient
Although we use well-known first-order gradient methods to solve the problem, we implement several highly efficient algorithms for each of the component steps. (See the Solver section for more details.) This allows us to scale up 20x over a naive implementation. Please see Ramanath et. al. (2021) for a comparative study.
- Modular Design
In our implementation, any problem can be formulated through a highly modular approach.
solver: We begin by choosing a first-order optimization solver. We currently support Accelerated Gradient Ascent, LBFGS and LBFGS-B.
projectionType: We implement several very efficient projection algorithms to allow for a wide class of constraint sets \(\mathcal{C}_i\). For a list of supported constraints sets see here.
Each of these components is highly flexible and can be easily customized to add new solvers, or new types of projections for different constraints sets \(\mathcal{C}_i\). New formulations can also be added by appropriately stitching together these different components.
- Infeasibility Detection
We have incorporated simple checks on infeasibility. This helps the end user to appropriately tweak the problem space.
- Extensive Logging
We have added extensive logging to help users understand whether the solver has converged to a good approximate solution. For more details, please see here.
- Warm start
We allow the user to input an initial estimate of the dual solution if they have one (e.g., if they have solved a highly related problem before). This can result in very efficient solving of the overall problem.
Usage
Building the Library
It is recommended to use Scala 2.12 and Spark 3.1.1. To build, run the following:
./gradlew build
This will produce a JAR file in the ./dualip/build/libs/
directory.
Tests typically run with the test
task. If you want to force-run all tests, you can use:
./gradlew cleanTest test --no-build-cache
Using the JAR File
Depending on the mode of usage, the built JAR can be deployed as part of an offline data pipeline, depended upon to build jobs using its APIs, or added to the classpath of a Spark Jupyter notebook or a Spark Shell instance. For example:
$SPARK_HOME/bin/spark-shell --jars target/dualip_2.12.jar
Copyright
Copyright 2022 LinkedIn Corporation All Rights Reserved. Licensed under the BSD 2-Clause License (the "License"). See License in the project root for license information.
Contributing
If you would like to contribute to this project, please review the instructions here.
Bug reports and support
Please report any issues via the Github issue tracker. All types of issues are welcome including bug reports, documentation typos, feature requests and so on.
Acknowledgments
Implementations of some methods in DuaLip were inspired by other open-source libraries. Discussions with several LinkedIn employees influenced various aspects of this library. A full list of acknowledgements can be found here.
References
DuaLip has been created on the basis of the following research papers. If you cite DuaLip, please use the following:
@inproceedings{ramanath:21, author = {Ramanath, Rohan, and Keerthi, Sathiya S. and Basu, Kinjal and Salomatin, Konstantin and Yao, Pan}, title = {Efficient Algorithms for Global Inference in Internet Marketplaces}, journal = {arXiv preprint arXiv:2103.05277}, year = {2021}, url = {https://arxiv.org/abs/2103.05277} } @InProceedings{pmlr-v119-basu20a, title = {{ECLIPSE}: An Extreme-Scale Linear Program Solver for Web-Applications}, author = {Basu, Kinjal and Ghoting, Amol and Mazumder, Rahul and Pan, Yao}, booktitle = {Proceedings of the 37th International Conference on Machine Learning}, pages = {704--714}, year = {2020}, volume = {119}, series = {Proceedings of Machine Learning Research}, month = {13--18 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v119/basu20a/basu20a.pdf}, url = {http://proceedings.mlr.press/v119/basu20a.html} } @misc{dualip, author = {Ramanath, Rohan, and Keerthi, Sathiya S. and Basu, Kinjal and Salomatin, Konstantin and Yao, Pan and Ghoting, Amol and Cheng, Miao}, title = {{DuaLip}: Dual Decomposition based Linear Program Solver, version 2.0.0}, url = {https://github.com/linkedin/dualip}, month = dec, year = 2022 }