BOAT Introduction

BOAT (OperAtion-level Toolbox for gradient-based BLO) is a compositional, operation-level framework designed to bridge the gap between theoretical modeling and practical implementation in Bi-Level Optimization (BLO). Unlike existing libraries that typically encapsulate fixed solver routines, BOAT factorizes the BLO workflow into atomic, reusable primitives. Through a unified constraint reconstruction perspective, it empowers researchers to automatically compose over 85+ solver variants from a compact set of 17 gradient operations.

• PyTorch-based: An efficient and widely-used version.

• Jittor-based: An accelerated version for high-performance tasks.

• MindSpore-based: Incorporating the latest first-order optimization strategies to support emerging application scenarios.

BOAT is designed to offer robust computational support for a broad spectrum of BLO research and applications, enabling innovation and efficiency in machine learning and computer vision.

🔑 Key Features

• 🧩 Compositional Operation-Level Abstraction: Deconstructs BLO solvers into three modular stages: Gradient Mapping (GM), Numerical Approximation (NA), and First-Order (FO).

• 🏭 Generative Solver Construction: Supports dynamic serialization of operations. Users can recover classical algorithms (e.g., DARTS, MAML) or discover novel hybrid solvers (e.g., NGD + DI + PTT) simply by changing configurations.

• 🛠 Configuration-Driven: Define complex optimization strategies via simple JSON configurations (boat_config & loss_config), decoupling algorithmic logic from model definitions.

• 🚀 Unified Computational Analysis: Offers a comprehensive complexity analysis of gradient-based BLO techniques to guide users in selecting optimal configurations for efficiency and accuracy.

• ✅ Comprehensive Testing: Achieves 99% code coverage through rigorous testing with pytest, ensuring software robustness and reliability.

🚀 Why BOAT?

Existing automatic differentiation (AD) tools primarily focus on specific optimization strategies, such as explicit or implicit methods, and are often targeted at meta-learning or specific application scenarios, lacking support for algorithm customization.

In contrast, BOAT expands the landscape of Bi-Level Optimization (BLO) applications by supporting a broader range of problem-adaptive operations. It bridges the gap between theoretical research and practical deployment, offering unparalleled flexibility to design, customize, and accelerate BLO techniques.

🌍 Applications

BOAT covers a wide spectrum of BLO applications, categorized by the optimization target:

• Data-Centric: Data Hyper-Cleaning, Synthetic Data Reweighting, Diffusion Model Guidance.

• Model-Centric: Neural Architecture Search (NAS), LLM Prompt Optimization, Parameter Efficient Fine-Tuning (PEFT).

• Strategy-Centric: Meta-Learning, Hyperparameter Optimization (HO), Reinforcement Learning from Human Feedback (RLHF).

🚩 Related Operations

Gradient Mapping Operation Library (GM-OL)

• Towards gradient-based bilevel optimization with non-convex followers and beyond (DI)

• Averaged Method of Multipliers for Bi-Level Optimization without Lower-Level Strong Convexity(DM)

• A General Descent Aggregation Framework for Gradient-based Bi-level Optimization (GDA)

• Bilevel Programming for Hyperparameter Optimization and Meta-Learning (NGD)

Numerical Approximation Operation Library (NA-OL)

• Hyperparameter optimization with approximate gradient (CG)

• Optimizing millions of hyperparameters by implicit differentiation (NS)

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks (IAD)

• On First-Order Meta-Learning Algorithms (FOA)

• Bilevel Programming for Hyperparameter Optimization and Meta-Learning (RAD)

• Truncated Back-propagation for Bilevel Optimization (RGT)

• DARTS: Differentiable Architecture Search (FD)

• Towards gradient-based bilevel optimization with non-convex followers and beyond (PTT)

• Learning With Constraint Learning: New Perspective, Solution Strategy and Various Applications (IGA)

First-Order Operation Library (FO-OL)

• BOME! Bilevel Optimization Made Easy: A Simple First-Order Approach (VFO)

• A Value-Function-based Interior-point Method for Non-convex Bi-level Optimization (VSO)

• On Penalty-based Bilevel Gradient Descent Method (PGDO)

• Moreau Envelope for Nonconvex Bi-Level Optimization: A Single-loop and Hessian-free Solution Strategy (MESO)

📜 License

MIT License

Copyright (c) 2024 Yaohua Liu

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.