研究

Informed Actor-Critic Method for Optimal Control in Economic Dynamics.
with Kenneth L. Judd, Bo Li, and Serguei Maliar 求职论文（Job Market Paper） Abstract: This paper applies deep reinforcement learning to high-dimensional optimal control in dynamic economic models. We introduce an informed actor–critic (iAC) framework that embeds structural equations and constraints directly into the policy optimization loop. Unlike standard model-free reinforcement learning algorithms that treat the economy as a black box—relying entirely on data-driven simulation—the iAC method leverages analytical economic transitions. Computational applications to benchmark consumption–savings, Krusell–Smith, and overlapping-generations economies show that iAC reduces the need for exploration noise and provides a precise, scalable, and sample-efficient framework for complex macroeconomic control problems.
[论文]

Discrete-Continuous Dynamic Choice: A Soft-to-Hard Reinforcement Learning Framework with an Application to Sovereign Default.
with Bo Li and Serguei Maliar Under review at Journal of Monetary Economics. Abstract: To solve discrete-continuous dynamic choice models, the quantitative literature has long relied on effective permanent structural smoothing to handle non-differentiable kinks. While traditional methods produce robust qualitative insights, their permanent modifications can alter quantitative predictions. To solve the true, unsmoothed economy, we introduce a soft-to-hard (StH) deep reinforcement learning framework. By systematically annealing temporary smoothing to zero, StH enables gradient propagation across decision boundaries before enforcing hard-switch operators. We implement this framework in two complementary ways. The grid-based StH-Q implementation delivers high numerical precision in the low-dimensional benchmark, whereas the continuous-state, continuous-action StH-AC implementation avoids storing a full state–action table and provides a scalable route to higher-dimensional applications.
[论文]

W-exp: A Routed Post-Decision Expectation Adapter for Dynamic Economic Solvers.
with Bo Li and Serguei Maliar Abstract: Conditional expectations are repeatedly evaluated while policies and value approximations move during the solution of stochastic dynamic models. We study W-exp, a fitted post-decision continuation adapter. Conditional on the current host-solver continuation, its training problem is a conventional conditional-mean regression. The contribution is the combination of a common adapter interface, online updating, explicit routing to a critic target or actor objective, and diagnostics for level error, derivative error, support, and drift. The interface is instantiated separately in tabular Q-learning, a deterministic actor–critic method, and soft actor–critic; trained weights and configurations are not transferred across solvers. We give conditional perturbation results for a contraction-based Bellman operator and a fixed-distribution deterministic actor, an idealized deterministic tracking lemma, and a one-call mean-squared-error comparison with fresh Monte Carlo. These results do not establish convergence of the nonlinear solvers or a computational break-even. In a descriptive consumption–saving study, tuned W-exp-equipped packages have lower mean final policy error than direct MC50 packages in all 12 solver–environment cells after one million outer steps, with reductions of 23.5%–84.1% across three seeds. The comparison does not match optimizer work: the primary IAC and SAC W-exp packages use additional actor or actor–critic updates, and neural runtimes are 1.57–3.96 times the direct baselines. Mechanism diagnostics indicate that routing and training support can matter, but the small and partly unbalanced designs do not identify universal causal rankings. The evidence supports W-exp as a testable adapter architecture, not as a generally faster or uniformly superior integration method.
[论文]

DSPG: Learning Distribution-Conditioned Policies for Heterogeneous-Agent Economies.
with Chiyuan Wang and Bo Li Abstract: Advances in household microdata and computation have shifted policy analysis beyond aggregate averages toward policy incidence across households in different economic states. Heterogeneous-agent models provide this distributional view, but they make the cross-sectional distribution of assets and income an infinite-dimensional recursive state: household choices move the distribution, and the distribution feeds back through equilibrium prices. Existing solvers trade distributional information against tractability. Moment methods impose a low-dimensional summary, Deep Equilibrium Nets retain a discretized histogram in a dense neural solver, and DeepHAM learns generalized moments. We introduce distribution-based structural policy gradients (DSPG), an algorithmic framework that adds direct distribution conditioning to structural reinforcement learning. DSPG learns consumption policies from the asset–income histogram while the model supplies budget accounting, market clearing, and distribution transitions. The policy architecture is modular. As a second contribution, we propose a one-dimensional U-Net that exploits the ordered asset grid, retains the full histogram without a prespecified moment map, and represents the policy with only 783 parameters. In a Huggett economy with aggregate shocks, DSPG produces smooth equilibrium policies and stationary distributions with small asset-market residuals. Using a common MLP-32 representation, DSPG has the lowest Euler diagnostic among four modern equilibrium algorithms in our benchmark. Separately, across 600 architecture and training configurations, the U-Net achieves lower Euler errors than the tested dense networks at the reference learning rate with far fewer parameters. The economic application shows why this capacity matters: greater idiosyncratic risk changes inequality and household asset buffers even when aggregate assets barely move.

Gradient Descent of the Discrete-Time Ramsey Growth Model.
with Bo Li, Yapeng Qian, and Shenghao Zhu Abstract: This paper provides an optimization-oriented study of the discrete-time Ramsey growth model, developed as a companion to our continuous-time analysis, replacing every continuous-time device by its discrete-time counterpart. We derive the discrete Euler–Lagrange residual as a functional gradient and use gradient-based procedures to validate the Ramsey dynamics: strict concavity and global optimality from the second variation, local strong convexity from a discrete Poincaré inequality, and exponential convergence of the sequence-space gradient flow from a Lyapunov–Grönwall argument. We construct a quadratic Lyapunov function and show, from strict monotonicity of the optimal capital policy rather than by linearization, that the optimal policy never overshoots, certifying global convergence to the steady state. We then relate the resulting schemes to deep deterministic policy gradient methods, identifying exact analytical counterparts for residual relaxation, target networks, the actor–critic split, and off-policy sampling.

Gradient Descent of Ramsey Growth Model.
with Bo Li, Yapeng Qian, and Shenghao Zhu Abstract: This paper provides an optimization-oriented study of the continuous-time Ramsey growth model, connecting classical stability analysis with residual-minimization approaches for solving Hamilton–Jacobi–Bellman (HJB) equations. We revisit the planner's primal formulation through a variational perspective and validate the Ramsey dynamics numerically. We construct a Lyapunov function and show that it decreases along saddle-point stable trajectories, certifying convergence to the steady state. We then formulate a residual loss functional induced by the HJB operator and study its analytical properties, including well-posedness, regularity, and convexity in an infinite-dimensional setting, enabling a principled comparison of first-order and second-order optimization schemes. We extend the construction to deterministic and stochastic environments, discussing how diffusion terms modify the HJB residual.

缴费基数约束与弹性退休：养老保险改革的政策权衡及其组合效应
李博、陆毅、张培元 摘要： 本文构建包含内生退休选择的异质性个体世代交叠模型，量化评估缴费基数调整与弹性退休的独立效应和组合效应。模型同时刻画缴费基数上下限、缴费历史和双账户待遇规则，将缴费端的非线性约束、待遇端的历史依赖与退休选择纳入同一家庭决策。量化结果表明，缴费基数下限是主要政策边际。降低下限虽能减轻低收入阶段的缴费负担，却会降低养老金替代率并扩大基金赤字，宏观总量和福利改善有限。弹性退休通过少数当前生产率较高、历史缴费基数较低的边际个体延长劳动和缴费期、推迟待遇领取，从而提高产出并改善基金收支。两项政策并行能够保留缴费减负效果，并基本抵消降低下限造成的待遇下降和基金压力。Shapley 分解显示，缴费减负主要来自降低下限，基金压力下降、产出和消费增量主要来自弹性退休。拓展分析发现，将缴费窗口前移至青年期后，弹性退休缩小赤字的幅度由 4.65% 降至 0.58%。本文在统一框架中识别了缴费减负与退休激励的政策分工，并揭示延迟退休的财政作用取决于继续工作是否继续缴费。
[论文]

AI Taxing AI: A Multi-Agent Reinforcement Learning Approach to Optimal Taxation.
with Bo Li Abstract: Artificial intelligence (AI) is improving productivity, yet its economy-wide diffusion may reshape the labor market and the distribution of income. To address these issues, we develop a Multi-Agent Reinforcement Learning (MARL) model featuring endogenous skill accumulation. We find that while AI boosts aggregate output via a rapid process of skill sharing, it displaces middle-skilled humans and increases inequality. However, optimal taxation can redistribute the technological surplus to achieve Pareto improvements. To find the optimal tax schedule, we employ Dual-Loop Multi-Agent Reinforcement Learning to account for both the government's fiscal policy and agent behaviors. Furthermore, we demonstrate that a “Digital Commons” regime—combining universal access with real-time knowledge sharing—maximizes social welfare by mitigating the trade-off between efficiency and equity.
[论文]

Forecasting Crude Oil Spot Prices Using a Transformer-BiLSTM Architecture with NRBO-Based Hyperparameter Optimization.
with Wenhui Huang, Zhihan Shen, Yu Lin, and Yi Zhang Abstract: A future vision of global crude oil markets is crucial for market participants and policy makers. However, making an accurate prediction of crude oil prices is always challenging, due to the difficulty in extracting both global and local information from historical price data and optimizing hyperparameter settings of the forecasting framework. In this research, a Transformer-BiLSTM-NRBO forecasting model architecture is proposed to overcome these challenges. First, a Transformer-based feature extraction module is adopted to enhance the capture of global information in the forecasting framework. Then, a BiLSTM-based feature extraction module is used to enhance the capture of local information. Finally, an NRBO algorithm is adopted to optimize the key hyperparameter settings of the previous modules. Experiments conducted on daily Brent and WTI crude oil spot prices from 2013 to 2024 illustrate the robust forecasting capability of the proposed method. Research limitations and future directions are given.
[论文]

Reduced-State Deep Learning for Heterogeneous Agent Models.
以低维矩向量近似分布状态，使含总量不确定性的异质性主体模型保持可计算。

Emergence of Universal Equivalents with Deep Reinforcement Learning.
构建多智能体强化学习环境，使主体从自给自足演化至物物交换并收敛到一般等价物，从而在不依赖强制度假设的前提下研究价格发现与货币涌现。

Asset Pricing in China's Stock Market.
with Shuaiyu Jiang
Finance & Economics Vision, 1(1), 2022. Abstract: This paper uses traditional machine learning methods and deep neural networks based on both firm-specific characteristics and macroeconomic variables to price China's A-share stock market. The main contributions are as follows. We give the stochastic discount factor a flexible form and compare different models' performances. Since the Chinese government adopts various policies to maintain financial stability, we borrow the idea from generative adversarial networks to find the true SDF by selecting moment conditions that minimize return volatility. Additionally, we compare this model's performance with Chen's work and find that this model can obtain a higher Sharpe ratio and R².
[论文]