Research

Informed Actor-Critic Method for Optimal Control in Economic Dynamics.
with Kenneth L. Judd, Bo Li, and Serguei Maliar Job Market Paper. Abstract: This paper applies deep reinforcement learning to high-dimensional optimal control in dynamic economic models. We introduce an informed actor–critic (iAC) framework that embeds structural equations and constraints directly into the policy optimization loop. Unlike standard model-free reinforcement learning algorithms that treat the economy as a black box—relying entirely on data-driven simulation—the iAC method leverages analytical economic transitions. Computational applications to benchmark consumption–savings, Krusell–Smith, and overlapping-generations economies show that iAC reduces the need for exploration noise and provides a precise, scalable, and sample-efficient framework for complex macroeconomic control problems.
[Paper]

Solving Discrete-Continuous Dynamic Choice Models: A Soft-to-Hard AI Solver with an Application to Sovereign Default.
with Bo Li and Serguei Maliar Abstract: To solve discrete-continuous dynamic choice models, the quantitative literature has long relied on effective permanent structural smoothing to handle non-differentiable kinks. While traditional methods produce robust qualitative insights, their permanent modifications can alter quantitative predictions. To solve the true, unsmoothed economy, we introduce a soft-to-hard (StH) deep reinforcement learning framework. By systematically annealing temporary smoothing to zero, StH enables gradient propagation across decision boundaries before enforcing hard-switch operators. We implement this framework in two complementary ways. The grid-based StH-Q implementation delivers high numerical precision in the low-dimensional benchmark, whereas the continuous-state, continuous-action StH-AC implementation avoids storing a full state–action table and provides a scalable route to higher-dimensional applications.
[Paper]

W-exp: A Routed Post-Decision Expectation Adapter for Dynamic Economic Solvers.
with Bo Li and Serguei Maliar Abstract: Conditional expectations are repeatedly evaluated while policies and value approximations move during the solution of stochastic dynamic models. We study W-exp, a fitted post-decision continuation adapter. Conditional on the current host-solver continuation, its training problem is a conventional conditional-mean regression. The contribution is the combination of a common adapter interface, online updating, explicit routing to a critic target or actor objective, and diagnostics for level error, derivative error, support, and drift. The interface is instantiated separately in tabular Q-learning, a deterministic actor–critic method, and soft actor–critic; trained weights and configurations are not transferred across solvers. We give conditional perturbation results for a contraction-based Bellman operator and a fixed-distribution deterministic actor, an idealized deterministic tracking lemma, and a one-call mean-squared-error comparison with fresh Monte Carlo. These results do not establish convergence of the nonlinear solvers or a computational break-even. In a descriptive consumption–saving study, tuned W-exp-equipped packages have lower mean final policy error than direct MC50 packages in all 12 solver–environment cells after one million outer steps, with reductions of 23.5%–84.1% across three seeds. The comparison does not match optimizer work: the primary IAC and SAC W-exp packages use additional actor or actor–critic updates, and neural runtimes are 1.57–3.96 times the direct baselines. Mechanism diagnostics indicate that routing and training support can matter, but the small and partly unbalanced designs do not identify universal causal rankings. The evidence supports W-exp as a testable adapter architecture, not as a generally faster or uniformly superior integration method.
[Paper]

DSPG: Learning Distribution-Conditioned Policies for Heterogeneous-Agent Economies.
with Chiyuan Wang and Bo Li Abstract: Advances in household microdata and computation have shifted policy analysis beyond aggregate averages toward policy incidence across households in different economic states. Heterogeneous-agent models provide this distributional view, but they make the cross-sectional distribution of assets and income an infinite-dimensional recursive state: household choices move the distribution, and the distribution feeds back through equilibrium prices. Existing solvers trade distributional information against tractability. Moment methods impose a low-dimensional summary, Deep Equilibrium Nets retain a discretized histogram in a dense neural solver, and DeepHAM learns generalized moments. We introduce distribution-based structural policy gradients (DSPG), an algorithmic framework that adds direct distribution conditioning to structural reinforcement learning. DSPG learns consumption policies from the asset–income histogram while the model supplies budget accounting, market clearing, and distribution transitions. The policy architecture is modular. As a second contribution, we propose a one-dimensional U-Net that exploits the ordered asset grid, retains the full histogram without a prespecified moment map, and represents the policy with only 783 parameters. In a Huggett economy with aggregate shocks, DSPG produces smooth equilibrium policies and stationary distributions with small asset-market residuals. Using a common MLP-32 representation, DSPG has the lowest Euler diagnostic among four modern equilibrium algorithms in our benchmark. Separately, across 600 architecture and training configurations, the U-Net achieves lower Euler errors than the tested dense networks at the reference learning rate with far fewer parameters. The economic application shows why this capacity matters: greater idiosyncratic risk changes inequality and household asset buffers even when aggregate assets barely move.

Gradient Descent of the Discrete-Time Ramsey Growth Model.
with Bo Li, Yapeng Qian, and Shenghao Zhu Abstract: This paper provides an optimization-oriented study of the discrete-time Ramsey growth model, developed as a companion to our continuous-time analysis, replacing every continuous-time device by its discrete-time counterpart. We derive the discrete Euler–Lagrange residual as a functional gradient and use gradient-based procedures to validate the Ramsey dynamics: strict concavity and global optimality from the second variation, local strong convexity from a discrete Poincaré inequality, and exponential convergence of the sequence-space gradient flow from a Lyapunov–Grönwall argument. We construct a quadratic Lyapunov function and show, from strict monotonicity of the optimal capital policy rather than by linearization, that the optimal policy never overshoots, certifying global convergence to the steady state. We then relate the resulting schemes to deep deterministic policy gradient methods, identifying exact analytical counterparts for residual relaxation, target networks, the actor–critic split, and off-policy sampling.

Gradient Descent of Ramsey Growth Model.
with Bo Li, Yapeng Qian, and Shenghao Zhu Abstract: This paper provides an optimization-oriented study of the continuous-time Ramsey growth model, connecting classical stability analysis with residual-minimization approaches for solving Hamilton–Jacobi–Bellman (HJB) equations. We revisit the planner's primal formulation through a variational perspective and validate the Ramsey dynamics numerically. We construct a Lyapunov function and show that it decreases along saddle-point stable trajectories, certifying convergence to the steady state. We then formulate a residual loss functional induced by the HJB operator and study its analytical properties, including well-posedness, regularity, and convexity in an infinite-dimensional setting, enabling a principled comparison of first-order and second-order optimization schemes. We extend the construction to deterministic and stochastic environments, discussing how diffusion terms modify the HJB residual.

Contribution Base Constraints and Flexible Retirement: Policy Trade-offs and Combination Effects in China's Pension Reform.
with Bo Li and Yi Lu Abstract: We build a heterogeneous-agent overlapping-generations model with an endogenous retirement choice to quantify the separate and joint effects of contribution base adjustments and flexible retirement, specifying contribution base bounds, contribution histories, and the benefit rules of the two pension accounts in a single household problem. The lower contribution bound is the main policy margin: lowering it relieves low-income contributors but reduces replacement rates and widens the pension deficit, with limited aggregate gains. Flexible retirement raises output and improves the pension balance through a small number of marginal individuals with high current productivity and low historical contribution bases. Combining the two policies preserves the relief effect while offsetting much of the benefit and financing pressure. A Shapley decomposition attributes relief mainly to the lower bound, and the output and financing gains mainly to flexible retirement. Shifting the contribution window forward to early in the working life reduces the deficit improvement from 4.65% to 0.58%. The analysis identifies the division of labor between contribution relief and retirement incentives and shows that the fiscal effect of delayed retirement depends on whether continued work also entails continued contributions.
[Paper]

AI Taxing AI: A Multi-Agent Reinforcement Learning Approach to Optimal Taxation.
with Bo Li Abstract: Artificial intelligence (AI) is improving productivity, yet its economy-wide diffusion may reshape the labor market and the distribution of income. To address these issues, we develop a Multi-Agent Reinforcement Learning (MARL) model featuring endogenous skill accumulation. We find that while AI boosts aggregate output via a rapid process of skill sharing, it displaces middle-skilled humans and increases inequality. However, optimal taxation can redistribute the technological surplus to achieve Pareto improvements. To find the optimal tax schedule, we employ Dual-Loop Multi-Agent Reinforcement Learning to account for both the government's fiscal policy and agent behaviors. Furthermore, we demonstrate that a “Digital Commons” regime—combining universal access with real-time knowledge sharing—maximizes social welfare by mitigating the trade-off between efficiency and equity.
[Paper]

Forecasting Crude Oil Spot Prices Using a Transformer-BiLSTM Architecture with NRBO-Based Hyperparameter Optimization.
with Wenhui Huang, Zhihan Shen, Yu Lin, and Yi Zhang Abstract: A future vision of global crude oil markets is crucial for market participants and policy makers. However, making an accurate prediction of crude oil prices is always challenging, due to the difficulty in extracting both global and local information from historical price data and optimizing hyperparameter settings of the forecasting framework. In this research, a Transformer-BiLSTM-NRBO forecasting model architecture is proposed to overcome these challenges. First, a Transformer-based feature extraction module is adopted to enhance the capture of global information in the forecasting framework. Then, a BiLSTM-based feature extraction module is used to enhance the capture of local information. Finally, an NRBO algorithm is adopted to optimize the key hyperparameter settings of the previous modules. Experiments conducted on daily Brent and WTI crude oil spot prices from 2013 to 2024 illustrate the robust forecasting capability of the proposed method. Research limitations and future directions are given.
[Paper]

Reduced-State Deep Learning for Heterogeneous Agent Models.
A reduced-state neural solution method that approximates the distributional state using a low-dimensional vector of moments, keeping heterogeneous-agent models with aggregate uncertainty computationally tractable.

Emergence of Universal Equivalents with Deep Reinforcement Learning.
A multi-agent reinforcement learning environment in which agents evolve from autarky to barter and converge on a universal equivalent, allowing price discovery and monetary emergence to be studied without strong institutional assumptions.

Asset Pricing in China's Stock Market.
with Shuaiyu Jiang
Finance & Economics Vision, 1(1), 2022. Abstract: This paper uses traditional machine learning methods and deep neural networks based on both firm-specific characteristics and macroeconomic variables to price China's A-share stock market. The main contributions are as follows. We give the stochastic discount factor a flexible form and compare different models' performances. Since the Chinese government adopts various policies to maintain financial stability, we borrow the idea from generative adversarial networks to find the true SDF by selecting moment conditions that minimize return volatility. Additionally, we compare this model's performance with Chen's work and find that this model can obtain a higher Sharpe ratio and R².
[Paper]