All projects
M.Eng. Aerospace Engineering project, Cornell University — advisor Prof. Jane Wang

Reinforcement Learning for Wing Kinematics in Hovering Flight

Nonlinear equations of motion, quasi-steady aerodynamics, and a PPO agent

Role
Sole author
Status
Completed
Reinforcement Learning for Wing Kinematics in Hovering Flight

Headline results

+75%
Episode reward mean improvement over training
196
PPO iterations in the extended training run
1.12 mm
Vertical drift over 10 s at the identified hover point
2 DOF
Per wing — flapping and pitching, pinned at the root
PythonSymPySciPy RK45Stable-Baselines3PPOBlade Element MethodGauss–Legendre
01

Why flapping flight resists analysis

Flapping-wing flight is governed by highly nonlinear, unsteady aerodynamic interactions that make both analytical modeling and control design difficult. Unlike a fixed wing, the aerodynamic environment a flapping wing sees changes continuously within a single stroke, and the body responds to those forces on the same timescale. There is no clean separation between the aerodynamics and the rigid-body dynamics.

This project builds a reduced-order dynamic model of a flapping system with an ellipsoidal body and rigid wings capable of symmetric flapping and pitching, then asks whether a reinforcement learning agent can discover wing kinematics that hold the body in place.

Figure 1: Rear view of the flapping system, showing the inertial, body, and wing reference frames.
Figure 1: Rear view of the flapping system, showing the inertial, body, and wing reference frames.
Figure 2: Upper-right lateral view of the same system.
Figure 2: Upper-right lateral view of the same system.
02

Deriving the equations of motion

I set up three reference frames — inertial, body, and wing — connected by direction cosine matrices, then applied Newton's second law to the coupled body-plus-wings system. Each wing carries two degrees of freedom, flapping angle φ and pitch angle ψ, pinned at the root. Because the wings carry mass and accelerate relative to the body, the body-frame acceleration terms have to be resolved back into inertial coordinates before the force balance closes.

Figure 3: Free-body diagram of the flapping system with aerodynamic loading distributed along the wing.
Figure 3: Free-body diagram of the flapping system with aerodynamic loading distributed along the wing.
Figure 4: Wing element definition used for the blade-element integration.
Figure 4: Wing element definition used for the blade-element integration.

Aerodynamic forces come from a thin-body quasi-steady model in the tradition of Andersen, Pesavento and Wang, applied strip-by-strip along the span and integrated numerically. The symbolic derivation was done in SymPy so the resulting expressions could be differentiated and verified rather than hand-transcribed, and the resulting system is integrated with SciPy's RK45.

φ(t) = A_φ · cos(ω_φ t) + φ_c ψ(t) = (ψ₊ − ψ₋) · sin(ω_φ t) + ψ_c , ψ_c = (ψ₊ + ψ₋) / 2
Figure 4b: Commanded wing angles over time — flapping angle φ and pitch angle ψ sharing a frequency but offset in phase.
Figure 4b: Commanded wing angles over time — flapping angle φ and pitch angle ψ sharing a frequency but offset in phase.

Splitting ψ between upstroke and downstroke — rather than holding pitch constant — is what gives the model control authority over translational motion. The sine form for ψ against the cosine form for φ places maximum pitch at the stroke reversals, where it does the most work.

03

Finding hover from first principles

Before handing anything to a learning agent, I validated the equations of motion by sweeping the kinematic parameters directly, using dimensionless coefficients scaled to a housefly. Hovering flight showed up at a specific combination:

A_φ = 60° φ_c = 30° ω_φ = 30.04 Hz ψ₊ = 87° ψ₋ = 4° ψ_c = 45.5°
Figure 5: Lateral position over time at the identified hover point — y varies by only 3 mm across 10 seconds.
Figure 5: Lateral position over time at the identified hover point — y varies by only 3 mm across 10 seconds.
Figure 6: Vertical position over the same interval — z falls by 1.12 mm in 10 seconds.
Figure 6: Vertical position over the same interval — z falls by 1.12 mm in 10 seconds.
Figure 6b: The resulting trajectory in the y–z plane. The body traces a tight closed loop within a few millimetres rather than drifting away — this is what the learning agent was later asked to rediscover.
Figure 6b: The resulting trajectory in the y–z plane. The body traces a tight closed loop within a few millimetres rather than drifting away — this is what the learning agent was later asked to rediscover.
04

The learning environment

I wrapped the simulation in a reinforcement learning environment where the agent adjusts the kinematic parameters — amplitudes, phase offsets, and wingbeat frequency — rather than commanding torques directly. Actions are smoothed between steps so the wing motion stays kinematically continuous instead of jumping discontinuously between wingbeats.

The reward is segment-based: within each evaluation window, a least-squares trend is fit to the body's y and z trajectories, and the agent is penalized on both the intercept difference and the slope difference from the target. Penalizing slope rather than only position is what makes the reward robust to the oscillation inherent in flapping flight — a body that bobs around a fixed point should not be punished the way a body that steadily drifts is.

  • Algorithm: Proximal Policy Optimization (PPO), continuous action space
  • Physics: full nonlinear EOM integrated in-loop at every environment step
  • Aerodynamics: quasi-steady blade element with Gauss–Legendre spanwise integration
  • Reward: least-squares trend extraction over trajectory segments, penalizing intercept and slope
05

Results

The first training attempts did not learn. Both the episode reward mean and the action-distribution standard deviation stayed nearly flat across iterations, which told me the policy was neither exploring nor improving. The problem was conditioning, not capability: the action space was too large and the reward too simply defined for the optimizer to find structure.

Tightening the action bounds and rescaling the reward produced the first genuine learning signal — episode reward rose over 25% and the policy visibly reduced lateral drift. With learning confirmed, I reverted to the originally intended reward formulation and launched a much longer run.

Figure 7: Wing kinematics the policy converged on during the extended run.
Figure 7: Wing kinematics the policy converged on during the extended run.
Figure 8: Body trajectory under the learned policy.
Figure 8: Body trajectory under the learned policy.

Across roughly 196 iterations of that extended run — about 17 hours of training — the episode reward mean increased by more than 75% while the action distribution standard deviation steadily narrowed. Both signals point the same way: the policy was consistently selecting wingbeat parameters that reduced body drift, and it was becoming more confident about them.

Figure 9: Final y and z displacements over the evaluation interval.
Figure 9: Final y and z displacements over the evaluation interval.
Figure 10: Final learned wing kinematics — note the amplitude the policy settles on.
Figure 10: Final learned wing kinematics — note the amplitude the policy settles on.
06

What it did and did not achieve

The PPO agent did not achieve sustained stable hover. It approached it — improving trajectory performance substantially and learning structured, physically plausible flapping motions — but the policy did not converge on the kind of stationary flight the hand-identified kinematics demonstrate.

The most likely cause sits in the reward shaping. Examining the learned kinematics, the reward appears to indirectly over-penalize large amplitudes and high frequencies, which are exactly the regions where hover lives. A policy that is punished for approaching the answer will stop short of it.

This was my first application of reinforcement learning to a physics-driven control problem. Coming from deterministic modeling, rigid-body dynamics, and numerical simulation, integrating policy-based learning and stochastic optimization required stepping well outside my usual analytical framework — which was much of the point.