We investigate an optimization problem governed by an elliptic partial differential equation with uncertain parameters. We introduce a robust optimization framework that accounts for uncertain model parameters. The resulting nonlinear optimization problem has a bilevel structure due to the min-max formulation. To approximate the worst case in the optimization problem, we propose linear and quadratic approximations. However, this approach still turns out to be very expensive; therefore, we propose an adaptive model order reduction technique which avoids long offline stages and provides a certified reduced order surrogate model for the parametrized PDE which is then utilized in the numerical optimization. Numerical results are presented to validate the presented approach.