This paper considers binary choice models with panel data. We extend the correlated random effects binary choice models for panel data in Chamberlain (1980) to semiparametric models in which the conditional expectation projection of the unobserved time-invariant heterogeneity onto the space of functions of time-varying covariates for all time periods is nonparametrically specified. This class of models is tractable for identification and estimation of the model parameters with short panel data. We provide a set of mild conditions under which the parameters are identified. We propose to use the penalized sieve minimum distance (PSMD) estimation and develop the asymptotic theory. The PSMD estimators of finite dimensional parameters are shown to be semiparametrically efficient when the weighting matrix is the optimal one. We also show the bootstrap validity. The Monte Carlo simulation results confirm that the proposed estimator performs well in finite samples.