In this thesis, we investigate developing machine learning models that can adapt to new tasks and environments rapidly with few observations. To this end, we propose to tackle the problem by learning an adaptive metric space, learning a set of more generalized priors, and learning additional parameters. Empirically, we also show that such adaptation methods can benefit the visual tracking task.