We study experiments and simulations of highly polydisperse emulsions under shear flow. By highly polydisperse, we mean with the largest droplet diameters being as much as ten times the smallest diameters. We quantify non-affine motion in these highly polydisperse samples — motion where droplets move with displacements distinctly different from their neighbors, or different from the overall imposed shear flow. The largest droplets typically move affinely, as if they are in a simple effective fluid formed by the other droplets. In contrast, the smallest droplets are often forced to move non-affinely by the larger droplets.  The behavior depends on the overall droplet size distribution; in general, the largest droplets are always the troublemakers.  Our main conclusion is that highly polydisperse samples behave qualitatively differently than weakly polydisperse samples.  This is both true microscopically (in terms of droplet motion) and macroscopically (in terms of rheology).