This paper proposes a color-based video analytic system for quantifying limb movements in epileptic seizure monitoring. The system utilizes colored pyjamas to facilitate limb segmentation and tracking. Thus, it is unobtrusive and requires no sensor/marker attached to patient’s body. We employ Gaussian mixture models in background/foreground modeling and detect limbs through a coarse-to-fine paradigm with graph-cut-based segmentation. Next, we estimate limb parameters with domain knowledge guidance and extract displacement and oscillation features from movement trajectories for seizure detection/analysis. We report studies on sequences captured in an epilepsy monitoring unit. Experimental evaluations show that the proposed system has achieved comparable performance to EEG-based systems in detecting motor seizures.