Just say "people with disabilities", because that's who we are.
The disability or impairment you have does not necessarily disable you, what disables you is the way society treats you and reacts to you because of your impairment. Society most often does exactly what you unwittingly did, and reduces a person with disabilities to being a member of "the disabled", someone whose identity is governed by their disability, rather than their disability merely being a single facet of their identity. That disadvantages you the most.
Acknowledging the fact of a person's disability doesn't turn them into a victim, it only does that if you treat them as if their disability/impairment is their defining characteristic, and so treat them differently than anyone else for reasons unrelated to their particular impairment (you'd be amazed how many people still do the old "slow talking very loudly" to just about any person with a visible physical disability, as though we're all deaf and/or stupid).