Yes, indeed. And that's where ideas such as those of Bjorn Merker come in. We construct our experience, generate it ourselves - but that includes the construction of the observer as well as the observed. We actually do observe qualia in our experience - because that's how our experience is constructed: with an observer and that which the observer is observing. Without both those elements, we wouldn't be able to do anything with our consciousness - we need to place ourselves in the scene as the one observing that cannot be observed - and the crucial point is that the whole thing is a self-generated construction, including the observer.
So, qualia are simply a part of the phenomenology of consciousness, and I defy anyone to come up with a phenomenology of consciousness that does not include them. You can't. They're there. We know they are there.