what helped me a lot in the beginning was starting with "lower density" of strums, and increasing it as needed...
e.g., when playing song (with 4/4 time signature) for first time
- i counted all the numbers only - 1,2,3,4, no &'s -, and strummed chord only on count of 1 (down strum);
- this way, i play the song as a sequence of simple chords, with no fancy rhythm/pattern
- when feeling confident, i added strums also to other numbers, e.g. yo 2, or to 2,3,4
Same applies to up-strums of course, if the rhythm contains them - adding them for the hand movements heading upwards right after specific numbers.
I think i never explicitly counted &'s in the rhythm, i just hard-mapped all the down movements of my hand to numbers, all the up-movements to &'s, and counted numbers only.
Hope i'm making some sense, some terms/descriptions are a bit clumsy
For a fancy patterns, i just learn them first with no sound - muting all the strings with my fretting hand. This helps to concentrate on the strumming, and no stress with chord changes and fingers swapping etc.