It matters quite a lot.
It's not a matter of a few mm of string height being hard to press down.
It affects the overall ease of fluidly playing, especially if you're trying more challenging instrumental solos.
It probably matters a bit less if you're just strumming a few chords, and not playing many barred chords.
There is no single unified height that is considered "ideal" as it differs from person to person and different heights come with different characteristics.
Broadly speaking, lower action (string height = "action") makes it easier to press down the strings, meaning you can be more fluid and accurate with the fretting hand (left hand). Higher action generally enables with louder tone, as the strings have more clearance to vibrate freely and also makes a bit easier to partially mute the strings while holding down chords. It also allows you to strike the strings with more force.
For ukulele, I'd say 1.5mm to 2.00mm is what most would consider "low" action.
2.0mm to 2.5mm would be "medium" and anything above 2.5mm would be a bit on the "high" side.
There's no official agreement on the matter, but this is just from my experience.
Going to the extremes of either low or high comes with problems - action being too low would make the strings susceptible to buzzing, poor tone and being difficult to fingerpick or strum at all due to how close the strings are hitting the soundboard and neck. Too high action makes it rather unplayable and gives you poor intonation, etc.
Also, the action itself isn't everything, as there are other factors at play too. Such as action at the nut. Your action measured at the 12th fret might be one reading, but if your nut slots are high, it would affect playability. Furthermore, there is also the curvature of the neck. If your neck has too much forward or back-bow, it will affect playability. Overall setup of an ukulele would depend on playing around with these factors:
-Nut action
-Action at the saddle (measured at 12th fret string height)
-Neck bend (relief)
and the type/tension of strings and also the state of your frets.
That being said, I personally prefer LOWER action of around 1.5mm to 2mm, with the 4th string a tad higher than the 1st string.
For example, 1.5mm at the A string and 1.75mm at the G string.
Some would say this is quite low, but it's what suits my play style.
More "typical" actions of 2.5mm and up feel uncomfortably high for me, and I end up sanding the saddle down and/or filing the nut a bit.