Do I need to let water sit out overnight?

No!

It used to be that if you let water sit out in a jug overnight, the chlorine would dissipate, saving your sensitive plants from absorbing it. Two things are wrong with this now, though!

First: As far as I'm able to tell in my research on the topic, chlorine does not adversely affect even "sensitive" plants". The acceptable level of chlorine in drinking water in any developed country I've found is an order of magnitude too low to adversely affect any plants, even so-called sensitive ones like spider plants or calatheas. (In those plants, generally brown tips are caused by watering or humidity issues!)

Second: While chlorine does dissipate (or "off-gas") in this way, most major cities (and a lot of minor cities) have switched instead to chloramine as an additive. This is a more complex molecule than straight-up chlorine, and doesn't off-gas the same way! So even if it were an issue, you wouldn't be doing anything by letting it sit out!