Well, HDR will also be bigger deal there, so my guess is you really do need 4K HDR telly to see the better quality of light and shadows and such. So yeah, I'm afraid regular HD pic will be somewhat poor.
I'm sure you will notice an upgrade, but you're not getting the benefits of what a 4k TV can bring aside from just resolution, such as HDR or even with some of the more advanced TVs, like variable refresh rates. You're also most likely not getting extremely dark blacks either due to HD TVs local dimming abilities (or lack of it). The lighting and color in games will be less accurate. So yes, it definitely won't look as sharp, but you'll see some benefits. For example, if ray tracing works well next-gen, you'll definitely see that. 4k is great, but it's not just the 4k that makes a 4k screen worth getting.