Well that’s your choice, but it’s a lot more rational to limit filesystem access by the browser, but allow a user unlimited access to the files downloaded by the browser (very easy to set that up asymmetrically with a symlink). Or at least restrict which directories a browser can or cannot access.