There are currently two potential answers. Yes and not really. Not that helpful I know but probably true at the moment. Naturally the faster your website responds to a server request and downloads to the visitor’s browser the less likely the visitor will be to become irritated and go instead to another site. So adopting obvious measures such as optimising images and videos, not using ‘off the shelf’ development platforms with ‘bloated’ code and hosting your site on a fast server etc. should be done. The benefits of the less straightforward measures you can take to ensure that your site keeps search engine speed measurement tools happy are less clear.
The question is does going to this effort dramatically improve site speed from the visitor’s perspective and by keeping Google’s tool happy improve the site’s Google search ranking? As far as the former consideration is concerned the answer is not particularly, yet! On the latter (based upon some not particularly scientific research), it would appear not to affect rankings to any discernible extent. This, however is very likely to change in the near future as Google adds greater weight to site speed within its ranking algorithm. The factor missing in all of this is the consideration of network latency – the speed of the network across which the data are being transferred. This is something that is not within the control of a website owner or developer.