Considering the performance of your responsive web designs is as important as the aesthetics. However, performance presents something of a moving target. For example, browsers update and improve the way they handle assets, new techniques are discovered that supersede existing 'best practices', technologies eventually get enough browser support that they become viable for widespread adoption. The list goes on.
There are however, some basic implementation details that are pretty solid (well, until HTTP2 is common place, more of which shortly). These are:
There are a number of great tools available to measure and optimize performance too. My personal favorite being http://webpagetest.org/. At its simplest, you pick a URL and click on START TEST. It will show you a complete analysis of the page but even more usefully, it shows a 'filmstrip' view of the page as it has loaded, allowing you to concentrate on getting the rendered page complete sooner. Here's an example of the filmstrip view of the BBC home page:

Whenever trying to optimize performance, ensure you take measurements before you begin (otherwise, you have no idea how effective your performance work has been). Then make amendments, test, and repeat.