Ran into this and felt like until the documentation is updated, I should call this out.  On this Technet article, it shows you how to configure port ranges for Edge Servers in Lync Server 2013. In the hopeful case that this page is updated, here is a static image:

The issue with this article is that it appears to tie the port ranges for the Edge server to QoS which is not the case. You need to read the article very carefully. The first sentence in it tells you that you do not need to configure separate port ranges for Audio/Video/Application Sharing on the Edge. It then goes on to tell you how to change the port ranges to match up with what you may have set for your front-end servers.

The problem with this is that you are changing the ports that the Edge will/can communicate on. If you are following Microsoft’s firewall guidance on ports, you should be allowing the 50,000-59,999 port range (TCP and UDP) outbound. If you follow this example, you would need to allow the range 40,803-65,533 (TCP and UDP) outbound.

The article claims you might do this to make administration easier but I will claim just the opposite. Based on what most Lync admins know and what Microsoft states are the default ports, without some really good documentation and knowledge transfer, you are probably setting up a future admin to fail.

If you are wondering what happens when you set this like this but only allow the 50k port range outbound from the Edge servers, here is your answer. When an outside user attempts to call a user who is inside or join a conference, the client will send an Invite that contains SDP candidates. Those candidates will have ports associated with them based on the configuration. The external client will attempt to connect on ports outside of the 50k range that is being allowed on the firewall (i.e. 40,080-49,999 or 60,000-65,533). These connections will fail and the call will fail to establish. On a conference call, this can be seen as the user connecting and disconnecting from the conference several times in just a few seconds.

Many kudos to @tompacyk for helping me see what was happening here.