The max_instance resource-limit limits the maximum number of instances a DDS_DataReader can manage. Instance state is one of the most expensive resources to maintain as well as often being the highest one.
The resources allocated by an instance is normally only released when an instance is disposed and all known owners have unregistered the instance, please refer to DDS_DataWriter_unregister_instance and DDS_DataWriter_dispose.