You are correct that even without the GCPs you should be getting a fairly accurate elevation (still remember the elevation will have the most error in it height wise) and using GCP will in crease your absolute accuracy but really shouldn't have a huge affect on the resolution. I Tend to bump my side and front lap up some more, but again should not really cause a huge issue. One thing you could do to increase the elevation accuracy is to fly at two heights for the one map, giving it some more data to work with. But lets go on the assumption that every thing on this end is fine and look at the maps.
A couple of things, since this is relative it would be good to know what the range of elevations in the map are (meaning what height is the darkest blue and what height is the red) While the drone does in theory have some more overlap in turns, that doesn't actually mean you are capturing any new data. [i am unsure if they take based on time, or actually based on positional overlap, i would think the second but unclear]. Really what i mean to say is to map a larger area so that you ensure anywhere you are trying to take data from is well within the bounds and wont suffer from decreasing accuracy at the edges where tie points invariably are less. In fact give that a try, in crease the mapped area and re-export to see if that decrease of elevation is really there. Also check the difference, it might be a difference of only a few cm, or a meter.
This is a side view of the first 3D map, it has some curvature, but again without knowing the area it is tough to tell if this is data related, or the actual topology of the area. Also might as well check the drone you are using, and its camera system. it looks fine detail wise, but it might help to know.