So I noticed there’s RMSE numbers now. I flew a site at 1.1in/pix GSD both Nadir 70% sidelap and 80% frontlap, and obliques in a criss cross pattern at 60% sidelap 70% frontlap which isn’t really required but helps improve accuracy.
THe RMSE numbers I got are:
RMSE 3.1ft
X 3.9ft
Y 3.6ft
Z 1ft
So these are obviously absolute position accuracy numbers - which don’t really help me much as everyone with half a brain knows GPS accuracy for most Drones (or any other GPS device) is typically in the 3-8ft range.
What most end users are interested in is knowing how accurate their distance and volume measurements are. I would think it would be much more helpful if DroneDeploy could give estimated relative measurement accuracy numbers based on the GSD, sidelap/overlap, how many angles, and if GCPs were used. I’m sure you could come up with a formula to take all those things into account and give estimated numbers.
I’ve seen lots of posts on here from people that verified volume calcs via other methods (laser scanner or having already known volumes) and found the numbers to more off than 2-5% like you see posted in many blogs posts as to the accuracy to expect. Obviously the GSD and whether you also took obliques has a significant impact.
It is widely published that relative horizontal distance accuracy will be 2X GSD and veritcal 3X. I think that’s just a map created only with nadirs, min 65% sidelap and 75% frontlap. However with the obervations about volumetric calcs being off more than expected I’m wondering how accurate those 2X and 3X GSD assumptions really are. I did a football field one time at 0.8in/pix, laid out GCPs and took a 300ft roll tape measure to verify the distance and the length of the football field measured in the resulting map was off by 5 inches, when using the 2X GSD, it should have not been more than 1.6 inches. There’s not that much curvature of the earth in 300 feet. Maybe one could argue the field wasn’t perfectly flat, so let’s give it 3 inches. Still the actual was almost twice.
So for my own estimates I use 4X GSD for horizontal and 6X GSD for veritical relative accuracy estimates. Anyone else doing something similar based on their own observations?
You opened a can of worms here.
Non-GCP RMSE’s are the average absolute difference of the camera location (nothing to do with the ground) as reported by the GPS versus where it ended up after processing. As you stated, these numbers are generally between 3 feet and 9 feet (1 and 3 meters for our metric friends) as reported by the specifications of the drone manufacturer or industry standard for that type of GPS. The relative accuracy (or error) is derived from the RMSE XYZ and the size of the measurement/mission XYZ. If the mission covered 10 acres (208 ft x 208 ft) then your relative accuracies are 1.5% X/Y (same because I used a square) and 1.2% Z if flown at 250 ft. This is for the overall map. Stockpiles are a condition of A versus B in a much smaller area, so the relative accuracies are much smaller, but are absolutely larger the bigger the stockpile gets.
With GCP’s the RMSE is the location of the GCP as measured by survey vs where it shows up after processing using those measurements. My RMSE’s are usually less than 1 inch.
Back to stockpiles. I have done comparisons using a laser scanner, GPS base/rover and robotic total station versus the drone. In all instances they were all within 5% of any other measurement. I am not quite sure why people keep harping on the absolute accuracy of stockpile measurements, because the way we have been taking off stockpiles in the field for the past 10 years is not as accurate unless you shoot every top and toe and 5 ft in between. I shot in a stockpile this way and at 10,000 cy it took me almost 2 hours. I flew it with the drone at 85/85 in 5 minutes and was within 200 cy (12,000 cy) of that. That would be 2%… Predicting trucking is a completely different topic with many variables which I will not go into here and usually leave up to experienced estimators.
As for obliques making a more accurate map, I personally do not believe that is true except for the fact that it models vertical faces and underneath objects better. It does not help ground accuracy and as a matter of fact it hurts it when using GCP’s. Never mix nadir with obliques for absolute ground accuracy unless you are isolating the areas and just want a nicer looking building. For even better results, make two maps and marry the point clouds in another piece of software like Carlson Precision 3D or Autodesk Recap then into Autodesk Navisworks for your BIM model.
Ahh, I can see the more usefulness of the RMSE numbers when uing GCPs but realistically, probably less than 1% of the jobs being run on DD are using GCPs so in my opinion, min 99%+ of the cases the RMSE numbers have no practical use. And even them their actual practical usefulness is questionable as I will explain below.
I believe the practical use of the photogrammetry output in the majority of applications is not absolute positioning but relative distance and volume measurements which we know can be achieved to a useful degree of accuracy without GCPs or RTK/PPK for that matter.
Very few drone service providers are currently providing Land Surveying services requiring cm level accuracy in which case the RMSE figures from DD would be of interest but should not be relied on in my opinion for verification. In addition most drone operators are NOT licensed Land Surveyors so their output has limited usage in the legal context.
So basically what I’m saying is if the application is such that the RMSE is critical to the success of the job, you’re probably going to be verifying your accuracy using other more accurate methods such as having the Land Surveyor go out and take their own measurements to compare to. In a nut shell, the addition of the RMSE numbers in DD seems more of a gimmick than of a real-world practical usefulness.
You are the first person I’ve ever seen argue that oblique imagery doesn’t increase the accuracy of vertical measurements (such as topographic mapping - of say a mine quarry.) You sure about that?
RMSE numbers are just as useful without GCPs as they are with and can be relied upon in my experience over hundreds of maps and point cloud edits. As I notated there are direct percentages that you can apply to any measurement that you make on each given map to define the accuracy you can expect.
I would love to hear DroneDeploy’s statistics on the number of maps run with GCP’s because I can almost assure you that it’s a much larger percentage than what you think.
Relative and absolute accuracy are not static terms and are useful in different ways for each individual’s use case. Those that have the capability and infrastructure to use absolute real-world accuracy will, but the majority don’t and won’t. Becoming an RPLS takes effort, time and money. LIDAR is exspensive so most aren’t going to do it. They think drones are is and it happens like magic. I dont see what difference any of these separations make on any of our scenarios since your can have it either or both ways.
RMSE numbers are a direct representation of the accuracy of your map and an instant visual cue as to whether or not your data is acceptable for your use case.
Surveying the ground is different that measuring vertical objects so my point was not to use obliques for measuring the ground especially if you are using GCPs. Including GCPs on stockpiles. Yes, I put GCPs on tops and toes, just like any surveyor would shoot them.
All I’m saying, is I’m confident in the real world, currently with the majority of applications, the accuracy of relative measurements is much more important than the absolute geolocation of points on the map. It sounds to me like you might have a biased perspective I’m guessing because you might be one of the few who are generating work products where absolute position better than what the consumer GPS provides is a requirement for a lot of your jobs. In most cases where it’s a requirement though, you probably need a Land Surveyor’s stamp. But as I said, you would not be in the majority of use case scenarios. I don’t have hard numbers but I would place a sizeable monetary bet on my educated guess based on my observations over the last two years of the subject of threads on a variety of forums and the responses I’ve got to subcontracting requests requiring GCPs or RTK/PPK. Most drone pilots are doing applications where all they need to do is take some photos, maybe some video (real estate, construction progress, roof inspection), and possibly run a DD mission to generate an ortho usually just for visual inspection. The majority of commercial drone pilots out there don’t even know how to properly use GCPs in my opinion, let alone RTK/PPK. So absolute position RMSE doesn’t even enter the equation so to speak. So the RMSE figures of DD I am confident are of no value to the majority of DD users.
The absolute geolocation is paramount for the accuracy of the overall map. If all you are interested in is stockpiles then relativity is fine.
To put the worms on hooks the answer to the original question, RMSE’s are useful and directly related to the accuracy of the data period. Regardless of use case. My biased perspective is achieving verifiable accuracy.