Your DTM algo is to put it mildly, less than perfect. But no AI based filter is going to be perfect. One of the big reasons is to the best of my knowledge these filters try to identify objects to filter out based on their geometry and don’t take their color into consideration.
Now go lookup/research/try the “color replace” feature in Photoshop. You can either select an area of the image or operate on the whole image, pick a color in the image with a picker, and then adjust how narrow or wide of color range to include in the selection.
Now imagine a fairly common site image where the ground is shades of brown, and there are patches of green bushes, trees, yellow colored heavy equipment, blue/red/black cars parked, white/grey shingled or commecial roofs, etc. Take a tool like that and add it to Map view (or model view) so you can pick out color ranges and delete all of the points in the point cloud that match that color without a specified range, or be able to draw around an object (like you do for area/volume calculation) and remove EVERYTHING within that boundary from the point cloud. And add a toggle to turn on or off the AI DTM filter as a first pass filter. Results = a much more accurate DTM given the human being more control to identify the objects to be removed from the DSM.