In general, a virtual fn means to start at the class of the object itself, not the type of the pointer/ref (‘do the right thing based on the actual class of’ is a good way to remember it). Virtual destructors (dtors) are no different: start the destruction process ‘down’ at the object’s actual class, rather than ‘up’ at the ptr’s class (ie: ‘destroy yourself using the correct destruction routine’).
Virtual destructors are so valuable that some people want compilers to holler at you if you forget them. In general there’s only one reason not to make a class’ dtor virtual: if that class has no virtual fns, the introduction of the first virtual fn imposes typically 4 bytes overhead in the size of each object (there’s a bit of magic for how C++ ‘does the right thing’, and it boils down to an extra ptr per object called the ‘virtual table pointer’ or ‘vptr’).