We need to use shutil.rmtree here since removedirs() only covers
directories. Make the exception for specific too to make errors
easier to catch.
(From OE-Core rev: 9d2a661e46123a2292f7887658e6fa54923dbcc0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>