Extend closet door by removing it and wall so I can get to both sides of the closet and add shelving.
So you are opening the wall, more? As long as it isn't load bearing, I don't see why not. Maybe a barn type door to close it when not in use?
I don't think it would be a problem as long as the studs in the closet aren't load bearing, which they usually aren't. My concern is once you take down the doors and the wall and build the shelves what will you put in place of the doors? The opening either needs to be a standard opening that pre made closet doors would fit or your will have to make your own somehow.
As long as it's not load bearing it should be good!