This was certainly a good 'un in my personal list of stories…
Today I was QA'ing a minor change. Literally just renaming a couple of models (+their tables), nothing significant.
However it just felt like performance had somewhat taken a hit. The system seemed to be twice as slow all of a sudden.
We all pretty much agreed that a minor object rename wouldn't have done this. It must have been that last minor upgrade… or when we moved SQL to in-memory Ruby, or perhaps the dynos are different at heroku, etc, etc.
Turns out… it WAS the object rename!!!
Basically doing this disabled some key indexes and that's what was causing performance to 'tank'.
Lesson learned (well reinforced at least):
Always test and never assume a 'minor change with no front end consequences to the user' will actually be that.