I believe that Scott Hendrix, in his book Luther and the Papacy, does a fine job in showing why Luther came to doubt the papacy’s fulfilling of its duty to take care of Christ’s sheep (all bold and italics mine):
“Although his study of the mass [in 1507 in preparation for his ordination] scarcely weakened his ties to the pope, in Luther’s memory his internal struggle with confession was very much related to papal authority. The writings of John Gerson (d. 1429), mystical theologian and reform-minded chancellor of the University of Paris, played an important role in helping Luther recognize this relationship. Several times in his Table Talk Luther identified Gerson as the one who began to relax the reins of papal tyranny. Gerson argued that it was not a mortal sin to disobey the laws of the church unless the disobedience was deliberate. When Luther applied this argument to the practice of confession, it meant that he and other Christians were not under pressure to confess every sin. This spelled relief for the scrupulous conscience of Luther and made Gerson, in Luther’s memory, the great consoler among the medieval doctors and a forerunner of Christian freedom under the gospel. From the perspective of the 1530s and 1540s, Gerson, as an advocate of conciliar authority in the church, also appeared to be a forerunner of Luther’s own opposition to the papacy. Luther added to his comments on Gerson that no one could really understand the necessity of opposing the pope unless one had lived under the darkness of the papacy.
In the period of 1505 to 1510, obligatory confession of every sin may not have raised doubts in Luther’s mind about papal authority per se. Potentially, however, the questions were there. The guidelines for confessing were conceivably one point at which Luther questioned the teaching of the church, especially since they touched the nerve of his own religious experience. Nevertheless, he was hardly the first to voice his doubts about the degree of thoroughness required in confession. In his Exposition, Biel, relying on Gerson, discussed quite openly the degree of certainty required of a priest who had confessed before celebrating mass. According to Gerson, one should be as diligent in examining the conscience for past sins as one could be in any business where a great gain or loss was at stake. Quoting Gerson in his dictionary of medieval theology, Altenstaig noted that repeated confession of the same sins might result in doubts about the efficacy of sacramental confession and thus pose a danger for scrupulous and timid consciences. Luther’s struggles were not unique. When, however, Luther came to recognize the connection between his internal religious struggle and the external authority of the church, his scruples sounded the keynote of his lifelong opposition to the papacy: the pope must be opposed as long as he tyrannized the consciences of faithful Christians (p. 9, 10)….
“…Luther’s critical recollections of this period indicate that he was not able to divorce his internal struggles from the issue of church authority. It is impossible to determine just what Luther meant by the “many errors in the papacy” which, he says, the naked text of Scripture revealed to him during his time in Erfurt [1501-1508]. The penitential practice of the church, however, was a likely candidate for one of those “errors”. In fact, this was the discipline imposed on him by the authorities of the church and of his order whom he was bound to obey. In his conscientious struggle to follow the discipline, questions about its wisdom and its effect on his life would spark further questions about the wisdom of the hierarchy and teachers who sanctioned and justified that discipline. The ambivalence which Luther felt toward that discipline was more than enough to spawn ambivalent feelings about its source, regardless of the loyalty that he felt toward the church to which he now committed his life as preacher and teacher (p. 11)….
“A sermon preached by Luther in 1517 gives support to Preuss’s suspicion that Luther’s lack of reference to the papacy [in regards to his 1513-1515 lectures on the Psalms in which he discussed Matthew 16:18 (scholars have noted Luther’s interpretation was not unique)] was intentional. In this sermon Luther stressed orally what he would write down months later in the Explanation of the Ninety-five Theses: the power of the priests to bind and loose sins based on Matt. 16:19 was given to them for our comfort and certainty. Their tongues are the keys of the kingdom of heaven. Whenever we hear the word of absolution from their mouths, we should firmly believe in that word and no longer trust in our own contrition and repentance. But, complained Luther, this word of comfort has been twisted into a tyrannical word. For a long time, Luther recalls, Matt. 16:19 tormented his soul because he had thought it meant that the pope could do with him whatever he wanted; and, indeed, the keys are still being used to terrify and vex the people. The Priests are badly mistaken if they think they absolve only those Christians whose genuine contrition can be proved. On the contrary, faith in Christ through the word of the priest brings forgiveness to whoever trusts in that word.
By 1515, Luther could have decided not to apply the rock in Matt. 16:18 and the power of binding and loosing in Matt. 16:19 to the pope because he felt himself to be a victim of the misuse of that power in the penitential practice of the church. Consequently, he decided to adhere to the safer traditional exegesis of which he was certain. Even if Luther made such a decision, there is no reason to assume that he was questioning anything more than the misuse of papal and priestly power. On the other hand, Luther’s neglect of the pope could mean that he was not yet concerned about any misuse of the keys prior to 1516 or 1517. Arguments from silence are notoriously unreliable. Silence may speak louder than words, but it is frequently impossible to determine what that silence is saying. Still, Luther’s handling of Matt. 16:18-19 [in his 1513-1515 Psalms lectures] does fit into the ambivalence about the papacy which his recollections about this period reveal.
No ambivalence can be detected in the importance Luther attached to the priesthood during these years. Throughout the early writings Luther extols both the power and the authority of the hierarchy. The mouth of the priest is the mouth of God, and even preachers of the gospel who do not possess the Spirit serve God. Priests and prelates are the seats from which Christ exercises his rule in the church. Obedience is owed to these seats when they sit in judgment no matter how inappropriate their judgment might be, as long as it is not against God. It is precisely because Luther valued the priesthood and obedience to the hierarchy so highly that he could criticize them as sharply as he did in his early works.” (p. 13 and 14)
Part I coming in two days