First, we read from John:
Jn 21:15-19 When they had finished breakfast, Jesus said to Simon Peter, "Simon, son of John, do you love me more than these?" He said to him, "Yes, Lord; you know that I love you." He said to him, "Feed my lambs." 16 A second time he said to him, "Simon, son of John, do you love me?" He said to him, "Yes, Lord; you know that I love you." He said to him, "Tend my sheep." 17 He said to him the third time, "Simon, son of John, do you love me?" Peter was grieved because he said to him the third time, "Do you love me?" And he said to him, "Lord, you know everything; you know that I love you." Jesus said to him, "Feed my sheep. 18 Truly, truly, I say to you, when you were young, you girded yourself and walked where you would; but when you are old, you will stretch out your hands, and another will gird you and carry you where you do not wish to go." 19 (This he said to show by what death he was to glorify God.) And after this he said to him, "Follow me."
Note that in vs. 15, Jesus asks Peter, "Do you love me more than these?" ("these" being the other apostles). Jesus is interested in singling out Peter from among the group. This is b/c Jesus has a great task for Peter: to "feed" and "tend" His sheep. These are the actions of a shepherd, and He's asking Peter to play this role.
When we look closer at this "feeding" and "tending" we find that there is actually much more at stake. First, the greek word for "feeding" is bovskw (or "bosko") and it denotes promoting the spiritual welfare of others (see here). Also, the greek word for "tending", poimaivnw (or "poimaino") actually has the alternative meaning of ruling a people (see here). This is seen in verses from Matthew's Gospel (cf. Mt 2:6) and Revelation (cf. Rev 2:27; 7:17; 12:5; 19:15) where the word is translated as "rule" or "govern." So, essentially, we have here Jesus setting Peter apart from the other apostles to be His shepherd, providing spiritual nourishment and rule for his fellow servants and for the whole Church.
From here we move to Luke's Gospel, where we again see Peter being set apart for a position of unique leadership:
Lk 22:31-32 "Simon, Simon, behold, Satan demanded to have you, that he might sift you like wheat, 32 but I have prayed for you that your faith may not fail; and when you have turned again, strengthen your brethren."
Notice that the "you" in vs. 31 is plural but the "you" in vs. 32 is singular. So, while satan wishes to deceive the entire group of 12, Jesus responds to this by praying specifically for one of them: Peter. Why would Jesus do this, unless He intends for Peter to be the one from among them who will govern them and keep them from error? As you can see, Peter's unique authority is closely tied to his primacy among them.
Besides the examples I have already provided, in which Peter alone is called the Rock (Mt 16:18), given the keys (Mt 16:19), instructed to feed and tend the rest (Jn 21:15-19) and specifically prayed for by Jesus (Lk 22:31-32), we have many other examples that, when taken as a whole, portray Peter as holding a position of primacy among the apostles. There are many examples in fact (you can read them all here and here), but I would like to mention only the ones that stand out the most in my mind.
For one, Peter's name appears more often than the names of all the other apostles combined (195 to 130). In the lists of the apostles, Peter is always named first (Mk 3:14-19; Mt 10:2-4; Lk 6:13-16; Acts 1:13). Often times his is the only name given (Mk 1:36; 16:7; Lk 9:32). What is particularly interesting is this verse from Matthew's gospel:
Mt 10:2 The names of the twelve apostles are these: first, Simon, who is called Peter, and Andrew his brother; James the son of Zeb'edee, and John his brother
There is more to this word "first" than meets the eye. Charles F. B. Allnat has this to say:
"Wherever the Apostles are enumerated in the Gospels, St. Peter is invariably named first. St. Matthew expressly calls him 'the first' (10:2), the same Greek word (protos) being rendered 'chief' in 20:27 and other passages. Mr. Allies remarks: 'Now, that second and third do not follow, shows that "first" is not a numeral here, but designates rank and pre-eminence. Thus in heathen authors this word "first" by itself indicates the mor excellent in its kind: thus in the Septuagint occur, "first friend of the king," "first of the singers," "the first priest," i.e., the chief priest (Nehem 12:46; 2 Chron 26:20). So our Lord: "Whichever among you will be first" (Mat 20:27); "Bring forth the first robe" (Luke 15:22); and St. Paul: "Sinners, of whom I am the first," i.e., chief (1 Tim 1:15). Thus "the first of the island" (Acts 28:7) means the chief magistrate; and "first" generally, in Latin phraseology, the superior or prince.' "Also, Peter often acts as a representative for the 12, or speaks on behalf of them (cf. Mt 17:24; Mk 10:27-28; Lk 8:45; Jn 6:67-69; Acts 2:14). Peter initiated the succession of Judas (Acts 1:15-26), enacted eternal judgment upon Anani'as and Sapphi'ra (Acts 5:1-11), and made the decision to admit Gentiles into the Church (Acts 10), a decision the apostles and the "circumcision party" later defer to (Acts 11:1-18). Peter also either led or held a prominent role in the council of Jerusalem (Acts 15), and it was Peter with whom Paul consulted after his conversion (Gal 1:18). All things considered, Peter definitely held a position of primacy among the apostles.
Now, we have so far established that Peter, who was the pre-eminent apostle, was given unique authority over the Church by Jesus Christ. But, this means nothing, and has little to do with the papacy today unless this same authority is passed on to Peter's successors, who are the popes.
In Acts 1:15-26, we see Peter instituting this very succession. If even Judas, the least of the apostles, is provided with a successor, then Peter, the greatest of them, surely was! Throughout Acts, we continue to see the apostles raising others to their place (almost always thru the laying on of hands) as the work required (Acts 6:1-7; Acts 13:1-3; Acts 15:22-27) and especially as new churches were formed (Acts 14:21-23; 1 Tim 4:14; 2 Tim 1:6; Titus 1:5). Note that God not only commissioned Paul, but Silva'nus and Timothy as well (2 Cor 1:19-22), and placed upon them the seal of his spirit, which is what takes place upon ordination.
What is also worth nothing is the term "office" that Paul uses to describe his position, which is essentially that of a bishop (Col 1:25; 1 Tim 3:1). This word is also used to describe the position of the steward, the head of the King's house, back in Isaiah:
Isa 22:19 I will thrust you from your office, and you will be cast down from your station.
This office too had successors (from Shebna to Eli'akim), just like its parallel in the New Testament. Note that an office by its very nature has successors. As long as the position is necessary, there will be a need for someone to fill it. Can we really be so bold as to say that the authority of the apostles died with them? It would be rather imprudent of the Lord to give so much authority to Peter and the rest if it was only meant to benefit their contemporaries.
[See Part 4 for the conclusion of this debate]