So, some theories:
1. The extra bit of viaduct carried a siding.
I don't think so. Any siding would have been very short, and I can't see any reason for one in this location.
Crispy noted upthread that there appears to be a similar abutment a bit to the east (above where 'signal box' is written on the map) and suggested this could also have been for a siding, but again I don't think so. However, I do think that other abutment is significant.
2. It was to allow for a tripling of track between here and Brixton, as suggested by
Puddy_Tat .
This proposed tripling was what also caught my eye reading the LCDR book. It's not clear exactly when this tripling was first discussed - would it already have been an idea when that first (green) section of track was constructed? It might have been, in which case it would make sense, when building that 2-track viaduct carrying the green lines, to provide for a widening from Barrington Rd eastwards. But the fact that the widening only starts at this point (that is, the curved viaduct round to Loughborough Junction was only built as 2 track) suggests that it was in anticipation of there being a junction, for what would become the red lines. Because the purpose of building an extra track would be to provide extra capacity on the part of the line used by both of those routes (red and green lines). Well, looking at what was built, that sort of makes sense, but the junction actually occurs somewhat to the west (it's known as Canterbury Rd junction). Why extend the extra track beyond this point, necessitating extra bridges over Gresham and Barrington Rds?
3. It was related to the fact that two different companies built the track east and west of Barrington Rd.
Reading the LCDR book the other thing that caught my eye was the agreement between the LCDR and the LBSCR that they build portions of track for each other (because they were both wanting to build a route between Brixton and Peckham). The agreement was that the LBSCR build the route to the east of Barrington Rd, and hand over one pair of tracks to the LCDR, and the LCDR would build tracks to the west of Barrington Rd. Initially I thought this was the answer - surely more than a coincidence that a seemingly mis-aligned bridge abutment was at the exact point where two railways' lines were to interface. Perhaps there was a disagreement, a change of plan on the part of one, something relating to land purchases each side of the boundary being carried out separately, which could result in this.
However - in fact, the interface between the two construction projects would only have concerned the four southernmost tracks where they cross Barrington Rd (the red and blue ones) because the LCDR had already built the 'green' route. In other words, on the section where the mystery abutment exists, the same company built the track on each side.
4. It was for some entirely new route diverging at this point
I've not come across anything that suggests a plan at any point in time for a line diverging to the north here.
5. It was built to allow for a junction that ended up in a different place.
This is what seems most likely to me. It's a kind of combination of theories 2 and 3. This is where the 'other' mystery abutment is significant, because it suggests there was a future section of track planned to head off to the west as well. Below I've drawn what I think they had in mind when they built that initial section of track in 1863 - they already had it in mind that there would be a junction here with a line heading off towards denmark hill, and they probably also had it in mind that the section towards Brixton would become 3 or even 4 track. Maybe this was before any kind of agreement with the LBSCR had been reached, and it's to do with that agreement, that the alignment of things ended up differently? Perhaps it was realised that if the tracks diverged here, the gradient rising to get the 'upper' viaduct over the Elephant-Herne Hill line would be too steep, and that's why the junction was eventually placed at Canterbury junction instead?
Perhaps it was to do with purchasing the necessary land to the west of Barrington Rd or perhaps it was a mixture of all of these things. When you read the history of how stuff was being developed at the time it's pretty clear that everything was constantly changing with deals being made and broken at quite a pace.
Below I've drawn out what I think they originally intended the alignment to be. For this alignement both of the abutments make sense. I've not shown the 'blue' LBSCR lines because perhaps these simply weren't the LCDR's concern at that stage.