On further investigation, I realized the routers were actually not compromised. This was determined by looking at the configs that are backed up daily via the archive feature available in Cisco IOS. This feature allow us to back up configs to the flash on the device, ftp, scp, http/https, rcp, or tftp. In my case I have set for the configs to be scp'd to a server every 24 hours and any time a configuration write is done.
Going over the latest backed up configurations, I realized when the changes causing the issue were made. It turns out the contractors were adding aaa new-model related configs and pointing authentication and command authorization to the ISP's tacacs+ servers. Another addition was an access list restricting access to vty lines via SSH to only IPs belonging to the ISP. Hence, the only way to restore my access via SSH was to get in touch with the ISP's team or figure out an altenative. This different way turned out to be SNMP as there was a RW community string configured on the routers. After an online search and a bit of learning, I figured out how to do a configuration change via SNMP. Below is how I did it and the links I used.
First of all, you need to install net-snmp if not already done. I had it already installed on my NMS running CentOS. What I was missing was the CISCO-CONFIG-COPY-MIB. Please note you will/might need to install some other MIBs in a specific order before installing this MIB. Otherwise, you will get some errors such as being unable to translate OIDs and of course, then you will not be able to achieve the end goal.
I used the order below as per Cisco's recommendation here (Link)
wget ftp://ftp.cisco.com/pub/mibs/v1/CISCO-SMI-V1SMI.my
wget ftp://ftp.cisco.com/pub/mibs/v1/CISCO-ST-TC-V1SMI.my
wget ftp://ftp.cisco.com/pub/mibs/v1/CISCO-CONFIG-COPY-MIB-V1SMI.my
wget ftp://ftp.cisco.com/pub/mibs/v2/CISCO-SMI.my
wget ftp://ftp.cisco.com/pub/mibs/v2/CISCO-ST-TC.my
wget ftp://ftp.cisco.com/pub/mibs/v2/CISCO-CONFIG-COPY-MIB.my
Please note these were being downloaded directly to my /usr/share/snmp/mibs/ folder. My snmp.conf has mibs ALL meaning all MIBs available are loaded. You can check the folders where net-snmp checks for MIBs by running the command, net-snmp-config --default-mib-dirs
snmptranslate -Td -m +CISCO-CONFIG-COPY-MIB -IR ccCopyDestFileType
The output should be as below without any errors.
Proceed to the next steps. As agreed, we want to use TFTP to copy our new config file to the running configuration of the router so we can log in remotely via SSH. We do this via snmpset commands sent to the router. N/B, you can use FTP, RCP, SCP or SFTP - just be sure to use the right integer in your snmpset command.
In our scenario below, our TFTP server IP is 192.168.1.2 and the IP of the router whose running configuration we want to modify is 192.168.202.1
Step 1
Set the protocol type (TFTP). As can be seen from the snmptranslate output below, the argument should be an integer. For TFTP it's 1, 2 for FTP, 3 for RCP, 4 for SCP and 5 for SFTP.
Our first snmpset command is,
snmpset -c ourRWString -v 2c 192.168.202.1 1.3.6.1.4.1.9.9.96.1.1.1.1.2.340 i 1
Step 2
Select the config file we want to copy from. In our scenario, it's a file residing in our tftp server root folder, hence it's of type networkFile.
Our second snmpset command is,
snmpset -c ourRWString -v 2c 192.168.202.1 1.3.6.1.4.1.9.9.96.1.1.1.1.3.340 i 1
Step 3
Set the type of config file we want to copy to. In our scenario, we want to modify the running config.
Hence, our 3rd snmpset command is,
snmpset -c ourRWString -v 2c 192.168.202.1 1.3.6.1.4.1.9.9.96.1.1.1.1.4.340 i 4
Step 4
Set the server address type. It's an integer type. The options are:
1 for ipv4
2 for ipv6
3 for ipv4z
4 for ipv6z
16 for dns
In our case, we are using ipv4 addresses, so our snmpset command will be,
snmpset -c ourRWString -v 2c 192.168.202.1 1.3.6.1.4.1.9.9.96.1.1.1.1.15.340 i 1
Step 5
Set the IP address of the server we are going to copy our file from (the TFTP server IP)
Our 5th snmpset command is,
snmpset -c ourRWString -v 2c 192.168.202.1 1.3.6.1.4.1.9.9.96.1.1.1.1.16.340 a 192.168.1.2
Step 6
Set the name of the config file we are going to copy from (I have named mine config.txt and put it in my tftp root folder, i.e. /var/lib/tftpboot/).
Our 6th snmpset command is,
snmpset -c ourRWString -v 2c 192.168.202.1 1.3.6.1.4.1.9.9.96.1.1.1.1.6.340 s config.txt
Step 7 (Final step)
Set the variable to cause the router to copy the configuration file to it's running configuration.
snmpset -c ourRWString -v 2c 192.168.202.1 1.3.6.1.4.1.9.9.96.1.1.1.1.14.340 i 1
Notes on Steps 4 and 5.
Kindly note that you can skip Step 4 if all you want to use is an ipv4 address, but then you will have to use a different oid object in step 5. See below image for the translation of this oid and below the image is the command
snmpset -c ourRWString -v 2c 192.168.202.1 1.3.6.1.4.1.9.9.96.1.1.1.1.5.340 a 192.168.1.2
Please take note of the number in red and italics. This is a random number but should be the same in all your commands. It creates a row instance on the device we are managing All the commands should also be run within 5 minutes as this random value is timed out after 5 minutes. However, you can reuse the number before 5 minutes expire by sending a destroy command as below.
snmpset -c ourRWString -v 2c 192.168.202.1 1.3.6.1.4.1.9.9.96.1.1.1.1.14.340 i 6
Also, it is possible to put all the commands we have used in one line or even in a script. See link 3 below to see how that can be done - the particular example uses snmpv3 which is even better because it ensures secure communication between the managing device and the router(s).
Links.
1. http://net-snmp.sourceforge.net/wiki/index.php/TUT:Using_and_loading_MIBS
2. https://www.ciscozine.com/send-cisco-commands-via-snmp/
3. https://serverfault.com/a/620553
