Make sure your Mi 5 is running stock MIUI and is updated to the latest.
Obtain permission from Xiaomi to unlock the bootloader of your device here. Once approved you’ll land at the unlock tool download page. Download the MI unlock tool from there or here is a direct link. Unzip it at a convenient location on your PC. You can download it without permission from Xiaomi, but without permission it won’t do what it’s supposed to do; unlocking will fail after reaching certain percentage. Also after obtaining unlock permission there will be a wait time of 72 hours, when you try to unlock with the tool it’ll fail and show your how much time is remaining. After expiry of the wait time use the tool to unlock it and it’ll work.
Make sure ADB is installed on your PC, if not, download the Google ADB from here and unzip it at a convenient location on your PC.
To unlock, power off the phone, keep volume down button pressed and press power button until the word “Fastboot” appears on the screen with a picture. Now connect the phone to the PC using its cable. Unlock tool will detect it then click the button to unlock.
The custom recovery I installed is TWRP. Download it for your device from here and save it in the same directory as ADB. Rename the file to ‘twrp.img’ for convenience.
On your phone go to Settings > About Phone > Tap MIUI Version several times until you see the message “now you are a developer”. Then go to Settings > Additional Settings > Developer Option and add the MI account that you’ve got unlock approval for. Then look for an option named ‘USB debugging’ under ‘Developer Option’ and enable it.
On the PC open the folder where you had extracted ADB earlier, keep ‘Shift’ pressed and right click inside the window and click ‘Open Command Prompt Here’ (In case of Windows 10 click ‘Open PowerShell Window Here’).
Reboot the phone into ‘Fastboot’ mode again and connect to the PC. Once connected go to the command window and run ‘adb devices‘, it should show the serial number of the phone. If nothing is shown then the device is not recognised, validate that all previous steps are done successfully. Or just kill the adb server and redo this step.
Once ‘adb devices’ is successful, run below command to flash TWRP recovery to the phone (assuming twrp.img is saved in the same directory):
fastboot flash recovery twrp.img
*Important: The next command is going to reboot your phone, make sure to immediately hold the volume up button depressed as soon as the phone’s display goes blank for reboot. Else the TWRP recovery you just flashed will be overwritten by the stock recovery and then you’ll have to go through the process all over again. The next command to run is:
If you kept the volume up button depressed in time then after some time you’ll see the TWRP screen after which release the button. Once in TWRP it’ll ask you to modify the partition or keep it read only, choose to keep it ‘read only’ so that you can revert to stock ROM later if required.
You’ll now see a screen like this:
Congratulations, you just successfully unlocked the bootloader of the phone and installed custom recovery TWRP on it. You may now go ahead and install any compatible custom ROM on your phone if you want.