This is cool. I’m learning a lot reading through the docs. Why is subversion so instructive? I’ve read about SSL certificates a gazillion times and it’s simple enough in theory but for some reason this is making everything click.
What’s the use most common use case for something like this, by the way?
Believe it or not, the most common use-case is simply discovering what your code is actually doing. Modern frameworks are often so far removed from the nuts and bolts of HTTP and TCP that it can be difficult to know what's actually happening on the wire without using a tool like this.
A similar frustration is the tendency for SDK / library vendors to document their product using code snippets in various languages, but without simply showing you what the raw HTTP requests look like. Typically the closest you can get is a set of curl commands.
When you show me a python/ruby/swift/js snippet, I have a guess as to what's happening. When you show me HTTP, I know exactly what's going on. And if you won't show me HTTP, time to fire up mitmproxy.
I recently used it to modify the response from the replay server for an autobattler game, to let me watch arbitrary replays for theory-crafting the ideal team. The game is called Super Auto Pets, here's the code with the mitmproxy extension: https://github.com/bspammer/super-auto-pets
You can use this to debug stuff happening over TLS connections, but there are also other practical ways to use it.
For example, install the CA certificate on your phone (with root if you have Android) and set up a WiFi network that transparently proxies everything through mitmproxy. Let it collect data for a while and you'll be surprised how many random servers even well known apps try to contact. You'll also be able to see what data they're trying to send! With modern TLS pinning that's a little hard, but there are Frida scripts out there for rooted/jailbroken devices that will let you bypass that. Just make sure to remove the cert afterwards and to reboot the device to make sure none of your mods remain in memory where they might affect your device's security.
You don't always need TLS, either. It's also useful as a general proxy for protocols like HTTP, for example to intercept, alter, and repeat requests. This can be very useful if you're running into a bug accessing a backend endpoint from code that doesn't easily allow you to iterate through the request code, like some mobile dev frameworks.
Since Android Nougat I think you cannot install user certificates unless you:
1. Root your device.
2. Add the certificate to the APK manifest.
I'm not well versed in Android changes, this is just the conclusion I came to when Nougat came out and haven't tried it again since.
If there's an easier way to use MITM with Android without modifying the APK / rooting your device, I'd love to know!
This is true, by default Android apps do not trust user-installed certificate authorities. IMO the easiest solution if you're doing security testing on a dedicated device is MagiskTrustUserCerts[1]. If you're not testing on a dedicated device or you don't want to root the device, I'd recommend using the objection[2] tool which has a guided mode for patching an apk, and you can modify the manifest to add your CA or to trust all user-installed CAs.
You can install certificates just fine, but apps need to opt in to use user certificates. You can MitM apps that have this flag enabled, but very few mobile developers seem to even know what TLS is, let alone how and why you should change trust store settings.
With root access, you can put your certificate in the root store, which most apps use for validation. I wrote a blog about it around the time Android 7 came out. Some apps (either not enough or too many, depending on your perspective) leverage Android's certificate pinning feature to connect a domain to a specific TLS certificate. You can't MitM those apps without either modifying the APK or hijacking them via Frida.
Lastly, if you want to MitM Firefox, you need to enable their secret debug menu (available on some builds) to enable the flag that enables user certificates. Flagging the relevant setting in about:config does not work and about:config isn't even accessible on stable builds of Firefox for Android.
Sadly, there's no good way to MitM an Android device without full system access. Android has always been quite annoying to use custom CA certs for, but since Android 7 it's pretty much impossible without root access. I'm not entirely sure why they went this route instead of sticking to their existing warnings (i.e. a notification every time a custom CA gets used, like in Android 4, and a constant "your network may be monitored" label in the notification tray like in Android 5+).
You are right, I didn't phrase it correctly, what I is meant that since Nougat I'm not aware of way to force system-wide use of a user certificate without gaining root access.
Thanks for the Firefox tip as well, it's been a while since I used Mitmproxy and I see things have not become easier since.
I haven't tried it for replacing TLS certificates, but you do not need a rooted phone to mess with 3rd party apps on android. You can unpack an APK and re-sign it using your own key, allowing you to make whatever modifications to the APK you want. In addition to making whatever changes you want to directly (such as just replacing the pinned certificates), you can update the manifest to set the android:debuggable flag to true, which allows you to access and modify the apps private files through ADB.
The main downside of this approach is you cannot do an in-place replacement of already installed app, as your version is signed by a different key. You need to uninstall the previous version first, causing you to lose existing app data. At one point, I was able to work around this using Android's backup feature, but I never bothered to retain data since then since it was too much of a hassle and turned out not to matter that much for any of the apps I was playing with.
You're right, but this does have side effects. APIs like in app purchases and DRM freak out about the rogue signature.
It's also a huge hassle if you just want to see what metadata your phone is leaking.
If you're only going after a few specific apps then sure, you can just alter the APK. It's not hard to disassemble and patch the custom validation code most apps use either. The downside is just that it's a huge bother.
Not sure about the most common, but I used it to inspect some requests that an app was making so I could find the needed parameters to get the data from the App’s backend servers in a tool I was developing. I was able to proxy the app’s connection and inspect the traffic as I used the app and was easily able to see what it was doing and replicated the parts I needed.
What’s the use most common use case for something like this, by the way?