I've been looking to use mitmproxy to capture traffic in a project, but the serialization format is something called TNetStrings [1], which postfixes types and is thus not streamable. There seem to be some efforts to switch flow captures to use SQLite [2] or Protobuf [3], although PCAP would also be nice, I think.
Sorry for not stating that in the first place! My use case is archival and preservation. It should be possible to stream the format, build indexes into it (ideally at runtime alongside the capture), it should be as complete as possible, and it should last a long time. I'm looking into pcapng together with a sslkeylog file as the formats of choice for this currently.
I've used several other apps like Mitmproxy, and some of them have nicer UIs... but Mitmproxy is the only one I've found that supports MITM'ng connections using client certificates.
Shameless plug: We (mitmproxy) have a somewhat reasonable web UI (React/Typescript), but I think there's a lot more potential here. We're also looking for Google Summer of Code students at the moment, so if someone wants to spend a few months hacking on cool FOSS stuff while getting a nice stipend for it, please check out https://github.com/mitmproxy/mitmproxy/issues/5048 and reach out to me. I'd love to work with good front end folks on this! :)
er. don't they all? at least all the sophisticated ones. otherwise you can't see the contents of https/etc requests. Charles does this as well for example, and it's reasonably popular: https://www.charlesproxy.com/
OSX-only: I think so. But google trends doesn't care about that, so the scale-of-support side still stands.
And yeah, I wasn't thinking of client certs for auth purposes. I assume that'll hide the encryption keys, so you need a proxy that can do its own exchange? I have no idea what support for that looks like, I've never tried.
Mitmproxy is so much fun. I used it recently to inspect the traffic between a `node --inspect` process and the Chrome devtools. I then used my understanding to "fix" a few things that annoyed me about the node debugging workflow - like how each time the debug server starts, it has a unique URL that you need to manually attach to in the debugger. It's very hacky but it works for my needs.
When I was a teenager I wrote generic proxy in Borland C++ Builder. I say generic because it was not protocol specific, at the time I called it a "bridge". I had lots of fun convincing 'friends' to try my "own IRC server" and watch they interact with other people.
These days I think it would be a simple matter of using tee and netcat, but it was fun at the time.
It is a shame getting a "real ip" is such a hassle nowadays.
No question, just a thanks. First used mitmproxy to build a demo CDN to show off for a customer that hadn't signed yet, so they wouldn't give me certs (reasonable). It was so simple to use, it scares people. I had to explain how CA trusts worked before the company would consider using it to show off.
This is cool. I’m learning a lot reading through the docs. Why is subversion so instructive? I’ve read about SSL certificates a gazillion times and it’s simple enough in theory but for some reason this is making everything click.
What’s the use most common use case for something like this, by the way?
Believe it or not, the most common use-case is simply discovering what your code is actually doing. Modern frameworks are often so far removed from the nuts and bolts of HTTP and TCP that it can be difficult to know what's actually happening on the wire without using a tool like this.
A similar frustration is the tendency for SDK / library vendors to document their product using code snippets in various languages, but without simply showing you what the raw HTTP requests look like. Typically the closest you can get is a set of curl commands.
When you show me a python/ruby/swift/js snippet, I have a guess as to what's happening. When you show me HTTP, I know exactly what's going on. And if you won't show me HTTP, time to fire up mitmproxy.
I recently used it to modify the response from the replay server for an autobattler game, to let me watch arbitrary replays for theory-crafting the ideal team. The game is called Super Auto Pets, here's the code with the mitmproxy extension: https://github.com/bspammer/super-auto-pets
You can use this to debug stuff happening over TLS connections, but there are also other practical ways to use it.
For example, install the CA certificate on your phone (with root if you have Android) and set up a WiFi network that transparently proxies everything through mitmproxy. Let it collect data for a while and you'll be surprised how many random servers even well known apps try to contact. You'll also be able to see what data they're trying to send! With modern TLS pinning that's a little hard, but there are Frida scripts out there for rooted/jailbroken devices that will let you bypass that. Just make sure to remove the cert afterwards and to reboot the device to make sure none of your mods remain in memory where they might affect your device's security.
You don't always need TLS, either. It's also useful as a general proxy for protocols like HTTP, for example to intercept, alter, and repeat requests. This can be very useful if you're running into a bug accessing a backend endpoint from code that doesn't easily allow you to iterate through the request code, like some mobile dev frameworks.
Since Android Nougat I think you cannot install user certificates unless you:
1. Root your device.
2. Add the certificate to the APK manifest.
I'm not well versed in Android changes, this is just the conclusion I came to when Nougat came out and haven't tried it again since.
If there's an easier way to use MITM with Android without modifying the APK / rooting your device, I'd love to know!
This is true, by default Android apps do not trust user-installed certificate authorities. IMO the easiest solution if you're doing security testing on a dedicated device is MagiskTrustUserCerts[1]. If you're not testing on a dedicated device or you don't want to root the device, I'd recommend using the objection[2] tool which has a guided mode for patching an apk, and you can modify the manifest to add your CA or to trust all user-installed CAs.
You can install certificates just fine, but apps need to opt in to use user certificates. You can MitM apps that have this flag enabled, but very few mobile developers seem to even know what TLS is, let alone how and why you should change trust store settings.
With root access, you can put your certificate in the root store, which most apps use for validation. I wrote a blog about it around the time Android 7 came out. Some apps (either not enough or too many, depending on your perspective) leverage Android's certificate pinning feature to connect a domain to a specific TLS certificate. You can't MitM those apps without either modifying the APK or hijacking them via Frida.
Lastly, if you want to MitM Firefox, you need to enable their secret debug menu (available on some builds) to enable the flag that enables user certificates. Flagging the relevant setting in about:config does not work and about:config isn't even accessible on stable builds of Firefox for Android.
Sadly, there's no good way to MitM an Android device without full system access. Android has always been quite annoying to use custom CA certs for, but since Android 7 it's pretty much impossible without root access. I'm not entirely sure why they went this route instead of sticking to their existing warnings (i.e. a notification every time a custom CA gets used, like in Android 4, and a constant "your network may be monitored" label in the notification tray like in Android 5+).
You are right, I didn't phrase it correctly, what I is meant that since Nougat I'm not aware of way to force system-wide use of a user certificate without gaining root access.
Thanks for the Firefox tip as well, it's been a while since I used Mitmproxy and I see things have not become easier since.
I haven't tried it for replacing TLS certificates, but you do not need a rooted phone to mess with 3rd party apps on android. You can unpack an APK and re-sign it using your own key, allowing you to make whatever modifications to the APK you want. In addition to making whatever changes you want to directly (such as just replacing the pinned certificates), you can update the manifest to set the android:debuggable flag to true, which allows you to access and modify the apps private files through ADB.
The main downside of this approach is you cannot do an in-place replacement of already installed app, as your version is signed by a different key. You need to uninstall the previous version first, causing you to lose existing app data. At one point, I was able to work around this using Android's backup feature, but I never bothered to retain data since then since it was too much of a hassle and turned out not to matter that much for any of the apps I was playing with.
You're right, but this does have side effects. APIs like in app purchases and DRM freak out about the rogue signature.
It's also a huge hassle if you just want to see what metadata your phone is leaking.
If you're only going after a few specific apps then sure, you can just alter the APK. It's not hard to disassemble and patch the custom validation code most apps use either. The downside is just that it's a huge bother.
Not sure about the most common, but I used it to inspect some requests that an app was making so I could find the needed parameters to get the data from the App’s backend servers in a tool I was developing. I was able to proxy the app’s connection and inspect the traffic as I used the app and was easily able to see what it was doing and replicated the parts I needed.
I've seen this software, misconfigured, become an open socks proxy sitting on the public internet being used by shady 'vpn' providers to proxy connections. (it bubbled up to me because dns operators were seeing a lot of very weird dns queries originating from our account where we ran a large shared kubernetes cluster. I was tasked with finding the pod/s that were causing the problem)
I was reverse engineering my microwave a few years ago and couldn't MITM on XMPP because of the TLS so I used a jailbroken iPhone and was able to "swizzle" the Objective-C methods to log the data at a method right before encryption started. Fun.
There was another XMPP MITM I could have used, but I think I was already invested in figuring out the first idea before I learned of the simpler, existing way:
Not yet. What works reasonably well is to iteratively work with a custom addon (addons are livereloaded on every change). But I'd love to see progress on that front, it'd make a great GSoC project! :)
Squid is meant to be used as a general proxy for things like caching to help your local network save bandwidth, etc. Mitmproxy is more of a pen testing/security tool. They’re very different things and not something you would be comparing benchmarks to try to choose one.
FWIW from experience MITMproxy eventually reaches a point on my 2019 MacBook Pro where the whole computer becomes unusable. I have to restart MITMproxy.
I don't think they are doing all the necessary GC when disposing of flows from the UI.
Nice idea but terrible implementation. Mitmproxy solves a lot of annoying problems, but, Python holds it back to such an extent that it's hard to make the case for deploying it in production due to massive inefficiencies.
I thought mitmproxy was designed to make it easier to debug stuff in a local environment. Why would you want to deploy it to production? How would you use it?
One of the things I like about mitmproxy is how easy it is to develop and use addons. You just pass a script file to it. How would you achieve such painless extensibility if it were written in Go?
It's very hard to do with Go; also see Hugo, where Go is in many ways "holding it back" compared to something like Jekyll, which you can customize and hack with great ease.
I love Go, but this kind of things is not where it shines.
mitmproxy is explicitly intended as an interactive MITM for debugging: "your swiss-army knife for debugging, testing, privacy measurements, and penetration testing".
What you're talking about here was never a goal. And being in Python means it's much easier to extend, which is also greatly useful.
FWIW: I'm building something in Go (https://github.com/dstotijn/hetty). But it's pretty early stage and not even near the featureset that mitmproxy or Burp Suite has. Also I wouldn't dare say it's more efficient (yet!). But Go has been great so far to build it.
As mentioned elsewhere: (dev friendly) extensibility/add-ons with Go will be an interesting challenge. Haven't looked into it yet.
[1] https://tnetstrings.info/
[2] https://github.com/mitmproxy/mitmproxy/issues/3075
[3] https://github.com/mitmproxy/mitmproxy/pull/3232