Wow, so many of these sound so much better than mine, but I'm going to share anyway.
We had an audit requirement to gather all AD accounts (computer, user, group) with group memberships in all domains[0], discover all servers, login and acquire their local user/group ACLs/config (win/linux), discover all databases (MSSQL, MySql, Postgres and Oracrap), acquire all of the ACLs in the databases, themselves, and a variety of application-specific credentials. This had to be completed once every 24-hours in order for other compliance operations to complete in time. We purchased a system to do this[1]. It managed to hit all AD domains in 4 days. A few more and it could get the Windows server account ACL information. We gave up adding anything else and a coworker took it upon himself to rewrite portions of it in a scripting language, which due to his shocking abilities in said language, netted Active Directory environments in 24-hours.
I wrote a plug-in based system that was as lockless as could be with a thread scheduler which spawned old-school threads, increasing count when capacity was available, decreasing when it wasn't. This had to be done because the options available to me in C# 2.0 wouldn't reach the required 24-hours and I had to substantially beat it if they were going to let me build this thing out. When I was done with the MVP, I was handling all of AD in under 4 hours.
The component that was going out and collecting data was the slowest and most difficult to optimize, partly because we'd flood a "pretty beefy for a link from the US to Argentina/Brazil/Venezuela" but that's not saying a lot, so I split things up, made the processor into two components, one with an authenticated web service that handed off to a processor when it received anything. I ended up using the oldest, most crotchety method to communicate between the two apps because the web service ran as a down-level account and the processor ran with incredible permissions (at least on the read-side).
It was horrible on every level. The scope was massive - more than 20,000 boxes with the desire to even collect file system ACLs at some point. The implications of screwing something up on the security side were daunting -- the firewall rules and layer upon layer upon layer that went into securing each of these components (not just on the network, but isolating them as much as possible using built-in OS ACLs and rules). Doing threading ... correctly ... in C# is easy to screw up and I think I managed to actually witness every single thing that goes wrong when you fail to protect shared data that's mutable. The available options for thread scheduling in .Net and Windows didn't fit well with my use case (I tried several with tweaked parameters in an attempt to bend them to my will). I ended up having to create a state machine to monitor and compare various counters in the application to decide, upon receiving a request, if it would improve speed to spin up a new thread in a thread pool that I also hand-rolled. These were certainly naive implementations, but I could find no other way to make this work the way it did. At the end of the day, I was able to scan the entire set of required environment components and grant, revoke, or apply a custom permissions rule to anything I could scan. Rather than schedule it to run daily, I just had it continue to repeat after it completed, meaning the completion time ranged from about 3-hours to about 12-hours depending on traffic/server loads. This meant when a new hire started, they received access around 4:00 AM to every system in the entire company that their job required, and when they left, their access was cut immediately, everywhere, due to a priority system I also added to the system.
This project came with a mountain of politics. Two, large, companies where one purchased the other and I was in a team on "the other" ("the other", thankfully, had said contractual requirements so the company who purchased us had no prior knowledge on how to do something like this). They strongly resisted rolling our own solution in this manner. In the end, it performed so ridiculously well, and was so easy to build a new "module" for (compared against the off-the-shelf product that we had both, coincidentally, purchased) that it was kept, upgraded, much of the threading complexity replaced with new features available in .Net 4.5 and hopefully a lot more (I left when .Net 4.5 was released).
[0] Accurately ... 14 domains with varieties of trusts, various misconfigurations, one case of three domains in the middle of migration to a single domain with the fun that SID history injects. sIDHistory, alone, represented a month of debugging to get it to return the correct account, Alice in Domain "A", is a member of Group "1" with her account in both Domain "B" and "A". Abstracted, Alice's two accounts are supposed to be seen as one account. Higher-level AD libraries will return either account regardless of what is joined to the group (there's ways to coerce it to return a specific domain, but not a way to coerce it to return the correct one without using a more painful library).
[1] It was one of those solutions where you buy a sort-of framework and pay the company to code modules for it that are custom to your environment in a DSL that is custom to the application. It works as well as you'd imagine.
We had an audit requirement to gather all AD accounts (computer, user, group) with group memberships in all domains[0], discover all servers, login and acquire their local user/group ACLs/config (win/linux), discover all databases (MSSQL, MySql, Postgres and Oracrap), acquire all of the ACLs in the databases, themselves, and a variety of application-specific credentials. This had to be completed once every 24-hours in order for other compliance operations to complete in time. We purchased a system to do this[1]. It managed to hit all AD domains in 4 days. A few more and it could get the Windows server account ACL information. We gave up adding anything else and a coworker took it upon himself to rewrite portions of it in a scripting language, which due to his shocking abilities in said language, netted Active Directory environments in 24-hours.
I wrote a plug-in based system that was as lockless as could be with a thread scheduler which spawned old-school threads, increasing count when capacity was available, decreasing when it wasn't. This had to be done because the options available to me in C# 2.0 wouldn't reach the required 24-hours and I had to substantially beat it if they were going to let me build this thing out. When I was done with the MVP, I was handling all of AD in under 4 hours.
The component that was going out and collecting data was the slowest and most difficult to optimize, partly because we'd flood a "pretty beefy for a link from the US to Argentina/Brazil/Venezuela" but that's not saying a lot, so I split things up, made the processor into two components, one with an authenticated web service that handed off to a processor when it received anything. I ended up using the oldest, most crotchety method to communicate between the two apps because the web service ran as a down-level account and the processor ran with incredible permissions (at least on the read-side).
It was horrible on every level. The scope was massive - more than 20,000 boxes with the desire to even collect file system ACLs at some point. The implications of screwing something up on the security side were daunting -- the firewall rules and layer upon layer upon layer that went into securing each of these components (not just on the network, but isolating them as much as possible using built-in OS ACLs and rules). Doing threading ... correctly ... in C# is easy to screw up and I think I managed to actually witness every single thing that goes wrong when you fail to protect shared data that's mutable. The available options for thread scheduling in .Net and Windows didn't fit well with my use case (I tried several with tweaked parameters in an attempt to bend them to my will). I ended up having to create a state machine to monitor and compare various counters in the application to decide, upon receiving a request, if it would improve speed to spin up a new thread in a thread pool that I also hand-rolled. These were certainly naive implementations, but I could find no other way to make this work the way it did. At the end of the day, I was able to scan the entire set of required environment components and grant, revoke, or apply a custom permissions rule to anything I could scan. Rather than schedule it to run daily, I just had it continue to repeat after it completed, meaning the completion time ranged from about 3-hours to about 12-hours depending on traffic/server loads. This meant when a new hire started, they received access around 4:00 AM to every system in the entire company that their job required, and when they left, their access was cut immediately, everywhere, due to a priority system I also added to the system.
This project came with a mountain of politics. Two, large, companies where one purchased the other and I was in a team on "the other" ("the other", thankfully, had said contractual requirements so the company who purchased us had no prior knowledge on how to do something like this). They strongly resisted rolling our own solution in this manner. In the end, it performed so ridiculously well, and was so easy to build a new "module" for (compared against the off-the-shelf product that we had both, coincidentally, purchased) that it was kept, upgraded, much of the threading complexity replaced with new features available in .Net 4.5 and hopefully a lot more (I left when .Net 4.5 was released).
[0] Accurately ... 14 domains with varieties of trusts, various misconfigurations, one case of three domains in the middle of migration to a single domain with the fun that SID history injects. sIDHistory, alone, represented a month of debugging to get it to return the correct account, Alice in Domain "A", is a member of Group "1" with her account in both Domain "B" and "A". Abstracted, Alice's two accounts are supposed to be seen as one account. Higher-level AD libraries will return either account regardless of what is joined to the group (there's ways to coerce it to return a specific domain, but not a way to coerce it to return the correct one without using a more painful library).
[1] It was one of those solutions where you buy a sort-of framework and pay the company to code modules for it that are custom to your environment in a DSL that is custom to the application. It works as well as you'd imagine.