When I was a CS prof, many, many years ago, our undergraduate lab had Macs with floppy disks. I asked the University to pay for installing 10MB Hard Drives in the Macs. I was asked to present my case to the deans council. At the meeting, I said that the students used the floppy to load their development environment. I said that, with a hard drive, it took 10 secs to load and be ready. With the floppy, I said it took 30 seconds. Then I said, "That does not sound like much difference, but this is 10 seconds ...." I paused and looked at my watch for 10 seconds. Then I said "And this is 30 seconds" - again I looked at my watch. At the 20 second mark, the VP Academic (chair of the meeting) said "You have your money".
I've used this trick many times when people underestimate how long a few minutes can be in certain contexts. It works astonishingly well. Happy to see I'm not the only one!
There's an anecdote [1] about Steve Jobs and the original Mac's boot time similar to this story.
One of the things that bothered Steve Jobs the most was the time that it took to boot when the Mac was first powered on. It could take a couple of minutes, or even more, to test memory, initialize the operating system, and load the Finder. One afternoon, Steve came up with an original way to motivate us to make it faster.
[...]
"Well, let's say you can shave 10 seconds off of the boot time. Multiply that by five million users and thats 50 million seconds, every single day. Over a year, that's probably dozens of lifetimes. So if you make it boot ten seconds faster, you've saved a dozen lives. That's really worth it, don't you think?"
If the request you are talking about actually gets executed once (maybe it gets cached or something), then you shouldn't be pitching the time difference anyway.
If it gets executed 100s of times in a day per person, you can say this is 50 ms * 100 = 5s, 200 ms * 100 = 20s. And that's just per user.
Yeah, actually doing this math tends to surprise people. We had an automated background process that took ~2s to run, which doesn't sound bad at all considering it includes an API call over the internet. But multiplying it out to the number of backlog items we actually had at the time, 30-40 hours doesn't sound so good.
Fair enough. Unless you're talking about lags for automated trading systems/ algos where even that single "I really can't measure this" sec difference counts.
I mean, sure, the precision matters for HFT but at the scale the point would be moot since the time is so minuscule. Unless you hyperscale it: “on 1,000,000 trades the 50ms difference becomes very pronounced and could cost us $z” or something of the sort. But I still think it loses “the spirit” of the method — best way I can phrase that.
Interesting. By coincidence last night (while playing with a new streaming-stick) I did a little quick calculation. If I watched, on average, only 18 minutes of advertising per day, and did that for 40 years, that adds up to 6 months of my life.
What else might I have done with that 6 months? I will never know. But I know this: I won't be doing that any more.
Left university and joined a startup, switched jobs to a different company, that company was acquired by Apple, worked there for about 12 or so years, now retired, but work on an App.
I somehow decided I needed to cheat to pass a certain exam because I was basically crap at memorizing stuff. So I used an analogue wireless headphone, an induction loop around my neck and a mobile phone. Since I lacked an accomplice to dictate, I read aloud the hundreds of pages and recorded myself, careful to preserve and properly serialize things like complex formulas.
This was before the era of iPods and SDCard players, so I had my mobile phone in a setup where I would call back another phone connected to my Pentium MMX 233MHz at home, that ran a sort of audio directory that would playback a certain lecture recording I would select from the menu, using DTMF tones.
I had a small keyboard sewn into my sleeve that connected to the customized mobile phone via a DB9 plug and then to the numeric keyboard, allowing DTMF codes to be sent by gently and invisibly moving my fingers. The whole setup was hideous and had it's own dedicated jacket, with wires, phone, keyboard, audio amplifier, neckloop, the earphone... a complete cyborg for academic fraud.
Back on the PC side, I wrote a C++ application from scratch that would capture audio via the soundcard using Windows Wave API, decode the DTMF pulses using a couple of IIR filters then navigate the menu and playback the required file to the mobile phone connected to the soundcard. The C++ program and menu system was scripted using an .INI file that defined the structure with links to various ADPCM-compressed .wav files that represented the menu headings or the leaf content itself (a good structure was necessary to quickly access the correct lecture after receiving the exam subjects).
Work-wise, it was a lot more difficult then putting the effort in memorizing the stuff, but I rejected memorisation on principle, that's not what an university should be about. The whole thing turned out to be a massive learning project, but I obviously could not speak about it at interviews. It's the first time I mention it to anybody.
I used the setup for two exams that I aced, was never caught but it was nerve-wracking to use in close proximity to a teacher.
One thing universities should teach is efficient problem solving. If it was more work to cheat than to memorize, I'd say you might want to re-evaluate whether "not memorizing things" is a principle you should cling so dogmatically to.
I'm sure he learned much more useful stuff by coding up that monstrosity, and in the real world he can just google to look up those formulas he didn't memorize.
The fully elaborated version of the underlying principle is probably something like: "I have much better ways to spend my time than on memorizing formulas that I could easily look up later".
If an individual's philosophy is that you can always google stuff so you don't need to know anything, I don't see why that person should waste time in school.
There's more to exams than memorization too, and cheating because you don't like memorization is childish. If the exams really test only memorization, drop the class.
In a perfect university teaching environment, yes. However, in real life sometimes cheating is necessary. Real university environments are flawed, rules and requirements often stacked against the student. They have to pay up, get a massive loan, endure narcis/dumb teachers and stupid administratory requirements, and educate themselves despite that environment. After a lot of invested time and money donated by state/parents, if faced with particularly nasty course exam they are unable to pass in a legal way, it would be crazy to fold and waste all that investment.
In the US the legal age of adulthood is 18. This is very far from the actual maturity point. It's not infantalizing to be accepting and unsuprised when a 18-22 year old does something childish.
Edit: Being a child, and acting childish is an important part of the life cycle that allows development. We seem to be bumping into some biological limits on how late we can extend it, but if it were possible I would.
When I worked in baseball concessions as a teenager, the employees who were the most responsible and dependable were the university students. The older workers tended to be more experienced, but some of them were a little sketchy and a couple were fired for showing up drunk.
There is a tremendous variation between individuals. Part of the point of the university degree as a credential is to identify those people who are responsible enough to complete the program. I graduated at the age of 21 and immediately became an Engineer in Training, working on code that could kill people if it behaved improperly. I took that responsibility very seriously, and I'm thankful that my employer and coworkers took me seriously in return.
I fear that a focus on age tends to lead to prejudice. If you just want to ensure young adults can be allowed to make mistakes without ruining their lives, I would encourage you to expand your goal. There are many adults who could benefit from a bit more compassion and forgiveness. We don't need to restrict it to the young.
I strongly remember a mandatory English (literature) exam in Highschool where you had to write an essay in response to a question that was known beforehand, in relation to one of three or so books you had studied beforehand (You didn't know which on the day). The way this test was practiced was to get every student to write the full essay for all 3 books and then to memorize that. Most other English tests I had to take in Highschool were similar to this, but not quite as bad. I think you have too high of expectations from education systems to think that all tests test more than memorization or that people can simply "drop the class".
I dislike open book exams so much that I took nothing into my last one. It was a unit without much memorisation that focused much more heavily on applying concepts, and I feel that having it as open book was a trap for ill-prepared students.
One of my highschool teachers mentioned that a book wouldn't help much before her first open-book test, then openly stated that afterwards that it was a small trick to see who wouldn't study. She was among the best and most-liked teachers in the school.
I'm not defending what they did, but it's worth pointing out that electives aren't exactly universal. During my university I was allowed to choose exactly 4 classes out of 6, in my last semester. Every single other course was compulsory. I never cheated, and looking back on it I'm glad I didn't, but I certainly did my share of useless learning.
Edit: that being said, in a system where there is such a thing as non-compulsory courses, and the critical path to graduation is sane, cheating on an exam on the critical path should definitely not be acceptable, even if it's "only" memorization. Some things really do need to be memorized.
Not every academic system allows to freely drop classes, to drop a class without an economic penalty, or to drop a class without falling behind in your studies.
I had a programmable calculator during A Levels (UK sixth form college). Was talking Computer Science, Maths, Further Maths, a couple of others. These were not AS Levels, full A Levels.
I also had the seemingly good idea it would be nice to program the calculator, initially just writing algorithms like bubble-sort for CS that could be referred to, then extending to Statistics showing working for various statistical methods (the exam required steps showing working steps, learnt a little about data structures myself for that, then extending to matrix algebra and curvature). I also didn't go to most classes, only attending the ones that would be more useful than time in the library.
Typing character-by-character on a numerical keypad, getting a series of working systems. Was pretty fun.
And in the end, having working 'cheating systems'... I could do it faster in my head and never even took the calculator to the exam.
That's learning, I suppose. Did very well on the exams.
I also wrote programs to run long equations on my calculator. This wasn't forbidden at the time because it was too early, I suppose. Regardless, you had to show your work to get credit anyway so they just functioned as a way to check my work instantly.
I had a lot of fun writing little programs to help out on long arduous problems though. I wish I had all my TI programs.
Unless the exam allows notes, it can make sense to forbid using programs that are already stored and require to write the program during the time needed for the exam if you want to do that (even to program it to show the work if you want to add that into the program too; I think someone once did this), then it can be OK to allow it, I suppose.
I wrote a raster graphics editor for my Casio programmable graphing calculator. I also wrote it exclusively on the calculator. Did it to one-up my class mate, who had done one previously--mine had better features (no dot in the center of circles, could also do ellipses, had a spray function), and it was half the LOC.
Casio BASIC had subroutines in IIRC 36 registers ([A-Z0-9], conditional jumps and labels.
I wish I had that source code. I can't see how I would ever do something like it again.
I would 100% talk about this in interviews depending on the interviewer. I can think of a couple interviews I had in SF that would probably be appreciative of this level of effort because you principally reject memorization.
Admitting to significant academic fraud during a job interview is a very, very bad idea. Furthermore, I would question the ethics and morality of any company that said "Wow, you solved an incredibly tough problem in order to commit fraud. Boy, are we the place for you!"
I'm not sure why you need that trait in software. You're (usually) not trying to beat other programmers. So I suppose the parent comment should have added "except in adversarial situations".
Which means it does apply to cyber security. You need hackers to outsmart hackers. If that's the kind of industry you're applying for, it probably is a good idea to mention this actually.
That's different. According to that page, Abagnale served 5 years in prison (6 months in solitary confinement), and had to work for the feds for a while "without pay". Then when he first got legitimate work, he was usually fired.
I'd be willing to hire someone who cheated and was caught and served time, but not someone who cheated without consequence and boasted about it.
The commenter here doesn't sound like they segued that into a career catching academic cheaters. They just used it as a stepping stone to their next personal accomplishment.
If you cheated on an exam, it's a very bad negative signal for integrity there's no question about it.
This is not very grey.
If you were caught you would probably be kicked out of your Uni, this is a serious thing.
I understand there are extenuating circumstances, people are young, they do crazy things, we all have. But it's not something we run around talking about, certainly not in an interview.
> Admitting to significant academic fraud during a job interview is a very, very bad idea.
Sure, it's a no go if you're looking for a job at an academic institution, and a big company will reject you because fraud of any kind is a
huge HR liability, but I bet nose people would be impressed.
It might be well received at a startup, though. As long as you recognize your action was highly unethical.
Like, I don't agree with you but I do see your point. I'd hope that a company would ask "and how do you feel about your choice to cheat?" open up a conversation about ethics and if you learnt from the experience. It's a can of worms and could result in a great interview.
As a public speaker and author mostly. He doesn't get access to state secrets and hasn't been employed by any other company. He wouldn't get a call back from McDonalds if he applied.
He's hired as a pen tester and gains access to companies, their secrets, etc. I have a sysadmin friend whose company hired him to pentest, and they kept in contact.
I would allow it (if these kind of skills to design all of this system is the skills that is useful for this job). You are honest to admit it, and if you can make such thing, it is good. Of course that was cheating, but that does not mean that such thing is only to make fraud; you might do similar thing with other thing too.
Once you admit to fraud, though, they have no idea what else you might be lying about -- including the admission itself. They can't exactly call his old professor to confirm that he built and used a cheating machine.
Plus, after you admit to doing it once, if I weren't the sort to reject you outright, I'd still call every single person and institution on your resume to confirm all of those were real. Right out of the gate, you're proving yourself to be a lot more trouble than anyone else I'm interviewing.
It sounds like mostly luck that they weren't caught before. Are they going to try again, at my company? I don't want my company's legal footing to rest on one person thinking they'll never get caught.
It adds a human component to your interview. As the saying goes, programmers are lazy. Based on the components you describe, I think it's safe to say, that you're not an entry-level hire and you have experience in your field. It'll make for a great story and demonstrates passion.
Amazing as this tale is, folks for anyone else who is sure that memorization is too hard don't do this. The real tip is:
Spaced repetition. Load up Anki or Mnemosyne and stick the info in there.
"Craig prepared for Jeopardy! by studying the online archive of past questions maintained on the J! Archive website. Using data-mining and text-clustering, he identified the topics most likely to occur in game questions,[9] then used the spaced repetition program Anki for memorization and tested himself using his own program.[10][11][12][13][14]
Craig played quiz bowl as a student at both Virginia Tech and the University of Delaware.[15] Before his Jeopardy appearances, he played numerous Jeopardy scrimmage matches against his friends with quiz bowl experience.[14]"
I was always tempted to modify a calculator to add Bluetooth, then I could stream over whatever data I wanted from a phone somewhere without any suspicion. Graphic calculators make this trivial as they often have a serial port, I just never got around to building it. I guess I didn't want to cheat that badly
That is unbelievable! How did you index the audio? What did you have to do to access them during a test, and how long did it take? Did you find that you had accidentally memorized some of the lectures after having read them aloud?
> I rejected memorisation on principle, that's not what an university should be about.
The lengths you went to to avoid doing it the easy way are incredible. You’re not alone, but I have to say I think this idea - that university should not involve memorization - is a common misconception. If you think about it, a good education requires memorization. What we’re learning in all fields in large amounts - even in math - is some degree of human history, and some degree of human notation and convention, neither of which can be derived by logic. Think about language, becoming a fluent speaker, especially the first time, is almost entirely learn by rote.
You rejected ethical behaviour 'on principle', you even seem proud of that. Have you kept doing that in life? Do you feel like an impostor, presumably with your degree that you didn't think you could acquire without cheating, or couldn't be bothered?
A surprising number of the replies to your comment also seem to think cheating like that at university is perfectly fine.
My humble opinion. University is about learning (not, for example, status). Learning is something strictly personal. There is nothing fundamentally universally wrong to cheat at an exam beside the fact you may be harming yourself.
Now since we need a little order for various secondary reasons, we promoted exam cheating to illegal and that's ok. We need order. But there is nothing someone should feel guilt for imho, assuming the person knows she may be harming herself.
I agree with this. I would further suggest that the modern conflation of two orthogonal functions (teaching and certification) in universities is rather misguided. These would be better served by two separate types of institution.
> These would be better served by two separate types of institution.
For a long time, they were. A technical school certifies someone to do something. A university teaches you how to think about hard things (supposed to, anyway).
I wasn't talking about law, I was talking about ethics.
"There is nothing fundamentally universally wrong to cheat at an exam beside the fact you may be harming yourself."
I don't understand that, I have no idea where you got that. Or what those words "fundamentally" and "universally" add. I say it's wrong, you say "Oh, but it's not fundamentally, universally wrong".. As if it's clear what that means.
For example: You may have harmed the people who didn't get good enough marks because you cheated your way into higher marks. Then you may harm people in your career that you're not qualified for, besides stopping properly qualified people from doing their jobs. I don't want an airlane pilot or doctor that bought their degree or cheated in exams, thanks. Anyway, it seems ridiculous that I have to explain to people why cheating's bad. Well, I don't know, maybe you are in a country where it's normal, perfectly fine, accepted, everyone does it. Where I come from, people don't have to have it explained to them why it's bad.
Yes, cheating is bad. But if you want to be interviewed to the job you need to see if is qualify to the job; some people can be good even if you have not went to the university or other schools, and even if you answered the exam it does not mean you are better at that particular job than another candidate but only that you know the answer of questions (or successfully cheating without being caught) and can be good at examsmanship.
If you are good at mathematics, you will invent a new theorem! If you are good at music, you will compose a new music! If you are good at chess, you can win! If you are good at exam answering, you can earn some more marks (guessing at answers if you do not know the answer)!
'Cheating at an exam' is transgressing a rule that doesn't "exist" in nature, it only exists in our social (if that's the right word) system. So if I cheat at an exam, I'm breaking a rule that's in the system that we made up (together), and therefore I can judge that, in some circumstances, I can break the rule without feeling morally wrong, without feeling guilt, because I, in some sense, made the rule myself. I'm breaking my own-making rule.
Another example of that could be: I want my kid to go to bed at 9pm. Sometimes I will break that rule. To some extend, because of the reasons I've advanced, I claim that cheating at an exam follows the same characteristic as the "kid go to bed at 9pm". Just not in the same magnitude if you will.
I then guessed that it may draws the limit between what's ethic and moral.
Hi again. I don't see how that draws any limit/distinction - it was 2 examples of rules that can be broken, not sure how that helps explain the difference, or why you said that. I wouldn't say bedtime is a moral rule/principle, or that breaking it is unethical. Maybe could make it clearer for me which one was supposed to illustrate what, if one was meant to be ethical, one moral, or something, I don't know. I really have never heard the words used with much or any difference. (I'm no expert, but have read dozens of ethics books, studied ethics/moral philosophy at uni etc)
I'm just guessing here, but maybe you have a religious value system, with absolute moral commandments or something? All I have (as an atheist) are ethical/moral principles exactly like 'cheating is wrong'.
You say 'cheating is wrong'. Fair enough, you can see 'right or wrong' as binary. Or you can live in the real world and understand that things are a little more complicated than that. With all the information you have out here (more than your hypothesis) you can make a fairer judgement. And you don't do judgement without introducing the living anyway because only what's living can judge and be judge-able. It's a social thing to judge right or wrong. So you have to take into account all the system. The living.
Yes. Learning is yourself. (University is also supposed to be about learning, although this is only partially true as it is implemented.) They have exam you can test, but that is difference from learning, although if you know the answer then you can see if you know the subject being learned. But you do not have to learn only that way; you can do many thing such as to read a book, figure out by yourself, or in this case, to attend the lecture. If you have a question because you do not understand the lecture, then you should ask, and that is how you can learn. Also, examsmanship is not same as learning the other subject, but, still you can learn examsmanship too. Noticed I mentioned before, I might to accept if the stuff you did for the cheating is subject of the class anyways such that it mean is good at it, then perhaps you can earn "SG" (meaning that, you pass regardless of what is your mark).
But still, memorization is not the same as learning. That is one thing that the test is not always so good; whether or not is "ethic" and/or legal is independence from that, because if you understand, then you can do, but if you know the word but don't actually know what is the significance, then you might answer the question same like that one but not the difference question that you can actually use.
(I remember once on one exam, the last question I did not know, and cheated off of someone who also didn't know and was cheating off of me (I don't know if they were cheating on other questions too or not, only that it is for last question), so in this case we cheat off of each other. Of course it is no good compared to all of the effort they mentioned above, but still you can see, you can be cheating off of each other the same question. I think this is the only instance of cheating on exams I have done, although once I tried to use the "coughing code" (without telling anyone!!!) just to see if I can, and not because I actually wanted to cheat, because I don't want to cheat.)
It was undoubtedly unethical behavior. But it's like... well it's like the story of Toshihide Iguchi, who accumulated over a billion dollars of trading losses and hid them successfully for 12 years. Sure you throw the guy in prison, but you have to hand it to him that what he managed to do is pretty damn impressive. After his release, finance CCOs might even want to hire him to audit their traders.
You can get off your high horse about not being able to acquire your degree without cheating, too. I never once cheated at any point in college, but if I had spent more time working on sick electronics projects and less time memorizing the years of significant 15th century battles, I'd be more qualified technically, not less. The ethics of it are problematic as I've said, but you seem to be going farther and implying something about his ability as well.
Why isn't it fine, though? What serious damage did his actions cause? Which important societal construct rests solely on the assumption that no students cheat and, once this is violated by a single student, comes crashing down? I would suggest that if such a construct exists, then maybe the problem is its very conception and this is what should be rethinked.
I think the implication that he should feel like an impostor for cheating on two exams is a very large over-reaction on your part.
"Which important societal construct rests solely on the assumption that no students cheat "
Society does not depend on wether or not a few students cheat.
But society definitely depends on the social and moral contract that we do not cheat, and that it is wrong. If most of us cheated, the system would definitely fall apart.
Have you ever visited a developing nation? Where outside of small villages it seems like everyone is cheating at everything, all the time?
It's like pouring sand into the gears of an engine: everything starts to break down.
I would not hire someone who admitted cheating during an interview unless they talked at length at their remorse, how they learned from it, how they grew from the experience, and there were exceptional circumstances.
> It's like pouring sand into the gears of an engine: everything starts to break down.
While true in general, cheating in that environment is also sometimes necessary to get on with your life unharmed. And sometimes cheating the system is actually regarded highly by the fellow citizens, because the people do not believe in the imposed system. Cheating is not beneficial to the system, but it often is to the people. Now, what is more ethical, doing what benefits the people or doing what benefits such system?
"And sometimes cheating the system is actually regarded highly by the fellow citizens, because the people do not believe in the imposed system. "
If you live a totally corrupt system, maybe.
But functional societies are based on the notion that cheating is bad, immoral, and nobody should do it.
"Now, what is more ethical, doing what benefits the people or doing what benefits such system?" - ha ha ... this sounds like how psychopaths in prison justify their crimes!
"I robbed the bank because the bank is evil, now who's more important, the people or the evil bank?"
People who rob banks are not necessarily psychopaths. For a good (fictitious but convincing) example, see Toby Howard in the film https://en.wikipedia.org/wiki/Hell_or_High_Water_(film) Robbing a bank is universally illegal, but it is not regarded as unethical in general.
Robbing average Joe is universally considered wrong. Robbing bankers, insurance companies or other powerful subjects, I don't think so. It depends on the circumstances - for a good example, see the movie, that is, if you like such a mental challenge.
Have you heard of the Robin Hood legend? It says the man was robbing the rich, and was very popular with the populace. I find it hard to believe that his robbing activity was regarded as 'unethical' by 'almost everyone'. Or a recent example, the sci-hub and libgen projects. They are robbing publisher shareholders of their profits. Do you think those projects are regarded as unethical by 'almost everyone'?
Ethics is not something that gets decided by the Dean and laid down in a code of academic ethics. The University is just a part of society and it has it's own perverse and quite unethical practices; for example, the tendency to grab one's money and time and not provide any substantial education in return. Furthermore, the university might be perfectly aware that it provides crap services and that fraud is rampant, yet enforce it's own rules selectively just to the point where fraud is hidden away and not damaging to it's reputation.
So while I agree with you in the abstract, I believe it's hard for people in the western system to relate and make moral judgements about the experiences of a student in post-communist Eastern Europe, where the rules were often gray. Fellow students were cheating on a grand scale, and up to that point, I had the same principled attitude towards cheating as yourself. Part of the reasons for creating "the system" was my revolt towards what I saw around me - the tacit acceptance that those with good cheating skills, which I lacked and did not develop early in my academic career, should be allowed to cheat their way to a degree, while I was supposed to memorize garbage.
Sure, my "rebellion" was unethical and compounded the social problem around me. But what could I realistically do, could I have changed anything? At least I saved me from myself, because otherwise I would have dropped-out. In the grand scheme of things, impeccable ethics is often a luxury and could have the exact opposite effect, letting only those with no ethics whatsoever to graduate.
If you sign a code of ethics swearing you won't cheat and you do anyway, that's unethical. Even without an actual signature, the social contract is implicit. The fact that the university exploits athletes and postdocs is unrelated. You can break out the "I did what I had to do to survive" argument but then you're just conceding the point that you abandoned ethics out of necessity or simple mercenary desire for a better position.
I don't think it's a huge deal and wouldn't personally let it negatively affect a hiring decision provided you made the right noises about how it was dumb in retrospect and you were young, etc, but your comment here is pretty much the opposite of that. Who could possibly employ someone with your attitude? Even a gas station wouldn't want to hire a cashier who steals money from the register when they can get away with it just because the petroleum industry is destroying the environment, or whatever.
As I anticipated, it's hard to relate to somebody living in a profoundly corrupt society. My gripe with the university was not that it was exploiting postdocs or killing the environment; rather, that it failed to setup and enforce a system of rules where my extant academic ethics mattered.
The social contract includes not just what's written or implied, it's also what people do in practice, what is acceptable. If cheating is acceptable and required to get a tech job, then I can do that, in fact I can do it better than most. That says very little about my profound sense of ethics.
To counter your analogy: what if you are indicted of a crime you did not commit in North Korea? You surely accepted their rules when entering the country, but would you trust their judicial system to do the right thing? Would you bribe your way out if you were given the chance? Would you ask your country to put diplomatic pressure on your behalf, a clearly unethical advantage no Korean has? Would you consider escaping from prison if wrongfully convicted?
In a narrow definition of ethics as "whatever the current rules are" (typical, I would say, for somebody living their whole life in a state with strong rule of law), the only ethical behavior is to subject yourself to whatever abuse NK decides for you.
That's throwing the baby out with the bathwater. It's true that law and ethics aren't always aligned, especially in authoritarian regimes. But basic things like integrity are essential for any human society.
Also, I'm not convinced that growing up in the former Soviet bloc necessarily means that honesty isn't the best policy. If everyone around you is corrupt, being the one dependable person around is a differentiator. Maybe I'm being naive, but it seems like that reputation could have some real world value to you.
By the way, it doesn't matter what your classmates are doing. Comparing myself to the average was horribly destructive to my college education. It turns out, everyone around me putting in average amounts of studying and getting average grades ended up getting average jobs with average pay! The one or two people in class that always aced everything, about whom I thought "well I don't have to be as good as them" - those were the ones who ended up having a real shot at grad school or dat $100k Facebook new-grad signing bonus. It's just like when you move from high school to college and suddenly you're not the smartest person in the universe anymore. Your cheating classmates are bozos - stomp them with harder work and keep moving up. Don't sink to their level.
> If you sign a code of ethics swearing you won't cheat and you do anyway, that's unethical
If you do not sign such a document, does it make a difference? Of course not. Getting massive advantage by extraordinary cheating is still unacceptable. However, cheating a little when stakes are very high, or cheating for 'greater good' may be acceptable. Cheating is a personal decision and whether it is acceptable depends on the situation the cheating person is in, and the person doing the judgement.
Thanks for explaining further, that's a finer reply than I deserved. Well, I didn't know where that was, or the wider story. What do I know, I've never been in that situation. Good luck, I hope to read your book one day. :-)
Wow. How that comment hasn't been flagged yet I have no idea. People who don't agree with you are 'nuts', yet you have a nuanced view of ethics? That's the most condescending, deluded thing I've heard in years. "For them.." blah blah.. you don't know us; in my case at least, your assumptions couldn't be more wrong, but what do you know or care; you've decided who the in- and out- groups are here.
Oh.. I just noticed this. I'm not sure why you are complaining about my comment but not the one I was replying to! At least I think you were talking to me. I'm not sure what in what I said was personal attack. At least, it seems you were implying something I said was a personal attack. I strongly criticized his view, words, assumptions, decisions. I'm kind of amazed to be threatened with banning about that. [reviews guidelines] Ah maybe "please reply to the argument instead of calling names" - I guess you mean "That's the most condescending, deluded thing I've heard in years." That seems to me a fact. From memory, I don't think there was an argument, just a string of false assumptions presented as fact. (Can't read his comment now) Or maybe it's a "shallow dismissal"? Not sure. What particularly about my comment deserved being threatened with banning?
I have noticed that the worse the comment, the harder it is to reply to decently/politely/etc; maybe it was impossible for me to reply to this one, or say what I thought without arguably breaking the guidelines, so OK, point taken, better not to reply in such cases. "Someone is wrong on the internet!" is real. I really did try to stay on-point, not hard enough apparently. I mean, I really do think it was the most condescending, deluded thing I'd read in years, definitely while on here. (I guess you've read much worse!) Again, not sure if that's the part you're objecting to. Apologies if you've read 10,000 replies just like this one. Kind of surprised (and depressed) I'm the problem here though.
ps I only saw your comment by chance, it's pages back in my comment history. Is there a way of getting alerts for replies on here?
I don't think I saw the parent comment; it's about as bad as yours was, though yours might have been a notch worse, since that one was attacking in the third person while you did it in the second.
The main thing to realize is that even if another comment is bad/uncivil, you still need to refrain from replying in kind. Believe me, I understand how hard that is! But our community depends on it.
Sometimes I think that HN threads are a big experiment in all of us learning how not to get triggered. If you give better than you "get" (by reading someone's comment), you'll be on the right track.
If you'd told me this in an interview, I'd have hired you on the spot. The biggest issue with cheating, IMO, is that it can indicate laziness, but in your case it definitely doesn't - the story is great and says you're a highly creative person who gets things done and doesn't have to follow norms.
The headphone was a very small analogue receiver that had it's own micro-battery, an inductor and amplifier - all about 4mm wide and 8mm long. You would place it deep inside the ear channel and it was invisible even to a person standing right next to you. The signal was induced by a coil you would wear around you neck and you would pipe raw audio into it.
Personally for theses kind of things I used to use one earphone in my left hand, wearing a long sleeve shirt and laying my head over my left hand, covering the whole operation :)
To work as a (commercial) movie, the stakes would have to be phenomenally high. I can't imagine the scene without the viewers saying "he did all that ... just to pass a test he objected to?".
IMO if you can get around a system that's as good as going through it. It usually takes a certain kind of ingenuity to be able to do, so I can't be mad at it.
It's interesting to me that everyone here seems hellbent on ethics yet at the same time all of the big companies that we work for are some of the most unethical beasts in modern society.
Cheating on an exam does not always mean one has 'lack of ethics'. It may mean the opposite - say, "my parents aren't paying out of their veins for me to waste time on this nonsense."
I am not talking about myself. Young people are strongly forced to pass the exam by multitude of strong incentives, including societal pressure and prospects for their financial well-being.
A few weeks in total, but it was a labor of love so it was quite fun. The hard part was reading all the lectures out loud, it took about a week during which my family, probably hearing my from my room, was impressed of my new found and never since matched learning zeal.
If it is exam about computer programming, then I might say, that is very good and I will not disqualify you for cheating in that way that you have managed that (although still the answers must be given correctly, like any exam).
That's quite impressive setup. I've used programmable calculator with text storage capability for my chemistry exam. I was correct that I will not need chemistry knowledge in all my life.
I was also cheating on my chemistry exam, but as your life is over yet I wouldn't be so sure.
As I'm getting older it's getting more and more important for me to understand the biochemical processes that are going on (or going wrong) in my body.
It might be a good test of a company to see if they were impressed and amused enough by this story to hire you because of it, rather than rejecting because of it.
This was more of an introspection, but i used to be rather depressed. When I started reflecting on life and all I could contribute or succeed at (this was actually years ago, not specifically this question). I realized that if someone 1000x more intellect / capable / knowledgeable came along, everything I could add or solved from an intellectual perspective would be useless.
After meditating / contemplating this for quite some time, we are talking the order of months. I realized that even if someone knows 1000x more, they still may not know your niche. More than that, everyone has niche expertise from their lives, which no one else can access (from their experiences).
I don't know how I got to that conclusion, or how to explain it. It was the hardest problem I solved, because it's something that had to do with a change in perspective and personality. It also is what helps me listen to others in debates and to be much more open minded. I wouldn't say it's that big of a deal, as I felt it was part of growing as a person, however many around me don't seem to recognize that. When I hear "they are naturally gifted" or "I'm not smart enough", I feel those are people who haven't had the same realization (i.e. experience).
Anyway, that was probably the most difficult and profound problem I solved for myself. What motivates me.
> When I hear "they are naturally gifted" or "I'm not smart enough", I feel those are people who haven't had the same realization (i.e. experience).
They may simply not agree with the conclusion, or the end. It's pretty hard to have this discussion without crashing into the nature-nurture debate, but one could be interested in a particular result, not merely _any_ result, and they may perceive a lack of something blocking their way towards that.
Not to mention, we do not really live in a world / society that recognizes this, and social approval + hireability dramatically affects one's psyche, more so than mere rational thinking.
I had one similar realization but more focused on learning.
"What's the difference between an expert and me?"
Like a tennis pro, or a recognized programmer (besides muscle memory)
"Information"
Even emotions could be seen as a state of certain information in your head. Even your ego.
"We are not that different. I can learn anything."
This doesn't count efficiency and optimization. It was just that given enough time and resources we are capable to "do something".
It's hard to learn that hard work will beat aptitude for a skill and that we actually have limited time in our lives though. But as you mention it makes a difference in the mind of people. The mental blocks many people have against learning new things really hinders how people interact with this rapid changing modern world and with their social groups. Also, there's these kind of realizations that humbles you (or lift your spirit up) and helps you being less egocentric, more empathic and forgiving.
I think bringing down these walls is a really hard problem.
I once told a room mate in college I would repair their laptop's charging port if they were willing to pay more per month on 50 megabit internet (this was a lot of bandwidth back then).
I ordered a new port online, waited for it to come, then spent like 12 hours trying to get the factory solder out of the original port. I ended up accidentally frying part of the power board. By this time it was the middle of the night and I had class the next day, and my room mate expected a working laptop in the morning.
So I started trawling Craigslist for similar laptop models, emailed every single one of them that I would buy the laptop at 6am (enough time before class). I found one with a broken screen, and got a decent deal, I think I still spent $150 or something (almost a whole paycheck at the time). I brought it home and tore it open, took the entire power connector module out of it and into the new one, it fit thankfully.
I handed my room mate her working laptop and never mentioned the ordeal... It was still working 2 years later when I moved out of that house. The problem wasn't so much technical as a problem of desperation and saving face :)
Fixing a SRAM 11 speed GX rear shifter on my mountain bike. I was on a downhill MTB holiday and fell off, snapping the thumb trigger on my rear gear shifter.
Replacing the shifter is usually like a £30 part, but I was in an alpine resort in summer so the shops that stocked them were charging nearly £100!
I had my tools and some super glue so sat down to repair it. The snapped part fixed in two minutes but getting that part in place required the complete disassembly of the internal mechanics of the shifter which was all tensioned with multiple hidden springs, the internals literally burst apart half way through carefully taking it apart.
We had no instructions, no YouTube video (since it is usually cheap enough just to swap out) and no diagrams.
It took me and a random guy staying in the chalet 3.5 hours to put it back together, we essentially had to spread everything out and think from first principles of how a shifter works, the specific features of that shifter (rapid fire, mutiple down shifts with one leave arch) and build up how those peices would match how it operates.
Must have been 30 part, all small and all tensioned with three(?) springs.
6 months later still working fine but man did we get a mental workout that day.
Next day I had to fix my 3 axis gimbal but that was a lot easier.
I empathize! These kinds of shifters are SUPER hard to put back together. Quite literally, so many moving parts in such a tight space. If you inadvertently bend the springs or coils too much, you could end up ruining the whole thing.
I had a similar situation happen to on a Shimano Deore shifter, which is even less complex than the SRAM you mention.
For others who have never had this "joy", it looks like there's at least one video at least partially showing what the OP had to go through: https://www.youtube.com/watch?v=nrfKQfXJgd0 Jump to ~2:00 to get a sense of how finicky this stuff is.
That's pretty funny and I'm glad to hear something other than another barely believable "I did this god mode hack" story... The mechanical realm doesn't get enough love. Indexing shifters have become better, but progressively more miniaturized (and ironically flimsier IMO) over the past 30+ years. Good work!
Curious if your hourly rate at work works out at more than £28.5 per hour, as that's what your time would have cost. Saying that I imagine the satisfaction was worth it. (I do a lot of my own bike repairs and a 5 minute youtube video usually works out as closer to an hour or two when I try.)
I see where you are coming from but since I was already on holiday it would have been pure cost. It’s not like I could have spent that time earning money, I would have just come back from France feeling over charged for something that it turned out I could fix myself.
Are you making YouTube videos of your mountain biking adventures or just videos for fun? I would be keen to see them as I am getting into it now myself.
We had been planning on doing a couples MTB channel as my wife was looking for something outside of work to make up for a lack of interesting projects at work. In the end she landed up getting a job at Aston Martin and now I just use the gimbal for fun with our friends when riding.
I recommend BKXC and Seth’s Bike Hacks as good channels that avoid all the Gnar downhill stuff
Those are a couple of my favourite channels! Mountain biking for the everyman. Singletrack Sampler and BCPov are a couple of my recommendations. BCPov does some pretty awesome bike treking videos too, with camping and all the other stuff involved with multiday rides.
I started with the glimmer of a hope that perhaps network sync could be made stateless, went down a months-long rabbit-hole of research, and ended up writing a novella-length article about CRDTs: http://archagon.net/blog/2018/03/24/data-laced-with-history/
So many days spent thinking, sketching, trying to swallow a concept that seemed far to big for my jaws—only to suddenly find myself on the other side, with this arcane knowledge fully internalized.
Going from a few wayward thoughts to a working proof-of-concept was the most professionally satisfying thing I've done. It felt like alchemy.
That might be mine as well, https://medium.com/@raphlinus/towards-a-unified-theory-of-op... . It's very much unfinished work though. Certainly wrapping my head around the OT and CRDT literature took huge amounts of brainpower exertion, comparable to what I did for my PhD.
Also, I have your essay in my queue to read more thoroughly, it's high on my list of stuff to study as we possibly rethink the way this stuff works in xi-editor.
Hey Raph, thanks for the vote of confidence! I spent a lot of time looking through your research on OT and CRDTs (plus your Rope Science series) in the course of writing my article, especially as documented and implemented in xi. Really fascinating stuff. I love how many different ways there are to approach this problem.
I found it a few months ago, read it through the end, and shared it with my team. It's the best survey of synchronization techniques that I've come across.
Dont know if this is the hardest, but it is one of the most impactful. During the 2010 recession I had a client that had about 700K in receivables past due and we were doing about 120K/month in additional work. I started reading their 10Q and 10K and decided that they were at risk for bankruptcy as their cash position was not great.
I told my sales guy that we had to collect our money somehow. He was like they are a big public company, there is no way they will go bankrupt. I insisted.
I told him we needed to start working their procurement and AP hard. We bought gifts, took them out to golf, bars, lunch, dinner, and told a ton of sob stories about how we needed the money. We managed to get our AR down to about 120K which we decided was acceptable losses.
They went into bankruptcy and I was contacted by a receivables purchase company who offered to buy our 120K for 40K, so I sold them immediately.
Many of the small companies that we worked with there went out of business as none of us could take such a huge receivables loss.
To this day I try to get my finance team to send gifts to the AP team of our clients.
> To this day I try to get my finance team to send gifts to the AP team of our clients.
1. Many, many big companies require vendors to agree to codes of conduct that explicitly prohibit such gifts.
2. If the customer were to file for bankruptcy protection, the trustee (or debtor-in-possession) likely would characterize recent payments as "avoidable preference" payments and seek to claw them back — with claw-back being the presumption and the vendors having the burden of proving that they're entitled to keep the payments (which is somewhat of a PITA). [0]
3. If the recipient of such a gift happens to be a foreign "official," then the criminal penalties of the (U.S.) Foreign Corrupt Practices Act might become salient. [1]
In my experience, the vast majority of companies will not be reporting 'dinner' as a gift, probably not even golf. Maybe 'material gifts' i.e. things that are directly gifted that have value, but even a small basket might not get reported.
When I was 14 or so, my mom installed one of those "telephone locks" that blocked outgoing calls and only allowed incoming calls, she did this to limit my dial-up internet. I wanted to see if I could bypass it without my mom noticing.
I noticed that the case for the lock could be pried open easily because it was just a plastic cover with 2 tabs. I examined the circuit and saw that if I could bridge the cables around the lock via a toggle switch, I could have an open phone line during the day and then toggle it at night when my mom came home and she wouldn't know. And so I did. I felt like Kevin Mitnick. I remember documenting the process and posting it to hackaday.com
Funny thing was that my dad found out but he didn't tell my mom. He would just go "hey can you make the phone work, I need to make a call". I later found out he was on phone call restrction too.
That's pretty good. With the device is there some sort of emergency way to split the lock in case of emergency? It would be unfortunate to be unable to make outgoing calls if you needed to call 911.
I eventually solved a bug that took about 1.5 years to figure out, since we were not able to reproduce it, and it only happened on a customer's system. Long story short, it ended up being a by-product of sending a (256*N)+1 byte packet through the system, and the fpga asserted a signal 2 clocks later that did not assert in time on those sizes. This resulted in a single buffer leaking, but eventually it built up exponentially. Many days and nights of a logic analyzer and other equipment led to a single accident in the lab where I was able to reproduce it.
It's hard to describe the feeling of reproducing something like that, after haunting me for so long.
I feel you. A bug in our recovery software would result in the live CD being unable to open the Windows kernel file after having located it on disk. No amount of logging we added to the code made sense. In the end, after more than a year, there was a customer in the UK that could reproduce the problem each time on a clean Windows 10 install (so no private info). I mailed him a new SSD to replace his old HDD and paid for him to overnight me his hard disk. Problem solved the very next day.
I think that's the craziest one I've heard so far. I can't imagine trying to reproduce a hardware bug in an FPGA with a logic analyzer.
How much time did you spend on the bug? Was it something that you ignored for a long time, and then you decided to dive in and spend a couple of weeks on it?
It was about 8 years ago, so my memory is a bit foggy. At one point we had everyone on our entire project working on it, which was about 20 people. We had daily calls with the customer since it was causing frequent outages on their network. I was full time on the bug for a while, and most of that time was spent making meticulous documentation for what happens in every clock cycle in the fpga.
I visited the customer's site about 3 times during that period with a very expensive logic analyzer. By that time I'd added a ton of debug code in the fpga, and the logic analyzer was set to trigger if any of those counters hit. A big breakthrough was we were counting packets at each point in the pipeline, and that debug was showing that at a certain point, the counters were off. This was huge since it was the first clue into what was actually happening. Prior to that we just knew that there was an exponential increase in retransmitted packets until the whole thing came crashing down.
Looking back on it, I'm really happy I ended up writing a large post-mortem with screen shots of the logic analyzer and everything. I can go back and see exactly what caused it, which is really interesting even today.
Wow, that blows my mind. I don’t even know how to ask the right question, but is there some kind of “formal verification” or “type-checking” that could have caught this issue earlier? Or is that just too difficult, and reserved for medical equipment, nuclear reactors, etc.?
It was a problem with a clock domain crossing with IP from another company into ours. There were testbenches/simulations for verification, but they didn't test every packet size. In hindsight, everything can be caught, but it's a matter of how extensive the testing is. For example, passing every packet size through would not have showed the symptom. Everything looked normal, and all packets succeed. The key was that the meta data saying whether a buffer was free after transmitting was incorrect and never showing that it was free. So to really see the issue you had to send enough packets of that size to totally deplete the buffers. With a standard packet distribution, such as imix, you never hit that size.
Thanks a lot for sharing this story! I started reading about clock domain crossing and metastability issues, and for the past few days I've been thinking a lot about circuit designs and FPGAs.
I posted on /r/FPGA on Reddit [1], and I asked about the best development board to start with. I was wondering if you had any thoughts on that? I think I'm going to order a snickerdoodle black [2] and a breakout board, and start learning VHDL or Verilog. I think it would be a lot of fun to eventually design my own soft-core processor and compile a Rust program that runs on it. And after reading about clock domain crossing issues, I think it would also be really interesting to experiment with asynchronous circuits and clock-less CPUs. Thanks for introducing me to this!
Sorry, I can't help much with the dev board side of things. I mostly work on sw these days, and the fpgas we use at work are some of the largest you can buy (stratix 10). I can ask around at work if that would help.
That looks really interesting. I've never used any high-level languages, though, so I can't comment. Most of the fpga developers I know somewhat despise those higher-level languages since you lose a lot of the control you have writing the HDL. They also tend to take up a lot more resources in terms of gates and memory, but things may have gotten better.
I mentioned in a previous comment that I actually have an entire detailed write-up of the bug. If my work lets me release it, I wouldn't mind transferring it to a blog.
I’m surprised and slightly disappointed that my memories don’t bring up a clear answer to this.
The hardest problems I’ve faced were never exactly solved, just moved past or muddled through. Things like loss of close friends and family, acknowledging my own limitations, and accepting the inertia of flawed institutions.
Any problem that eventually found a solution I remember as feeling relatively simple in retrospect.
Agreed on the technical problems- the "hard" problems in my career have been massive degradations/failures with no error messages, which are typically solved with liberal application of strace, gdb/pdb, iproute2/ss and Wireshark. Once you find the smoking gun in the kernel or on the wire then it becomes simple again.
I guess the hardest problems I've ever "solved" (mitigated) were mental (depression/anxiety/impostor syndrome). But those aren't one and done happy endings- they're a bundle of general anxieties that are never fully resolved. You just learn how to manage.
Finding a solvable hard problem is itself very difficult. If the problem is within your confort zone, you will never feel that is an hard problem. If it is too far from your confort zone you will not be able to solve it.
I was working on a calendaring system. We supported recurring events, and each event could also have a piece of equipment associated with it (one or more). Events could also be changed, either the current event or this and every event. Event start/stop times were stored in UTC, but the event was displayed in the users timezone, and could cross daylight savings time boundaries. This was done with a mix of javascript on the front end and ruby on the backend.
When first implemented, the equipment was reserved for the entire time of the event. A feature request came in to allow for partial reservations of equipment (we want to reserve space A for four hours, but equipment B for only the last hour and equipment C for the first two hours).
Solving it on initial event creation was pretty easy. The UX was difficult. But handling updates, especially across recurring events, in a way that was maintainable and correct was the hardest technical work I've done. I wrote a lot of tests.
The difficulty was compounded by the fact that this was a startup and I had no technical peers to discuss the issue with. The code implementing the work wasn't the cleanest either. I could have reached out to friends, but they wouldn't have had the understanding of the issue.
How did you handle the database storage for recurring events? Did you have just one entry that represents the recurring event as a abstract whole, or a database entry for each instance of the recurring event?
The former seems "better", but you also run into a whole lot of complications:
1) The user can delete specific instances of a recurring event. Eg: delete the event for thanksgiving Thursday but leave it intact for all other Thursdays
2) The user can make instance-specific edits. Eg: edit the event description with custom meeting-notes that's specific to that week's instance
3) The user can invite/dis-invite people for specific instances. Eg: invite Dave only for the event this Thursday, but not for the following weeks'
Creating a database entry for each instance avoids the above complications, but comes with its own drawbacks: if a user creates a recurring event with no end-date, you have to populate a large number of instances, all the way until some arbitrary MAX_DATE.
I was asked this question in an interview once and I recommended the latter option as the lesser evil, and the interviewer was visibly displeased with my recommendation. I'm wondering what the better solution would be.
Matterlist's recurring task descriptors ("Schedule Items") essentialy define infinite sequences of recurring events, which are not represented in the database, but are visible and editable via the same UI, just as the regular, non-recurring tasks.
Here's how we deal with the problems you outlined:
1) When a user deletes a specific instance, we create an "exclusion" record for that day, so that the function that generates an endless sequence of recurrences doesn't generate one for that day.
2) When a user edits a specific instance of a recurring task, we also create an "exclusion" record for that day (because otherwise we would have two tasks on that day, the edited instance, now with a database record, and the virtual instance), and we create a database record for the edited instance, so it just becomes a regular task.
3) We don't have shared tasks yet, but that would be handled via the same framework, as outlined in 1 and 2 above. We'd just de-virtualize an instance and add/remove users to it.
Creating a database entry for each instance of a recurring task was unacceptable for us, because we wanted infinite forward visibility, and because when a user edits the parameters of a recurring task (e.g. changes it from daily to weekly), we'd have to delete all the old records and create the new records.
BTW, we have finally released an Android app, so you can see our recurring tasks in action: https://matterlist.com/#apps
I went with the latter. We limited the total number of events to something like 5 or 10 years to avoid it [edit: the max age problem]. The only real issues that came up from that approach were:
* UX was hard when people want to change individual events that are also part of a recurring event (who wins)
* daylight savings time is an issue when you store datetime in UTC (you need to shift the UTC value in Nov vs in June to have it be the same "8am every Friday" event)
* saving a change across all of a large number of events created a large, slow transaction that caused slow responses (in some cases triggering heroku timeouts)
I was lucky that we didn't need to implement inviting guests, as that wasn't needed.
There were times when I wish I'd implemented the RFC standard: https://www.ietf.org/rfc/rfc2445.txt but never got to that. (We also evaluated just building on top of Google Calendar, but I was worried about building on top of an API that we weren't paying for, and there was the issue of tying in child equipment events.)
Some very quick thoughts.. Have something to resemble the event series as a single instance - then have 1...n relation to actual occurrences for occasions when there has been a change, added notes or cancellation. These should be created when the change/added notes/etc. are made.
I would say store a default event together with the pattern it repeats in. Then create exceptions that hold additional info or a cancellation when needed. When they finally add an end date in one way or the other, add the end date to the default.
On load/view time, generate events from the list of defaults and amend with the exceptions as needed. The user should not see the default/exceptions system, just events and the pattern.
I’d say working with exceptions in this way is more efficient than storing many copies of the same event.
Outlook/exchange uses the former approach, a pattern with a list of instance-specific exceptions. One reservation system I worked on used the latter approach, with every instance saved to the database. I was tasked with building an outlook plugin to smoothly integrate the two, and it was probably the most difficult piece of software to get working reliably I ever wrote. The prototype took a few days, the fully reliable version compatible with all outlook versions took two years. Outlook’s internals are pure madness.
I have worked with recurring events, and the answer that I've seen is generally both. You have one record which is the "template" and then populate the database out with specific instances to some arbitrary date. Every so often then run a cron to populate further and further out.
That's pretty much what I did, but the original record wasn't stored in the DB, just the specific instances. Our use case was such that folks were moving around events periodically (it was a scheduling app), so we rarely had anyone "run off the end" of their repeated events.
The hardest thing I ever worked on was a calendaring system for scheduling ads on/off. The UI was great, but I just told them "oh, send me a cron format string for scheduling". It was easy for them to do, but very hard to handle on the back end. Cron format strings aren't fun to reverse engineer. Why did I do that?
File under "things programmers assume about time". It kept doing "weird" things from the user perspective and I ended up writing more tests for that subsystem than anything else I've ever done.
Years ago I wrote an emulator for the Intel 8086 processor in C++. It's deceivingly difficult because the instruction encoding is complex and the emulation of each instruction has to have a very high fidelity. In a sense, software at the CPU instruction level is a chaotic system, as each instruction can influence the system state to a critical degree, so if there is a slight deviation from the spec/hardware, a snowball effect of deviations happens that leaves the system in a completely botched state where your emulator won't boot at all. Because the deviation can happen anywhere within an execution path of millions of instructions, it's very hard to debug, too.
Eventually I got it working and I could boot DOS and play games.
The aim of my project was to create a programmable emulator that could be used for the semi-automated analysis of malware, and sell it. Eventually this didn't really go anywhere as this was a too ambitious goal to do all by myself. See here [1] for a demonstration where I load tweets from Twitter and send them to the DOS text editor by triggering a keyboard interrupt via a Python API. Fun...
Later, the Unicorn CPU emulator framework implemented my idea much more effectively by creating Python bindings to an already mature emulator (QEMU).
For me it turned out to be jettisoning all of the stuff from my life that didn’t feel meaningful and reorienting my life such that all of my major time commitments positively impact things that I consider to be personal values.
This includes: hobbies, non-profits, and my job.
This does not mean I live an ascetic life. Instead, I live one that I personally find meaningful.
Probably not hardest, but one that sticks in my mind was that I downloaded an old copy of Jedi Knight II onto my computer and was excited to play it again... but it crashed on launch. Or rather it would throw a modal dialog (complaining about graphics something-or-other) and go into a loop of system beeps.
So I attached to it with gdb... and started going down a rabbit hole of stack traces and assembly code. I don't remember ultimately what the problem was (I think the code had been (partially?) stripped of symbols) but I do remember ultimately flipping a bit in some conditional that finally got it to launch!
One of the hardest challenges I have had in my career is convincing our company to move to Continuous Delivery. 90% of the challenges weren't technical, but emotional. Shipping software comes with lots of feelings, fear, politics, etc. I had to personally work with various leaders across the org to help them through these feelings and perceived blockers.
We aren't 100% there yet, but we are shipping numerous times per day across 20 or so services and quality has gone _up_, not down
That is interesting because I am still trying to figure out how people do CD. Of course I know CI, we have it all set up. But we still work in sprints with manual testing and release every 2 weeks.
I could spend time on marking features 'frontend only, low impact' which we could deploy pretty much the same day. Still there are quite some features that need bigger amount of work where they might be 'done' by dev but I am sure they are not actually done, because security, because error checking. It of course is usually that one dev has his not tested feature merged to develop and then also some other dev has production ready one, then if one feature is production ready, I would have to put also some time to make release and pick only changes for accepted change. I am not sure that additional work to check what we can release ' right now' will pay off vs just taking time for fixes on acceptance by people who were working on code and then release it (after 2 weeks o 1 depending how fast it is done in sprint).
So are you having people who work only on picking changes that are low impact, or working on making stuff production ready by picking from develop?
Maybe you pick changes by yourself, or you just defer manual testing to end users and rely on automated tests unit/integration?
p.s. Funny thing with automated tests is they are good at keeping old stuff working but not for testing newly developed features where actual tester can test new GUI/ new features. If you have a lot of GUI changes you cannot automate first tests...
A couple obvious things stand out from your situation. First, only allow merges of code that's been reviewed and has enough tests along with it. Second, have a pipeline that automatically runs all the tests whenever you merge, and if they pass, then goes on to automatically deploy. It's really that simple. It's not easy to get to that point, but it's simple.
Mostly it comes down to organizational changes and everybody getting used to what constitutes "enough" tests.
It was pretty important to work with the Project Managers and "scrum-masters" to decouple our release cycle from our sprint cycle. In our case, like yours, they were coupled together for no technical reason. It's hard to sum up all the changes we made to allow it to happen, but mostly it boiled down to a few technical decisions -
* Every PR is treated as "production" ready. This means if it isn't ready for user eyes, it gets feature flagged or dark shipped. Engineers have to assume their commit will go into prod right away. Feature flags become pretty important
* Product Owners and QA validate code is "done" in lower environments (acceptance or staging, or even locally during PR phase). This helped us decouple code being in prod and "definition of done" in our sprints.
* All API changes and migrations follow "expand and contract" models that makes the code shippable. e.g. even if we are building new features, we can ship our code at anytime cause the public api is only expanding.
* More automated quality checks at PR time. Unit tests, integration tests, danger rules, etc. These vary from codebase to codebase. A key part of this is trusting the owners of that code or service. To a degree, if they are happy with their coverage, then they can ship. (Within limits of course. Not having unit tests at least would be a red flag)
Also, we still ship numerous times a day without a full automation test, we just make sure each release is really small (1-3) commits. The smaller the release the easier it is to manually QA it. So nothing fancy is needed, just smaller releases.
So to answer your question, we don't pick and choose work that is "low" impact or "high" impact, all code gets shipped the same and with the same cadence. It is our responsibility to ensure that when it goes to prod, it won't break anything
I have come to be against continuous delivery - in 'whole product terms' there are vast hidden expenses.
In particular - documentation and support.
Using complicated interfaces is very challenging sometimes, to the point of obscene - Google searches for help turn up a variety of outdated answers - and it's impossible to know what's what.
I'm using Facebook Ads right now quite a lot - it's complex system that just shifts like quicksand under your feet.
Users make incredible efforts to learn the product, only to have it shift away from them like a ghost.
Documentation may or may not be up to date.
Locations of things change.
And who can you ask? Where do you search for support? Facebook has never really provided me answers to many questions in their documentation.
Tiny example - an advisor used to know how to list people that have commented on a post, so that you could 'invite them to like your page'. But it's changed and now he can't find it. One small thing now that he doesn't know how to do, a tool list from his tool-chest.
And who really gains from all of this? Seriously? I don't think anyone.
Is that 'new feature' really that important? It needs to be rolled out 'now' instead of in a major/minor delivery?
These changes are seldom well communicated either.
I suggest the total opposite might be better: release major iterations every year, minor iterations quarterly, and patches as needed.
Every time there is a release - provide users friendly release notes - something they can read at a quick glance and get up to speed. A little 10 second video for every change: "Oh, now you can do XYZ like this"
This way - you have predictability and consistency so users can know how to 'keep up'.
Also - the ability to use the OLD interface where possible for at least 1 year, or something like that - that we users are not forced onto the quicksand.
"Ok it's Jan 1 - FB Ads 2019 is released in 1 month - let's go over the changes - Mary, you can be responsible for highlighting the major changes and communicating them to the rest of the team, and highlighting any risks for us"
Otherwise, you're trolling along, these companies make changes that could feasibly have major impact on your business and you're out to lunch.
Maybe internally continuous delivery could be a useful thing ... but for the world at large the downsides are real and the upsides are limited.
For my thesis project back in 2011, I wanted to create an interactive installation, the kind you see at museums. Since multitouch tables were insanely expensive to buy or rent, I decided to build one myself. Learned the basics of woodworking over a semester, and built a table after two failed attempts. Then I bought an LCD TV, took it apart, connected it to a playstation camera with a modified infrared lens and developed an Adobe Air application. The thing was so unresponsive to touches, I decided to ditch the whole multitouch table idea. Started from scratch again ... learned objective-c (iOS 5 or was it 6), and created an iPad app for my thesis. Don't think I've ever worked so hard in my life.
There's actually a quite nifty principle that makes it relatively easy to build your own multitouch table: Frustrated total internal reflection.
The idea is based on internal reflection: Whenever lights hits the boundary between two materials with different density it gets refracted. If it goes from dense to less dense and the angle is flat enough it will reflect back. This is the principle behind fibre optics. It's also the reason why the water's surface looks "silvery" (like a mirror) when you're submerged and look up at an angle (i.e. not straight up).
Now imagine you have a plane of glass and you put LEDs around the edges that shine into it from the side. The light will zig-zag trough the glass and come out at the other edge. However, if you touch the glass you inhibit that total internal reflection because your finger is alot denser than the air and so the light will "leak" out the glass where you touch it, illuminating your finger. If you look at the glass from behind you'll see a bright spot.
Use a camera to detect that spot and you basically have a touchpad. To make it a screen you can put a translucent sheet behind the glass and project an image onto it (and use infrared LEDs).
See e.g. http://wiki.nuigroup.com/FTIR for some helpful images. Just google "FTIR touch screen" or similar for build instructions and blob-detection software.
The hardest thing I’ve ever solved is finding 30 years of motivation. I co-founded a family business, or it co-founded me, because I was only 14. From inception to exit, it was an interesting intellectual and emotional challenge to keep 50-100 people motivated at any one time, including me. Coding was fun but talking to an angry customer was less so. Eventually, survival instincts kicked in and taught me that creativity and learning was the solution. Everything happened for a reason, and that reason was usually hidden under multiple layers—for staff, customers, or me. I found I became motivated by avoiding reaction, and instead seeing that there was already a motivation behind every interaction, like boredom, anger, and ambivalence. Understanding these individual motivations provided clarity as to what needed to change to maximize team motivation. This made hard problems palatable (and motivating) for an old-school techie, like me.
>> Everything happened for a reason, and that reason was usually hidden under multiple layers—for staff, customers, or me. I found I became motivated by avoiding reaction, and instead seeing that there was already a motivation behind every interaction, like boredom, anger, and ambivalence. Understanding these individual motivations provided clarity as to what needed to change to maximize team motivation.
At the risk of over generalizing it sounds like you learned to empathize. Sounds like you found that to be a key to success!
I had designed a PCI chip. Our driver team reported that the PC would hard hang under certain conditions. They suspected a bug in the chip.
Armed with a PCI analyzer, I figured out that it was a race condition where the firmware on the chip would raise an interrupt to the PC. And they would clear the interrupt right at the moment where the firmware wanted to issue a new interrupt.
If performed in the wrong order, the PCI interrupt stayed high even after the PC thought it was already serviced.
The solution was to just switch around 2 lines of code in the firmware, but it took two weeks to figure that out.
Not that long, but it was an insidiously subtle mistake.
This isn’t the hardest problem I’ve solved but I was damned proud of solving it when I did...
Was about to tape out a chip when we got a last minute change order... a 100kohm resistor needed to be added. This was an already laid-out, routed and optimized chip design that was ready to be fabbed.
A few hours later (including at least an hour of shouted obscenities) I found that I was able to snake through 100kohms of poly and diffusion resistor into our design without moving anything, violating any design rules or changing any operating points even after parasitic extraction.
I don't know if it's the hardest, but very recently I had a problem that drove me nuts: I was doing some consulting for a company that designs chips, and they assigned me a project to take a nearly completed design and augment it with additional wires to route extra power to some parts of the chip that were running on the hairy edge of not having enough current. So I had to read in the design spec, search it for available space, and generate new wire geometries for the power grid. This sounds straightforward, but there were two things that made is incredibly hard. First, the design that I was reading in was the output of a routing tool that was provided by a third-party vendor, and it was producing some really crazy shit. Not even the senior design engineers could understand some of the things that the router was doing. And second, there are hundreds of design rules that the new geometries had to conform to. Most of them are not relevant, but about a dozen or so are, and some of them involve incredibly complicated interactions of multiple geometries, occasionally across multiple metal layers. Trying to keep track of all that and make it all run in a reasonable amount of time was quite challenging, to say the least. But the worst part was that the development cycle involved a manual step where the output of my code had to fed into yet another third-party tool to see if I had violated any of the design rules, and the output of that tool was graphical. And I was not able to use the tool because of licensing restrictions! So I had to take my output, give it to someone else, have them run the tool, look at the results, and send me a report of how many rule violations there were, where they were, and what rules were being violated. That took several hours, sometimes multiple days.
1. Debugging a non-deterministic crash in a huge and complex C++ app. It took me a few weeks of super-focused hard work (kudos to my then team & management at Sabre office in Kraków for giving me this time, and understanding and appreciating what I was doing) to solve, with lots of low level debugging down to raw disassembly, as well as code reading and intense thinking and thinking and thinking and thinking. Eventually I managed to narrow it down to a memory race in advanced template code. Some time later I realized it scarred me to C++ (my first love in programming) for life, also opening my eyes to what horror Undefined Behaviors in C++ mean and how they loom ominously over every single seemingly simple character of C++ source code anyone writes. I'm now working in Go and appreciate it so much, as well as Rust etc. It also was one of things that made me appreciate what good management can mean to an employee.
2. Debugging Go runtime in pre-1.0 era, to find out why it was not working stably on Windows. It involved a lot of digging through Go internals (also in Go's old C compiler & assembly), as well as WinDbg disassembly-level debugging. I eventually found out that the Windows call convention was not preserved in some aspect, in that one of the registers was being mangled on return. I see it as my most important contribution to Go, though formally I was not listed as a Contributor through it, though I got a Thank You mention in a commit message I think. But some time later I did some much simpler contribution which got me a then coveted by me (and still valued) entry on the Go contributors list.
* Improving that design to make it work reliably and conveniently enough for mass production and deployment in 3rd world countries (still haven't figured it out :(
* Porting an entire automated Windows PC system repair suite from Windows to Linux (still to fix Windows PCs) in two weeks when Microsoft pulled the rug out from under us and abruptly informed me that they would no longer be licensing Windows PE to ISVs: https://neosmart.net/EasyRE/ (now powered by FreeBSD)
Thank you. It's an ongoing struggle that I haven't solved yet. It can be very tempting and fun to just switch to a new project when you reach the point where you know what needs to be done but it's an insane mountain of work that you either dread (e.g. you need to rewrite an entire project from scratch to take a different approach) or can't even do at the moment (e.g. you need precision machining beyond what is available to you).
I get a thrill out of besting myself, and find that helps extremely well when I've all but given up hope but conversely that makes it really hard to motivate yourself when it's something you know you can do (perhaps you've done it before) but just aren't looking forward to. Also it's really hard to do something that takes so much insane time/effort (literally years) when you know there's a good chance the world just won't care.
When you're younger, the problems are harder, and the success sweeter.
I earned my spurs (well, perhaps an enlisted stripe) back in the 1980's when I inherited a non-functional mess of an HP-85 Basic program to control an antenna test system. Many days in the Hanscom AFB RADC EE Propagation shack to get the RS-232 mag tape and IEEE HP-IB/GPIB/IEEE 488 custom hardware working. We kept bumping up against the HP-85's program storage capacity, so I had to remove most program comments, and actually shorten the variable names to get it all loaded into memory.
Then we got to fly that sucker on a pair of C-141B's from Wright Pat AFB, to Elmendorf AFB, then a flight over the North Pole, then to Thule AB, Greenland (9 days in November - do not recommend), then to RAF Lakenheath.
30 years on, I still make a living fixing crappy code.
Joining my current company and saving a relationship with a key client from the brink of termination.
The previous engineer on the account had overpromised and under delivered... in a rather big way. Not only that, but he also managed to misconfigure the enterprise automation software we use and accidentally delete the customer’s entire network share, then managed to get one of those cryptolocker viruses on the share after it was restored from backups, prompting a second restore. Regardless, the customer was pissed as hell and was on the verge of firing us.
After I was hired and put on the account (which I knew going in), I spent two days taking stock of all the shitty stuff the previous engineer had built, deciding what could be salvaged and what had to be discarded. Then, for the next two months, I went over the original contracts and built (or rebuilt) the originally promised system. I would basically work for 10 hours everyday, 7 am to 5 pm, then write an extremely detailed report to the customer explaining what I had done that way, which would take another 2 hours.
For the first week they ignored my reports. Then the CFO started taking interest, first by nitpicking, then asking questions, then praising. The rest of the customer’s team slowly came on board as well, especially once I started delivering the bits that were user-facing.
Today this customer is our flagship account, and has helped us land many other large accounts. But those two months were probably the most stressful of my life, not just because of the technical challenges but also due to having to simultaneously navigate the tricky political situation.
I solved some interesting technical problems this year. One was optimizing a script that took over eight hours to run, to just twelve seconds (validation code brought that back up to several minutes, but still a lot better than eight hours). Another was taking several map pins whose coordinates were based on the same physical address (just different suites/floors, and having the same exact coordinates returned from the API) and staggering them. I ended up using a spiral pattern to do the staggering.
An ongoing soft-skill problem I am learning to solve is the effective customer development.
An interesting UI/UX problem I am currently thinking about is how to allow people to draw sun/shade patterns, wind current patterns, and essentially contour lines that show elevation on their property (the last is to ultimately learn how water flows on their property, and where there are drainage problems) in a fun, minimal-effort way. If anyone has any suggestions on this, please feel free to share.
The last two types of problems are for my side project, AutoMicroFarm.
IMO the most fun way to enter elevation data for your property would be to ride around on your farm bike with an altimeter lashed up to a smartphone, gathering data as you go.
Thanks for the idea! I found a variety of altimeter apps for the phone; I'll have to try just walking around my yard with one (or several), then see if it's possible to get elevation data on a map from such an app.
I'd suggest getting an app like Sensors Multitool for Android which gives you the raw output of your phone's sensors. I tried one of the altimeter apps a few years ago to see how tall a big hill was, and the value that the app reported turned out to be far off the actual value. There are various techniques for mixing and cleaning data from the GPS and barometric pressure sensors and a premade altimeter app (if it's not open source) obfuscates what's really going on with the values you'll actually end up seeing if you create an app for your microfarming project.
Also be aware that different phones have different levels of support for non-GPS location services. I was surprised to learn recently that my Pixel 3 can use data from the US, EU, and Russian satellite geolocation systems for better coverage/precision.
Thanks for the tips! I don't need accurate readings, just precise ones (so the map can be internally consistent; here's what I drew manually for my property: https://i.imgur.com/2ZGmB1M.png). I'll keep in mind what you suggest when I am ready to develop that part of the app.
Well, I tried both Accurate Altimeter and Sensors Multitool, and I must say I'm disappointed. The values varied by 20 meters! In reality, my property varies by 2 to 3 meters at most.
I'll keep thinking about how to solve the problem.
I was working at an IT agency as an ops guy. One Saturday morning our alerting went of for one of our customer's sites (a somewhat high profile public sector company). Turned out we were being DDOS-ed.
The hosting company was trying to mitigate, but the attack was "real traffic" so hard to just black hole. I dove into one of the Apache logs and was looking for some way to sift out the bad traffic and keep the real traffic.
Then I noticed a lot of Nintendo Wii user agent headers. That was suspicious. Who in hell is using this service 100 times per second from their Wii from Pakistan and Russia?
I wrote a quick regex that blocked all IP's from requests with Wii user agent headers. This took care of 99% of the DDOS traffic. After 2 hours the whole attack stopped.
It's just an hypothesis, but I think saying CF when you actually have an atypical form of it is a bit dishonest. I have no doubt that diet and lifestyle do wonders for many conditions including yours, but the CF everyone thinks about is the one discovered in infancy and the full set of symptoms, not the one with late onset (at an age which many CF patients do not even reach) and milder symptoms.
If you were willing to lift up that imprecision, I bet you wouldn't have such a hard welcome.
It's a case of damned if I do, damned if I don't. People with CF don't like me saying "mild" CF. They have a cow about that. I spent years on CF forums and got schooled on that detail quite thoroughly. There are more than 1600 alleles that can lead to a diagnosis of CF and the exact presentation of the condition varies from one person to the next in part because of that.
My condition is not late onset. I have had it my whole life. Being diagnosed late doesn't mean it "came on" late. I spent years being treated by people like I was some kind of hypochondriac and asking doctors "Can we test me for something? My body doesn't seem to work normally." and being blown off.
It's not imprecision. I'm as precise as I know how to be or you wouldn't know that my actual diagnosis is atypical cystic fibrosis. I in no way hide that.
So from where I sit, that just looks like yet another BS excuse or justification for people on the internet to be jerks to me.
I knew it because I looked up your bio for the first time You don't hide it, but I don't remember reading it in your posts here, not as much as the complaints about other people's behavior at least.
>My condition is not late onset.
And the only person I knew with CF had been diagnosed as a toddler and had a 20 y. life expectancy because of genetic bad luck. That you could live 30 years without heavy treatment sure says that your condition was not as bad as that person, and maybe there's a state of affliction that cannot be conservatively managed.
I've spent years trying to figure out how to explain it to other people. I actually joined HN in hopes of learning to program so I could write a simulation to more effectively convey the information, but that hasn't happened. Getting well and other life drama has taken all my time.
Trying to explain it with mere words seems to be something that needs like a million of them and still isn't really adequate.
But, in a nutshell (or an attempt at a nutshell explanation):
First, I did things I knew were helpful from before having a diagnosis and tried to then go figure out why it was helpful. Having gotten diagnosed late in life, I had my own mental models and ideas that didn't jibe with our current official understanding of the condition. I respected my innate knowledge rather than just agreeing with the experts.
Second, I tried to understand the pathology. We know the mechanism behind the condition: A malfunctioning (or sometimes missing) cell channel called the CFTR that handles trafficking of certain molecules into and out of the cell. So I tried to understand how we get from there to all the symptoms typically associated with CF, such as malabsorption and lung infections. That seems to be a different approach from what medicine currently does.
Some of my general conclusions:
Malabsorption leads to malnourishment. Many of my issues were rooted in malnourishment. Improving my nutritional status was a major cornerstone of my efforts to get healthier.
I found ways to compensate for the defective cell channel. One thing that is known is that CF involves misprocessing of fats. I got very picky about what fats I eat and tried to develop some understanding of why some fats are helpful and others are problematic. This seems to be rooted in chemistry and at least partly in how complex the molecule is. My body generally handles long chain triglycerides poorly and I favor medium chain triglcerides, though that is likely a wholly inadequate explanation of the phenomenon.
Most genetic disorders involve a misfolded protein. I learned what promotes misfolds. Chemical derangement in the cell promotes misfolds. I believe this is an important mechanism behind the progression of CF, where you steadily get worse. I think reversing the chemical derangement stopped the positive feedback loop (aka vicious cycle) and this was a huge, huge factor in being able to get healthier rather than just try to slow or halt the progression.
I'm currently trying to figure out what I want to say on my blog about my understanding of "the immune system." When we speak most systems, we can name specific organs. The digestive system involves the stomach, small intestine and large intestine. The circulatory system involves the heart and an extensive system of blood vessels. But I can't list for you the organs involved in the immune system.
So I think better understanding exactly what we mean by that -- what is going on when the body successfully fights off infection -- is a major thing and I am still trying to figure out what I want to say about that. I think we generally have a poor concept for how that happens.
That's probably several hundred thousand words short of conveying much of anything, but it's an attempt to reply to your question.
I wrote a tool[0] that converts Erlang style Dialyzer messages to enough of Elixir to where the Elixir formatter can pretty print it. This isn't an intractable problem, but the source tool does really bizarre things so the output so it wound up being really annoying to deal with the quirks. Still not quite done, but it's in a workable state to where messages can be dealt with individually when the errors are poor.
Hey, I'm also working on something similar, namely automatic Elixir to Erlang code transformation at the AST level [0].
Despite it being really interesting, some features like scope analysis for variables detection and conversion, the possiblity of multiple modules in an Elixir file and the Elixir pipes which need to be translated to sequential function calls are honestly a pain in the ass.
I'm about to finish my PhD in the field of optical communications and applied machine learning to increase the data rate. More specifically, I used an autoencoder with an embedded fiber channel model, to learn something called a geometric constellation shape or modulation format. (if you ever heard of QAM then you're on the right track, you can think of it as a modern Morse code for transmitting bits)
Thereafter, we took the learned constellations to the lab and conducted an experiment, transmitting the learned constellations through actual fiber. I've learned so much during the whole process
Although, the gains are marginal, I like the method a lot since it combines physics (nonlinear schroedinger equation derived fiber channel model), information theory (optimizing for mutual information) and machine learning. It wouldn't have been possible without the people who published the fiber channel model, my colleagues and in particular the colleagues who could help me in the lab.
The debugging was hell, there are so many dimensions where stuff can go wrong (besides the usual bugs): physical parameters with the wrong unit, the implementation of the fiber model in tensorflow, the machine learning parts with its training process. Plus the things that can go wrong in the lab.
There are still pieces where I'm not 100% sure, and would love to speak to someone with some background in autoencoders.
I've open sourced the autoencoder and the fiber channel model, checkout my github with the same username as here!
That reminds me of when my (OpenBSD) software raid failed.. I think a power delivery issue took out the first disk for a moment, leading to degraded state and a long rebuild. I panicked and ran some smart self test on the other disk. This wasn't a good idea; the test caused some commands to that disk timeout and eventually softraid figured the disk is toast so now my raid is completely down halfway through a rebuild.
I figured the disk is actually ok, it just went south because of a temporary glitch caused by me. So I read the source code of the softraid implementation, flipped a bit on disk in softraid metadata with dd, rebooted, and got my data back.
Oh man, if Minecraft counts, I would say implementing a fully automated resource processing and storage system in FTB pre-Applied Energistics with Red Power and Buildcraft.
Recognizing face expressions of babies born before the expected date and put in incubators with oxygen masks.
The problem is really hard and the collection of video material available prevents the usage of machine learning techniques, so old school algorithms perform much much better.
Some guesses: to alert nursing staff in a preemie ward immediately (instead of waiting for a scheduled round of checks) when a baby is in pain or hungry or needs a diaper change, etc.
Without going into details, doing things where someone else you know will be devastated by your decision, and may even beg you to change your mind. Technically easier than opening a door. Emotionally like climbing a mountain. Can take a decade to get over it.
How to reduce time spent on incidents that were somewhat similar when managing an SRE team. It took seven months of full-time work writing documentation to get off the ground.
The hard bit of it was the leap of faith - I had to stop working on live issues to invest the time in getting the documentation to critical mass. Then we had to get the process right to maintain their usefulness. It resulted in _massive_ savings.
It also inspired this, which I've just started (so very early stages):
Creating a user customizeable webportal in ACT3 as my first real programming job. An undocumented template engine that grew into a full fledged programming language that compiled down to into a perl-dialect which inturn compiled down into native C. (ACT3 was used by just 2 companies APA and DPA)
Nearly every (but not all) instruction began and ended like HTML comments. It mixed HTML with code logic and code logic with HTML. Oh, I mean DHTML. Target plattforms: IE 5.5. & IE 6. and Netscape
No stack traces, no debugging tools other than alert().
In the end I had a whole meetingroom as my office as ever wall was full of print out sheets and hand written dependency & inheritance graphs. It was a procedural language, nearly functional but every function had mandatory side-effects (as it started as a template engine) with object oriented approaches sprinkeled in.
The project was a major success, but as the aspect of maintanance came up we switched to something sane: PHP 3 at that time.
I sometimes feel I am far from even being able to start on that problem. I think it is often life's biggest difficulty, especially if you have great loves (for activities, places, etc) that do not coincide. Unfortunately, it is easy for it to become a situation where nobody wins and everyone loses.
Ya, I hate this question as an interview one. I’ve solved lots of hard problems, but only the ones I haven’t solved yet stick around in my head. It’s even worse if they aren’t interested in design problems, but rather technical ones.
Ha, this is actually why I thought to ask it. Sadly, "I read the docs and wrote sample code for days until it became obvious" isn't what interviewers are looking for.
I know, but I debugged a lot of use cases, tried to narrow the scope of the bug, thought really hard about the execution in my head, and then fixed it all using an explicit semicolon....
Asking an experienced dev this question is like asking someone what was the hardest food they had to eat and how did they manage to swallow it. What is the interviewer hoping to learn by that?
Hardest, IDK. Most clever, the first that springs to mind is a power supply unit (PSU) replacement on an old HP server. It went down over the weekend, and HAD to be back up. It also had a proprietary pin setup with more than the standard ATX plugs. I popped in an ATX replacement, and it wouldn't go.
If you've never seen a PSU plug, it just has a bunch of cables placed into a plastic block for socket standardization - you're plugging in a chunk of plastic, which is physically assuring that each cable will go in place. Since I had little to lose, I severed the plastic block on the OEM PSU as to create two plugs: one ATX/standard, one proprietary. I then tried firing up the system with a standard ATX PSU in the ATX sockets, and the proprietary plugs (connected to a dead and not even power-connected) into the proprietary sockets. And it worked! I let the system run like this with a dead OEM PSU plugged into it with an open case as I waited for a proper part replacement.
I'm still not sure what those cables actually did. I posted about this on Usenet, and got 1-2 emails over the years about how it had saved someone else's bacon. Sadly, I can't even find it after the Deja News -> Google Groups -> offline transition of Usenet search.
My company is building a device which collects data from a car while you drive. We wanted to group those in "trips" on a very simple logic. If the device sends any data for the first time, start a trip and if there's none for more than 5 minutes, close the last trip.
Problem was that the source of the time is coming from the device which had a lot of bugs (e.g. time was in future, time is from year 1970, time is not in order...). Management also decided that it's too hard to fix them on the device. At the end it took me 3 full weeks +/- to make it work. At the end I was able to convince the management that some bugs were too serious and required fixing
A guy from the device developers and me talked with them about it. We argued that it's probably far easier and faster to fix the problems in the device code than to make some workarounds on server side.
I think the thing that it's faster and we were 2 convinced them since I tried it with the easier part already before.
On other parts we decided to not process the data.
When our education startup built out its first product in 2012 (real time student content delivery and live chat etc on iPads) the internet in many schools was terrible.
To counter that awful bandwidth issue we created a local server version of our cloud based application.
In each classroom we created an intranet with one web server and a wireless access point.
When students used the system their ipad directly connected to that local access point and intranet (ie for chatting via node/socket.io etc.)
It did not matter if the internet went down the full system still worked perfectly well as all the kids were operating 100% locally.
The hard part was making an entire system that could be online or offline at any time sync with the mothership cloud. Especially hard was synching, well everything. Node chat, Rabbit, mySQL, linux updates etc.
The app allowed the teacher to drive the content via node, live chatting between students, live answers/questions/assignemnts etc.
We needed to write a lot of software from the ground up that allowed for 100% asynchronous provisioning of machines, content synching, live chat syncing, message bus routing etc.
Crazy thing was, a kid could be playing and accidentally pull the plug out! We needed to account for all sorts of shenanigans.
Essentially, it was a bit like creating dropbox that was also a full stack architecture and a ridiculously distributed data center with servers that could switch on or off at any time.
It worked pretty well.
The teachers couldn't understand why our app always worked lighting fast when all the other internet stuff they did was so slow!
As time moved on, we realized that single point solutions would not solve the main goal we were targeting in education which is to modernize all aspects of districts.
So, we built out new software that helps districts take the 5 year journey of moving from a traditional district to a modern learning environment. Ironically the platform that does that is pretty simple by comparison. Essentially it's systametized consulting.
I think we still have one district still using that old software (5+ years old now) and we are about to sunset it.
We did that whole thing with a team of 3 devs in 12 months.
It was the third semester of grad complex analysis. I dread analysis to this day... but it was worse then. I didn't know my stuff. I hadn't done well the previous semesters; my homework scores were continuing to droop... no way that I could pass this final. I didn't have a clear idea of what I wanted out of my next action: to cry in my professor's office. He had some questions for me. Asked me how I'd gotten into grad school. Not in a mean way, just, we all have our own things that work. He gave some nice words of encouragement, and a kleenex, but he was right to send me back to my own toolshed.
My first real math class was undergraduate differential equations. The teacher was legendary; gave a speech on the first day saying it's normal to repeat the class and you can drop without penalty, even after the final. I bricked his first exam; didn't even smudge the back page with a sweaty fingerprint -- he tells me with a tsk, my handwriting is too sloppy: I should be deliberate in my thoughts, not rushed. So I did speed drills before subsequent exams. Later in life, I'd write sloppier answer keys for my own students.
But you can't speed drill for a graduate course in complex analysis -- proving an unfamiliar theorem is quite a bit different than applying a standard method for a familiar diff.eq problem with carefully-chosen coefficients.
The problem wasn't just believing in myself -- my prof boosted me over that fence. It was believing in abstractions hidden even from mathematical lore. Back to undergrad: I wrote a note sheet that night. I summarized every important theorem from the class into the most compact statement possible, along with a cliffnote about the proof. And then came the drills, copying the notesheet again and again, with as few peeks as possible.
The actual hard problem that I grappled with on the exam is memorable only for its beauty. The 24h transition from paralyzed fear, to justified confidence, was where the real work happened.
A simplified universal parsing format (AST) that equally describes all languages. It needs to allow seamless switching between different languages in the same file and also allow recursively nesting of different languages within each other without resulting in a deviating from the simplified parse format.
The solution is written and it appears correct and viable. As in all things related to parsers the devil's in the details in attempts to prove edge cases with exotic tests and use cases.
Please keep in mind this is a single person's pet project, so it doesn't cure world hunger or eliminate all disease just yet. Drop in some JSX or HTML with embedded JavaScript to see it go.
The structure is defined using the begin and stack data fields of a given token. The begin points to the index of the current token's parent and the stack describes the structure contained by that parent. Items in the global space have no parent, so I assign a begin value of -1 and stack value of global. I use -1 because there is no such index in an array.
I honestly don't know. I have only written lexers, the part that does the analysis, for a few languages. The output format is completely language agnostic though. Go read through the docs or play with the demo to see what the output format looks like.
Multiplayer engine for an open-world RPG that doesn't require developer to run a dedicated server. Think Skyrim, GTA or Kingdom Come, but with a much smaller budget for content - aside from very fast movement and physics-based mechanics of GTA, this engine could've been used for any of these games. Previous developers did the single-player logic first, and then added very ad-hoc multiplayer sync, hoping that everything will be fine - but if you played long enough, you inadvertently ran into desyncs.
So, I had to write first the network engine itself (which was used both for networking and load/save), and then rewrite the whole game logic to take advantage of this mechanism. Now, if you want to write any game mechanic, you have to define it in proper MVC terms (although, due to amount of legacy code, I didn't separate the view and controller as well as I should have). Dozens and hundreds of small scripts for all the unique quest content and game mechanics working on top of a unified platform, relying on it to sync their data, one-off events (think "play sound" command that doesn't change any data but has to be replicated) and not even knowing on which one of connected clients is it run with authority, while all others run this particular object in "slave" mode.
I honestly tried to research if any other game developer have used a similar solution, but found nothing - it seems as if all open-world games that have multiplayer rely on dedicated authoritative servers, operated by developer, to run. Here's the project itself, although this update is not yet released: https://store.steampowered.com/app/526160/The_Wild_Eight/
Too bad that at the end of this project I got the worst case of burn out in my career, spiraled into the worst depression episode that I've ever experienced, and now quit the company while struggling to put my life back together. I have no idea if I'll be able to publish anything on this tech, but I still feel that it's been worth it.
I solved a performance bug that was probably going to cost my company the majority of our customers if we didn't sort it out as it made the app unusable. It had been going on for months. None of the dev team had any idea. APM was throwing out a bunch of red herrings. I haven't written a line of code in the app or a line of production Java (the codebase language) in my life. In the end I tracked it down to a single line of configuration code. Performance went from being atrocious to being one of the best performing apps I've ever seen.
This was not technically "hard" as in complicated but more so hard in the sense that it was very high stakes and the time pressure was high.
I do this a lot but it wouldn't have helped in this case unfortunately. Also the code base is over 700k lines which added another parameter to the mix.
I once wrote an app for Windows Mobile that implemented the L2CAP bluetooth protocol, and implemented the HID spec. This allowed phones to appear as bluetooth mouse/keyboards and be paired. It used the accelerometer and touchscreen to allow for cursor manipulation and had an onscreen keyboard for typing. Took me months to figure out the Bluetooth driver and to debug the HID issues, but when it worked, was pretty exciting. Just dug up an old analysis of it too https://forum.xda-developers.com/showthread.php?t=504730
Hey Leo, You may be interested in Dan Ariely’s short book on Motivation, which is based on his Ted talk here: https://www.ted.com/talks/dan_ariely_what_makes_us_feel_good... Part of it covers the motivational impact of having years of work cancelled. If you’ve struggled with this, I just want to wish you the best.
Just watched, great motivational content, thank you for sharing! Cake example was really weird and still making me think.
I dont struggle as I used to since I've learned it is about environment we are in (meaning as in video) and tomorrow is uncertain (because many people are involved)
Yes this is generally the idea. Didn't know about OpenCV though. Was a font-matching and trial and error by hand in MS Notepad + some scripts to help brute force the thing kind of solution... Turns out there are actually at least a couple dozen possible solutions based on what is visible so some small amount of information was lost through the redaction, but not tons.
Yes, it's a monospace font. The start, end and pattern of visible ascenders behind the redaction, along with the word list in electrum would allow you to reduce the search space by a lot, I suppose.
I'll give you a hint. Download the image. Because it is PNG that means it is lossless... Zoom in very closely on the top row of pixels above the redacted area and you may discover you have all that you need...
Thanks. I already did that, but there were quite a few letters that were completely covered.
Programming isnt' my day job - but I have an interest in it, so was curious to see what tools / approach you used to automate this and whether it was fully automated or whether your solution recovered the partially covered letters and your 'wetware' recovered the rest ;)
Cyph0n already provided an approach, which you confirmed - so thanks again.
A few years ago I solved a computer vision problem involving getting the spin of a golf ball as it came off the tee; a fuzzy image of a ball going up to 150mph.
Because the client didn’t have a have a system that could truly “freeze” the ball (“make it work with our crappy images”) I had jump through a lot of hoops to get an accurate spin. In the end I was getting rotational accuracy of around 1/2 a degree I think. I remember calculating that it was 1/2 the thickness of a credit card as measured on the surface of the ball.
I came up with a recursive layout (ie graph drawing) algorithm for directed acyclic graphs (DAGs). I was trying to draw program flowcharts and the hierarchical layout graphvis creates are very ugly and unbalanced. If you reverse the back edges of while and for loops a flowchart is a DAG. I did a python implementation.
I research graph drawing for a while and the closest algorithm to mine was a “delta drawing”, I also found a paper about delta with force reduction steps.
Then I stopped working on program static analysis and didn’t do anything with it.
> If you reverse the back edges of while and for loops a flowchart is a DAG.
Depending on language there could be also end recursions that never return or - the horror- GOTOs.
Not _the_ hardest, but among them. Back to 2011~12, cross-platform mobile application development with Titanium SDK (JavaScript).
We have this ~50K sloc application which has several performance bottlenecks. At the time connecting the Chrome Inspector was not a thing. There was a half functional profiler in the paid version of the platform, but it was cluncky and the UI simply unusable (the whole IDE was a customized Eclipse distro).
Since at the time ES6 definitely was not a thing yet, everyone rolled their own class system in projects. In our case the OOP library was more or less Backbone’s, with the typical Child = Parent.extend({ ... }).
This was a godsend: I had a place for every class definition of our codebase, and there I placed an hook which wrapped every damn method of every class with profiling instrumentations.
I then built a small profiling UI (shake the device to start/stop collecting + profiling chart in the logs) and we were good to go.
The funny thing is that I never actually used a profiler before that, so I literally re-invented it from the ground up — only having a collegue of mine staring at the results and saying «what a nice profiler you built over night!».
Only then I started looking at real profilers and introducing “correct” features in my own: real time, self time, flame chart.
Working as a junior analyst at a large pharmaceutical company we had a tender document arrive due in 2 weeks for a massive amount of money. Being generally super keen I was given the brief 'can we do anything in this time period? If so, do it!' - usually our models take 4 months to build, and cost ~$100k.
I used meta-analysis our our trials vs the main competitors for each age group (children, adults, elderly), cross referenced to the age structure of the population i.e. number at risk in each group. This showed because of the numbers at risk our drug looked better.
I then included data on the harms and connsequences in each group, using past data, and added in monte carlo analysis around the uncertainty, which gave a massive chance of of ours being the optimum strategy.
It was only a part of the case we made, but was delivered in 10 days at a cost of $600 (software licenses), and really helped show what I could do. People still mention it now (it was 10 years ago). My boss at the time was also amazing in giving me freedom, but then helping communicate it all.
I've worked on some really cool projects in my career, but that's probably the one I'm proudest of.
As far as technical problems, there are plenty of optimization sorts of things that come to mind. But I'll go with a different sort.
A while back I was working on an embedded 3D scanner at school. Part of this was using a freescale ARM processor, and a USB interface to transfer scanned models back to the computer. But a weird problem was happening: only when trying to use the USB, every once in a while the processor would have a fault, and the fault was often different and seemingly unrelated.
After a week of constantly banging my head on the problem, I came upon a stroke of luck. We were smart enough to have built in a programmer on the board that supported gdb debugging (we built the circuit and populated the PCB). While in a gdb session, I read from the code flash section and noticed that there was a zeroed-out word (4-bytes) that definitely shouldn't have been. More intriguing, a subsequent read was correct, or had a different word zeroed! So the random faults were when one of these zeroed words occurred while code was read (and would have a bad effect depending in the instructions).
Now that I had found the problem, the solution wasn't too far, but still took some fiddling. Eventually I found that reducing the flash memory clock fixed the issue. We had all the clocks set to a valid configuration according to the documentation, and they did work fine when the USB module wasn't enabled. But as soon as it was enabled, it apparently interacted with the flash in weird ways.
I was glad to find a fix, and luckily we didn't necessarily need the flash to be fast. Embedded systems (and other complex black boxes) can cause some real frustration at times. But it's always fun, and I still found it rewarding :)
What a fun question :) And there are some very cool answers!
A few things that come to mind from my life, from back when I was still a kid/teenager:
- Building a three-way marble-track switch. There are many marble tracks that have these two-way switches that alternate between two tracks, like this one here: https://theworkbenchutah.files.wordpress.com/2013/04/06-marb...
I wanted to build one that would evenly distribute marbles along three tracks. Took me quite a few prototypes until I finally had a working one. And all I had to work with was paper, tape, glue etc. No wood or metal, no bearings or precision mechanics, only a kid's craft stuff.
- A LEGO technics robot leg. This is a bit hard to explain without images. I wanted to build a legged robot (something spider-like) and needed legs that could move a foot both borward and back and also up and down. As the LEGO motors where quite big and heavy I didn't want to mount them on the leg itself (e.g. at the "knee") but have them all fixed inside the body and transmit their power via gears and shaft.
The problem here was, that the up-and-down-motion had to go trough the forward-and-back-joint which meant that whenever the leg moved forwards or backwards it would also move a bit up or down, even if the up-and-down-motor didn't move at all. So I wanted to decouple these two motions. This would be trivial to do in software but I was just a kid playing with LEGOs so I wanted a mechanical solution. I managed it by using a mechanical differential to actually subtract the one motion of the other.
Unfortunately, I then realised I didn't have enough motors for a full robot...
- Years later I was learning C++ and I wanted some kind of "linked variables" (I'm sure there's an official term for this): I.e. variables that would depend on others and would update whenever one of their dependencies changed. I though this would be a cool way of writing a GUI. With a lot of operator overloading and some abuse of the type system I could actually write things like
x = a + b * 2;
y = x / b;
And 'y' would update whenever a or b (or x) was changed.
I'm not sure if they really are the hardest problems I've ever solved, but it certainly felt so at the time ;)
This was 15 years ago. I signed up to a correspondence course to study company law after my under graduation(not CS). Was bored so wanted to pick a hobby and was fascinated by robotics/automation. Self thought 8051 by reading Kenneth J. Ajala and this hobby soon turned lucrative.
I signed up with an institute which was helping college students with their final year projects. I had a college junior(Amazing guy who is still a buddy) to help me with the pcb fabrication and i would handle the software side. All was well until a project which required a heat sensor and i couldn't source it. My mode of sourcing the chip is via a friend who was in Chennai and i would send the list of items and he would go into Ritchie street and get it for me and mail it. Good old days. Sorry forgot the name of the chip. All i could get was an alternative one with single pin instead of double and it uses a different protocol. I had the alternative chip delivered one day before the girl student's viva voce. Got it working and the examiner took the pain to visit the lab in the institute to examine and conduct the viva voce. Phew.
Another incident happened with a company where i was the second technical hire and there was just me and the CTO on tech team. The CTO had a working POC and it was quite a complex product. Even before i could start writing a single line of code the CTO left the company after a spat and without proper hand-off. I wouldn't wish this scenario even upon my worst enemy. It was quite a significant challenge to even get it running. One simple example - In the initial days after he left, i found out Opennebula Virtualbox driver is not attaching the context disk on windows. I had to write a patch in ruby and submit a pull request. I didn't knew ruby and haven't even wrote a single line of ruby code before that. But after quite a struggle - got the code base to start making significant money.
> Even before i could start writing a single line of code the CTO left the company after a spat and without proper hand-off. I wouldn't wish this scenario even upon my worst enemy.
I know that feel. My first full-time job right out of college, the lead developer left only a week after I joined. I individually inherited a 70k+ SLOC project (in Java, but still...) of his that sorta kinda worked, in a domain I had no experience in. Especially as a new grad there was a massive amount of anxiety trying to get up to speed with nobody to ask questions to. But as the months went by I learned every nook and cranny of every class in the codebase, and was able to make big sweeping changes. It turned out to be an incredibly valuable lesson in soldiering on and not getting intimidated.
I had loads of experience when i encountered the scenario so your story is much more impressive. Agree that the schooling this kind of scenario provides is harsh but an enriching one.
I once had to fix a rendering bug in the heart of Mozilla that prevented a XUL-based code editor from working properly on Linux - this was back in 2001. The bug was a one off error about eight layers deep in the XML layout system. It took me about two months of full time digging and nearly destroyed my life. But I found it!
Oh, and debugging a DOS TSR (Terminate and Stay Resident) API for realtime communications to mainframes and other information sources (async and X.25) while running under Windows 386 - Windows 3.11. Every different network stack (IPX, Netbeui, LanMan, etc.) presented a new brainfucker of a problem.
The hardest problem I've ever solved is learning to sit down and collage. Like, where you cut out bits of magazine and paste them onto paper, etc..
My friends do it pretty often. I find it super relaxing, and know I'll be happy I did it, but the engineer / researcher in me wants to always have fingers on a keyboard.
My task: Take a top 500 website, all written in smalltalk (no documentation), and port it to a modern scripting language. The company has: no money, a hostile customer base, angry previous engineers, angry management, and hostile current employees. The site generated billions of webviews and was all unstructured (no database) user generated content.
It was grueling. There was only one smalltalk engineer that would talk to us (the others quit, or weren't paid). Getting the dev environment running was impossible due to license issues with cincom. Luckly, I was able to get 3 smalltalk endpoints that would dump XML -- using the last remaining favors we had with the last small talk dev. Using this and layers of hacks we were able to start porting in 6 months.
The site was sold (3?) times and was ported again to something else.
I had to build an app where you could draw a trend line over a stock chart, and it would set up an alert notification that would alert your phone if the price tripped the line. The math that goes into this is absolutely insane.
I should add that I never really solved it, as the solution didn't work very well.
I convinced my wife to marry me. She was very pessimistic, thinking that I would leave her. It took six months to get a positive response to my proposal and another year of reassurance before we married. We have been married 41 years. The courtship was helpful experience for the marriage.
Back in 2008 during the start of the economic downturn. I was fresh out of MBA studies, working as a management consultant.
I’d studied options and derivatives in my second year. From this, I further became interested in “real options”, which provides ways to value real decisions under conditions of uncertainty.
Anyway, we (as in the firm I worked for) were trying to figure out what was going with the economy, what it meant, and how we could make money off of it advising customers. So, we had a few group ideation meetings, stuff like that.
At this point in time in September-ish 2008, volatility in many prices for many basic things, like fuel, even some food like milk, started noticeably increasing. Like, fuel prices would go up and down $1 in a week.
I noticed, people started delaying purchases until they absolutely had no more time to delay. Like, car sales dramatically dropped. Many people stopped buying cars. Because, of the tangible levels of uncertainty they were beginning to feel.
One day, a connection just clicked with me, that option valuation theory could be really useful to help explain and predict, what was happening in the economy.
As volatility increases, option value increases. In the real world, volatility manifests itself as uncertainty. On a microeconomic level, the volatility linked uncertainty gave consumers and businesses a more valuable real option of postponement.
On a macroeconomic level, the individual decisions of consumers and businesses to postpone spending, contributed to a death spiral for the overall economy.
I’m no economist. But, I put these ideas together in a deck, centered around a pitch for 3M company, showed it to some directors. They loved it, formed a team around it, continued developing it, pitched to 3M, and it helped to sell millions in project work.
Meanwhile, after I had made the connection, it was really hard to stop thinking about it. It ties together a lot of branches, old schools of economics (which, by the way, you will be flayed for questioning) with finance (has a lot of assumptions built in, which you will also be flayed for questioning) with human behavior.
Continuing to think about these connections, led me to quit my job at the consulting firm, after I had made a good name for myself.
Ultimately led me to become an entrepreneur, to build a company more closely aligned with writing real options, which is what I feel like I do, every time we give a quote at BOM Quote Manufacturing.
I think the hardest problems I've encountered are the ones without a specific right answer but tend to be more something that finds a balance between many complex and often contradictory requirements.
These are typically problems that involve interaction with very complex meat-space concerns.
What was supposed to be a weekend project consisting of re-grouting bathroom shower tile turned out to be a two-month side project replacement of wall from the studs out. It was the kind of project that got worse at every step.
The contractors who built the bathroom cut corners by creating a bathroom in a single day. They didn't sufficiently mortar each tile to the wall (back butter) and wait a couple of days to set prior to grouting, as they should have. As I cleared out grout between the tile using a multi tool, vibrations agitated the tile, causing several to fall from the wall, crashing into pieces in the bathtub. Evidently, many of the tile were hardly, if at all, mortared to the substrate. Some tile weren't even fastened to the wall at all but were rather being held in place by grout.
For those unfamiliar with the business of ceramic bathroom tile, the industry does not make standard tile in perpetuity. Consequently, the odds that you will find a match for 10 year old tile are extremely low. I could not find an even remotely close style of tile to match against.
So, realizing the futility of this work, it seemed that I had no other option but to remove the three walls of tile surrounding the bath area and start new.
I did all of this from a condominium unit four flights from ground. I cut tile from the ground floor, outside, without appropriate workspace nor wet tile cutting machine but rather an angle grinder and some buckets. If you're not laughing, you really ought to try this some time.
Lessons learned:
1. Never, ever, regrout anything beyond a very small area. Avoid re-grouting entirely if possible.
2. Buy at least twenty extra tile for any new project and store them somewhere safe for the long term.
3. Professional contractors who tile for a living are worth their fee. Same applies for mortar and stone specialists.
4. DIY projects are a great distraction from cerebral day jobs but are minefields
This isn't the hardest problem I ever solved, by a stretch, but it's a horror story I like to share.
I was always told to make sure you keep extra of any kind of material like that around. A handful of extra tile, some extra wood flooring, some carpet that we pulled out to replace with wood floor at one point, even some extra molding just because why not?
Worst case scenario, it sits up in the attic and the 15 minutes spent taking it up there is a waste, best case scenario it makes it so that a single cracked tile or a deep scratch in a wood floor isn't as big of a deal. And I have found that I pretty much always have at least some left over normally anyway.
I left my wife and my two kids of 6 and 8, because I couldn't adjust to my role as a father. A few months later I realized this was the biggest mistake of my life, but my wife wouldn't allow me to return - can't blame her.
For a few years I was eaten by feelings of guilt, and the only thing that kept me alive was my two sons. Today, I still get guilt attacks, but I somehow managed to move on. I have a good relationship with my soon-to-be ex wife (she's the best person I ever met), and I'm seeing my sons as often as possible.
So this basically isn't a "problem" per se, but it's the hardest thing I ever had to overcome.
Teaching a team of hardware engineers moonlighting as software guys to replace ZIP files and "version[1..n]_backup_2_reallyfinal.c" type filenames on a network share with proper version control.
Then static analysis and CI.
Then the supportive manager left, to be replaced with a new manager (one of the aforementioned hardware guys "promoted out of mischief) hellbent on going back to the "old, better way".
"Why should you get £1000 a year from capital expenditure just to keep our static analysis licence! You should write good code to begin with! Otherwise why did you get hired? Are you incompetent?"
I wrote a transactional object store for an early PDA that could use both vanilla battery-backed RAM and flash media. Took about six back-to-back hundred hour weeks (and had a surprisingly short tail of bugs).
Later, I hooked the store up to the NewtonScript garbage collector in a way that let us have garbage-collected semantics for persistent memory-mapped objects. This let us do things like "receive a FAX" on a system with not a bunch of free RAM.
I wrote a 3-axis CNC simulator using a new algorithm and published it as Open-Source. See https:/camotics.org/ More recently I created an S-curve path planner for CNCs. Also Open-Source.
Built a package manager to transfer user selected data from one installation of the system to another. The data was stored in a mysql database, and used foreign keys everywhere, so installing the exported records in the right order was necessary. Some data was local to the machines though, and should absolutely not be exported. But exported entities need to be able to reference these local settings, so these references were exported using a symbolic value - the records had a name field which was used. Exportable entities included packages, so the whole thing was recursive and all requirements also needed to handle that.
Sounds fun right? Well, everything should also be installed keeping the integrity of the system intact, and if something didn't work out (say, an entity of the same name as one that is being installed already exist), then everything should fail in a controlled manner and the user should be given a good error message and there shouldn't be any crap left behind in the system.
Then the people in charge realized that this is nice, but as they built a big package hierarchy that was then installed and modified slightly in the customer systems, it would be nice to be able to serialize, and uninstall the local changes to packaged objects, uninstall the packages, install a new version and reapply the changes. These should also be possible to package, so that each customer system was possible to recreate. Supported serializable changes was not only fields, but also above mentioned symbolic references as well as whole new records.
This is easily the task that has formed me the most as a developer.
On a modded Minecraft server, I wanted to use an opencomputers 3d printer to print out pngs from the internet. It had a pallette system that let you choose colors at specified dimensions, but I wanted to constrain myself to only use colors that not only existed on existing blocks, but also in the same position.
1. Had to build a Minecraft computer and figure out its interface. The computer came with git and curl commands, so that's how I downloaded files.
2. Every block had to be pulled apart and every pixel had to be mapped to it's voxelspace dimensions.
3. Some blocks had extra logic which caused random transparent pixels to show up, so I had to find and exclude those from samples (flowers were the biggest contributor). This was a bulk of the work, because there was no simple way to find which blocks had extra logic. I even pulled out jad to find any sort of useful pattern, but there really wasn't any.
4. Had to do some k-map stuffs to find the closest color to each pixel from all images.
5. Create a file for each block, which all had to include the original block's color for that specified dimension.
6. Import each file one by one because the computer had limited diskspace.
7. Place each block manually.
It ended up working surprisingly well and could be scaled to any size. Problem was, even looking at a printed block used a LOT of resources at both server and client side. So I scrapped the project.
Fun stuff, especially with that being the first sort of image processing work I've ever done.
As far as CS goes the hardest problem I'd somehow "solved" involves designing program that produces reasonable approximations of optimal solutions for TSP/VRP. The actual implementation took a weekend of wall time (which includes several weeks of single-core CPU time), but that was preceded by two months of reading papers with abstracts like "We present practical approach to solving intractable problems".
Trying to get a Glowing Black Stone drop broke me on that game. The final straw was after camping the npc that drops it as a rare drop for days, with like 6-8 hours between spawns (Pyzjn), she finally spawned again and midway through casting a spell to kill her, she spawned because it was morning. :/
That’s the thing. I miss being broken by a game. I miss that struggle. I’ve tried to experience it again but maybe nostalgia is superior. I’m hoping for pantheon to bring that feeling back. I genuinely miss being addicted to the struggle and joy Everquest was. It helped mold me into who I am today and I love it.
Removing from my budding startup team a highly motivated guy, bright, experienced in big companies... without damaging his ego AND without lying AND while being clear. If I can re-orient everyone I need to, as well as this guy until the end, I'd be so proud and happy! And I won't ever make enemies that way.
That was a very charged conversation emotionally, I was vulnerable and highly respectful, but still decisive and honest.
I refrained from exposing the case I had 'against' him to make the decision to move him from a potential cofounder role to an external advisory role. Instead, once decision was made, I committed for this to be as good as possible for everyone. I started bravely with the clear news, pointing out it had been my mistake to bring him on at that point.
I had taken serious time imagining his viewpoint. I talked about all the good he had brought to the project, for real, and all the good I valued about him, for real, and asked him to be an advisor (yes, he's been of value in that role, despite the hurt feelings).
Oh, and I gave birth to my daughter in silence all by myself on the couch. That was pretty intense too. While living in a squat, I also unfolded a dead guy starting to harden, to make sure we could later lay him down in a coffin. Life is full of strange adventures, I guess!
I wrote an inliner once. It might seem like an easy problem, but a proper inliner is a rabid animal you can just barely restrain on a leash.
The simple part is making it work without clashing. The hard part is making inlining always be an optimisation with reasonable code growth with regards to compilation time and cache use.
I built a slot machine game with multiple lines, odds, payouts in unity. The slot machine math was suprisingly complex ( calculating the odds of lines and payouts ), and making the app performant was even more difficult ( 3D animations and UI in NGUI ).
Toolpath equidistant to mill around an object designed with a CAD system. Math is not my strength and it took forever to come up with a way of doing it that would not amplify subtle errors t in the underlying model. The basic idea is that you have an outline of what you want to make, then you have to offset that outline by your tool radius. And preferably offset it to the outside otherwise you end up destroying the workpiece. Now anywhere your original input has overshoot (which is easy enough with a CAD package, too small to see without zooming in in extreme detail) the direction will reverse and your outside suddenly becomes insides.
Convincing a department to graduate me with an ABET accredited degree without finishing all coursework. It was significantly easier than deriving the time-dependent wave equation for hydrogen, writing a Java-to-MIPS compiler with two implementations, one in C++ and one in Java, and making an UNIX FORTRAN reactor simulator compile on Windows 95 (lol!)... all with crushingly, untreated depression. Also, survived the Loma Prieta earthquake and Paradise Camp Fire... nothing special there though.
My friends: getting an ASW account while being neither rich nor famous; another, coming into possession of a MiG-21. /unhumble-brag
Develop a tool to boost the productivity of a software engineering team and convince all members to adopt it. Software engineers (myself included) can be very resistant to changes (especially when it comes to their tooling) and an in-house tool with a single developer behind it didn't contribute to the ease of adoption, but it was adopted regardless. I guess that the tool was indeed very good, or maybe I'm a sweet talker :)
Note: the team was very specialized, so there really were no competing tools around for that specific task.
The team had to work with tens of different embedded devices with all kinds of weird compilers and flashing tools.
My tool would help with the context switching between projects, providing wrapper scripts and inline documentation. It was some sort of literate programming in bash :)
Building a profiler for .NET. Learning the .NET profiling APIs is a massive learning curve and extremely complex. But, we figured it out! This is for Stackify, BTW.
Summary: Qualcomm wanted me to devise a computer vision solution that was nearly three orders of magnitude power-efficient than what they had then. There was a clear justification existing as to why such a drastic improvement was needed. Nobody had a solution in spite of trying for a long time. Most laughed it as impossible. I started by looking for a proof as to why it could not be done if it indeed could not be done. After some three months of pulling my hair, I started getting glimpses of how to do it. Some three months later, I could convince myself and a few others that it is doable. Some three months later, the local team was fully convinced. Some three months later, the upper management was convinced that the research phase was over.
Details:
I was told to reduce power consumption for an existing computer vision solution by nearly three orders of magnitude. This included end-to-end power consumption including camera as well as power for computer vision computations.
No one in the industry knew how to do this. Many industry experts laughed at the goal itself.
To make it worse, I by myself had no prior expertize in computer vision, nor in camera design, though I knew image processing. I was assigned to the project solely because of my brand name with the company. Being ignorant perhaps helped since else I may not even have accepted the assignment.
I was told this is to be a five-years type research program, given the agressive goal.
I was told that the company was ready to do change whatever is needed to make this happen, be it the camera design (including pixel circuit), processor architecture, computer vision algorithms, even optics and packaging.
With no starting point in hand (and in fact a false start assumed based on half-baked ideas from one university), I had head-aches daily for more than a month. Most of this was to learn and deep dive into the prior art.
The goal I assumed was to either solve the problem, or show a theoretical basis for why it cannot be solved. (In hindsight, to solve a challenging problem, try to prove why it cannot be solved, since even that is usually hard, and you may prove yourself wrong in the process by solving it!)
There was no assigned team, given that there was no solution in mind. There were people with varied skillsets (optics, packaging, digital circuits) to help as per need.
Three months in, I had some ideas for how it could work. Six months in, there were three more engineers on the project working under me on a near full-time basis. And some were complaining that we don't know what we were doing. Seven months in, I had Excel-level calculations to show that it can work, and actually a simpler solution than what I had in mind at the three-month point. Nine months in, with about seven people working, the team was convinced of the solution, as validated via rough circuit design and simulations. By the end of the year, senior management was convinced that we have solved the problem and the research phase is over. We had an estimated five years to solve the problem, and the solution achieved three times less power than the target!
The solution was a mix of several new inventions (one was given gold rating by the company) and significant power savings obtained just by careful design. Unfortunately, I cannot talk much about the solution itself.
We started showing early preview of the capabilities (under NDA and without disclosing how we solved it) at multiple Tier 1 companies, and saw their jaws drop. At one company, one whispered to their team members, "Are these guys kidding?!". At another company, someone smiled silently for half-an-hour till we started showing the system emulator.
Soon, MIT Technology Review talked about the work [1].
I have since then left the company, so do not know much about the current state of the project, other than additional news coverage received over the time. Additional references are there in my LinkedIn profile.
PS: I have solved many hard problems through my career, have written about the one that gave me the most head-aches. Here is a list of some other ones [2].
> (In hindsight, to solve a challenging problem, try to prove why it cannot be solved, since even that is usually hard, and you may prove yourself wrong in the process by solving it!)
This is how I approach almost any hard problem if I get stuck at some point. Attempt to prove that the next step is impossible - a missing link in the attempted proof usually reveals some deeper structure or invariant that you can use for solving the problem.
It's a great way to force yourself into a different perspective and distill the problem down further.
A deceptively easy looking obscure disentanglement brain teaser, that I found in a remarkable book on brain teasers from 80's. When you start solving it, you think "this has to be very easy". Hours and days go, and slowly a feeling sets in that your brain is not up to the level of this crazy problem. Then you suddenly find the solution, no idea how. Banal stuff, but when you solve it, you feel like million bucks.
I think it was something like this: https://image.ibb.co/nsz4x0/puzzle.png The goal is to separate the string with the beads from the rubber piece. The rubber has 5 holes in it. The beads cannot go through the hole.
I built an olap cube for manipulating time series of financial models, and made an API to serve it through a query language that I created just for this.
Probably one of my harder ones was finding a memory leak in one of my servers. Didn't happen often, never in development and only leaked a fraction of a byte per request. Turns out it was a bug in the logger, where if on rolling the log and it couldn't open a new file, the logger would just queue up messages. Was a real pain to track down as we normally had plenty of disk for logs.
The fraction of a byte was amortorized over the number of requests (e.g. the mean bytes lost per total requests), which is what made it puzzling and hard to track down. It only happened under specific conditions that were not evident running under valgrind.
Derived the Riemann Tensor[0] on a whiteboard being a tutor.
Obviously I wasn't the first person to 'solve' it, but being relatively unprepared in class I was shocked and very proud about my personal achievement.
I once had to design a function to open a series of nodes in a displayed tree from one given node down to another. Not so hard in general, but because of how the system was set up I had to do it using callbacks. So tell the first node to open itself and when done call the callback that tells the second node to open itself and when done call the callback that tells the third node ...
Piecing together my polyamorous orientation and getting over my poor self-esteem when it comes to romantic or sexual relationships. The journey from being confused, lonely and depressed to understanding that there actually are people that could be interested in me and desire similar things from a relationship.
Definitely building a design language that had a minimal learning curve, yet the expressiveness to build any website or shop imaginable. The MECE analysis of CSS, the development of a flexible JS framework for users to mix and mash without ever seeing the code. Took a few nights of thinking :)
All of the trickiest software I ever wrote was preceded by an exhaustive explanation to management to let me write it. In the end, the trick was political, not technical. I could single-line-summary any of the changes and they would sound absolutely banal to this crowd.
Why Olympic tickets for the 2008 Games wouldn't print on the Chinese version of Windows.
The short version it came down to Java on Chinese Window XP had a different $CLASSPATH than the versions in the US & Europe.
The longer version:
Ticketing for the 2008 Games was done by Ticketmaster. At that point, the core ticketing system was largely written in VAX assembler. For a longtime, most ticketing agents used a command line interface directly into the backend. Imagine, if you can, something even more user hostile than MS-DOS & you'd be close.
That wouldn't work in 2008. Tickets were going to be sold across China, at the Bank of China, including some very remote locations. Obviously they didn't speak English, nor was it feasible to train them on an English-based command line system.
So we created a web-based point of sale. It actually went really well. We had a set up process to make sure all the basics where in place -- the client certificates were installed correctly, they could reach the relevant servers, the location had sufficient bandwidth & so on. We literally caught thousands of problems before onsale.
Months later the ticket printers landed & everyone was to make sure they worked. For almost everyone, it failed & it failed silently. Debugging it from the United States was a nightmare. The time different, the language different, non technical people trying to relay technical information, the game of telephone from the bank agents to our Chinese staff to our American staff to us.
One thing to mention here is we need to print without any user interaction. The print prompt offered too many options to screw up tickets that had to be printed exactly.
So we had a Java applet to do the printing. We'd ship a PDF to the applet & it would take it from there.
It worked flawlessly in the US. We had EU folks test & it worked. We had our colleagues in our China office try it & it worked.
But for the Bank of China, no luck.
The Bank of China also didn't want us to have access to the machines out of security concerns. They were on the other side of the teller wall & we couldn't get on the sider, despite lots of escalations.
Eventually they brought a machine to the glass, turned the monitor so we could see it, and connected a keyboard through the slot at the bottom used to pass money back & forth.
We quickly found out the applet was crashing for them. It took the better part of a day, but eventually we figured it out: The $CLASSPATH was different. On top of that, applets could request classes remotely. With the different classpath it couldn't find one of the classes (I believe it was font related), would fall back to asking the server the applet was served from. If the server had returned a 404 that would have been fine, but since this wasn't expected & this server had other uses, it first wanted to authenticate the user. And to authenticate them, it would 302 to a login page (which Java would follow), that would return a 200. But instead of it being a .class file, it was the HTML of a login page.
Java, thinking the HTML was a .class file, would barf & crash.
Mystery solved.
There wasn't time to get all the machines to change the .class file, so we had the server 404. Everything worked from there.
We later found out our office in China has the US version of XP, but several people had the language set to Chinese. When they purchased the version of "China" version of XP, we had no problem reproducing the problem. It was somewhat moot, but we wanted to be ready for other problems.
This was sometime ago, so my memory might be off on some of the details, but that's the best as I remember it.
I worked on an online test with different types of question with video viewing.
The hard part is the video viewing progress is tracked by system. I have to implement my own video server and player. Youtube was new and wanted to use it but I can't track their view progress.
Recently, it's slicing of triangle meshes for a DLP 3D printer. The result is FullHD or 4K * up to 16x16 for anti-aliasing * a few thousands of Z layers = ~1E+12 voxels, while users only have ~4E+9 bytes of RAM to store them.
Maybe my first program in Pascal, my first programming course.
The school had never taught programming before (yeah, I'm that old), and it was a business teacher who didn't know how to program or use a Mac, and the incomplete syllabus was for a class from another school that used a Honeywell and punch cards.
The first Pascal program was to write a calculator that allowed a user to input arithmetic problems in words and get the correct answer.
>five plus seventeen
22
>two hundred fifty six divided by three
85 r 1
The professor hadn't taught us strings and the textbook Pascal was different from the Mac.
However, the school required me to maintain minimum credits to stay in a welding program I needed for a raise at work, so that by the time I realized how bad the class was, it was too late to drop.
I welded all day at school, programmed for 1-2 hours, went to class, then worked on the program all night at libraries at the UW. One of the UW students complained to a librarian about my smell from the welding; he knew I was junior college scum.
It seems ridiculous today, but it took me a week to pull it together. I sometimes cried from the frustration. I wrote everything out longhand on yellow legal pads at the UW libraries, welded 8 hours, then computer lab, and class, bus back to the UW. I was struggling in my arc welding class, too. I couldn't get the feel of striking an arc, and the men helped each other, but initially refused to help the only woman in the class.
My first solution was clumsy, but I got all the components to run, figured out reading strings, input/output, handling spelling errors. The text had a rigorous theoretical description of top down programming, and after I got the syntax and op system nailed, I used that to write an elegant and robust algorithm. I didn't realize until years later what I was doing, writing an algorithm.
Then late one night, library almost closing, I had it! Next day I entered the hand written program into a Mac at school, and it ran perfectly.
It was the solving, not the problem that was hard, and I bet most here can't imagine all the missing components I had to find for myself, but the lack of tools, prior knowledge, and information made solving it almost impossible. Today, I could Google all the help I needed for a zero-to-program in two hours, but I'd never feel the level of victory as I did that afternoon when all my classmates entered arithmetic problems into my calculator and got answers.
I went from a software engineer to starting my own CBD company. Most of what I do is sales now. Making the transition has been very difficult. It's a completely different world.
The biggest thing I've learned with sales is that the logic of an engineering mindset doesn't apply. With sales, you have to understand people. This sounds sort of basic and I wish I could communicate it better but it's true. Sales are all about social proof, making your customer feel special, and dealing with people problems.
For capital, I've self-funded from working in the software industry in San Francisco.
For CBD sourcing, initially, I was buying CBD to create products. This was a good idea to test products but a bad way to grow a business. Since realizing this, I've moved everything to Southern Oregon and partnered with a hemp farm. The combination of my understanding of the internet plus his high quality CBD rich hemp has allowed me to expand into other products while dropping my prices by 50%.
One time I figured out how to double integrate acceleration from a MEMS accelerometer to get displacement/distance. I did this for a fitness tracker for lifting.
Building healthy responses to mental ailment among some people who previously had no idea how such ailments manifest or affect people. The effort is still on!
As usual, debugging. The Intel 82599 Ethernet controller issue (featured here on HN and elsewhere) is easily the hardest issue I've ever worked through.
Everyone loves a good bug hunt story so I'll share one of the hardest technical problems I've ever solved, and certainly the one that makes the best story. Strap in; this is a long one.
The year was 2015 and the Christmas break was fast approaching. Unfortunately, while the rest of the office thinned out, another engineer and I were stuck debugging an increasingly urgent production issue. What had started weeks prior as some random intermittent failures in a few of our microservices had slowly escalated into a crisis where more and more services were experiencing failures. We didn't have great monitoring back then to trace any given request through the system's various microservices and figure out where the bottlenecks were - all we knew was that lots of requests were getting backed up somewhere.
We soon found Nagios metrics indicating that one particular critical service on a few boxes had been seeing steadily increasing CPU usage over the past days and weeks. It had reached a point where the service, which is normally heavily IO-bound, was actually now CPU constrained. Our suspicions therefore quickly centered on this service. Failures here could very well lead to the cascading failures we were seeing across our system. A small but increasing percentage of requests to this service were timing out. This made upstream services time out, which in some cases made their upstream services time out.
Once we had sorted through the chaos of cascading failures, we were pretty sure that this one critical service was the root cause of all of the trouble, so we restarted it, one slave at a time so as not to cause downtime for the whole product. Each instance came back up 100% healthy, with completely normal CPU usage. Odd. But sure enough, within a day, CPU usage was spiraling out of control again.
We knew the problem would just come back again if we kept restarting it, so we enabled JMX on the JVMs and attached VisualVM to take some thread dumps and run the profiler. After plenty of head-scratching at the stack traces and close examination of the code, we finally figured out what was going on...
One of our developers had helpfully provided an implementation of java.io.OutputStream for writing data back from the server to the client. The one thing you should know about OutputStream is that it's a blocking interface - if you write data to it, the data is written, and if there's a failure then it should throw an exception right then and there, before the method returns, so that the caller knows there was a failure. The one problem with this is that our Java services were based on Netty, which is based on java.nio, which is asynchronous. When you write to a Netty channel, you don't get feedback right away on whether the data was successfully written to the underlying socket. Instead, you get a java.util.concurrent.Future which will eventually tell you whether the write succeeded. It should be obvious that there's a major impedance mismatch between trying to implement a blocking I/O interface using nonblocking I/O primitives. Our developer had decided to handle this by kicking off the I/O and then simply completely discarding the Future that the write call returned!
What would happen is that sometimes a client would disconnect while we were in the process of returning a response, but the application would never find out because it never checked the status of those discarded Futures. So the application would happily keep streaming data through this OutputStream back to the client. Every time the buffer was flushed, the data would make its way through the Netty pipeline all the way to the bottom, where the write would fail. This generated a rather large stack trace. This stack trace was written to disk - and because it was written directly to standard error rather than via the normal logging infrastructure, we never saw it. But it was being written nonetheless, and every flush of the buffer would cause a new stack trace to be generated and written out. It turns out that this is a rather expensive thing for the JVM to do in a tight loop for dozens or hundreds of concurrent connections.
Our solution was to do the obvious thing that should have been done in the first place and check the results of the damn futures! Every API in the service was depending on there being a blocking OutputStream to write data to, and we didn't particularly want to do a major refactor over to async I/O and push out such a high-risk change right before the Christmas break. So we made a seemingly-harmless change, very minor, which should have fixed the issue and cleared us for a well-deserved vacation. Where before the code was letting the Future fall out of scope unused, now we made it block on its result, so that it could throw an exception to stop the application if there was a failure.
When we deployed this fix and restarted the servers, the CPU usage remained normal. We breathed a sigh of relief. Then, a few hours later, things got interesting.
On one of my monitors I happened to be tailing the logs on one of the servers and noticed, all of a sudden, a cascade of these messages flowing down my terminal:
WARN c.s.j.rep.utilint.ServiceDispatcher - Server accept exception: class java.io.IOException : Too many open files
Sure enough, netstat showed thousands upon thousands of open TCP connections, enough to exhaust all of the file handles that the Linux kernel was willing to allocate to the JVM.
netstat reported that these connections were almost all stuck in a CLOSE_WAIT state. What the hell did that mean? I had taken a couple of networking courses in college, one lab course from a tech's perspective and another programming course from an engineer's perspective, so I was pretty handy with the tools and the general theory. I went home to get a textbook and found the TCP state diagram:
We took some packet dumps with tcpdump to make sure that the client wasn't misbehaving. It wasn't. After noodling over the packet dumps, the state diagram, and RFC 793 for a bit, it became clear that the client was sending a FIN segment, but our application was never acknowledging that by calling close on the socket. The TCP stack would hold the connection open until it reached a timeout, at which time it would close the socket for the application. But the application quickly supplied more stuck sockets to replace the ones killed by the networking stack.
Christmas Eve arrived and I needed to get on a flight back to the East Coast to visit family. We decided that over the break we'd conduct rolling restarts of the servers to clear out stuck sockets before they reached the maximum, and pick the problem up after the new year.
After the holiday, we were able to locally reproduce the issue by introducing a lengthy sleep stage inside our Netty pipeline. We went through the Netty library's source code line by line to see exactly what was happening. As we picked through the code one of us stumbled upon the following Javadoc comment:
Thank Christ for the Java community's pathological love of absurdly prolix Javadoc. That footnote broke the case wide open. If you don't see the issue yet, I'll lay it out... in the next comment - HN won't let me post the whole thing in one comment.
* Netty has a boss thread that accepts incoming connections and assigns each successfully-opened socket to a single particular I/O thread.
* Each I/O thread runs an infinite loop that repeatedly waits for activity on its assigned sockets (using epoll/kqueue/select) and runs each received TCP segment through our Netty pipeline on that thread.
* Usually when an I/O thread writes to one of its own sockets, the write takes place synchronously. However, crucially, it may defer at least part of the write until later, for example if the kernel’s buffer is full. The write would then be performed on a later loop when the selector (epoll/kqueue/select) reports that the socket is ready for writing.
* We were writing to a channel and then blocking on the result Future to see if it succeeded. Since we were writing from the I/O thread, the write would usually be performed synchronously so the returned future would already be complete and the application would continue. However, sometimes the write wouldn’t fully complete and the future could not be completed until the next time around the selector loop. But the I/O thread was blocked, so the loop couldn’t proceed, so the future would never complete.
* Since the I/O thread was stuck, it couldn’t respond to any further messages from any of its managed sockets. Eventually the client would give up and send a FIN segment, and the kernel’s TCP/IP stack would put the socket in the CLOSE_WAIT state. Usually the I/O thread handles this in its worker loop by calling close on the socket, but it was stuck so this code never ran and the socket would never close.
* The boss thread was still running, so the system continued to accept new connections and assign some of them to the stuck I/O thread.
To add insult to injury, Netty actually has a deadlock detector to prevent this scenario from occurring. Unfortunately, one of the libraries that we were using at the time is also based on Netty, and they explicitly disabled the deadlock checker, globally!
So there you have it. Don't block in your Netty worker threads. This should have been obvious at the time - I had worked with Netty before - but we were too busy investigating to sit back and just think. If I had stood in front of a whiteboard for an hour, I suspect I could have worked it out, but instead we spent anxious days tearing our hair out attacking the problem by stepping through multiple threads of code in a debugger (racing timeouts is always fun) and taking endless stack traces and thread dumps.
The short-term mitigation ended up being trivial: we could still block the worker thread; we just had to check to see if the channel was already closed first! This is not a perfect fix, but it let us preserve the illusion that the application code had a real bona-fide blocking java.io.OutputStream. The long-term solution is to switch the API handlers over to using real async I/O.
To this day, my fellow engineer investigating this issue refers to this saga as the Christmas Miracle bug. :)
Here's one that stands out from my previous career in cellular telecom.
The story goes along the lines of, our engineering team purchased a vendor supported BIND solution, to replace our unmaintained linux bind servers. Sometime after that, our GRX provider (one of the companies that connect various cellular networks together), had one of their DNS servers down for replacement, and an outage on 1 of 2 working DNS servers. Our problem was, we didn't failover to the last working server, and all inbound roaming (users from other countries coming to Canada) was down (cached stuff still worked until the caches started expiring)
In cellular roaming for IP services (circuit switched voice works differently), the routing per APN back to the home network is done using DNS. However, in investigating this and similar problems, I eventually found that this wasn't fully standards compliant. And even when talking to our DNS vendor about this problem, it was difficult because they weren't familiar with the quirks of the GSMA recommendations.
The outage was eventually recovered, but the question remained why did it happen?
This took place over a 2 or 3 weeks period, and I don't remember the exact sequence of events, conference calls, etc now that it's been several years. But I do know I ended up building a simulation of 3 networks (My Own, the GRX Provider, and a roaming partners network). Using dig to datafill each server with the exact response from the partner. This allowed me to control the startup / failure of each server across the simulation of the 3 companies.
Using this simulation I eventually found the root cause.
When BIND started, it would take our root hints, load them into it's cache, and begin to perform AAAA queries for the root servers. One of the upstream servers would respond with 0 records, the other would respond with NXDOMAIN. The server that responded with NXDOMAIN, would subsequently get deleted from our BIND servers cache, and would no longer be used as a root.
The next question was why?
After some sleuthing through the DNS RFCs, I eventually found the answer. There are two ways for a DNS server to return that an answer to a query doesn't exist. Returning 0 records, and returning NXDOMAIN, and they have slightly different meaning. Returning 0 records, means that the label (think example.cm) exists, but the type of record does not (AAAA doesn't exist, but A/SRV/TXT/etc might). Returning NXDOMAIN means the the label doesn't exists, for any type of record, so don't bother querying me again for a different record type (There may have been some vagueness around this, I don't remember).
The second discovery, is that we had a typo in our configuration, what we configured as the name of that root server, didn't match what our GRX provider had configured, which is why we were getting NXDOMAIN on one but not all servers we had configured as our roots.
The next question was why were our old servers working? This typo was actually duplicated from our older servers... which still worked during that outage.
So using my simulation, I tested every version of BIND released across something like a 3 year period, until I found it. Older version of BIND interpreted NXDOMAIN the same as 0 record answer, and at some point, I can only assume they fixed a bug, that updated this interpretation of NXDOMAIN.
Yay for not finding out you're redundancy doesn't work, until it's actually triggered.
Anyway's, there were many challenging issues like this in my telco days. This probably isn't close to the hardest, but hopefully made sense to those who don't have a background in telco standards.
Wow, so many of these sound so much better than mine, but I'm going to share anyway.
We had an audit requirement to gather all AD accounts (computer, user, group) with group memberships in all domains[0], discover all servers, login and acquire their local user/group ACLs/config (win/linux), discover all databases (MSSQL, MySql, Postgres and Oracrap), acquire all of the ACLs in the databases, themselves, and a variety of application-specific credentials. This had to be completed once every 24-hours in order for other compliance operations to complete in time. We purchased a system to do this[1]. It managed to hit all AD domains in 4 days. A few more and it could get the Windows server account ACL information. We gave up adding anything else and a coworker took it upon himself to rewrite portions of it in a scripting language, which due to his shocking abilities in said language, netted Active Directory environments in 24-hours.
I wrote a plug-in based system that was as lockless as could be with a thread scheduler which spawned old-school threads, increasing count when capacity was available, decreasing when it wasn't. This had to be done because the options available to me in C# 2.0 wouldn't reach the required 24-hours and I had to substantially beat it if they were going to let me build this thing out. When I was done with the MVP, I was handling all of AD in under 4 hours.
The component that was going out and collecting data was the slowest and most difficult to optimize, partly because we'd flood a "pretty beefy for a link from the US to Argentina/Brazil/Venezuela" but that's not saying a lot, so I split things up, made the processor into two components, one with an authenticated web service that handed off to a processor when it received anything. I ended up using the oldest, most crotchety method to communicate between the two apps because the web service ran as a down-level account and the processor ran with incredible permissions (at least on the read-side).
It was horrible on every level. The scope was massive - more than 20,000 boxes with the desire to even collect file system ACLs at some point. The implications of screwing something up on the security side were daunting -- the firewall rules and layer upon layer upon layer that went into securing each of these components (not just on the network, but isolating them as much as possible using built-in OS ACLs and rules). Doing threading ... correctly ... in C# is easy to screw up and I think I managed to actually witness every single thing that goes wrong when you fail to protect shared data that's mutable. The available options for thread scheduling in .Net and Windows didn't fit well with my use case (I tried several with tweaked parameters in an attempt to bend them to my will). I ended up having to create a state machine to monitor and compare various counters in the application to decide, upon receiving a request, if it would improve speed to spin up a new thread in a thread pool that I also hand-rolled. These were certainly naive implementations, but I could find no other way to make this work the way it did. At the end of the day, I was able to scan the entire set of required environment components and grant, revoke, or apply a custom permissions rule to anything I could scan. Rather than schedule it to run daily, I just had it continue to repeat after it completed, meaning the completion time ranged from about 3-hours to about 12-hours depending on traffic/server loads. This meant when a new hire started, they received access around 4:00 AM to every system in the entire company that their job required, and when they left, their access was cut immediately, everywhere, due to a priority system I also added to the system.
This project came with a mountain of politics. Two, large, companies where one purchased the other and I was in a team on "the other" ("the other", thankfully, had said contractual requirements so the company who purchased us had no prior knowledge on how to do something like this). They strongly resisted rolling our own solution in this manner. In the end, it performed so ridiculously well, and was so easy to build a new "module" for (compared against the off-the-shelf product that we had both, coincidentally, purchased) that it was kept, upgraded, much of the threading complexity replaced with new features available in .Net 4.5 and hopefully a lot more (I left when .Net 4.5 was released).
[0] Accurately ... 14 domains with varieties of trusts, various misconfigurations, one case of three domains in the middle of migration to a single domain with the fun that SID history injects. sIDHistory, alone, represented a month of debugging to get it to return the correct account, Alice in Domain "A", is a member of Group "1" with her account in both Domain "B" and "A". Abstracted, Alice's two accounts are supposed to be seen as one account. Higher-level AD libraries will return either account regardless of what is joined to the group (there's ways to coerce it to return a specific domain, but not a way to coerce it to return the correct one without using a more painful library).
[1] It was one of those solutions where you buy a sort-of framework and pay the company to code modules for it that are custom to your environment in a DSL that is custom to the application. It works as well as you'd imagine.
Missing notifications for Android
I was the lead for Flock Android, a team messaging app which competes with the likes of Slack. We started receiving some weird complaints from users about missing notifications. Now since it was a business messaging app, notifications were critical. We tried out everything to reproduce this issue, including bombarding devices with notifications, trying out different networks but we just could not reproduce the issue. Worse we had a couple of incidents with devices in our team but logs showed absolutely nothing. Google GCM said that the notification was sent but there was no trace of the notification in the app's logs, it was as the the OS was swallowing notifications. And somehow popular apps like whatsapp, facebook were immune to these issues. The final nail in the coffin was the CEO himself missing notifications on some occassions. Google was unhelpful as usual with their non existent dev customer support. We were able to find a bunch of support articles on the internet where other apps had essentially listed down steps to turn off battery optimizations and that seemed to work ( Interestly whatsapp, facebook are usually added by default in these lists). But there was no way to do it automatically and we could also do it in response to a support ticket, which is painful for users.
After a couple of days we were able to encounter a few missed notifications on a oneplus device which we had connected to a machine taking system logs. And voila we got a few lines of what was happening.
Apparently some process called BgDetect figured out that our app was taking too much CPU and decided to kill it. Android being open source, we figured that we should be able to get source for it. But not only could we not find BgDetect in Android sources, it was non existent on oneplus sources too. We then got in touch with the marketing team and reached out to some contacts at oneplus, who directed us to their dev team. They asked for our apk and voila , in a week we were whitelisted along with the likes of Whatsapp and Facebook.
But the issue persisted on Xiaomi, Oppo and a bunch of chinese manufacturers and for those we still had to dish out steps to whitelist Flock in battery optimizations. We eventually figured out that moving from http to xmpp for android fcm notifications gave us delivery receipts for push notifications on device. On google phones and reputable manufacturers these delivery receipts also meant that app received the notification. On affected devices apparently the OS would give us a delivery receipt but not deliver it to the application. We built a double ack mechanism which also sent delivery receipts from the app. Based on the time difference between app delivery receipts and fcm delivery receipts we were able to figure out affected devices and send them directed bot messages to preemptively ask users to whitelist flock. You can read more about the double ack mechanism architechture here: https://hackernoon.com/notifications-in-android-are-horribly....
Google still refuses to even acknowledge the issue while problems like these are widespread: https://bit.ly/2QSzTKK
The most elusive bug I ever investigated was also one of my first, in my first job after graduation, back in the early 90s.
TL;DR; spend months investigating a problem that didn't actually exist.
I was put on an investigation into why the data acquisition interface we were developing was not working. It was designed to plug into the serial bus on a Coast Guard icebreaker. Since we couldn't develop on the ship, one of the other engineers made an hour long recording of the bus traffic, which we were able to play back in a simulated setup in our lab.
Unfortunately, the traffic on the bus was not as reliable as we expected. There were supposed to be fixed number of data channels being broadcast every second. But frequently, seemingly at random, channels would disappear and then reappear a few seconds later. This made it impossible to identify the different channels, since they were order dependent. It usually wasn't even easy to tell which channel was missing. Usually multiple channels would missing at the same time. Our data readings would get corrupted and everything would fall apart. Since the end-product was to be a real-time engine diagnostics system, this was unacceptable.
When I was hired, the company had already been working at the issue for some time. I was tasked to try to find a pattern in the drop-outs, so that they could be predicted and our interface parsing adjusted as necessary. I tried to find correlations in the intervals between drop-outs, and the length of the drop-outs. I tried to apply smoothing functions so the effect of the drop-outs would be dampened. I tried doing predictive analysis on the data channels, so that when a drop-out occurred we would at least be able to make a pretty good guess which channel was missing, and realign all the other channels. Months went by. Nothing worked.
Then, one day while I was mulling over an printout of the data recording (literally 100s of pages of nothing but columns of numbers), I noticed something odd: almost every time a drop-out occurred, a channel would transition from positive to negative, or from negative to positive. That is, before the drop-out a channel would be positive, and after the drop-out had ended, the channel would be negative. Or vice versa. But a channel would never be 0. I poured over the entire recording, and there was not a single 0 value to be found.
The data recording had been made by plugging into the serial port on the ice breaker's data bus and using some modem software to record the traffic on the bus. We quickly put together a test setup using the same modem software, and sure enough, when a 0 value was received the software would not record it. It would just bloop right over it. There was nothing at all wrong with our data acquisition interface. The problem was with the recording we were using the test it. We had a new recording made using different modem software, and the problem was gone.
Not the hardest and the exact technical details are a bit murky now, but super annoying nonetheless.
I had an external disk for extra data with an APFS volume as well as an NTFS partition for data when I was in Windows. Once I plugged it in under Windows, it must have nuked the partition table because of an 'unrecognized' partition or something, as I then rebooted back into macOS and noticed my APFS container wouldn't show up.
I opened disk utility and saw that it was marked as NTFS too. No my data!
I contemplated what to do and realized that it may be just the partition table gone and my data may still be intact, since I haven't actually formatted the drive myself, or anything similar.
The problem was, this was an APFS container, not a simple partition and the tooling was not very good in the Sierra days. So I had to manually calculate the sector at which I then created a new APFS container, but without formatting and indeed my data was there.
That episode ended up reducing my already little time spent in Windows to practically nothing these days. (As primarily a Linux user, I don't spend much time in macOS either, but Windows is forbidden on real-hardware since that fiasco).
Digging myself the fuck out of multiple compound interest debt traps, multiple times, while generally managing to subsist and have what I'm reluctant to refer to as "A Life."
A high tech toy for pets. I have a lot of experience in software at Unicorn telcos and also some other big names ...
But my god 'hardware is hard'.
Just some quick points for the uninitiated:
There are 'creative' aspects both to design, and to play, and then of course issues with pets. Industrial design to make the physicality work, some high tech secrets, a lot of prototyping on Arduino/Raspi. Hardware customization, PCBs, optimizing for manufacturing ... getting many pieces to fight just right together (I mean product/market mix, i.e. features, price - not just physical) with long dev cycles, waiting for parts, trying to predict the cost of various things at various quantities. Setting up relationships with overseas manufacturers - which is partly a product issue ... but mostly - working capital requirement.
If you're well funded up front, you can get 'very expensive' units built, put them in the hand of journalists, bloggers who will write about it. Even better - get your first production run in place and give them those to coordinate with first sales run. But who invests in hardware startups enough to support working capital? Not many - it's hard enough just operating on regular Angel ... but coming up with the money for the first 'x units' is a huge bet. If you can't get prototypes in the hands of buyers, bloggers (and good ones) it increases risk quite a lot.
So the biggest issue with factories is not so much 'getting it made' because ultimately if you're careful you'll definitely find someone competent to make it ... but it's how they get paid, which is related to the effect of the 'hotness' of your product. Factories are not interested in 'one off' runs, they want something that's hot, the hotter, the better the terms. Things like moulding etc. which drive up the initial unit costs - factories will eat the cost of this if they really want your business. And of course paying for the production ... if you have something white-hot or an operational history - they might give you credit or much better terms, which makes getting to market more accessible.
Finally there are the channel issues - buyers that want guarantees, the ability to return product ... retailers who don't put those terms of front but return a bunch of inventory anyhow. Just because. Every buyer wants a 'unique' aspect to differentiate their offer, but it's hardware, so that's hard, you have to get creative etc..
Online channels are quite fundamentally different than retail - you can sell at higher price points online to cash flush buyers, but in-store, things are extremely price sensitive. It hardly matters what you are selling: once you're past $30 with a retail physical thing ... it gets risky to the buyers who want to see big ad spends, lots of social media activity, yada yada.
This later part is why so many really cool products fail. They're just too expensive for physical retail channels: plastic needs to have 'velocity' or it's a big problem, and most 'cool things' are quite expensive. As soon as you have a decent camera, or wifi, or a high end processor ... it starts to get very hard to get prices down to a retail happy point.
And so many products tend to be 'hit driven' meaning it's hard to turn all of that into something more long lived. Consider how many toy companies come and go - they are so often 'betting the whole company' on every new line.
Unless you have some kind of well oiled machine with all of those things in place, it's hard to break into - you kind of need a hit, and serious backing/believers in order to create the initial momentum. There are a few SV companies who've done that, kudos to them.
Very fun but very hard, and very 'business' i.e. a lot less about 'fun factor' 'design', many more raw operational concerns than a software startup.
These are just some quick thoughts, by no means comprehensive.
Abstract: The Green Bay Packers are the only community owned team in the NFL. Legally they are a very unique organization - a non profit with stock-stockholders - in fact the NFL even considers them at a competitive advantage to all other teams because of that and has control over Packers right to offer/issue new stock to raise capital. But when the NFL does permit the Green Bay Packers to raise money by issuing new stock, the Packers file a No Action Letter to the SEC setting out their legal argument that Packers Stock is not a security or does not need to register. We published their original documents, our redline versions and our final version of the SEC letter and Public Offering Document. Our Delaware Certificate of Formation and Operating Agreement are also worth a look because they memorialize the LLC units being maintained and issued on the Ethereum Blockchain and include the addresses for the “Blockchain Stock” (ERC20) and public offering smart contract.
The SEC does not accept hypothetical No Action Letters so behind the cloak of anonymity of The Satoshi Nakamoto Trust we launched a Delaware LLC and submitted our legal argument. Initially intended to be a safe/sandbox startup to legitimize our legal framework, we memorialized the duration/powers/duties of the LLC into a decision tree - which isn’t but could further be memorialized in smarts contracts in future iterations - which begins to look a lot like a Delaware DAO.
SEC No Action Letter- Delaware LLC & Operating Agreement
1. The Green Bay Packer’s Proposed Stock Offering “No Action” Letter dated June 16, 1997 [1]
2. A redline version of the Green Bay Packer’s Proposed Stock Offering “No Action” Letter dated June 16, 1997 [2]
3. The Blockchain Stock Company’s Proposed Blockchain Stock Offering “No Action” Letter dated 31 October 2018[3]
a. Exhibit A - The Blockchain Stock Company’s Certificate of Formation (Articles) [4]
b. Exhibit B - The Blockchain Stock Company’s Operating Agreement [5]
Public Offering Document:
1. The Green Bay Packer’s Public Offering Document [6]
2. A redline version of the Green Bay Packer’s Public Offering Document [7]
3. The Blockchain Stock Company’s Public Offering Document [8]
Sent. Thanks we understand it’s extremely easy to blow off an idea that sounds like “legit ICO” especially when the SEC warns to look out for that language.
If you don’t find all the law to boring and get through it we would love feedback and/or questions.
We previously posted with the links, it was flagged before anyone read our work. Through email exchange Dang agreed we could repost but suggested not post the links to our work. Flagged again, which is strange because it is upvoted.
I would love to know what you say when someone asks you where you would prefer to eat out. I’m imagining something like “the tastebuds attached to my mind have not and can not experience the entire range of culinary experiences that humanity has and will produce so I can not answer your question.”
