• tl;dr sec
  • Posts
  • Hasnain Lakhani - fmr Software Engineer at Meta

Hasnain Lakhani - fmr Software Engineer at Meta

Author

Hasnain Lakhani
September 02, 2022

Hasnain Lakhani: personal site, Twitter, LinkedIn

Tell us a little about your experience as a Staff Engineer: where did you work, your title and generally the sort of work you and your team did.

I was most recently at Meta (though I’ll always call it Facebook).

I appreciated that we didn’t really do titles for engineers, so I was a Software Engineer. I worked on the Dynamic Analysis team within the Product Security org. The org has a mix of both security and software engineers. My team was one of several responsible for building tools to find security bugs at scale. Facebook has a number of archetypes for Staff+ engineers, which were written about publicly. They start at the Senior Staff level, but some of this bleeds into the Staff level. I was somewhere between the generalist and the tech lead, depending on who you ask.

For the last few years the main thing we worked on was fuzzing. Fuzzing is a pretty open ended problem - the team’s work was all over the place: from working on docs and guiding engineers on how best to get started, to fighting the compiler and build system, to building large scale infrastructure to run fuzzers at scale, to creating pretty UIs to view graphs, to writing advanced libraries to make harnessing easier. I’m leaving a lot off here, but you get the general idea.

Broadly, my mandate was to ensure the team hit its goals. I spent a decent chunk of my time on communication and collaboration. This included talking to our partner teams to ensure we met their needs and understanding what was coming down the pipe, and to engineers on my team to share that context, learn about how work was progressing, etc.

My engineering work was all over the place: I bounced between design reviews, code reviews, debugging, implementing prototypes/tricky features, or hopping on urgent things. As a rule, I tried to focus on unblocking others so they could be the most productive.

As a Staff engineer, I worked with my manager to ensure the team was successful, including on the people side. I was responsible for providing detailed feedback about other team members in real time.

It worked out quite well: at the start of a planning cycle we’d sync up on development opportunities, and we’d work on the right project assignments and support to ensure everyone was happy. I’d also provide technical feedback/vision/direction to my manager on key decisions – and it felt like a partnership, with us working together as peers. It was also really great to know we had each other’s backs: sometimes I’d need to focus on technical stuff and have them cover a collaboration; or they’d need to go off into a bunch of performance review meetings and I’d help keep the boat afloat.

TL;DR: Together, my manager and I were responsible for the people and technical side of things for our team respectively, but we worked together on that and the lines sometimes blurred, which all worked out.

What two or three factors were most important in you reaching Staff? How have the companies you joined, your location, or your education impacted your path?

My path to Staff was atypical. I “grew up” as an engineer at Facebook. I started off working in Ads, and very quickly learned how to be productive and build scalable systems. Growing to staff, however, took a mindset shift. I had to learn (the hard way!) about the importance of direction, vision, and people and becoming a well rounded engineer rather than just someone that wrote a bunch of code.

The first key insight was just that: realizing that the job effectively changes. A Staff engineer isn’t graded on how much code they write. They’re graded on the business outcomes their team is able to achieve. That took me quite a while to get; and only came after many failed attempts.

I also did a stint as a manager for a few years. Management isn’t for everyone, but if you’re open to it – it’s a great way to become a better engineer. I now joke (only half joking!) with my team-members that being an engineer after being a manager is like engineering on easy mode. Collaboration, leadership, vision-setting, and prioritization become so much easier after you’ve been forced to do that as your primary job for a while.

Management was a great forcing function for me to stop thinking about code and think about the bigger picture. It also helped me figure out how to successfully work with my management chain. I learned the importance of regular, effective communication: syncing over chat multiple times a day (sharing feedback/ideas/concerns in real time) and we would quite often run out of time in our 1:1s.

Lastly, I was very lucky throughout this process - Facebook has a ton of resources for career growth. I was able to talk to a bunch of amazing engineers, get their advice, and get some hands-on mentorship - I’m eternally grateful to my mentors for being so patient with me.

This is a whirlwind summary of pivotal experiences I’ve had. I’ve written about this in much more detail than you likely want on my blog

Some of Hasnain’s Public Work

You’ve mentioned that you feel like “a software guy who happens to have worked on security problems” - do you think there was any relationship between the domain you worked in and your path to Staff?

I definitely think it’s easier to grow into the Staff role when working in the security domain. One key aspect of working in security is that you have to get good at collaboration. It’s not just that you have to be great at explaining what security is and why people should care - you have to interact with pretty much every engineering team at the company. In general, I found that software engineers within our org - regardless of level - tended to be a bit better at collaboration than equivalently-leveled engineers in other orgs. We just had a lot more exposure and opportunities to get better at it.

At the same time, working in security gives you easy access to problems that are of the right ‘scope’ for a Staff engineer. Security is a critical priority; and a lot of the problems are technically challenging and quite impactful - so the opportunities are there, and you have to spend a little less time coming up with problems to solve (to be clear: you still have to spend a lot of time coming up with a precise problem definition). That means you have your pick of areas to invest in.

Where do you feel most impactful as a Staff-plus Engineer? A specific story would be grand.

I think I’m the most impactful (and happiest) when I’m working on open-ended problems that meaningfully move the needle on our team’s goals. It’s so satisfying to be able to identify a problem, motivate people to solve it, assemble a team to do the work, and then go in and be part of the team that nails the problem.

Unfortunately most of my favourite stories are under NDA – I can talk about a story that exemplified some of these traits (but didn’t quite meet the Staff scope): rewritting our fuzzing scheduler in Rust . I identified that our team’s needs had grown beyond our existing scheduler – from there, I went in and collaborated with our partners to understand what their needs would be 3+ years out, looked at our own team’s projections, and came up with a proposal. With that proposal in hand, I did a bunch more research to come up with a feasible system design and prototype, then worked with a few engineers on my team to build the new system and migrate us over. It was a great little exercise and it’s hard to beat that feeling of shipping something new that lets you do so much more.

Can you think of anything you’ve done as a Staff-plus engineer that you weren’t able to or wouldn’t have done before reaching that title?

Answering the question literally: no. I’ve been lucky that (my part of) Facebook didn’t focus too much on titles so even as a new grad engineer I was able to have my proposals taken seriously - if there was something I wanted to do, I got the chance to do it.

Zooming out a little, though, there are definitely things I can now do that I couldn’t before. I pay a lot more attention to the project as a whole rather than the specific engineering problem we’re solving. This means I try to ensure the business outcomes are met (rather than exclusively focusing on a tech solution); that people are happy with their work and the work split allows people to grow and play to their strengths, and that we use collaborations as an opportunity to build healthy relationships with other teams.

How have you sponsored other engineers? Is sponsoring other engineers an important aspect of your role? We’d love to hear a story.

I’ve spent a lot of time mentoring and growing engineers - both folks from my team and from the broader org. This is something I enjoy; but also it’s a requirement of being a Staff+ engineer. Sponsorship is a much higher bar, though, and I haven’t had the chance to do it right - there were more qualified sponsors around. I’m hoping to do it in the future, though!

What are some resources (books, blogs, people, etc) you’ve learned from? Who are your role models in the field?

I have too many role models to list! A lot of them are people that don’t like to be known too publicly unfortunately.

I do read a lot. Some pieces of advice that I’ve gone to over and over again:

Lastly, if I can do some shameless self promotion - for those that want to grow more as a Tech Lead, I wrote a series of posts based on a course I wrote which may provide some tactical advice.

What about advice for someone who has just started as a Staff-plus engineer?

Focus on the defining and delegating the problem at hand - resist the urge to solve it yourself!

As a final shameless plug, I wrote a lot more about this here: https://mhlakhani.com/blog/2022/08/on-being-a-staff-engineer/.