Team Management

GenAI is everywhere you look, and organizations across industries are putting pressure on their teams to join the race – 77% of business leaders fear they’re already missing out on the benefits of GenAI. Data teams are scrambling to answer the call. But building a generative AI model that actually drives business value is hard. And in the long run, a quick integration with the OpenAI API won’t cut it. It’s GenAI, but where’s the moat? Why should users pick you over ChatGPT? That quick check of the box feels like a step forward. Still, if you aren’t already thinking about how to connect LLMs with your proprietary data and business context actually to drive differentiated value, you’re behind. That’s not hyperbole. This week, I’ve talked with half a dozen data leaders on this topic alone. It wasn’t lost on any of them that this is a race. At the finish line, there are going to be winners and losers: the Blockbusters and the Netflixes. If you feel like the starter’s gun has gone off, but your team is still at the starting line stretching and chatting about “bubbles” and “hype,” I’ve rounded up five hard truths to help shake off the complacency. 1. Your Generative AI Features Are Not Well Adopted, and You’re Slow to Monetize “Barr, if generative AI is so important, why are the current features we’ve implemented so poorly adopted?” Well, there are a few reasons. One, your AI initiative wasn’t built to respond to an influx of well-defined user problems. For most data teams, that’s because you’re racing, and it’s early, and you want to gain some experience. However, it won’t be long before your users have a problem that GenAI best solves, and when that happens – you will have much better adoption compared to your tiger team brainstorming ways to tie GenAI to a use case. And because it’s early, the generative AI features that have been integrated are just “ChatGPT but over here.” Let me give you an example. Think about a productivity application you might use every day to share organizational knowledge. An app like this might offer a feature to execute commands like “Summarize this,” “Make longer,” or “Change tone” on blocks of unstructured text. One command equals one AI credit. Yes, that’s helpful, but it’s not differentiated. Maybe the team decides to buy some AI credits, or perhaps they just simply click over on the other tab and ask ChatGPT. I don’t want to completely overlook or discount the benefit of not exposing proprietary data to ChatGPT. Still, it’s also a smaller solution and vision than what’s being painted on earnings calls across the country. That pesky middle step from concept to value. So consider: What’s your GenAI differentiator and value add? Let me give you a hint: high-quality proprietary data. That’s why a RAG model (or sometimes, a fine-tuned model) is so important for Gen AI initiatives. It gives the LLM access to that enterprise's proprietary data. (I’ll explain why below.) 2. You’re Scared To Do More With Gen AI It’s true: generative AI is intimidating. Sure, you could integrate your AI model more deeply into your organization’s processes, but that feels risky. Let’s face it: ChatGPT hallucinates and can’t be predicted. There’s a knowledge cutoff that leaves users susceptible to out-of-date output. There are legal repercussions to data mishandling and providing consumers with misinformation, even if accidental. Sounds real enough, right? Llama 2 sure thinks so. Your data mishaps have consequences. And that’s why it’s essential to know exactly what you are feeding GenAI and that the data is accurate. In an anonymous survey, we sent to data leaders asking how far away their team is from enabling a Gen AI use case, one response was, “I don’t think our infrastructure is the thing holding us back. We’re treading quite cautiously here – with the landscape moving so fast and the risk of reputational damage from a ‘rogue’ chatbot, we’re holding fire and waiting for the hype to die down a bit!” This is a widely shared sentiment across many data leaders I speak to. If the data team has suddenly surfaced customer-facing, secure data, then they’re on the hook. Data governance is a massive consideration and a high bar to clear. These are real risks that need solutions, but you won’t solve them by sitting on the sideline. There is also a real risk of watching your business being fundamentally disrupted by the team that figured it out first. Grounding LLMs in your proprietary data with fine tuning and RAG is a big piece to this puzzle, but it’s not easy… 3. RAG Is Hard I believe that RAG (retrieval augmented generation) and fine-tuning are the centerpieces of the future of enterprise generative AI. However, RAG is a simpler approach in most cases; developing RAG apps can still be complex. Can’t we all just start RAGing? What’s the big deal? RAG might seem like the obvious solution for customizing your LLM. But RAG development comes with a learning curve, even for your most talented data engineers. They need to know prompt engineering, vector databases and embedding vectors, data modeling, data orchestration, data pipelines, and all for RAG. And, because it’s new (introduced by Meta AI in 2020), many companies just don’t yet have enough experience with it to establish best practices. RAG implementation architecture Here’s an oversimplification of RAG application architecture: RAG architecture combines information retrieval with a text generator model, so it has access to your database while trying to answer a question from the user. The database has to be a trusted source that includes proprietary data, and it allows the model to incorporate up-to-date and reliable information into its responses and reasoning. In the background, a data pipeline ingests various structured and unstructured sources into the database to keep it accurate and up-to-date. The RAG chain takes the user query (text) and retrieves relevant data from the database, then passes that data and the query to the LLM in order to generate a highly accurate and personalized response. There are a lot of complexities in this architecture, but it does have important benefits: It grounds your LLM in accurate proprietary data, thus making it so much more valuable. It brings your models to your data rather than bringing your data to your models, which is a relatively simple, cost-effective approach. We can see this becoming a reality in the Modern Data Stack. The biggest players are working at a breakneck speed to make RAG easier by serving LLMs within their environments, where enterprise data is stored. Snowflake Cortex now enables organizations to analyze data and build AI apps directly in Snowflake quickly. Databricks’ new Foundation Model APIs provide instant access to LLMs directly within Databricks. Microsoft released Microsoft Azure OpenAI Service, and Amazon recently launched the Amazon Redshift Query Editor. Snowflake data cloud I believe all of these features have a good chance of driving high adoption. But, they also heighten the focus on data quality in these data stores. If the data feeding your RAG pipeline is anomalous, outdated, or otherwise untrustworthy data, what’s the future of your generative AI initiative? 4. Your Data Isn’t Ready Yet Anyway Take a good, hard look at your data infrastructure. Chances are, if you had a perfect RAG pipeline, fine-tuned model, and clear use case ready to go tomorrow (and wouldn’t that be nice?), you still wouldn’t have clean, well-modeled datasets to plug it all into. Let’s say you want your chatbot to interface with a customer. To do anything useful, it needs to know about that organization’s relationship with the customer. If you’re an enterprise organization today, that relationship is likely defined across 150 data sources and five siloed databases…3 of which are still on-prem. If that describes your organization, it’s possible you are a year (or two!) away from your data infrastructure being GenAI-ready. This means if you want the option to do something with GenAI someday soon, you need to be creating useful, highly reliable, consolidated, well-documented datasets in a modern data platform… yesterday. Or the coach will call you into the game, and your pants will be down. Your data engineering team is the backbone for ensuring data health. A modern data stack enables the data engineering team to monitor data quality continuously in the future. It’s 2024 now. Launching a website, application, or any data product without data observability is a risk. Your data is a product, requiring data observability and governance to pinpoint data discrepancies before they move through an RAG pipeline. 5. You’ve Sidelined Critical Gen AI Players Without Knowing It Generative AI is a team sport, especially when it comes to development. Many data teams make the mistake of excluding key players from their Gen AI tiger teams, and that’s costing them in the long run. Who should be on an AI tiger team? Leadership, or a primary business stakeholder, to spearhead the initiative and remind the group of the business value. Software engineers will develop the code, the user-facing application, and the API calls. Data scientists consider new use cases, fine-tune their models, and push the team in new directions. Who’s missing here? Data engineers. Data engineers are critical to Gen AI initiatives. They will be able to understand the proprietary business data that provides the competitive advantage over a ChatGPT, and they will build the pipelines that make that data available to the LLM via RAG. If your data engineers aren’t in the room, your tiger team is not at full strength. The most pioneering companies in GenAI are telling me they are already embedding data engineers in all development squads. Winning the GenAI Race If any of these hard truths apply to you, don’t worry. Generative AI is in such nascent stages that there’s still time to start back over and, this time, embrace the challenge. Take a step back to understand the customer needs an AI model can solve, bring data engineers into earlier development stages to secure a competitive edge from the start, and take the time to build a RAG pipeline that can supply a steady stream of high-quality, reliable data. And invest in a modern data stack. Tools like data observability will be a core component of data quality best practices – and generative AI without high-quality data is just a whole lot of fluff.

Junior, Middle, and Senior are how a Software Engineer (SWE) career looks, right? But what does this mean? Different companies have different definitions, so borders are blurred. In this article, I’m going to share with you my considerations regarding levels in software engineering and try to rethink what the path might look like. A kind of disclaimer: this is only my vision and not the ultimate truth, so I’m happy to hear your feedback. What Is Wrong With Current Levels They are polysemantic. From what I can see on the market, from my experience, and those I tracked, different companies have different definitions of Junior/Middle/Senior engineers. Some of them have even more: Staff, Principal, and Distinguished engineers to have a better expression of seniority of highly experienced individual contributors. One of the key problems with “Senior SWE” is that people with absolutely different experiences might get this title. Technically, a Senior Mobile Engineer is not the same as a Senior Frontend Engineer or a Senior Backend Engineer. There are different specializations, and in general, that would not be correct to move from SME to SBE without any downgrade, but why? Logic dictates soft skills are the same, and life experience is the same as well (because it is still the same person). Only one thing changed - the ability to solve problems. You might be an extremely experienced Mobile Developer, but you have never solved issues within a web browser, problems with distributed systems, etc. So, let me take this particular criterion as a separator between levels. The first milestone is simple problem-solving. 1. Pathfinder: Random-Way Simple Problem Solver Originally, a pathfinder was someone who found or created a path through an unexplored or wild area. This term was often used to describe explorers or scouts who ventured into unknown territories, paving the way for others to follow. They were crucial in mapping new lands and navigating through difficult terrains. Sometimes, I hear, “You have to look after Junior dev, but the Middle one is working fully on one’s own." Is that true? Not. People on the Middle level usually do not care about the wider picture of the world, so they cannot make the best decision just by design. In any way, you need to look after every developer to help to stay within the project’s range of norms and not to allow to leak of over-engineered solutions. So, Random-way Simple Problem Solver (author: RwSPS short scares me as well), a.k.a Pathfinder, is able to solve atomic problems (not decomposable in a relevant way). Think about one task that someone else prepared for Pathfinder. Would you name it Junior? Middle? It doesn’t matter because you measure the results of these guys by their ability to solve business problems. Ok, Pathfinder will solve your problems. SOMEHOW, is it just enough? It depends. If you are creating a short-living project and all tasks are straightforward, a group of pathfinders will probably be enough. But for a long-living project, you need to solve problems, especially in a simple way. Otherwise, maintenance becomes a nightmare. 2. Specialist: Simple-Way Simple Problem Solver Specialist has deep, extensive knowledge and expertise in a specific field. Specialists are highly skilled in their area, often focusing on a narrow aspect of a discipline. Simple-way Simple Problem Solver (SwSPS), a.k.a Specialist, is the next level after Pathfinder. The key difference is that Specialists have enough experience to solve simple problems predictably in a simple way. That might be a proper framework/library usage, assembling solutions from existing components. For example: If Pathfinder tries to handle nulls by IFs, the Specialist will use nullable types to strict nullability by design. If Pathfinder might add logging at the start and the end of every method explicitly, Specialists will just use Aspect Oriented Programming (those who believe that AOP is unacceptable should throw tomatoes in the comments) If Pathfinder might refactor ten lines of code one by one. Specialists will use IDE’s multi-cursor to introduce changes in many places simultaneously. With experience and seeing more and more code that works, mastering tooling Specialists will provide more reliable solutions faster. This level is limited to atomic tasks only. Sounds like the next growth point! 3. Generalist: Random-Way Complex Problem Solver Generalist solves problems by synthesizing and applying knowledge from various domains. They are often effective in dealing with new or unforeseen challenges due to their adaptable and flexible approach. What is outside of a simple problem? Other simple problems! The source of all these simple ones is one or more complex issues that software engineers have to decompose before they start working. Let’s define a complex problem. In the context of this article, a complex problem is a problem that might be decomposed for the sake of improved predictability of implementation time. Also, a complex problem might consist of other complex problems that should be decomposed eventually. The key difference between a Random-way Complex Problem Solver (Generalist) and a Simple-way Simple Problem Solver (Specialist) is scale. A generalist is still able to solve simple problems in a simple way, but experience relevant to complex problems is not enough to follow the same approach for complex tasks. Here are a few examples: Generalists might design a new complex system starting with microservices and ignoring the fact that the customer-facing systems have <10 unique users in total yet. Generalists might start from on-premise db instead of just relying on managed services even if requirements do not specify that need and the key motivation is past experience. Generalists might bring redundant complex technologies from the previous company, ignoring the fact that the previous and the current ones are at different stages of business maturity. Getting experience in solving complex problems, Generalist starts finding ways to quickly get simpler solutions, and it means that the next level is coming. 4. Navigator: Simple-Way Complex Problem Solver Historically, navigators were crucial on ships and aircraft. In modern contexts, the term is used in roles requiring strategic planning and direction-setting, like in project management or leadership positions in companies. Counterintuitive that solving problems in a simple way is more complex, but the nature behind this fact is ignorance. At the beginning of your path, you have a high level of unawareness about already available solutions and ready-to-go components. Sometimes, they appear while you develop your own. Simple-way Complex Problem Solvers (Navigator) can deeply and seamlessly dive into an unknown environment, map their experience, find a simple solution, and basically have this expectation of something available instead of reinventing the wheel. A few examples: Navigator would never start by creating a marketplace if the business is about selling things but not SaaS. Navigator would research available opportunities before planning and designing. Navigator is fine with Google Sheets + Forms to launch the business. Navigator provides relevant solutions to the current business stage. Profits of the Alternative Level Classification Relative to the classical level set, this gradation: Is more transparent in terms of specific business requirements. Is measurable in practical tasks. Does correlate with experience. Does align expectations of a particular company. Conclusion The past “Senior” job title doesn’t say anything about your real ability to solve complex problems in the new company. People should align their skills with reality and not pretend to be seniors only because they already have this title. Movement through Pathfinder -> Specialist -> Generalist -> Navigator requires constant self-educating, so don’t waste your time. And please, don’t tell me that I showed myself as Pathfinder when describing such a simple topic in such a complex classification :)

Meetings are a crucial aspect of software engineering, serving as a collaboration, communication, and decision-making platform. However, they often come with challenges that can significantly impact the efficiency and productivity of software development teams. In this article, we will delve deeper into the issues associated with meetings in software engineering and explore the available data. The Inefficiency Quandary Meetings are pivotal in providing context, disseminating information, and facilitating vital decisions within software engineering. However, they can be inefficient, consuming a substantial amount of a software engineer’s workweek. According to Clockwise, the average individual contributor (IC) software engineer spends approximately 10.9 hours per week in meetings. This staggering figure amounts to nearly one-third of their workweek dedicated to meetings. As engineers progress in their careers or transition into managerial roles, the time spent in meetings increases. One notable observation is that engineers at larger companies often find themselves in even more meetings. It is commonly referred to as the “coordination tax,” where the need for alignment and coordination within larger organizations leads to a higher volume of meetings. While these meetings are essential for keeping teams synchronized, they can also pose a significant challenge to productivity. The Cost of Unproductive Meetings The impact of meetings on software engineering extends beyond time allocation and has financial implications. Research by Zippia reveals that organizations spend approximately 15% of their time on meetings, with a staggering 71% of those meetings considered unproductive. It means that considerable time and resources invested in discussions may not yield the desired outcomes. Moreover, unproductive meetings come with a substantial financial burden. It is estimated that businesses lose around $37 billion annually due to unproductive meetings. On an individual level, workers spend an average of 31 hours per month in unproductive meetings. It not only affects their ability to focus on critical tasks but also impacts their overall job satisfaction. The Impact on Software Engineering In the realm of software engineering, the inefficiencies and challenges associated with meetings can have several adverse effects: Delayed Development: Excessive or unproductive meetings can delay project timelines and hinder software development progress. Reduced Productivity: Engineers forced to spend a significant portion of their workweek in meetings may struggle to find uninterrupted “focus time,” which is crucial for deep work and problem-solving. Resource Drain: The coordination tax imposed by meetings can strain resources, leading to increased overhead costs without necessarily improving outcomes. Employee Morale: Prolonged or unproductive meetings can decrease job satisfaction and motivation among software engineers. Ineffective Decision-Making: When meetings are not well-structured or attended by the right participants, critical decisions may be postponed or made without adequate information. Meetings are both a necessity and a challenge in software engineering. While they are essential for collaboration and decision-making, the excessive time spent in meetings and their often unproductive nature can hinder efficiency and impact the bottom line. In the following sections, we will explore strategies to address these challenges and make meetings in software engineering more effective and productive. The Benefits of Efficient Technical Meetings in Software Engineering In the fast-paced world of software engineering efficient technical meetings can be a game-changer. They are the lifeblood of collaboration, problem-solving, and decision-making within development teams. In this article, we’ll explore the advantages of conducting efficient technical meetings and how they can significantly impact the productivity and effectiveness of software engineering efforts. Meetings in software engineering are not mere formalities; they are essential forums where ideas are exchanged, decisions are made, and project directions are set. However, they can quickly become a double-edged sword if not managed effectively. Inefficient meetings can drain valuable time and resources, leading to missed deadlines and frustrated teams. Efficiency in technical meetings is not just a buzzword; it’s a critical factor in the success of software engineering projects. Here are some key benefits that efficient meetings bring to the table: Time Savings: Efficient meetings are succinct and stay on topic. It means less time spent in meetings and more time available for actual development work. Improved Decision-Making: When meetings are focused and well-structured, decisions are made more swiftly, preventing bottlenecks and delays in the development process. Enhanced Collaboration: Efficient meetings encourage active participation and open communication among team members. This collaboration fosters a sense of unity and collective problem-solving. Reduced Meeting Fatigue: Prolonged, unproductive meetings can lead to fatigue, hindering team morale and productivity. Efficient meetings help combat this issue. Knowledge Sharing: With a focus on documentation and preparation, efficient meetings facilitate the sharing of insights and knowledge across the team, promoting continuous learning. We will delve into a five-step methodology to achieve these benefits to make technical discussions more efficient. While not a silver bullet, this approach has proven successful in many scenarios, particularly within teams of senior engineers. This methodology places a strong emphasis on documentation and clear communication. It encourages team members to attend meetings well-prepared, with context and insights, ready to make informed decisions. By implementing this methodology, software engineering teams can balance the need for collaboration and the imperative of focused work. In the following sections, we will explore each step of this methodology in more detail, understanding how it can revolutionize the way software engineers conduct technical meetings and, ultimately, how it can drive efficiency and productivity within the team. Step 1: Context Setting The initial step involves providing context for the upcoming technical discussion. Clearly articulate the purpose, business requirements, and objectives of the meeting. Explain the reasons behind holding the meeting, what motivated it, and the criteria for considering it a success. Ensuring that all participants understand the importance of the discussion is critical. Step 2: Send Invitations With Context After establishing the context, send meeting invitations to the relevant team members. It is advisable to provide at least one week’s notice to allow participants sufficient time to prepare. Consider using tools like Architecture Decision Records (ADRs) or other documentation formats to provide comprehensive context before the meeting. Step 3: Foster Interaction To maximize efficiency, encourage collaborative discussions before the scheduled meeting. Share the ADR or relevant documentation with the team and allow them to engage in discussions, provide feedback, and ask questions. This approach ensures that everyone enters the meeting with a clear understanding of the topic and can prepare with relevant references and insights. Step 4: Conduct a Focused Meeting When it’s time for the meeting, maintain a concise and focused approach. Limit the duration of the meeting to no longer than 45 minutes. This time constraint encourages participants to stay on track and make efficient use of the meeting. Avoid the trap of allowing meetings to expand unnecessarily, as per Parkinson’s law. Step 5: Conclusion and Next Steps After the meeting, clearly define the decision that has been made and summarize the key takeaways. If the discussion led to a decision, conclude the Architecture Decision Record or relevant documentation. If further action is needed, create a list of TODO activities and determine what steps are required to move forward. If additional meetings are necessary, return to Step 2 and schedule them accordingly based on the progress made. By following these key steps, software engineering teams can streamline their technical discussions, making them more efficient and productive while preserving valuable product development and innovation time. This approach encourages a culture of documentation and collaboration, enabling teams to make informed decisions and maintain institutional knowledge effectively. Conclusion In the fast-paced world of software engineering, efficient technical meetings play a crucial role, offering benefits such as time savings, improved decision-making, enhanced collaboration, reduced meeting fatigue, and knowledge sharing. To harness these advantages, a five-step methodology has been introduced emphasizing documentation, clear communication, and preparation. By adopting this approach, software engineering teams can balance collaboration and focused work, ultimately driving efficiency, innovation, and productivity.

By Otavio Santana CORE

The Agile Manifesto, a revolutionary document in the world of software development, emerged as a response to the inadequacies of traditional, rigid development methodologies. This article explores its origins, applications, and misuses, offering insights for engineering managers on how to effectively interpret and implement its principles. Origins of the Agile Manifesto In February 2001, seventeen software developers met at Snowbird, Utah, to discuss lightweight development methods. They were united by a common dissatisfaction with the prevailing heavyweight, document-driven software development processes. This meeting led to the creation of the Agile Manifesto, a concise declaration of four fundamental values and twelve guiding principles aimed at improving software development. Key Values Individuals and Interactions: The focus is more on the individuals involved and their interactions, rather than simply relying on processes and tools. This highlights the importance of team dynamics and interpersonal communication in achieving success. Working Software: The emphasis is on delivering a working software as the principal measure of progress, as opposed to creating comprehensive documentation. This does not undermine the importance of documentation, but rather stresses the need for a functioning product. Customer Collaboration: Instead of focusing solely on contract negotiation, there is a greater emphasis on collaboration with the customer. This fosters better understanding of the customer's needs, leading to a product that better fulfills those needs. Responding to Change: The ability to respond to change is prioritized over sticking rigidly to a plan. This emphasizes the need for adaptability and flexibility in the face of changing requirements or circumstances. These values represented a radical shift from the traditional waterfall approach, emphasizing flexibility, customer satisfaction, continuous delivery, and team collaboration. Application in Software Development The Agile Manifesto quickly gained traction in the tech world, leading to the development of various Agile methodologies like Scrum, Kanban, and Extreme Programming (XP). These methodologies share the core values of the manifesto but differ in practices and emphasis. Scrum, for instance, focuses on short, iterative cycles called sprints, with regular reassessments of tasks and goals. Kanban emphasizes continuous delivery and efficiency, while XP prioritizes technical practices to enhance software quality. Misuse in Software Development In spite of its widespread popularity and adoption in many industries, particularly in the realm of software development, the Agile Manifesto is frequently subject to misinterpretation or misuse. This is often due to a lack of understanding of its core principles or an attempt to apply it in contexts for which it was not originally designed. The common instances where the Agile Manifesto is not used as intended include: Overemphasis on Speed: A common misconception about Agile is that it is solely about accelerating the delivery process. This interpretation often leads to compromised quality and sustainability, resulting in burnout among team members and building up technical debt that may hinder future development. Agile is indeed about swift delivery, but not at the expense of quality or the well-being of the team. Ignoring the Importance of Documentation: Agile methodologies do favor working software over comprehensive documentation. However, this does not mean that documentation should be entirely neglected. Misunderstanding this principle can lead to a lack of essential documentation, which is critical for maintaining the software in the long run and ensuring scalability. It's important to strike a balance between creating working software and maintaining adequate documentation. Dogmatic Adherence to Specific Methodologies: Agile is often synonymous with methodologies like Scrum. However, treating Scrum or any other methodology as a one-size-fits-all solution can be counterproductive. Agile is fundamentally about flexibility and adaptation to the unique needs and circumstances of each project. Strict adherence to a particular method without considering the specific context can defeat the very purpose of Agile, which is to promote adaptability and responsiveness to change. Engineering Managers and the Agile Manifesto For those in leadership roles within the field of engineering, having a comprehensive understanding and ability to effectively implement the Agile Manifesto is of utmost importance. Here’s how engineering managers can approach, internalize and execute Agile principles within their teams: Embrace a Mindset of Flexibility and Adaptation: Agile is much more than a mere set of practices; it's an entire mindset. Managers should strive to foster a conducive environment that values and appreciates the ability to adapt with an openness to change. This involves cultivating a culture that encourages innovation and flexible thinking, positioning the team to quickly respond to any shifts or changes that may occur. Focus on People and Interactions: Building a culture within the team that is centered around collaboration is absolutely vital. It's essential to encourage open communication, regular feedback, and collective problem-solving. This not only involves dealing with issues as they arise but also proactively working to prevent potential problems through effective communication and teamwork. Balance Agility With Discipline: While embracing the fluidity and flexibility that comes with change, it's equally important to maintain a disciplined approach to development. This includes maintaining critical documentation, steadfastly adhering to quality standards, and not compromising on sustainable development practices. Balancing agility with discipline ensures that while the team is adaptable, the quality of work does not suffer. Customer-Centric Approach: Regular interaction and engagement with customers and stakeholders are key. Agile methodology is fundamentally about delivering value to the customer, and this necessitates continuous feedback and collaboration. Regular check-ins, updates, and discussions with customers ensure that the development process is aligned with customer needs and expectations. Tailor Agile to Your Context: There is no one-size-fits-all model in Agile. Agile principles are meant to be adapted, not adopted verbatim. Therefore, engineering managers should tailor Agile principles to their specific project, team, and organizational context. This involves understanding the unique needs and constraints of each project and making necessary adjustments to ensure that the Agile principles are applied in a way that is most effective for the given context. Conclusion The Agile Manifesto marked a paradigm shift in software development, advocating for more flexible, iterative, and collaborative processes. While its principles have significantly influenced modern software development practices, it’s important for engineering managers to understand and apply these principles judiciously. Misinterpretations and rigid methodological adherence can lead to the very pitfalls Agile seeks to avoid. Ultimately, Agile is about creating better software, fostering better teamwork, and satisfying customers, and should be seen as a flexible guide rather than a rigid doctrine.

Embarking on the exciting journey of bringing a product from idea to market requires careful planning and storytelling. Product managers play a crucial role in defining and guiding the success of a product. From the inception of an idea to its market launch, product managers have to navigate through various challenges and make strategic decisions. As a product manager, crafting compelling narratives and strategies is key to success. As the LLM is disrupting the market PMs can use LLMs to build effective strategies at each stage of the product lifecycle to improve their productivity. This article is all about identifying the life cycle from ideation to market and how we can use prompt engineering to query an LLM model and increase productivity as a product manager. Product management is the art of turning ideas into experiences, challenges into opportunities, and dreams into tangible realities. A great product manager crafts not just solutions but stories that resonate with the hearts and needs of users. A Large Language Model (LLM) are advanced artificial intelligence system, primarily based on transformer architectures. and are powerful language models known for their broad language understanding and generation capabilities. Trained on vast datasets, they excel in understanding and generating human-like language. They offer powerful capabilities in natural language processing, enabling tasks such as crafting compelling narratives, generating creative content, and interpreting user queries with high accuracy. Leveraging LLMs empowers product managers to streamline communication, enhance user experiences, and extract valuable insights from textual data throughout the product development lifecycle. Integrating these models requires understanding their architecture, training processes, and the art of effective prompt engineering. Prompt engineering is the practice of crafting text in a way that a generative AI model can interpret and comprehend. A prompt, which is natural language text outlining the task for the AI, can take various forms, such as a query, command, feedback, or a detailed statement with context and instructions. The process involves formulating queries, specifying styles, offering context, or assigning roles to the AI. Additionally, prompts may include examples for the model to learn from, employing a few-shot learning approach. Effective prompt engineering is like giving clear instructions to a clever robot friend. It's about crafting questions or tasks in a way that helps the robot understand exactly what you want. It's like speaking their language to get the best results! Let's discuss a few prompt examples for PMs in different phases of product development to get help in documenting and crafting quality stories with meaningful tasks. The prompts are just hints, a little more effort and you can be creative and add more to add it for sure get the most productive results. We will consider a "To-Do" list as a product for all examples. Idea Exploration: At the inception of the product journey, we can dive into the story of how the idea was born. We have to discover the issue it wants to address, identify possible challenges for users, and imagine how it fulfills the demands of the market. At the beginning, unravel the story behind your product's idea. Imagine a time when someone realized how hard it was to remember all the tasks for the day. This realization led to the idea of a smart to-do list app that not only stores tasks but also sends friendly reminders. Prompt: "Generate a detailed narrative outlining the initial spark of the product idea. Describe the problem it solves, potential user pain points, and how it addresses market needs." User Persona Development: Create a story around the primary user of your product. Define their characteristics, goals, and challenges. Illustrate how your product becomes an essential part of their daily life, addressing their specific needs and concerns. Create a story about the primary user of your product. Picture a friendly neighbor named Tapan, a busy professional who struggles to stay organized. Describe how your smart to-do list app becomes Tapan's assistant, making his life easier and more enjoyable. Prompt: "Create a story around the primary user persona for the product. Define their characteristics, goals, and challenges. Illustrate how the product will enhance their daily life or address specific pain points." Competitive Landscape Analysis: Navigate through the competitive landscape, telling a strategic tale of key competitors, their strengths, and weaknesses. Share how your product will stand out and carve its path in the market. Navigate through the competitive landscape by telling a story. Consider a marketplace where various to-do list apps exist. Define how your app stands out by offering a delightful experience, combining simplicity with powerful features. Prompt: "Compose a strategic analysis of the competitive landscape. Identify key competitors, their strengths and weaknesses, and how our product will differentiate itself in the market." Value Proposition Crafting: Craft a compelling story that highlights the core value your product offers. Showcase unique features and benefits that distinguish it from other solutions, aligning seamlessly with the needs of your target audience. Craft a compelling story around the core value your product offers. Imagine a conversation with a user named Shree. She loves how your app not only keeps her organized but also adapts to her preferences, creating a personalized and stress-free experience. Prompt: "Articulate the core value proposition of the product. Describe the unique features and benefits that set it apart from existing solutions. Consider how it aligns with the target audience's needs." MVP Definition: Focus on the story of your Minimum Viable Product (MVP), detailing key features prioritized for the launch. Consider scalability and user adoption as you show the way for a successful introduction to the market. Visualize a small group of users who eagerly try out your app's basic features. Their feedback becomes a crucial part of the story, helping you shape the app into something that truly meets their needs. Prompt: "Outline the Minimum Viable Product (MVP) for the initial launch. Define the key features and functionalities that will be prioritized to deliver value quickly. Consider scalability and user adoption." User Journey Story: Map out a user's journey from discovery to becoming a loyal customer. Focus on the touch points, anticipate challenges, and reveal how your product provides solutions at every step of the way. Map out a user's journey with a story. Follow Tapan as he discovers the app, starts using it daily, and eventually becomes a loyal user. Narrate how the app seamlessly fits into different aspects of his life, from work meetings to weekend plans. Prompt: "Map out the user journey from discovering the product to becoming a loyal customer. Describe touchpoints, potential challenges, and how the product addresses user needs at each stage of the journey." Go-to-Market Strategy: Craft a strategic narrative for your go-to-market plan. Define your target audience, outline promotional channels, and share key messaging. Explore pre-launch activities, launch day execution, and post-launch engagement. Imagine Shree sharing her experience with friends and colleagues, creating a buzz around the app. Describe how your marketing campaign spreads the word, making the app a must-have for busy professionals like Tapan and Shree. Prompt: "Develop a comprehensive go-to-market strategy. Define the target audience, channels for product promotion, and key messaging. Consider pre-launch activities, launch day execution, and post-launch engagement." Iterative Development Plan: Tell the story of how your product evolves with an iterative development plan. Discuss how user feedback is collected and integrated into future releases, embracing an Agile approach for continuous improvement. Picture a scenario where user feedback leads to new features, making the app even more user-friendly. Describe an iterative journey where each update brings joy to users like Tapan and Shree. Prompt: "Create a phased plan for iterative development. Outline how user feedback will be collected and incorporated into future releases. Consider the Agile methodology and continuous improvement." Marketing Campaign Concept: Paint a vibrant picture of your marketing campaign. Set the theme, outline key messaging, and choose the most effective channels for promotion, both online and offline. Imagine creating a fun and relatable video featuring Tapan and Shree. Picture them sharing how the app has become an essential part of their lives, adding a personal touch to your marketing strategy. Prompt: "Craft a concept for a captivating marketing campaign. Define the theme, key messaging, and channels for promotion. Consider both online and offline strategies for maximum reach." Metrics for Success: Conclude your journey by defining metrics for success. Share how these indicators align with overarching business goals, reflecting user satisfaction, adoption, and retention. Picture a dashboard filled with positive numbers – increasing user engagement, high satisfaction rates, and a growing community of users who appreciate the simplicity and effectiveness of your smart to-do list app. Prompt: "Specify key performance indicators (KPIs) to measure the success of the product. Outline how these metrics align with overarching business goals and reflect user satisfaction, adoption, and retention." For product managers, these prompts become the tools to shape a compelling narrative that guides their products to success. As we conclude, remember: "Embarking on the product journey is like telling a captivating story. With these simple yet powerful prompts, you have the tools to shape your narrative and guide your product to success. Let your story unfold, embracing each stage of the journey with enthusiasm, adaptability, and a keen eye for the needs of your audience." Example of a Story Written by Chat-GPT With a Good Prompt. Prompt: I want you to act as a Product Manager. Define clear and concise Gherkin-style stories for the "Where is My Order" feature on an online retail platform, ensuring a seamless and user-friendly order tracking experience. Make sure that you write the story and various sub-tasks associated with it and write the functional requirements as an objective for completing the story. Gherkin is a format for writing executable specifications, commonly used with behavior-driven development (BDD) tools. Remember that we've only scratched the surface. We've explored how these language models can be productive in crafting compelling product narratives, and we've taken baby steps at effective prompt engineering. There's so much more to discover! The landscape of language and AI is vast, and as we continue this adventure, there's always more to learn, experiment with, and explore. Keep that curiosity alive! LLM is causing quite a disruption in the market scene. AI is the co-pilot in the journey of product management, navigating through complexities, predicting user needs, and ensuring a smooth ride to success. Further Learning: ChatGPT Prompts for Agile Practitioners Ethical Prompt Engineering: A Pathway to Responsible AI Usage Prompt Engineering: Unlocking the Power of Generative AI Models Let's keep the conversation going! I'd love to hear your thoughts on Large Language Models and prompt engineering. Feel free to share your experiences, ask questions, or even share your tips. Your feedback is like a compass guiding me through this language adventure! Drop your thoughts below, and let's explore the world of words together.

By Manas Dash CORE

The Dynamic Squad model, a software development model, is the modern way of organizing software development teams focusing on a specific set of goal(s). A squad is a small group of people focusing on a particular goal or purpose. The size and lifespan of each squad will be different and will be based on the goal; hence, it's referred to as dynamic. Structure Each Dynamic Squad consists of at least four roles : Squad Structure Role Description Squad Lead Development leader who leads the squad Product Lead Provides Product Guidance Team Member(s) Developers, QA Tester Project Manager Manages the delivery plan for the squad Developer(s) are dedicated to one squad at a given time, whereas Squad Lead, Product Lead, and Project Managers can be involved in multiple Dynamic Squads depending on the needs in a particular organization. Life Cycle of Squad The life cycle of Squad consists of the following three phases. Create Senior Leadership appoints Squad Lead, Product Leads for Squad Squad Lead & Product Lead deep dives into the goal(s) and identify resources needed and appropriate team members from the available resource pool Squad is formed Function The squad starts detailed analyses of requirements associated with goal(s) and defines high-level solutions. The project manager outlines the overall delivery strategy and plan Squad now started putting the plan into motion using Agile methodology. Kanban is more suitable with this model, but depending on the nature of the work, Squad can decide and use Kanban or Scrum agile methodology. For proper execution highly recommend the following ceremonies: # Ceremony Frequency Purpose Participants 1 Squad huddle Daily To keep everyone in the squad aware of the progress Squad, PjM, PL 2 Requirements Refinement Weekly once or twice To refine the requirements on an ongoing basis Squad, PL 3 Retrospective Bi-weekly To come up with lessons learned and improvements Squad 4 Squad Leadership sync Weekly To monitor the squad's progress, Dependencies and risks and appropriate adjustments are made as needed. SL, PL, PjM Dissolve The squad enters into this phase once goals are achieved, and deliverables are pushed to production. The squad does a retrospective to identify what went well and what didn't go as per the plan and identifies potential improvements for upcoming squads. The squad gets dissolved, and members are released back to the resource pool so that they can be part of upcoming squads. Pros of the Model Focus on goals results in focus on delivering value Developer(s) are focused on one or more related goal(s), which reduces context switching and improves productivity, and also time to market is less The agility of the model allows quick adaptation to changes in scope, priorities, or business demands. The dynamic nature of this model allows leaders to do dynamic resource allocation as compared to strong team boundaries. Developer(s) get the opportunity to work on different things in a systematic order instead of multiple projects simultaneously, which boosts their morale. Squad members take ownership of their tasks, leading to higher accountability and commitment. Cons of the Model It's a cultural shift; hence, implementation from the traditional hierarchical model to this model could be challenging. Scaling this model with for large organization with a matrix structure needs an open mindset, continuous learning, and adaptation for successful implementation. Case Study I used this model for a group of 35+ developers. The group was responsible for Client Customization and integration development. For a while, operating in a scrum team-based model resulted in different challenges, as listed below. Developers need to switch context more often while simultaneously working on multiple client-specific projects, which is painful, less productive and takes a longer time to market. Difficult to maintain the team's backlog Due to strong team boundaries, it is hard to move resources between scrum teams based on need. To improve team efficiency and for a shorter time to market, there is a need for the group to operate in an innovative model that allows dynamic resource allocation. Developers have to focus on one thing at a given time. Started adapting the dynamic squad model. At a given point, there were 5-6 Squads, as shown below, with specific goals: After adoption, observed the following advantages Time to market improved by 50%, i.e., before, it used to take six weeks to deliver, whereas in the Dynamic Squad model team was able to deliver a similar type of work in about three weeks. The team happiness survey indicated that individual member’s happiness has increased by 25% since day-to-day operations became more structured and organized. Overall, leadership observed the success of the group after adopting the Dynamic Squad Model. Other examples where this model can be used are in Startups and big companies focusing on next-generation product development in a fast-paced start-up-like environment. Also, the latest emerging technologies complement this model.

Are you an Individual Contributor (IC) aspiring to step into a leadership role? The journey might seem daunting, but in this article, we'll explore the qualities and qualifications essential for a successful career as an engineering leader and how to actively cultivate them. Drawing from personal experiences and insights, my goal is to provide clarity on the expectations and responsibilities of an Engineering Manager (EM). Hope it will help you! Qualities and Qualifications Engineering management demands a unique skill set, combining technical prowess with interpersonal abilities learned through hands-on experience. EMs orchestrate, supervise, and streamline engineering projects, teams, and operations, requiring an intricate understanding of the software engineering lifecycle and business dynamics. Core Responsibilities Project Management: Overseeing project ideation, development, and implementation within defined constraints. Team Management: Leading day-to-day team operations, including recruitment, training, and mentorship for enhanced productivity. Technical Expertise: Proficiency in relevant technologies, tools, and methodologies. Cross-functional Collaboration: Aligning engineering endeavors with organizational objectives through collaboration. Risk Management: Identifying and mitigating risks associated with engineering projects, including cost and safety considerations. Continuous Learning: Staying updated on emerging technologies and trends for optimized engineering processes. Strategic Planning: Establishing clear goals, allocating resources, and managing deadlines based on data-driven insights. Thriving in this position goes beyond just technical knowledge. It entails a balance between leadership, communication, strategic thinking, and adaptability. Let’s go over some Key Traits of Exceptional EMs. Effective Communication: The ability to convey complex technical concepts to both technical and non-technical stakeholders creates a shared understanding within the team and the organization. Empathy and Emotional Intelligence: Understanding and empathizing with team members create a positive and collaborative work environment, building trust and support. Delegation Skills and Trust: Effective delegation empowers the team, emphasizing the importance of trust in leadership. Flexibility and Negotiation: Navigating trade-offs and being open to compromise are crucial in challenging situations with multiple stakeholders. Leadership and Vision: Providing a clear vision, aligning goals, inspiring the team, delivering transparent and constructive feedback, and having a strategic mindset contribute to long-term success. For a video version of this article: Conclusion Success as an Engineering Manager goes beyond technical knowledge; it requires a balance of leadership, communication, strategic thinking, and adaptability. By blending these qualities, an EM can navigate the complexities of team management and contribute to both individual and organizational achievements.

Development teams today face increasing complexity as they strive to deliver innovations quickly while ensuring quality, security, and scalability. Markus Eisele, Head of Developer Tools Marketing at Red Hat, discusses how an internal developer platform (IDP) can optimize workflows to boost productivity. Q: What are the main pain points developers face today that hinder productivity and innovation? A: DevOps has arrived in many companies. The need to achieve “more with less” and bring applications to market quickly leads to companies spreading the existing trends. Parallel to the “more” responsibility for the DevOps teams and individual developers, the toolbox of the full-stack developers is expanding nearly on a monthly basis with new tools and approaches. This inevitably leads to fewer real specialists who are able to master the complexity of modern applications and their landscapes. The training of new colleagues is becoming increasingly difficult, and developers see the increasing responsibility coupled with the growing technological stack as more of a burden and blocker for productivity. This is commonly referred to as cognitive load and negatively influences developer happiness, leading to burnout and dissatisfaction with work and even workplaces. Q: How specifically can an internal developer platform help standardize processes and tools to streamline work? A: What is necessary to solve the challenges starting with cognitive load can be twofold. One approach is to reduce the choices and create strong reference architectures and master solutions. While this is a suitable solution for bringing down technology complexity, it is also seen as a hurdle and roadblock for innovation. A better approach is to create best practices and golden paths from existing solutions and give teams the ability to capture them in a way that other teams can easily replicate them. Coupled with everything developers need to not only find them but also quickly access documentation enhanced with direct system access in a self-service fashion, you have the core building blocks of a developer portal. This portal ideally comes with a standardized set of technologies, for instance, ArgoCD and Tekton Pipelines, which should be available pre-configured and ready to roll. Both portal and underlying capabilities make up the so-called developer platform. Basically, it provides an optimized experience on a well-known and established platform that is curated by a dedicated platform engineering team working towards the best possible experience for their users. Q: What collaborative benefits can developers gain from an IDP approach? How does it improve teamwork? A: This depends on the capabilities of the IDP. Something broadly found is the open-source project Backstage. Backstage's core features can be changed and adapted through a plug-in mechanism. There are plenty of opportunities for platform teams to help developers with teamwork and collaboration. The most simplistic approach is to help teams adopt a centralized documentation standard for services and applications funneled through IDP mechanisms. But there are more specialized plug-ins available that can address every upcoming need. Q: How can an IDP simplify and improve onboarding and self-service for developers? A: Most obvious is the single point of access to all the tools, resources, and documentation needed to work on their projects. Instead of having to screen endless documents and different bookmarks, the portal allows for quick, centralized access to everything relevant. Beyond that, it enables developer autonomy and productivity by allowing them to request resources, spin up fully provisioned environments, deploy, and roll back without depending on the platform or ops team. Onboarding is also accelerated through access to standards, best practices, and workflows for development, testing, and deployment that are encapsulated in templates for an easy project start. A side effect of centralization is improved visibility and governance of the software development life cycle by tracking and monitoring the performance, quality, and security of applications and services. Q: In what ways can an IDP provide more scalability for development teams and projects? A: Frequently mentioned is the ability to provision resources and environments more quickly and easily. Reducing the time and waiting for the cost of setting up and ultimately maintaining infrastructure. The standardization aspect during provisioning, as well as monitoring of the total amount of provided components, can’t be neglected here. IDPs can also support multiple languages, frameworks, and, ultimately, platforms, allowing developers to use the best tools for their needs and preferences while still complying with company standards and policies. Ideally, all this happens in a highly automated fashion implemented with the latest standards and tools suitable for handling large-scale environments. Q: How does an IDP approach help consolidate documentation and make it more usable for developers? A: The unique and key point here is the unified way developers can navigate and explore documentation relevant to their projects. An IDP typically offers powerful search capabilities, allowing developers to quickly locate specific information. Backed by version control features, they enable developers to track changes and updates to documentation over time. Collaboration is facilitated through built-in tools that allow team members to contribute to and maintain documentation collectively. This collaborative environment ensures documentation stays up-to-date and accurately reflects the current state of a project. Q: Can you provide some real-world examples or use cases of companies seeing better productivity/innovation after implementing an IDP? A: A common theme is the ability to adapt developer portals exactly to developer use cases and company infrastructures. Especially the plug-in system, and in particular, the new dynamic plug-in system, which allows companies to quickly integrate new capabilities into their development toolchain and maximize their IDP implementation advantages. Q: What advice would you give development leaders considering an IDP approach – where should they start? A: When development leaders start thinking about developer productivity, they are quickly tempted to look for productivity enhancements coupled with the measurable success of their initiatives. This leads to misguided approaches adopting certain metrics and pushing teams hard to follow a strict approach across the entire organization. An IDP is built on the idea of guardrails instead of roadblocks and is intended to standardize through best practices and collaboration. I strongly suggest starting from the minimum viable product approach and making sure teams get an opportunity to shape the IDP the way it maximizes their productivity before applying any metrics and stringent rules. I strongly believe that developer productivity is at the core of successful organizations and is almost a guarantee for a motivated team. Treating IDPs as a part of a process to make people more productive through the removal of overgrown processes and regulations is a good place to start. Q: What are some common pitfalls or mistakes to avoid when implementing an internal developer platform? A: The guarantee to failure is treating an IDP as “another tool” to add to the toolbox. This is the one thing that will absolutely not work. The concept is meant to provide teams with the individual solutions necessary to get their work done more efficiently and not as another corset they have to fit their daily routines into. Do not treat an IDP as an extended ticketing or control system. Treat it as a helper to scale best practices and successful efforts to bring them to more teams across your organization. Q: Where do you see IDPs headed in the future? How will they continue to evolve to meet developer needs? IDPs may become the centerpiece of developer workflows. While we typically talk about inner- and outer-loop development tasks during software projects, the outer-loop has been somehow focussed on operationalizing and deployment. With IDPs, the source code and deployment configuration, coupled with all the relevant documentation and configuration, can be accessed and worked on in a central repository that becomes the heart of initiatives. Independent of whether it is a service or a complete application. While the number of technologies involved doesn’t necessarily change, the teams can work more focused and securely on the basis of established patterns and proven approaches, turning a loose documentation place into an active collaboration platform. Ultimately, I expect IDPs to become the new approach for application development. Another entry point with applications and services as a first-class object providing context and information tailored to individual developers on the basis of a potentially complex multi-cloud environment, abstracting away the unnecessary complexity.

By Tom Smith CORE

I’ve noticed two danger zones that organizations run into. Today, I’ll describe these two danger zones, and give some advice for navigating them. I’ll talk about this for engineering organizations. But I suspect it’s applicable to any group of humans working at these scales. Why Are These Danger Zones Important? I’ve seen several companies waste years getting trapped in these danger zones. That time is precious for a startup and can result in the business failing. I’ve seen people who are smarter than me get into these traps over and over. I believe the reason for that is that these are structural problems. Solving them requires some deep refactoring of your organization, and most people haven’t done this type of work before. So, they fail, and their company and employees suffer. The growth traps in these danger zones interact with other leadership and organizational problems in harmful ways. For example, if you have a leadership team that tends to micromanage, these growth traps will make it worse. Or if you have a lack of leadership alignment, that will be more prominent. So the traps in these danger zones have a disproportionate impact. Why Listen to Me on This? I have both breadth and depth of experience in these danger zones. At New Relic, we had an experienced leader who helped us navigate the first danger zone. However, I saw areas where we struggled and was involved throughout the process. I spent years grappling with the second danger zone at New Relic. It was the most critical deficiency in our engineering organization for years. It was something we continuously grappled with. We worked with Jim Shore, author of The Art of Agile Development, to successfully address it. I was part of the team that fixed it, and it shaped my thinking about the patterns behind how groups of humans can operate at increasingly larger scales. It has become a major theme in my leadership work ever since. I’ve since worked at almost twenty-five startups, and any of them that have grown through the danger zones have run into these same growth traps. I’ve focused a lot of my consulting practice on helping organizations get through these traps and avoid much of the suffering you would expect. Danger Zone #1: The Team Trap The first of these two danger zones is what I call the “team trap." It generally happens sometime between ten and twenty people. You start with a co-founder or leader who is in charge of engineering. The engineers all report to this person. There are lots of projects going on, and they get busier and busier as the team grows. Often, you’ll have each individual focusing on their own project! Because the team is small enough, everyone has a pretty decent sense of what is going on. You know what projects are being worked on, and how things are progressing. Priorities are fairly clear, and communication is uncomplicated. Usually, it all happens in one place, with everyone reading everything or in the same room. Communication is many to many. The future is bright. What Happens With the Team Trap As the team grows, however, things start to break down. The leader works longer and longer hours. Yet, it seems harder and harder to get work done, and execution and quality seem to be slipping. Communication seems muddled, and you hear more people wondering about priorities or what the strategy is. This is the Team Trap. You’re heading towards failure, and unless you reshape things, it will get worse and worse! Why It’s Hard To Fix the Team Trap You will face a few obstacles to make the right changes. First of all, the founders and early people are often really smart, incredibly dedicated people. They will work harder and harder, and try to brute force themselves through the Team Trap. That won’t work – it will only delay the solution. Second, some of the solutions to the Team Trap will feel like bureaucracy to the founders and early people in the company. They’ll resist the changes because they want to preserve the way the company felt early on – everything could happen quickly and effortlessly. They will often have an aversion to the changes they might need to make. What To Do About the Team Trap The changes you typically need to make at this phase are structural changes: You need to set up cross-functional teams with ownership. This is a hard thing to do well, and if done incorrectly, can actually make everything worse! I advise you to read Team Topologies, and to get help with this. You can also try an alternative approach: FAST agile collectives. You’ll need to start thinking about your organization’s design. You may need managers. But you may need a different type of manager than you think. You’ll need to think about how to design your meetings. You may need some lightweight role definition. Ultimately, someone has to start thinking about the way your organization is structured, and how all the pieces will fit together. Part of this organizational design is to also think through your communication design. You probably want to start segmenting communication, so that people know what they need to, but aren’t flooded with a lot of information they don’t need. There is a balance to this. You don’t want to over tilt towards structure, but you also don’t want to avoid necessary structure. All of this is pretty hard, and I’ve built a business helping engineering organizations with this (so definitely reach out if you need help with it, or find someone else to help you). Why Does the Team Trap Happen? Incidentally, I believe the reason this seems to happen at between ten and twenty engineers is because that’s when one person can no longer reasonably manage everyone in engineering. You have to start to split the world. And once you do that, it forces a lot of other changes to happen at the same time. It’s a little like when you have a web server that is delivering content over the internet. As soon as you want a second server for redundancy or scaling reasons, all of a sudden, you need a lot more in place to make things work. You may need a load balancer. You may need to think about state and caching since it is done independently for each server. All of these concerns happen at the same time. If you’re successful in your design, you’ll have a structure that will take you pretty far. Your teams will be autonomously creating value for the company. And things should go pretty well until you hit the second danger zone. Danger Zone #2: The Cross-Team Project Trap The next trap is with how teams work with each other. You reach a level of complexity where the primary challenge for your organization is how to ensure that anything that crosses team boundaries can be successful. As the number of teams grows, each of them delivers value. But they aren’t perfect encapsulations of delivery. Teams need things from each other. And as your product grows, you’ll need things from multiple teams. The themes for this stage are coordination and dependencies. How do you get teams to coordinate, to deliver something bigger than themselves? And how do you deal with the fact that dependencies often aren’t reasonable? How do you sort through those dependencies, and minimize them? The cross-team project danger zone occurs somewhere after about forty people. I often see it happen between forty and sixty people in an organization. At New Relic, we tried valiantly to fix cross-team projects, but we didn’t really succeed until we worked with Jim Shore, and at that time there were probably 200 engineers in the organization. It was long overdue. As an aside: It’s plausible that our failure to address this earlier was instrumental in Datadog’s ascendance. Why? We were much less effective in engineering at the time, and this slowed our ability to succeed in the enterprise market. Most of the bigger projects were enterprise features. Our focus on growing into the enterprise distracted us from Datadog’s rise and prevented us from addressing shifts in the way developers were working with microservices. It’s possible that handling this earlier could have resulted in a completely different outcome, though there were a lot of factors involved. What Happens With the Cross-Team Project Trap To understand the cross-team project trap, consider a couple of examples. First, you may want to do some work that affects many teams. For example, let’s say your customers are asking for role-based access controls. This is work that many teams will need to focus on. Yet the enabling work might be done by one team. This can require coordination. Another need you’ll see increasingly is that multiple teams need similar functionality. They might both want a similar table user interface. Or they might both require a similar API. Or they might depend on similar data. This type of work tends to require teams to depend on other teams to do work for them. This is a growth in dependencies. At some point, coordination and dependencies grow to become your most serious obstacle to delivery. You’ll know you’re in this second danger zone if you see some of these symptoms: There is a lack of confidence from the rest of the company that engineering can deliver large, important initiatives. The general track record is that engineering ships late, if at all. You see lots of heroic effort to deliver anything that crosses team boundaries. You may have a few people who are unusually good program managers, but even they have failures. You might see opposing instincts to add more structure, or operate more “like a startup." A few really experienced engineers who can get things done are held up as saviors. But the general default is that things don’t ship well. You have areas of the organization that are such hot spots that they go through waves of failure - often because they are the hot spot for dependencies. Why It’s Hard To Fix the Cross-Team Project Trap When I was at New Relic, I was leading up the engineering side of a new analytics product. It was an ambitious product. We had widespread agreement that it was a top priority. However, we needed things from many parts of engineering in order to deliver on the complete vision for the product. Those dependencies didn’t seem optional. So the way I attempted to handle this was by acting as a program manager. I tried to organize my dependencies as projects. Each of them would be updated on progress and risks. Fairly standard stuff for program management. But what I found is that the structure of the organization didn’t make the execution on this type of project possible. It was mathematically impossible. Every team had their own priorities. Even if they thought we were the top priority, that was subject to change. If they had a reliability problem, they had to do work to address that problem. Sometimes something new would come up, and bump our priority back. I was essentially making plans that weren’t based on anything structurally sound. It was difficult to fix at that point. I couldn’t tell all of engineering how to operate! At the time, I didn’t know what would fix the situation. We had tried for years at New Relic to crack the “cross-team project” problem. We rewarded people who were good at project and program management. We hired people that were good at it. We even made delivery of cross-team projects part of our promotion criteria! But ultimately, we didn’t make the structural changes we needed. The challenges you face to fixing these at this stage are mostly that there is a lot of organizational inertia. Changes can feel threatening. People can feel demotivated to make changes when they are under so much stress. Working within the existing system can take all of your bandwidth, so people will be reluctant to work extra to extricate themselves from the mess. And structural fixes can intrude on leadership turf, so you need a high level of support from the very top of the organization to make your changes. This is not something you can just ignore. It will continue to get worse and worse until any project that crosses team boundaries ends up being impossible to complete. What To Do About the Cross-Team Project Trap These are the types of changes I recommend if you run into the cross-team project trap: Centralize cross-team priorities (possibly with a product council) and teach teams how to work with those central priorities. Define organizational and team coordination models. For example, move platform teams from service model teams to self-service. Make your product teams act as independent executors, assuming no dependencies in their projects. Carefully design which teams act in an embedded model. Use program managers for cross-team initiatives. Limit the number of cross-team initiatives. Reorganize your teams mostly along cross-functional lines. Reduce dependencies between teams. Or, you might experiment with FAST agile teams. Reduce the size of projects (using milestones or increments). Ideally, get help! Why Does the Cross-Team Project Trap Happen? I have a hunch these growth traps are the result of complexity jumps that occur when you add a layer of management. The first danger zone corresponds to when you add managers, the second danger zone is often when you add directors. When you have this jump in complexity, you have to shift the structure of the organization. Otherwise, you have a mismatch between what is necessary for that structure to be successful, and the way it really is. This is not to say that adding managers or directors is what causes the problem. You can run into these growth traps even if you have managers or directors. It’s just that you need them both at the same time. Incidentally, this is why I am bullish on FAST Agile. I think it may allow you to have a simpler organizational structure for a longer period of time. Combined with some of these other structures, I think the potential benefits outweigh the fact that it is a new, less well-developed practice. Thank You I try to credit people who have influenced my thinking or directly affected my approach. Much of the first danger zone is through my own observation. I’m not sure I’ve seen anyone else articulate it. But I would guess others have seen the same thing – when I talk with other leaders or venture capitalists, they have a look of “yeah, that sounds familiar." For the second danger zone, there isn’t a place I can point to (that I remember) that highlights this as a scaling barrier for organizations. I’m pretty sure something must exist! For how to address it, my biggest source of credit goes to Jim Shore. His work at New Relic was quite effective, and it was a career highlight to work with him and the Upscale team to design and implement the solutions. While the coordination models have been my own pattern language for organizational and cross-team work, you’ll notice he is credited on many of them.

By Jade Rubick CORE

In the realm of professional pursuits, there exists a common misconception that managing software development is akin to riding a bike – a static skill that, once acquired, can be smoothly pedaled forward with minimal adjustments. However, in the fast-evolving landscape of technology, such a comparison is not only overly simplistic but can lead to profound misjudgments in leadership. Unlike the steadfast predictability of a bicycle ride, software development is a dynamic and ever-changing process that defies the static nature of traditional analogies. As we celebrate the first birthday of our software endeavors, it's imperative to address the fallacy that managing software projects is as straightforward as steering a two-wheeler down a familiar path. This misapprehension often stems from leaders who, having once mastered coding or project management, find themselves trapped in a mindset that underestimates the fluidity of the software development journey. In this article, we unravel the intricacies of why software development is fundamentally distinct from riding a bike, shedding light on the pitfalls that managers and CTOs may encounter when they cling to static paradigms in a world that thrives on adaptability and innovation. Join us as we explore the dynamic nature of software development and challenge the notion that it can be steered with the simplicity of a handlebar. In the not-so-distant past, the scarcity and costliness of data storage spurred a focus on normalizing databases to conserve every precious byte. However, as technology advanced, we witnessed a paradigm shift. The advent of NoSQL databases prompted a reevaluation of our practices, challenging the once-unquestioned norms of normalization. Today, we find ourselves navigating the complexities of denormalization and replication, leveraging the databases' capabilities to handle the deluge of data in the age of information abundance. As access to computing power expanded with the rise of cloud platforms, the architectural landscape underwent a metamorphosis. Traditional monolithic structures gave way to the nimble and scalable world of microservices. With the cloud offering a buffet of resources, developers embraced a distributed approach, empowering them to create systems that are not only resilient but also capable of seamlessly scaling to meet the demands of modern applications. The software development life cycle has witnessed its evolution, from the rigidity of waterfall methodology to the agility of modern development practices. The cloud-native methodology has emerged as a flexibility champion, enabling teams to iterate rapidly and respond to changing requirements. Today, we stand in the era of Agile, where collaboration, adaptability, and continuous delivery reign supreme, ushering in an age where the pace of development matches the speed of technological innovation. Gone are the days when users patiently queue in lines to make a purchase. The digital age has ushered in a new era of seamless experiences, where transactions occur at the tap of a screen. The evolution of software has not only transformed the way we develop applications but has fundamentally altered user expectations, demanding intuitive interfaces and instant gratification. Artificial intelligence (AI) stands as the next frontier as we peer into the future. The integration of AI and generative AI has the potential to revolutionize how we conceive, build, and optimize software. Algorithms that learn and adapt, coupled with the ability to generate code, hint at a future where development becomes an even more symbiotic dance between human creativity and machine intelligence. In this ever-shifting landscape, software development remains a dynamic canvas where each stroke of innovation leaves an indelible mark. As we navigate the currents of change, it's crucial to recognize that the journey is far from over – the horizon holds new technologies, methodologies, and challenges, inviting us to continuously adapt, learn, and redefine the future of software development. A Brief History of Software Development Embarking on a journey through the epochs of software development is akin to navigating a landscape that constantly redefines itself. This session explores the dynamic evolution that has shaped the essence of how we conceive, craft, and deliver software solutions. As we traverse the annals of time, we'll unveil the intricate tapestry of changes woven together to form the contemporary fabric of software development. From the early days when data was a precious commodity to the present era of information abundance, from the rigid structures of waterfall methodologies to the agile dance of cloud-native development, each phase has left an indelible mark on the software development saga. Join us as we delve into the database dilemmas, architectural ascents, methodology metamorphoses, and user experience unleashing that define the narrative of our digital evolution. As we stand on the cusp of an era where artificial intelligence and generative AI promise to reshape the very foundations of our craft, it becomes imperative to reflect on the past, understand the present, and anticipate the future. The history of software development is not merely a chronological progression; it is a story of adaptation, innovation, and resilience. So, let us journey together through the corridors of time, where each twist and turn reveals a new facet of this ever-evolving realm. Welcome to exploring the dynamic symphony that is the history of software development. In the not-so-distant past, the scarcity and costliness of data storage spurred a focus on normalizing databases to conserve every precious byte. However, as technology advanced, we witnessed a paradigm shift. The advent of NoSQL databases prompted a reevaluation of our practices, challenging the once-unquestioned norms of normalization. Today, we find ourselves navigating the complexities of denormalization and replication, leveraging the databases' capabilities to handle the deluge of data in the age of information abundance. Historical cost of computer memory and storage As access to computing power expanded with the rise of cloud platforms, the architectural landscape underwent a metamorphosis. Traditional monolithic structures gave way to the nimble and scalable world of microservices. With the cloud offering a buffet of resources, developers embraced a distributed approach, empowering them to create systems that are not only resilient but also capable of seamlessly scaling to meet the demands of modern applications. New architecture using microservices The software development life cycle has witnessed its evolution, from the rigidity of waterfall methodology to the agility of modern development practices. The cloud-native methodology has emerged as a flexibility champion, enabling teams to iterate rapidly and respond to changing requirements. Today, we stand in the era of Agile, where collaboration, adaptability, and continuous delivery reign supreme, ushering in an age where the pace of development matches the speed of technological innovation. Agile methodology process Gone are the days when users patiently queue in lines to make a purchase. The digital age has ushered in a new era of seamless experiences, where transactions occur at the tap of a screen. The evolution of software has not only transformed the way we develop applications but has fundamentally altered user expectations, demanding intuitive interfaces and instant gratification. Artificial intelligence (AI) stands as the next frontier as we peer into the future. The integration of AI and generative AI has the potential to revolutionize how we conceive, build, and optimize software. Algorithms that learn and adapt, coupled with the ability to generate code, hint at a future where development becomes an even more symbiotic dance between human creativity and machine intelligence. In this ever-shifting landscape, software development remains a dynamic canvas where each stroke of innovation leaves an indelible mark. As we navigate the currents of change, it's crucial to recognize that the journey is far from over – the horizon holds new technologies, methodologies, and challenges, inviting us to continuously adapt, learn, and redefine the future of software development. Why Past Successes May Lead to Future Failures In the dynamic realm of software development, the adage "what worked in the past will work in the future" is a dangerous oversimplification that can potentially steer leaders and C-level executives into turbulent waters. This session aims to unravel why a deep understanding of the industry's evolution is beneficial and imperative for those guiding the ship. While the fundamental principles of computer science serve as a bedrock, the landscape in which they are applied undergoes perpetual transformation. Managers, CTOs, and executives who once thrived as hands-on engineers may be treading on thin ice if they believe their past achievements grant them a timeless understanding of the field. The danger lies in assuming that what was influential in the past remains applicable in an industry where change is the only constant. As software development evolves, so do the methodologies, tools, and paradigms that govern it. Leaders who cease to code and detach from the front lines risk becoming obsolete in their understanding of current practices. The disconnect between the executive suite and the development trenches can lead to misguided decisions, as what may have been best practice a decade ago might now be an antiquated approach. To remain relevant and practical, leaders must embrace the ethos of lifelong learning. It includes staying abreast of emerging technologies, methodologies, and trends. Arrogance and an unwillingness to adapt can hinder progress, whereas humility and a willingness to learn from younger, less experienced team members can foster a collaborative and innovative environment. In the evolving landscape, leadership roles have transformed as well. The emergence of positions like Staff Engineer exemplifies a harmonious convergence of coding proficiency and strategic thinking. This hybrid role acknowledges the value of technical prowess while emphasizing the strategic vision necessary for leadership positions. It's a testament that one need not abandon the code editor to ascend the career ladder. Recognizing that the history of software development is a dynamic narrative, not a static manual, is crucial for effective leadership. Managers and executives must acknowledge that the very fabric of the industry has changed, and what led to success in the past may not be a blueprint for the future. By staying curious, embracing continuous learning, and fostering a culture of collaboration, leaders can navigate the currents of software development and guide their teams toward success in an ever-evolving landscape. Summary As our journey through the dynamic history of software development comes to a close, it’s crucial to distill the essence of our exploration into actionable insights for leaders and visionaries. 1. Embrace the Current: Leaders must internalize the fluid nature of software development. Acknowledge that what worked yesterday might not work tomorrow, and be prepared to adapt swiftly to the evolving currents of technology and methodologies. 2. Continuous Learning is Key: The heartbeat of effective leadership in software development is a commitment to continual learning. Staying curious, remaining open to new ideas, and fostering a culture of shared knowledge ensures that leaders don’t just lead; they inspire growth. 3. Humility Fuels Innovation: A humble leader is an influential leader. Recognizing the value of diverse perspectives, including those of younger team members, fosters an environment where Innovation can flourish. Arrogance, on the other hand, creates blind spots that hinder progress. 4. The Hybrid Leader: The emergence of roles like the Staff Engineer signals a departure from traditional hierarchies. Leaders need not forsake coding to ascend the ladder; instead, they can integrate technical expertise with strategic vision, creating a harmonious synergy that propels teams forward. 5. Navigate With Purpose: Purposeful navigation is paramount in the dynamic seas of software development. Leaders must define clear goals, inspire their teams, and foster an environment where adaptability is not a reaction but a proactive stance. As we chart the course ahead, remember that leadership in software development is not about steering a static vessel but mastering the art of sailing through ever-changing waters. Embrace the dynamism, learn continually, lead with humility, and set sail towards a future where Innovation and adaptability are the guiding stars. The dynamic journey continues, and effective leadership will always be the compass for success in software development. Safe travels!

By Otavio Santana CORE