Software Design and Architecture

Managing your secrets well is imperative in software development. It's not just about avoiding hardcoding secrets into your code, your CI/CD configurations, and more. It's about implementing tools and practices that make good secrets management almost second nature. A Quick Overview of Secrets Management What is a secret? It's any bit of code, text, or binary data that provides access to a resource or data that should have restricted access. Almost every software development process involves secrets: credentials for your developers to access your version control system (VCS) like GitHub, credentials for a microservice to access a database, and credentials for your CI/CD system to push new artifacts to production. There are three main elements to secrets management: How are you making them available to the people/resources that need them? How are you managing the lifecycle/rotation of your secrets? How are you scanning to ensure that the secrets are not being accidentally exposed? We'll look at elements one and two in terms of the secrets managers in this article. For element three, well, I'm biased toward GitGuardian because I work there (disclaimer achieved). Accidentally exposed secrets don't necessarily get a hacker into the full treasure trove, but even if they help a hacker get a foot in the door, it's more risk than you want. That's why secrets scanning should be a part of a healthy secrets management strategy. What To Look for in a Secrets Management Tool In the Secrets Management Maturity Model, hardcoding secrets into code in plaintext and then maybe running a manual scan for them is at the very bottom. Manually managing unencrypted secrets, whether hardcoded or in a .env file, is considered immature. To get to an intermediate level, you need to store them outside your code, encrypted, and preferably well-scoped and automatically rotated. It's important to differentiate between a key management system and a secret management system. Key management systems are meant to generate and manage cryptographic keys. Secrets managers will take keys, passwords, connection strings, cryptographic salts, and more, encrypt and store them, and then provide access to them for personnel and infrastructure in a secure manner. For example, AWS Key Management Service (KMS) and AWS Secrets Manager (discussed below) are related but are distinct brand names for Amazon. Besides providing a secure way to store and provide access to secrets, a solid solution will offer: Encryption in transit and at rest: The secrets are never stored or transmitted unencrypted. Automated secrets rotation: The tool can request changes to secrets and update them in its files in an automated manner on a set schedule. Single source of truth: The latest version of any secret your developers/resources need will be found there, and it is updated in real-time as keys are rotated. Role/identity scoped access: Different systems or users are granted access to only the secrets they need under the principle of least privilege. That means a microservice that accesses a MongoDB instance only gets credentials to access that specific instance and can't pull the admin credentials for your container registry. Integrations and SDKs: The service has APIs with officially blessed software to connect common resources like CI/CD systems or implement access in your team's programming language/framework of choice. Logging and auditing: You need to check your systems periodically for anomalous results as a standard practice; if you get hacked, the audit trail can help you track how and when each secret was accessed. Budget and scope appropriate: If you're bootstrapping with 5 developers, your needs will differ from those of a 2,000-developer company with federal contracts. Being able to pay for what you need at the level you need it is an important business consideration. The Secrets Manager List Cyberark Conjur Secrets Manager Enterprise Conjur was founded in 2011 and was acquired by Cyberark in 2017. It's grown to be one of the premiere secrets management solutions thanks to its robust feature set and large number of SDKs and integrations. With Role Based Access Controls (RBAC) and multiple authentication mechanisms, it makes it easy to get up and running using existing integrations for top developer tools like Ansible, AWS CloudFormation, Jenkins, GitHub Actions, Azure DevOps, and more. You can scope secrets access to the developers and systems that need the secrets. For example, a Developer role that accesses Conjur for a database secret might get a connection string for a test database when they're testing their app locally, while the application running in production gets the production database credentials. The Cyberark site boasts an extensive documentation set and robust REST API documentation to help you get up to speed, while their SDKs and integrations smooth out a lot of the speed bumps. In addition, GitGuardian and CyberArk have partnered to create a bridge to integrate CyberArk Conjur and GitGuardian's Has My Secrets Leaked. This is now available as an open-source project on GitHub, providing a unique solution for security teams to detect leaks and manage secrets seamlessly. Google Cloud Secret Manager When it comes to choosing Amazon Web Services (AWS), Google Cloud Platform (GCP), or Microsoft Azure (Azure), it's usually going to come down to where you're already investing your time and money. In a multi-cloud architecture, you might have resources spread across the three, but if you're automatically rotating secrets and trying to create consistency for your services, you'll likely settle on one secrets manager as a single source of truth for third-party secrets rather than spreading secrets across multiple services. While Google is behind Amazon and Microsoft in market share, it sports the features you expect from a service competing for that market, including: Encryption at rest and in transit for your secrets CLI and SDK access to secrets Logging and audit trails Permissioning via IAM CI/CD integrations with GitHub Actions, Hashicorp Terraform, and more. Client libraries for eight popular programming languages. Again, whether to choose it is more about where you're investing your time and money rather than a killer function in most cases. AWS Secrets Manager Everyone with an AWS certification, whether developer or architect, has heard of or used AWS Secrets Manager. It's easy to get it mixed up with AWS Key Management System (KMS), but the Secrets Manager is simpler. KMS creates, stores, and manages cryptographic keys. Secrets Manager lets you put stuff in a vault and retrieve it when needed. A nice feature of AWS Secrets Manager is that it can connect with a CI/CD tool like GitHub actions through OpenID Connect (OIDC), and you can create different IAM roles with tightly scoped permissions, assigning them not only to individual repositories but specific branches. AWS Secrets Manager can store and retrieve non-AWS secrets as well as use the roles to provide access to AWS services to a CI/CD tool like GitHub Actions. Using AWS Lambda, key rotation can be automated, which is probably the most efficient way, as the key is updated in the secrets manager milliseconds after it's changed, producing the minimum amount of disruption. As with any AWS solution, it's a good idea to create multi-region or multi-availability-zone replicas of your secrets, so if your secrets are destroyed by a fire or taken offline by an absent-minded backhoe operator, you can fail over to a secondary source automatically. At $0.40 per secret per month, it's not a huge cost for added resiliency. Azure Key Vault Azure is the #2 player in the cloud space after AWS. Their promotional literature touts their compatibility with FIPS 140-2 standards and Hardware Security Modules (HSMs), showing they have a focus on customers who are either government agencies or have business with government agencies. This is not to say that their competitors are not suitable for government or government-adjacent solutions, but that Microsoft pushes that out of the gate as a key feature. Identity-managed access, auditability, differentiated vaults, and encryption at rest and in transit are all features they share with competitors. As with most Microsoft products, it tries to be very Microsoft and will more than likely appeal more to .Net developers who use Microsoft tools and services already. While it does offer a REST API, the selection of officially blessed client libraries (Java, .Net, Spring, Python, and JavaScript) is thinner than you'll find with AWS or GCP. As noted in the AWS and GCP entries, a big factor in your decision will be which cloud provider is getting your dominant investment of time and money. And if you're using Azure because you're a Microsoft shop with a strong investment in .Net, then the choice will be obvious. Doppler While CyberArk's Conjur (discussed above) started as a solo product that was acquired and integrated into a larger suite, Doppler currently remains a standalone key vault solution. That might be attractive for some because it's cloud-provider agnostic, coding language agnostic, and has to compete on its merits instead of being the default secrets manager for a larger package of services. It offers logging, auditing, encryption at rest and in transit, and a list of integrations as long as your arm. Besides selling its abilities, it sells its SOC compliance and remediation functionalities on the front page. When you dig deeper, there's a list of integrations as long as your arm testifies to its usefulness for integrating with a wide variety of services, and its list of SDKs is more robust than Azure's. It seems to rely strongly on injecting environment variables, which can make a lot of your coding easier at the cost of the environment variables potentially ending up in run logs or crash dumps. Understanding how the systems with which you're using it treat environment variables, scope them, and the best ways to implement it with them will be part of the learning curve in adopting it. Infisical Like Doppler, Infisical uses environment variable injection. Similar to the Dotenv package for Node, when used in Node, it injects them at run time into the process object of the running app so they're not readable by any other processes or users. They can still be revealed by a crash dump or logging, so that is a caveat to consider in your code and build scripts. Infisical offers other features besides a secrets vault, such as configuration sharing for developer teams and secrets scanning for your codebase, git history, and as a pre-commit hook. You might ask why someone writing for GitGuardian would mention a product with a competing feature. Aside from the scanning, their secrets and configuration vault/sharing model offers virtual secrets, over 20 cloud integrations, nine CI/CD integrations, over a dozen framework integrations, and SDKs for four programming languages. Their software is mostly open-source, and there is a free tier, but features like audit logs, RBAC, and secrets rotation are only available to paid subscribers. Akeyless AKeyless goes all out features, providing a wide variety of authentication and authorization methods for how the keys and secrets it manages can be accessed. It supports standards like RBAC and OIDC as well as 3rd party services like AWS IAM and Microsoft Active Directory. It keeps up with the competition in providing encryption at rest and in transit, real-time access to secrets, short-lived secrets and keys, automated rotation, and auditing. It also provides features like just-in-time zero trust access, a password manager for browser-based access control as well as password sharing with short-lived, auto-expiring passwords for third parties that can be tracked and audited. In addition to 14 different authentication options, it offers seven different SDKs and dozens of integrations for platforms ranging from Azure to MongoDB to Remote Desktop Protocol. They offer a reasonable free tier that includes 3-days of log retention (as opposed to other platforms where it's a paid feature only). 1Password You might be asking, "Isn't that just a password manager for my browser?" If you think that's all they offer, think again. They offer consumer, developer, and enterprise solutions, and what we're going to look at is their developer-focused offering. Aside from zero-trust models, access control models, integrations, and even secret scanning, one of their claims that stands out on the developer page is "Go ahead – commit your .env files with confidence." This stands out because .env files committed to source control are a serious source of secret sprawl. So, how are they making that safe? You're not putting secrets into your .env files. Instead, you're putting references to your secrets that allow them to be loaded from 1Password using their services and access controls. This is somewhat ingenious as it combines a format a lot of developers know well with 1Password's access controls. It's not plug-and-play and requires a bit of a learning curve, but familiarity doesn't always breed contempt. Sometimes it breeds confidence. While it has a limited number of integrations, it covers some of the biggest Kubernetes and CI/CD options. On top of that, it has dozens and dozens of "shell plugins" that help you secure local CLI access without having to store plaintext credentials in ~/.aws or another "hidden" directory. And yes, we mentioned they offer secrets scanning as part of their offering. Again, you might ask why someone writing for GitGuardian would mention a product with a competing feature. HashiCorp Vault HashiCorp Vault offers secrets management, key management, and more. It's a big solution with a lot of features and a lot of options. Besides encryption, role/identity-based secrets access, dynamic secrets, and secrets rotation, it offers data encryption and tokenization to protect data outside the vault. It can act as an OIDC provider for back-end connections as well as sporting a whopping seventy-five integrations in its catalog for the biggest cloud and identity providers. It's also one of the few to offer its own training and certification path if you want to add being Hashi Corp Vault certified to your resume. It has a free tier for up to 25 secrets and limited features. Once you get past that, it can get pricey, with monthly fees of $1,100 or more to rent a cloud server at an hourly rate. In Summary Whether it's one of the solutions we recommended or another solution that meets our recommendations of what to look for above, we strongly recommend integrating a secret management tool into your development processes. If you still need more convincing, we'll leave you with this video featuring GitGuardian's own Mackenzie Jackson.

In this tutorial, we will learn how to build and deploy a conversational chatbot using Google Cloud Run and Dialogflow. This chatbot will provide responses to user queries on a specific topic, such as weather information, customer support, or any other domain you choose. We will cover the steps from creating the Dialogflow agent to deploying the webhook service on Google Cloud Run. Prerequisites A Google Cloud Platform (GCP) account Basic knowledge of Python programming Familiarity with Google Cloud Console Step 1: Set Up Dialogflow Agent Create a Dialogflow Agent: Log into the Dialogflow Console (Google Dialogflow). Click on "Create Agent" and fill in the agent details. Select the Google Cloud Project you want to associate with this agent. Define Intents: Intents classify the user's intentions. For each intent, specify examples of user phrases and the responses you want Dialogflow to provide. For example, for a weather chatbot, you might create an intent named "WeatherInquiry" with user phrases like "What's the weather like in Dallas?" and set up appropriate responses. Step 2: Develop the Webhook Service The webhook service processes requests from Dialogflow and returns dynamic responses. We'll use Flask, a lightweight WSGI web application framework in Python, to create this service. Set Up Your Development Environment: Ensure you have Python and pip installed. Create a new directory for your project and set up a virtual environment: Shell python -m venv env source env/bin/activate # `env\Scripts\activate` for windows Install Dependencies: Install Flask and the Dialogflow library: Shell pip install Flask google-cloud-dialogflow Create the Flask App: In your project directory, create a file named app.py. This file will contain the Flask application: Python from flask import Flask, request, jsonify app = Flask(__name__) @app.route('/webhook', methods=['POST']) def webhook(): req = request.get_json(silent=True, force=True) # Process the request here. try: query_result = req.get('queryResult') intent_name = query_result.get('intent').get('displayName') response_text = f"Received intent: {intent_name}" return jsonify({'fulfillmentText': response_text}) except AttributeError: return jsonify({'fulfillmentText': "Error processing the request"}) if __name__ == '__main__': app.run(debug=True) Step 3: Deploy To Google Cloud Run Google Cloud Run is a managed platform that enables you to run containers statelessly over a fully managed environment or in your own Google Kubernetes Engine cluster. Containerize the Flask App: Create a Dockerfile in your project directory: Dockerfile FROM python:3.8-slim WORKDIR /app COPY requirements.txt requirements.txt RUN pip install -r requirements.txt COPY . . CMD ["flask", "run", "--host=0.0.0.0", "--port=8080"] Don't forget to create a requirements.txt file listing your Python dependencies: Flask==1.1.2 google-cloud-dialogflow==2.4.0 Build and Push the Container: Use Cloud Build to build your container image and push it to the container registry. Shell gcloud builds submit --tag gcr.io/YOUR_CHATBOT_PRJ_ID/chatbot-webhook . Deploy to Cloud Run: Deploy your container image to Cloud Run. Shell gcloud run deploy --image gcr.io/YOUR_PROJECT_ID/chatbot-webhook --platform managed Follow the prompts to enable the required APIs, choose a region, and allow unauthenticated invocations. Step 4: Integrate With Dialogflow In the Dialogflow Console, navigate to the Fulfillment section. Enable Webhook, paste the URL of your Cloud Run service (you get this URL after deploying to Cloud Run), and click "Save." Testing and Iteration Test your chatbot in the Dialogflow Console's simulator. You can refine your intents, entities, and webhook logic based on the responses you receive. Conclusion You have successfully built and deployed a conversational chatbot using Google Cloud Run and Dialogflow. This setup allows you to create scalable, serverless chatbots that can handle dynamic responses to user queries. This foundation allows for further customization and expansion, enabling the development of more complex and responsive chatbots to meet a variety of needs. Continue to refine your chatbot by adjusting intents, entities, and the webhook logic to improve interaction quality and user experience.

Drawing from my extensive experience over the past several years dedicated to cloud adoption across various applications, it has become apparent that attaining a mature state — often termed the "nirvana" state — is neither immediate nor straightforward. Establishing a well-structured and effectively governed cloud footprint demands thorough planning and early investment. In this article, I aim to share insights and practical tips garnered from firsthand experience to assist in guiding your teams toward proactive preparation for cloud maturity, rather than addressing it as an afterthought. Establish a comprehensive cloud onboarding framework for all teams and applications to adhere to. This framework will serve as a roadmap, guiding teams in making well-informed architectural decisions, such as selecting appropriate services and SKUs for each environment. Encourage teams to carefully consider networking and security requirements by creating topology diagrams and conducting reviews with a designated cloud onboarding committee. Implement cost forecasting tools to facilitate budget estimation and planning. By adhering to this framework, teams can minimize rework through informed decision-making and prevent unnecessary cost wastage by making early, accurate estimates. Establish a structured framework for onboarding new services and tools into your cloud environment. Prior to deployment, conduct a comprehensive assessment or a proof of concept to understand the nuances of each service, including networking requirements, security considerations, scalability needs, integration requirements, and other relevant factors like Total Cost of Ownership and so on. By systematically evaluating these aspects, you can ensure that new services are onboarded efficiently, minimizing risks and maximizing their value to the organization. This could be a repeatable framework, providing efficiency and faster time to market. Make business continuity and disaster recovery a top priority in your cloud strategy. Implement robust plans and processes to ensure high availability and resilience of your cloud infrastructure and applications. Utilize redundant systems, geographic replication, and failover mechanisms to minimize downtime and mitigate the impact of potential disruptions. Regularly test and update your disaster recovery plans to ensure they remain effective in addressing evolving threats and scenarios. By investing in business continuity and disaster recovery measures, you can preserve your cloud operations, prevent data loss, and maintain continuity of services in the face of unforeseen events. Implement a controls and policies workstream to ensure adherence to regulatory requirements, industry standards, and internal governance frameworks. This workstream should involve defining and documenting clear controls and policies related to data privacy, security, compliance, and access management. Regular reviews and updates should be conducted to align with evolving regulatory landscapes and organizational needs. By establishing robust controls and policies, you can mitigate risks, enhance data protection, and maintain compliance and governance across your cloud environment. Some example controls could include ensuring storage is encrypted, implementing TLS for secure communication, and utilizing environment-specific SKUs, such as using smaller SKUs for lower environments. Invest in DevOps practices by establishing pre-defined environment profiles and promoting repeatability through a DevOps catalog for provisioning and deployment. By standardizing environment configurations and workflows, teams can achieve consistency and reliability across development, testing, and production environments. Implement automated deployment pipelines that enable continuous integration and continuous deployment (CI/CD), ensuring seamless and efficient delivery of software updates. Embrace a CI/CD framework that automates build, test, and deployment processes, allowing teams to deliver value to customers faster and with higher quality. By investing in DevOps practices, you can streamline software delivery, improve collaboration between development and operations teams, and accelerate time-to-market for applications and services. Promote cost awareness and early cost tracking by establishing or enforcing FinOps principles. Encourage a culture of cost awareness by emphasizing the importance of tracking expenses from day one. Implement robust cost-tracking measures as early as possible in your cloud journey. Utilize automated tools to monitor expenditures continuously and provide regular reports to stakeholders. By instilling a proactive approach to cost management, you can optimize spending, prevent budget overruns, and achieve greater financial efficiency in your cloud operations. Provide guidance through your cloud onboarding framework about cost-aware cloud architecture. To save costs, periodically review resource utilization and seek optimization opportunities such as right-sizing instances, consolidating environments, and leveraging pre-purchasing options like Reserved Instances. Regular reviews to assess the current state and future needs for continuous improvement. Establish a practice of periodic reviews to evaluate the current state of your cloud environment and anticipate future needs. Schedule regular assessments to analyze performance, security, scalability, and cost-efficiency. Engage stakeholders from across the organization to gather insights and identify areas for optimization and enhancement. By conducting these reviews systematically, you can stay agile, adapt to changing requirements, and drive continuous improvement in your cloud infrastructure and operations. These are some considerations that may apply differently depending on the scale or size and the nature of applications or services you use or provide to customers. For personalized advice, share details about your organization's structure and current cloud footprint in the comments below, and I'll be happy to provide recommendations. Thank you for reading!

As a quick recap, in Part 1: We built a simple gRPC service for managing topics and messages in a chat service (like a very simple version of Zulip, Slack, or Teams). gRPC provided a very easy way to represent the services and operations of this app. We were able to serve (a very rudimentary implementation) from localhost on an arbitrary port (9000 by default) on a custom TCP protocol. We were able to call the methods on these services both via a CLI utility (grpc_cli) as well as through generated clients (via tests). The advantage of this approach is that any app/site/service can access this running server via a client (we could also generate JS or Swift or Java clients to make these calls in the respective environments). At a high level, the downsides to this approach to this are: Network access: Usually, a network request (from an app or a browser client to this service) has to traverse several networks over the internet. Most networks are secured by firewalls that only permit access to specific ports and protocols (80:http, 443:https), and having this custom port (and protocol) whitelisted on every firewall along the way may not be tractable. Discomfort with non-standard tools: Familiarity and comfort with gRPC are still nascent outside the service-building community. For most service consumers, few things are easier and more accessible than HTTP-based tools (cURL, HTTPie, Postman, etc). Similarly, other enterprises/organizations are used to APIs exposed as RESTful endpoints, so having to build/integrate non-HTTP clients imposes a learning curve. Use a Familiar Cover: gRPC-Gateway We can have the best of both worlds by enacting a proxy in front of our service that translates gRPC to/from the familiar REST/HTTP to/from the outside world. Given the amazing ecosystem of plugins in gRPC, just such a plugin exists — the gRPC-Gateway. The repo itself contains a very in-depth set of examples and tutorials on how to integrate it into a service. In this guide, we shall apply it to our canonical chat service in small increments. A very high-level image (courtesy of gRPC-Gateway) shows the final wrapper architecture around our service: This approach has several benefits: Interoperability: Clients that need and only support HTTP(s) can now access our service with a familiar facade. Network support: Most corporate firewalls and networks rarely allow non-HTTP ports. With the gRPC-Gateway, this limitation can be eased as the services are now exposed via an HTTP proxy without any loss in translation. Client-side support: Today, several client-side libraries already support and enable REST, HTTP, and WebSocket communication with servers. Using the gRPC-Gateway, these existing tools (e.g., cURL, HTTPie, postman) can be used as is. Since no custom protocol is exposed beyond the gRPC-Gateway, complexity (for implementing clients for custom protocols) is eliminated (e.g., no need to implement a gRPC generator for Kotlin or Swift to support Android or Swift). Scalability: Standard HTTP load balancing techniques can be applied by placing a load-balancer in front of the gRPC-Gateway to distribute requests across multiple gRPC service hosts. Building a protocol/service-specific load balancer is not an easy or rewarding task. Overview You might have already guessed: protoc plugins again come to the rescue. In our service's Makefile (see Part 1), we generated messages and service stubs for Go using the protoc-gen-go plugin: protoc --go_out=$OUT_DIR --go_opt=paths=source_relative \ --go-grpc_out=$OUT_DIR --go-grpc_opt=paths=source_relative \ --proto_path=$PROTO_DIR \ $PROTO_DIR/onehub/v1/*.proto A Brief Introduction to Plugins The magic of the protoc plugin is that it does not perform any generation on its own but orchestrates plugins by passing the parsed Abstract Syntax Tree (AST) across plugins. This is illustrated below: Step 0: Input files (in the above case, onehub/v1/*.proto) are passed to the protoc plugin. Step 1: The protoc tool first parses and validates all proto files. Step 2:protoc then invokes each plugin in its list command line arguments in turn by passing a serialized version of all the proto files it has parsed into an AST. Step 3: Each proto plugin (in this case, go and go-grpcreads this serialized AST via its stdin. The plugin processes/analyzes these AST representations and generates file artifacts. Note that there does not need to be a 1:1 correspondence between input files (e.g., A.proto, B.proto, C.proto) and the output file artifacts it generates. For example, the plugin may create a "single" unified file artifact encompassing all the information in all the input protos. The plugin writes out the generated file artifacts onto its stdout. Step 4: protoc tool captures the plugin's stdout and for each generated file artifact, serializes it onto disk. Questions How does protoc know which plugins to invoke? Any command line argument to protoc in the format --<pluginname>_out is a plugin indicator with the name "pluginname". In the above example, protoc would have encountered two plugins: go and go-grpc. Where does protoc find the plugin? protoc uses a convention of finding an executable with the name protoc-gen-<pluginname>. This executable must be found in the folders in the $PATH variable. Since plugins are just plain executables these can be written in any language. How can I serialize/deserialize the AST? The wire format for the AST is not needed. protoc has libraries (in several languages) that can be included by the executables that can deserialize ASTs from stdin and serialize generated file artifacts onto stdout. Setup As you may have guessed (again), our plugins will also need to be installed before they can be invoked by protoc. We shall install the gRPC-Gateway plugins. For a detailed set of instructions, follow the gRPC-Gateway installation setup. Briefly: go get \ github.com/grpc-ecosystem/grpc-gateway/v2/protoc-gen-grpc-gateway \ github.com/grpc-ecosystem/grpc-gateway/v2/protoc-gen-openapiv2 \ google.golang.org/protobuf/cmd/protoc-gen-go \ google.golang.org/grpc/cmd/protoc-gen-go-grpc # Install after the get is required go install \ github.com/grpc-ecosystem/grpc-gateway/v2/protoc-gen-grpc-gateway \ github.com/grpc-ecosystem/grpc-gateway/v2/protoc-gen-openapiv2 \ google.golang.org/protobuf/cmd/protoc-gen-go \ google.golang.org/grpc/cmd/protoc-gen-go-grpc This will install the following four plugins in your $GOBIN folder: protoc-gen-grpc-gateway - The GRPC Gateway generator protoc-gen-openapiv2 - Swagger/OpenAPI spec generator protoc-gen-go - The Go protobuf protoc-gen-go-grpc - Go gRPC server stub and client generator Make sure that your "GOBIN" folder is in your PATH. Add Makefile Targets Assuming you are using the example from Part 1, add an extra target to the Makefile: gwprotos: echo "Generating gRPC Gateway bindings and OpenAPI spec" protoc -I . --grpc-gateway_out $(OUT_DIR) \ --grpc-gateway_opt logtostderr=true \ --grpc-gateway_opt paths=source_relative \ --grpc-gateway_opt generate_unbound_methods=true \ --proto_path=$(PROTO_DIR)/onehub/v1/ \ $(PROTO_DIR)/onehub/v1/*.proto Notice how the parameter types are similar to one in Part 1 (when we were generating go bindings). For each file X.proto, just like the go and go-grpc plugin, an X.pb.gw.go file is created that contains the HTTP bindings for our service. Customizing the Generated HTTP Bindings In the previous sections .pb.gw.go files were created containing default HTTP bindings of our respective services and methods. This is because we had not provided any URL bindings, HTTP verbs (GET, POST, etc.), or parameter mappings. We shall address that shortcoming now by adding custom HTTP annotations to the service's definition. While all our services have a similar structure, we will look at the Topic service for its HTTP annotations. Topic service with HTTP annotations: syntax = "proto3"; import "google/protobuf/field_mask.proto"; option go_package = "github.com/onehub/protos"; package onehub.v1; import "onehub/v1/models.proto"; import "google/api/annotations.proto"; /** * Service for operating on topics */ service TopicService { /** * Create a new sesssion */ rpc CreateTopic(CreateTopicRequest) returns (CreateTopicResponse) { option (google.api.http) = { post: "/v1/topics", body: "*", }; } /** * List all topics from a user. */ rpc ListTopics(ListTopicsRequest) returns (ListTopicsResponse) { option (google.api.http) = { get: "/v1/topics" }; } /** * Get a particular topic */ rpc GetTopic(GetTopicRequest) returns (GetTopicResponse) { option (google.api.http) = { get: "/v1/topics/{id=*}" }; } /** * Batch get multiple topics by ID */ rpc GetTopics(GetTopicsRequest) returns (GetTopicsResponse) { option (google.api.http) = { get: "/v1/topics:batchGet" }; } /** * Delete a particular topic */ rpc DeleteTopic(DeleteTopicRequest) returns (DeleteTopicResponse) { option (google.api.http) = { delete: "/v1/topics/{id=*}" }; } /** * Updates specific fields of a topic */ rpc UpdateTopic(UpdateTopicRequest) returns (UpdateTopicResponse) { option (google.api.http) = { patch: "/v1/topics/{topic.id=*}" body: "*" }; } } /** * Topic creation request object */ message CreateTopicRequest { /** * Topic being updated */ Topic topic = 1; } /** * Response of an topic creation. */ message CreateTopicResponse { /** * Topic being created */ Topic topic = 1; } /** * An topic search request. For now only paginations params are provided. */ message ListTopicsRequest { /** * Instead of an offset an abstract "page" key is provided that offers * an opaque "pointer" into some offset in a result set. */ string page_key = 1; /** * Number of results to return. */ int32 page_size = 2; } /** * Response of a topic search/listing. */ message ListTopicsResponse { /** * The list of topics found as part of this response. */ repeated Topic topics = 1; /** * The key/pointer string that subsequent List requests should pass to * continue the pagination. */ string next_page_key = 2; } /** * Request to get an topic. */ message GetTopicRequest { /** * ID of the topic to be fetched */ string id = 1; } /** * Topic get response */ message GetTopicResponse { Topic topic = 1; } /** * Request to batch get topics */ message GetTopicsRequest { /** * IDs of the topic to be fetched */ repeated string ids = 1; } /** * Topic batch-get response */ message GetTopicsResponse { map<string, Topic> topics = 1; } /** * Request to delete an topic. */ message DeleteTopicRequest { /** * ID of the topic to be deleted. */ string id = 1; } /** * Topic deletion response */ message DeleteTopicResponse { } /** * The request for (partially) updating an Topic. */ message UpdateTopicRequest { /** * Topic being updated */ Topic topic = 1; /** * Mask of fields being updated in this Topic to make partial changes. */ google.protobuf.FieldMask update_mask = 2; /** * IDs of users to be added to this topic. */ repeated string add_users = 3; /** * IDs of users to be removed from this topic. */ repeated string remove_users = 4; } /** * The request for (partially) updating an Topic. */ message UpdateTopicResponse { /** * Topic being updated */ Topic topic = 1; } Instead of having "empty" method definitions (e.g., rpc MethodName(ReqType) returns (RespType) {}), we are now seeing "annotations" being added inside methods. Any number of annotations can be added and each annotation is parsed by the protoc and passed to all the plugins invoked by it. There are tons of annotations that can be passed and this has a "bit of everything" in it. Back to the HTTP bindings: Typically an HTTP annotation has a method, a URL path (with bindings within { and }), and a marking to indicate what the body parameter maps to (for PUT and POST methods). For example, in the CreateTopic method, the method is a POST request to "v1/topic " with the body (*) corresponding to the JSON representation of the CreateTopicRequest message type; i.e., our request is expected to look like this: { "Topic": {... topic object...} } Naturally, the response object of this would be the JSON representation of the CreateTopicResponse message. The other examples in the topic service, as well as in the other services, are reasonably intuitive. Feel free to read through it to get any finer details. Before we are off to the next section implementing the proxy, we need to regenerate the pb.gw.go files to incorporate these new bindings: make all We will now see the following error: google/api/annotations.proto: File not found. topics.proto:8:1: Import "google/api/annotations.proto" was not found or had errors. Unfortunately, there is no "package manager" for protos at present. This void is being filled by an amazing tool: Buf.build (which will be the main topic in Part 3 of this series). In the meantime, we will resolve this by manually copying (shudder) http.proto and annotations.proto manually. So, our protos folder will have the following structure: protos ├── google │ └── api │ ├── annotations.proto │ └── http.proto └── onehub └── v1 └── topics.proto └── messages.proto └── ... However, we will follow a slightly different structure. Instead of copying files to the protos folder, we will create a vendors folder at the root and symlink to it from the protos folder (this symlinking will be taken care of by our Makefile). Our new folder structure is: onehub ├── Makefile ├── ... ├── vendors │ ├── google │ │ └── api │ │ ├── annotations.proto │ │ └── http.proto ├── proto └── google -> onehub/vendors/google └── onehub └── v1 └── topics.proto └── messages.proto └── ... Our updated Makefile is shown below. Makefile for HTTP bindings: # Some vars to detemrine go locations etc GOROOT=$(which go) GOPATH=$(HOME)/go GOBIN=$(GOPATH)/bin # Evaluates the abs path of the directory where this Makefile resides SRC_DIR:=$(shell dirname $(realpath $(firstword $(MAKEFILE_LIST)))) # Where the protos exist PROTO_DIR:=$(SRC_DIR)/protos # where we want to generate server stubs, clients etc OUT_DIR:=$(SRC_DIR)/gen/go all: createdirs printenv goprotos gwprotos openapiv2 cleanvendors goprotos: echo "Generating GO bindings" protoc --go_out=$(OUT_DIR) --go_opt=paths=source_relative \ --go-grpc_out=$(OUT_DIR) --go-grpc_opt=paths=source_relative \ --proto_path=$(PROTO_DIR) \ $(PROTO_DIR)/onehub/v1/*.proto gwprotos: echo "Generating gRPC Gateway bindings and OpenAPI spec" protoc -I . --grpc-gateway_out $(OUT_DIR) \ --grpc-gateway_opt logtostderr=true \ --grpc-gateway_opt paths=source_relative \ --grpc-gateway_opt generate_unbound_methods=true \ --proto_path=$(PROTO_DIR) \ $(PROTO_DIR)/onehub/v1/*.proto openapiv2: echo "Generating OpenAPI specs" protoc -I . --openapiv2_out $(SRC_DIR)/gen/openapiv2 \ --openapiv2_opt logtostderr=true \ --openapiv2_opt generate_unbound_methods=true \ --openapiv2_opt allow_merge=true \ --openapiv2_opt merge_file_name=allservices \ --proto_path=$(PROTO_DIR) \ $(PROTO_DIR)/onehub/v1/*.proto printenv: @echo MAKEFILE_LIST=$(MAKEFILE_LIST) @echo SRC_DIR=$(SRC_DIR) @echo PROTO_DIR=$(PROTO_DIR) @echo OUT_DIR=$(OUT_DIR) @echo GOROOT=$(GOROOT) @echo GOPATH=$(GOPATH) @echo GOBIN=$(GOBIN) createdirs: rm -Rf $(OUT_DIR) mkdir -p $(OUT_DIR) mkdir -p $(SRC_DIR)/gen/openapiv2 cd $(PROTO_DIR) && ( \ if [ ! -d google ]; then ln -s $(SRC_DIR)/vendors/google . ; fi \ ) cleanvendors: rm -f $(PROTO_DIR)/google Now running Make should be error-free and result in the updated bindings in the .pb.gw.go files. Implementing the HTTP Gateway Proxy Lo and behold, we now have a "proxy" (in the .pw.gw.go files) that translates HTTP requests and converts them into gRPC requests. On the return path, gRPC responses are also translated to HTTP responses. What is now needed is a service that runs an HTTP server that continuously facilitates this translation. We have now added a startGatewayService method in cmd/server.go that now also starts an HTTP server to do all this back-and-forth translation: import ( ... // previous imports // new imports "context" "net/http" "github.com/grpc-ecosystem/grpc-gateway/v2/runtime" ) func startGatewayServer(grpc_addr string, gw_addr string) { ctx := context.Background() mux := runtime.NewServeMux() opts := []grpc.DialOption{grpc.WithInsecure()} // Register each server with the mux here if err := v1.RegisterTopicServiceHandlerFromEndpoint(ctx, mux, grpc_addr, opts); err != nil { log.Fatal(err) } if err := v1.RegisterMessageServiceHandlerFromEndpoint(ctx, mux, grpc_addr, opts); err != nil { log.Fatal(err) } http.ListenAndServe(gw_addr, mux) } func main() { flag.Parse() go startGRPCServer(*addr) startGatewayServer(*gw_addr, *addr) } In this implementation, we created a new runtime.ServeMux and registered each of our gRPC services' handlers using the v1.Register<ServiceName>HandlerFromEndpoint method. This method associates all of the URLs found in the <ServiceName> service's protos to this particular mux. Note how all these handlers are associated with the port on which the gRPC service is already running (port 9000 by default). Finally, the HTTP server is started on its own port (8080 by default). You might be wondering why we are using the NewServeMux in the github.com/grpc-ecosystem/grpc-gateway/v2/runtime module and not the version in the standard library's net/http module. This is because the grpc-gateway/v2/runtime module's ServeMux is customized to act specifically as a router for the underlying gRPC services it is fronting. It also accepts a list of ServeMuxOption (ServeMux handler) methods that act as a middleware for intercepting an HTTP call that is in the process of being converted to a gRPC message sent to the underlying gRPC service. These middleware can be used to set extra metadata needed by the gRPC service in a common way transparently. We will see more about this in a future post about gRPC interceptors in this demo service. Generating OpenAPI Specs Several API consumers seek OpenAPI specs that describe RESTful endpoints (methods, verbs, body payloads, etc). We can generate an OpenAPI spec file (previously Swagger files) that contains information about our service methods along with their HTTP bindings. Add another Makefile target: openapiv2: echo "Generating OpenAPI specs" protoc -I . --openapiv2_out $(SRC_DIR)/gen/openapiv2 \ --openapiv2_opt logtostderr=true \ --openapiv2_opt generate_unbound_methods=true \ --openapiv2_opt allow_merge=true \ --openapiv2_opt merge_file_name=allservices \ --proto_path=$(PROTO_DIR) \ $(PROTO_DIR)/onehub/v1/*.proto Like all other plugins, the openapiv2 plugin also generates one .swagger.json per .proto file. However, this changes the semantics of Swagger as each Swagger is treated as its own "endpoint." Whereas, in our case, what we really want is a single endpoint that fronts all the services. In order to contain a single "merged" Swagger file, we pass the allow_merge=true parameter to the above command. In addition, we also pass the name of the file to be generated (merge_file_name=allservices). This results in gen/openapiv2/allservices.swagger.json file that can be read, visualized, and tested with SwaggerUI. Start this new server, and you should see something like this: % onehub % go run cmd/server.go Starting grpc endpoint on :9000: Starting grpc gateway server on: :8080 The additional HTTP gateway is now running on port 8080, which we will query next. Testing It All Out Now, instead of making grpc_cli calls, we can issue HTTP calls via the ubiquitous curl command (also make sure you install jq for pretty printing your JSON output): Create a Topic % curl -s -d '{"topic": {"name": "First Topic", "creator_id": "user1"}' localhost:8080/v1/topics | jq { "topic": { "createdAt": "2023-07-07T20:53:31.629771Z", "updatedAt": "2023-07-07T20:53:31.629771Z", "id": "1", "creatorId": "user1", "name": "First Topic", "users": [] } } And another: % curl -s localhost:8080/v1/topics -d '{"topic": {"name": "Urgent topic", "creator_id": "user2", "users": ["user1", "user2", "user3"]}' | jq { "topic": { "createdAt": "2023-07-07T20:56:52.567691Z", "updatedAt": "2023-07-07T20:56:52.567691Z", "id": "2", "creatorId": "user2", "name": "Urgent topic", "users": [ "user1", "user2", "user3" ] } } List All Topics % curl -s localhost:8080/v1/topics | jq { "topics": [ { "createdAt": "2023-07-07T20:53:31.629771Z", "updatedAt": "2023-07-07T20:53:31.629771Z", "id": "1", "creatorId": "user1", "name": "First Topic", "users": [] }, { "createdAt": "2023-07-07T20:56:52.567691Z", "updatedAt": "2023-07-07T20:56:52.567691Z", "id": "2", "creatorId": "user2", "name": "Urgent topic", "users": [ "user1", "user2", "user3" ] } ], "nextPageKey": "" } Get Topics by IDs Here, "list" values (e.g., ids) are possibly by repeating them as query parameters: % curl -s "localhost:8080/v1/topics?ids=1&ids=2" | jq { "topics": [ { "createdAt": "2023-07-07T20:53:31.629771Z", "updatedAt": "2023-07-07T20:53:31.629771Z", "id": "1", "creatorId": "user1", "name": "First Topic", "users": [] }, { "createdAt": "2023-07-07T20:56:52.567691Z", "updatedAt": "2023-07-07T20:56:52.567691Z", "id": "2", "creatorId": "user2", "name": "Urgent topic", "users": [ "user1", "user2", "user3" ] } ], "nextPageKey": "" } Delete a Topic Followed by a Listing % curl -sX DELETE "localhost:8080/v1/topics/1" | jq {} % curl -s "localhost:8080/v1/topics" | jq { "topics": [ { "createdAt": "2023-07-07T20:56:52.567691Z", "updatedAt": "2023-07-07T20:56:52.567691Z", "id": "2", "creatorId": "user2", "name": "Urgent topic", "users": [ "user1", "user2", "user3" ] } ], "nextPageKey": "" } Best Practices Separation of Gateway and gRPC Endpoints In our example, we served the Gateway and gRPC services on their own addresses. Instead, we could have directly invoked the gRPC service methods, i.e., by directly creating NewTopicService(nil) and invoking methods on those. However, running these two services separately meant we could have other (internal) services directly access the gRPC service instead of going through the Gateway. This separation of concerns also meant these two services could be deployed separately (when on different hosts) instead of needing a full upgrade of the entire stack. HTTPS Instead of HTTP However in this example, the startGatewayServer method started an HTTP server, it is highly recommended to have the gateway over an HTTP server for security, preventing man-in-the-middle attacks, and protecting clients' data. Use of Authentication This example did not have any authentication built in. However, authentication (authn) and authorization (authz) are very important pillars of any service. The Gateway (and the gRPC service) are no exceptions to this. The use of middleware to handle authn and authz is critical to the gateway. Authentication can be applied with several mechanisms like OAuth2 and JWT to verify users before passing a request to the gRPC service. Alternatively, the tokens could be passed as metadata to the gRPC service, which can perform the validation before processing the request. The use of middleware in the Gateway (and interceptors in the gRPC service) will be shown in Part 4 of this series. Caching for Improved Performance Caching improves performance by avoiding database (or heavy) lookups of data that may be frequently accessed (and/or not often modified). The Gateway server can also employ cache responses from the gRPC service (with possible expiration timeouts) to reduce the load on the gRPC server and improve response times for clients. Note: Just like authentication, caching can also be performed at the gRPC server. However, this would not prevent excess calls that may otherwise have been prevented by the gateway service. Using Load Balancers While also applicable to gRPC servers, HTTP load balancers (in front of the Gateway) enable sharding to improve the scalability and reliability of our services, especially during high-traffic periods. Conclusion By adding a gRPC Gateway to your gRPC services and applying best practices, your services can now be exposed to clients using different platforms and protocols. Adhering to best practices also ensures reliability, security, and high performance. In this article, we have: Seen the benefits of wrapping our services with a Gateway service Added HTTP bindings to an existing set of services Learned the best practices for enacting Gateway services over your gRPC services In the next post, we will take a small detour and introduce a modern tool for managing gRPC plugins and making it easy to work with them.

In this tutorial, we’ll learn how to build a website for collecting digital collectibles (or NFTs) on the blockchain Flow. We'll use the smart contract language Cadence along with React to make it all happen. We'll also learn about Flow, its advantages, and the fun tools we can use. By the end of this article, you’ll have the tools and knowledge you need to create your own decentralized application on the Flow blockchain. Let’s dive right in! What Are We Building? We're building an application for digital collectibles. Each collectible is a Non-Fungible Token (NFT). (If you are new and don’t understand NFT, then take a look here.) Our app will allow you to collect NFTs, and each item will be unique from the others. To make all this work, we’ll use Flow's NonFungibleToken Standard, which is a set of rules that helps us manage these special digital items (similar to ERC-721 in Ethereum). Prerequisites Before you begin, be sure to install the Flow CLI on your system. If you haven't done so, follow these installation instructions. Setting Up If you're ready to kickstart your project, first, type in the command flow setup. This command does some magic behind the scenes to set up the foundation of your project. It creates a folder system and sets up a file called flow.json to configure your project, making sure everything is organized and ready to go! Project Structure The project will contain a cadence folder and flow.json file. (A flow.json file is a configuration file for your project, automatically maintained.)The Cadence folder contains the following: /contracts: Contains all Cadence contracts. /scripts: Holds all Cadence scripts. /transactions: Stores all Cadence transactions. Follow the steps below to use Flow NFT Standard. Step 1: Create a File First, go to the flow-collectibles-portal folder and find the cadence folder. Then, open the contracts folder. Make a new file and name it NonFungibleToken.cdc. Step 2: Copy and Paste Now, open the link named NonFungibleToken, which contains the NFT standard. Copy all the content from that file and paste it into the new file you just created ("NonFungibleToken.cdc"). That's it! You've successfully set up the standards for your project.Now, let’s write some code! However, before we dive into coding, it's important for developers to establish a mental model of how to structure their code. At the top level, our codebase consists of three main components: NFT: Each collectible is represented as an NFT. Collection: A collection refers to a group of NFTs owned by a specific user. Global Functions and Variables: These are functions and variables defined at the global level for the smart contract and are not associated with any particular resource. Smart Contract Structure Smart Contract Basic Structure Create a new file named Collectibles.cdc inside cadence/contracts. This is where we will write the code. Contract Structure JavaScript import NonFungibleToken from "./NonFungibleToken.cdc" pub contract Collectibles: NonFungibleToken{ pub var totalSupply: UInt64 // other code will come here init(){ self.totalSupply = 0 } } Let's break down the code line by line: First, we'll need to standardize that we are building an NFT by including the so-called "NonFungibleToken." This is an NFT standard built by Flow which defines the following set of functionality that must be included by each NFT smart contract. After importing, let's create our contract. To do that, we use pub contract [contract name]. Use the same syntax each time you create a new contract. You can fill in the contract name with whatever you’d like to call your contract. In our case, let’s call it Collectibles. Next, we want to make sure our contract follows a certain set of functionality and rules of NonFungibleToken. To do that, we add NonFungibleToken interface with the help of `:`.Like this (`pub contract Collectibles: NonFungibleToken{}`) Every single contract MUST have the init() function. It is called when the contract is initially deployed. This is similar to what Solidity calls a Constructor. Now let’s create a global variable called totalSupply with a data type UInt64. This variable will keep track of your total Collectibles. Now initialize totalSupply with value 0. That's it! We set up the foundation for our Collectibles contract. Now, we can start adding more features and functionalities to make it even more exciting. Before moving forward, please check out the code snippet to understand how we define variables in Cadence: Resource NFT Add the following code to your smart contract: JavaScript import NonFungibleToken from "./NonFungibleToken.cdc" pub contract Collectibles: NonFungibleToken{ // above code… pub resource NFT: NonFungibleToken.INFT{ pub let id: UInt64 pub var name: String pub var image: String init(_id:UInt64, _name:String, _image:String){ self.id = _id self.name = _name self.image = _image } } // init()... } As you have seen before, the contract implements the NFT standard interface, represented by pub contract Collectibles: NonFungibleToken. Similarly, resources can also implement various resource interfaces. So let’s add NonFungibleToken.INFT interface to the NFT Resource, which mandates the existence of a public property called id within the resource.Here are the variables we will use in the NFT resource: id: Maintains the ID of NFT name: Name of the NFT. image: Image URL of NFT. After defining the variable, be sure to initialize the variable in the init() function. Let’s move forward and create another resource called Collection Resource. Collection Resource First, you need to understand how Collection Resources work. If you need to store a music file and several photos on your laptop, what would you do? Typically, you’d navigate to a local drive (let’s say your D-Drive) create a music folder, and photos folder. You’d then copy and paste the music and photo files into your destination folders.Similarly, this is how your digital collectibles on Flow work. Imagine your laptop as a Flow Blockchain Account, your D-Drive as Account Storage, and Folder as a Collection. So when interacting with any project to buy NFTs, the project creates its collection in your account storage, similar to creating a folder on your D-Drive. When you interact with 10 different NFT projects, you’ll end up with 10 different collections in your account. It's like having a personal space to store and organize your unique digital treasures! JavaScript import NonFungibleToken from "./NonFungibleToken.cdc" pub contract Collectibles: NonFungibleToken{ //Above code NFT Resource… // Collection Resource pub resource Collection{ } // Below code… } Each collection has a ownedNFTs variable to hold the NFT Resources. JavaScript pub resource Collection { pub var ownedNFTs: @{UInt64: NonFungibleToken.NFT} init(){ self.ownedNFTs <- {} } } Resource Interfaces A resource interface in Flow is similar to interfaces in other programming languages. It sits on top of a resource and ensures that the resource that implements it has the required functionality as defined by the interface. It can also be used to restrict access to the whole resource and be more restrictive in terms of access modifiers than the resource itself. In the NonFungibleToken standard, there are several resource interfaces like INFT, Provider, Receiver, and CollectionPublic. Each of these interfaces has specific functions and fields that need to be implemented by the resource that uses them. In this contract, we’ll use these three interfaces from NonFungibleToken: Provider, Receiver, and CollectionPublic. These interfaces define functions such as deposit, withdraw, borrowNFT, and getIDs. We’ll explain each of these in detail as we go. We will also add some events that we’ll emit from these functions, as well as declare some variables we’ll use further along in the tutorial. JavaScript pub contract Collectibles:NonFungibleToken{ // rest of the code… pub event ContractInitialized() pub event Withdraw(id: UInt64, from: Address?) pub event Deposit(id: UInt64, to: Address?) pub let CollectionStoragePath: StoragePath pub let CollectionPublicPath: PublicPath pub resource interface CollectionPublic{ pub fun deposit(token: @NonFungibleToken.NFT) pub fun getIDs(): [UInt64] pub fun borrowNFT(id: UInt64): &NonFungibleToken.NFT } pub resource Collection: CollectionPublic, NonFungibleToken.Provider, NonFungibleToken.Receiver, NonFungibleToken.CollectionPublic{ pub var ownedNFTs: @{UInt64: NonFungibleToken.NFT} init(){ self.ownedNFTs <- {} } } } Withdraw Now, let's create the withdraw() function required by the interface. JavaScript pub resource Collection: CollectionPublic, NonFungibleToken.Provider, NonFungibleToken.Receiver, NonFungibleToken.CollectionPublic{ // other code pub fun withdraw(withdrawID: UInt64): @NonFungibleToken.NFT { let token <- self.ownedNFTs.remove(key: withdrawID) ?? panic("missing NFT") emit Withdraw(id: token.id, from: self.owner?.address) return <- token } init()... } With the help of this function, you can move the NFT resource out of the collection. If it: Fails: Panic and throws an error. Successful: It emits a withdraw event and returns the resource to the caller. The caller can then use this resource and save it within their account storage. Deposit Now it’s time for the deposit() function required by NonFungibleToken.Receiver. JavaScript pub resource Collection: CollectionPublic, NonFungibleToken.Provider, NonFungibleToken.Receiver, NonFungibleToken.CollectionPublic{ // other code pub fun withdraw(withdrawID: UInt64): @NonFungibleToken.NFT { let token <- self.ownedNFTs.remove(key: withdrawID) ?? panic("missing NFT") emit Withdraw(id: token.id, from: self.owner?.address) return <- token } pub fun deposit(token: @NonFungibleToken.NFT) { let id = token.id let oldToken <- self.ownedNFTs[id] <-token destroy oldToken emit Deposit(id: id, to: self.owner?.address) } init()... } Borrow and GetID Now, let’s focus on the two functions required by NonFungibleToken.CollectionPublic: borrowNFT() and getID(). JavaScript pub resource Collection: CollectionPublic, NonFungibleToken.Provider, NonFungibleToken.Receiver, NonFungibleToken.CollectionPublic{ // other code pub fun withdraw(withdrawID: UInt64): @NonFungibleToken.NFT { let token <- self.ownedNFTs.remove(key: withdrawID) ?? panic("missing NFT") emit Withdraw(id: token.id, from: self.owner?.address) return <- token } pub fun deposit(token: @NonFungibleToken.NFT) { let id = token.id let oldToken <- self.ownedNFTs[id] <-token destroy oldToken emit Deposit(id: id, to: self.owner?.address) } pub fun borrowNFT(id: UInt64): &NonFungibleToken.NFT { if self.ownedNFTs[id] != nil { return (&self.ownedNFTs[id] as &NonFungibleToken.NFT?)! } panic("NFT not found in collection.") } pub fun getIDs(): [UInt64]{ return self.ownedNFTs.keys } init()... } Destructor The last thing we need for the Collection Resource is a destructor. JavaScript destroy (){ destroy self.ownedNFTs } Since the Collection resource contains other resources (NFT resources), we need to specify a destructor. A destructor runs when the object is destroyed. This ensures that resources are not left "homeless" when their parent resource is destroyed. We don't need a destructor for the NFT resource as it doesn’t contain any other resources. Let’s look at the complete collection resource source code: JavaScript import NonFungibleToken from "./NonFungibleToken.cdc" pub contract Collectibles: NonFungibleToken{ pub var totalSupply: UInt64 pub resource NFT: NonFungibleToken.INFT{ pub let id: UInt64 pub var name: String pub var image: String init(_id:UInt64, _name:String, _image:String){ self.id = _id self.name = _name self.image = _image } } pub resource interface CollectionPublic{ pub fun deposit(token: @NonFungibleToken.NFT) pub fun getIDs(): [UInt64] pub fun borrowNFT(id: UInt64): &NonFungibleToken.NFT } pub event ContractInitialized() pub event Withdraw(id: UInt64, from: Address?) pub event Deposit(id: UInt64, to: Address?) pub let CollectionStoragePath: StoragePath pub let CollectionPublicPath: PublicPath pub resource Collection: CollectionPublic, NonFungibleToken.Provider, NonFungibleToken.Receiver, NonFungibleToken.CollectionPublic{ pub var ownedNFTs: @{UInt64: NonFungibleToken.NFT} init(){ self.ownedNFTs <- {} } destroy (){ destroy self.ownedNFTs } pub fun withdraw(withdrawID: UInt64): @NonFungibleToken.NFT { let token <- self.ownedNFTs.remove(key: withdrawID) ?? panic("missing NFT") emit Withdraw(id: token.id, from: self.owner?.address) return <- token } pub fun deposit(token: @NonFungibleToken.NFT) { let id = token.id let oldToken <- self.ownedNFTs[id] <-token destroy oldToken emit Deposit(id: id, to: self.owner?.address) } pub fun borrowNFT(id: UInt64): &NonFungibleToken.NFT { if self.ownedNFTs[id] != nil { return (&self.ownedNFTs[id] as &NonFungibleToken.NFT?)! } panic("NFT not found in collection.") } pub fun getIDs(): [UInt64]{ return self.ownedNFTs.keys } } init(){ self.CollectionPublicPath = /public/NFTCollection self.CollectionStoragePath = /storage/NFTCollection self.totalSupply = 0 emit ContractInitialized() } } Now we have finished all the resources. Next, we’ll look at the global function. Global Function Global Functions are functions that are defined on the global level of the smart contract, meaning they are not part of any resource. These are accessible and called by the public and expose the core functionality of the smart contract to the public. createEmptyCollection: This function initializes an empty Collectibles.Collection into caller account storage. checkCollection: This handy function helps you discover whether or not your account already has a collection resource. mintNFT: This function is super cool because it allows anyone to create an NFT. JavaScript // pub resource Collection… pub fun createEmptyCollection(): @Collection{ return <- create Collection() } pub fun checkCollection(_addr: Address): Bool{ return getAccount(_addr) .capabilities.get<&{Collectibles.CollectionPublic}> (Collectibles.CollectionPublicPath)! .check() } pub fun mintNFT(name:String, image:String): @NFT{ Collectibles.totalSupply = Collectibles.totalSupply + 1 let nftId = Collectibles.totalSupply var newNFT <- create NFT(_id:nftId, _name:name, _image:image) return <- newNFT } init()... Wrapping up the Smart Contract And now, FINALLY, with everything in place, we’re done writing our smart contract. Take a look at the final code here. Now, let’s look at how a user interacts with smart contracts deployed on the Flow blockchain.There are two steps to interact with the Flow blockchain: Mutate the state by running transactions. Query the blockchain by running a script. Mutate the State by Running Transactions Transactions are cryptographically signed data that contain a set of instructions that interact with the smart contract to update the Flow state. In simple terms, this is like a function call that changes the data on the blockchain. Transactions usually involve some cost, which can vary depending on the blockchain you are on. A transaction includes multiple optional phases: prepare, pre, execute, and post phase.You can read more about this in the Cadence reference document on transactions. Each phase has a purpose; the two most important phases are prepare and execute. Prepare Phase: This phase is used to access data and information inside the signer's account (allowed by the AuthAccount type). Execute Phase: This phase is used to execute actions. Now, let’s create a transaction for our project. Follow the steps below to create a transaction in your project folder. Step 1: Create a File First, go to the project folder and open the cadence folder. Inside it, open the transaction folder and make a new file with the name Create_Collection.cdc and mint_nft.cdc Step 2: Add the Create Collection Transaction Code JavaScript import Collectibles from "../contracts/Collectibles.cdc" transaction { prepare(signer: AuthAccount) { if signer.borrow<&Collectibles.Collection>(from: Collectibles.CollectionStoragePath) == nil { let collection <- Collectibles.createEmptyCollection() signer.save(<-collection, to: Collectibles.CollectionStoragePath) let cap = signer.capabilities.storage.issue<&{Collectibles.CollectionPublic}>(Collectibles.CollectionStoragePath) signer.capabilities.publish( cap, at: Collectibles.CollectionPublicPath) } } } Let's break down this code line by line: This transaction interacts with the Collectibles smart contract. Then, it checks if the sender (signer) has a Collection resource stored in their account by borrowing a reference to the Collection resource from the specified storage path Collectibles.CollectionStoragePath. If the reference is nil, it means the signer does not yet have a collection. If the signer does not have a collection, then it creates an empty collection by calling the createEmptyCollection() function. After creating the empty collection, place it into the signer's account under the specified storage path Collectibles.CollectionStoragePath. This establishes a link between the signer's account and the newly created collection using link(). Step 3: Add the Mint NFT Transaction Code JavaScript import NonFungibleToken from "../contracts/NonFungibleToken.cdc" import Collectibles from "../contracts/Collectibles.cdc" transaction(name:String, image:String){ let receiverCollectionRef: &{NonFungibleToken.CollectionPublic} prepare(signer:AuthAccount){ self.receiverCollectionRef = signer.borrow<&Collectibles.Collection>(from: Collectibles.CollectionStoragePath) ?? panic("could not borrow Collection reference") } execute{ let nft <- Collectibles.mintNFT(name:name, image:image) self.receiverCollectionRef.deposit(token: <-nft) } } Let's break down this code line by line: We first import the NonFungibleToken and Collectibles contract. transaction(name: String, image: String) This line defines a new transaction. It takes two arguments, name, and image, both of type String. These arguments are used to pass the name and image of the NFT being minted. let receiverCollectionRef: &{NonFungibleToken.CollectionPublic} This line declares a new variable receiverCollectionRef. It is a reference to a public collection of NFTs of type NonFungibleToken.CollectionPublic. This reference will be used to interact with the collection where we will deposit the newly minted NFT. prepare(signer: AuthAccount) This line starts the prepare block, which is executed before the transaction. It takes an argument signer of type AuthAccount. AuthAccount represents the account of the transaction's signer. It borrows a reference to the Collectibles.Collection from the signer's storage inside the prepare block. It uses the borrow function to access the reference to the collection and store it in the receiverCollectionRef variable. If the reference is not found (if the collection doesn't exist in the signer's storage, for example), it will throw the error message “could not borrow Collection reference.” The execute block contains the main execution logic for the transaction. The code inside this block will be executed after the prepare block has successfully completed. nft <- Collectibles.mintNFT(_name: name, image: image) Inside the execute block, this line calls the mintNFT function from the Collectibles contract with the provided name and image arguments. This function is expected to create a new NFT with the given name and image. The <- symbol indicates that the NFT is being received as an object that can be moved (a resource). self.receiverCollectionRef.deposit(token: <-nft) This line deposits the newly minted NFT into the specified collection. It uses the deposit function on the receiverCollectionRef to transfer ownership of the NFT from the transaction's executing account to the collection. The <- symbol here also indicates that the NFT is being moved as a resource during the deposit process. Query the Blockchain by Running a Script We use a script to view or read data from the blockchain. Scripts are free and don’t need signing. Follow the steps below to create a script in your project folder. Step 1: Create a File First, go to the project folder and open the cadence folder. Inside it, open the script folder and make a new file with the name view_nft.cdc. Step 2: View the NFT Script JavaScript import NonFungibleToken from "../contracts/NonFungibleToken.cdc" import Collectibles from "../contracts/Collectibles.cdc" pub fun main(user: Address, id: UInt64): &NonFungibleToken.NFT? { let collectionCap= getAccount(user).capabilities .get<&{Collectibles.CollectionPublic}>(/public/NFTCollection) ?? panic("This public capability does not exist.") let collectionRef = collectionCap.borrow()! return collectionRef.borrowNFT(id: id) } Let's break down this code line by line: First, we import the NonFungibleToken and Collectibles contract. pub fun main(acctAddress: Address, id: UInt64): &NonFungibleToken.NFT? This line defines the entry point of the script, which is a public function named main. The function takes two parameters: acctAddress: An Address type parameter representing the address of an account on the Flow blockchain. id: A UInt64 type parameter representing the unique identifier of the NFT within the collection. Then we use getCapability to fetch the Collectibles.Collection capability for the specified acctAddress. A capability is a reference to a resource that allows access to its functions and data. In this case, it is fetching the capability for the Collectibles.Collection resource type. Then, we borrow an NFT from the collectionRef using the borrowNFT function. The borrowNFT function takes the id parameter, which is the unique identifier of the NFT within the collection. The borrow function of a capability allows reading the resource data. Finally, we return the NFT from the function. Step 3: Testnet Deployment Now, it's time to deploy our smart contract to the Flow testnet. 1. Set up a Flow account Run the following command in the terminal to generate a Flow account: Shell flow keys generate Be sure to write down your public key and private key. Next, we’ll head over to the Flow Faucet, create a new address based on our keys, and fund our account with some test tokens. Complete the following steps to create your account: Paste in your public key in the specified input field. Keep the Signature and Hash Algorithms set to default. Complete the Captcha. Click on Create Account. After setting up an account, we receive a dialogue with our new Flow address containing 1,000 test Flow tokens. Copy the address so we can use it going forward. 2. Configure the project. Now, let’s configure our project. Initially, when we set up the project, it created a flow.json file. This is the configuration file for the Flow CLI and defines the configuration for actions that the Flow CLI can perform for you. Think of this as roughly equivalent to hardhat.config.js on Ethereum. Now open your code editor and copy and paste the below code into your flow.json file. JavaScript { "contracts": { "Collectibles": "./cadence/contracts/Collectibles.cdc", "NonFungibleToken": { "source": "./cadence/contracts/NonFungibleToken.cdc", "aliases": { "testnet": "0x631e88ae7f1d7c20" } } }, "networks": { "testnet": "access.devnet.nodes.onflow.org:9000" }, "accounts": { "testnet-account": { "address": "ENTER YOUR ADDRESS FROM FAUCET HERE", "key": "ENTER YOUR GENERATED PRIVATE KEY HERE" } }, "deployments": { "testnet": { "testnet-account": [ "Collectibles" ] } } } 3. Copy and paste Paste your generated private key at the place (key: “ENTER YOUR GENERATED PRIVATE KEY HERE”) in the code. 4. Execute Now execute the code on the testnet. Go to the terminal and run the following code: Shell flow project deploy --network testnet 5. Wait for confirmation After submitting the transaction, you'll receive a transaction ID. Wait for the transaction to be confirmed on the testnet, indicating that the smart contract has been successfully deployed. Check your deployed contract here. Check the full code on GitHub. Final Thoughts and Congratulations! Congratulations! You have now built a collectibles portal on the Flow blockchain and deployed it to the testnet. What’s next? Now you can work on building the frontend, which we will cover in part 2 of this series. Have a really great day!

By John Vester CORE

Have you ever wondered what gives the cloud an edge over legacy technologies? When answering that question, the obvious but often overlooked aspect is the seamless integration of disparate systems, applications, and data sources. That's where Integration Platform as a Service (iPaaS) comes in. In today's complex IT landscape, your organization is faced with a myriad of applications, systems, and data sources, both on-premises and in the cloud. This means you face the challenge of connecting these disparate elements to enable seamless communication and data exchange. By providing a unified platform for integration, iPaaS enables you to break down data silos, automate workflows and unlock the full potential of your digital assets. Because of this, iPaaS is the unsung hero of modern enterprises. It can play a pivotal role in your digital transformation journey by streamlining and automating workflows. iPaaS also enables you to modernize legacy systems, enhance productivity, and create better experiences for your customers, users, and employees. Let's explore some key tenets of how iPaaS accelerates digital transformation: Rapid integration building: iPaaS reduces integration building time, allowing you to save resources and focus on other strategic initiatives. iPaaS accesses a list of pre-built connectors for various applications that accelerate integration and eliminate the need for custom coding to connect to a new application, service, or system. It also commonly offers a simple drag-and-drop user interface to ease the process of building the connections. Often, the user can start with a reusable template, which cuts down on development time. iPaaS can enhance the developer experience by providing robust API management tools, documentation, and testing environments. This promotes faster development and more reliable integrations. API management: iPaaS facilitates API management across their entire lifecycle — from designing to publishing, documenting, analyzing, and beyond — helping you access data faster with the necessary governance and control. iPaaS acts as a centralized hub for managing and monitoring APIs. iPaaS platforms offer robust security features like authentication, authorization, and encryption to protect sensitive data during API interactions. They also facilitate automated workflows for triggering API calls, handling data transformations, and responding to events. Modernizing legacy systems: Connecting your on-premises environment to the newer SaaS applications can significantly hinder the modernization process. iPaaS allows you to easily integrate cloud-based technologies with your legacy systems, giving you the best of both worlds and enabling a smooth transition to modern processes and technologies. iPaaS helps virtualize the entire environment, making it easy to replace or modernize your applications, irrespective of where they reside. Automation and efficiency: iPaaS helps automate repetitive complex processes and reduce manual touchpoints, ultimately improving operational efficiency and providing better customer experiences. For example, you can define a trigger in your workflow, and your functions will be automatically executed once the trigger is activated. The more you reduce human intervention, the better it gets at providing consistent results. Enabling agile operations: iPaaS enables you to rapidly integrate new applications and services at your organization as and when required, allowing you to remain agile and flexible in a quickly digitizing business market. Enhanced productivity with generative AI (Gen AI): Modern iPaaS solutions offer advanced Gen AI capabilities for rapid prototyping, error resolution, and FinOps optimization, helping you become more data-driven. It provides recommendations based on history, which makes it easier for a citizen integrator to get started without depending on the experts. Scalability and performance: One of the biggest reasons to use an integration platform on the cloud is its ability to scale up and down almost instantaneously to accommodate unpredictable workloads. Depending on the configuration you choose, you can ensure that performance does not dip even when the workload drastically increases. iPaaS enables you to scale your cloud systems seamlessly, supporting growing data volumes, increasing transaction volumes, and evolving business processes. Security and compliance: Last but not least, iPaaS helps you implement stringent security standards — including data encryption, access controls, and compliance certifications — to ensure the confidentiality, integrity, and availability of sensitive information. iPaaS as a Catalyst for Digital Transformation iPaaS is not just a technology solution; it's a strategic enabler of digital transformation as it empowers organizations to adapt, innovate, and thrive in the digital age. In that way, it acts as a catalyst for digital transformation. By embracing iPaaS, you can break down barriers, enhance collaboration, and create a connected ecosystem that drives growth and customer satisfaction.

In the dynamic world of cloud-native technologies, monitoring and observability have become indispensable. Kubernetes, the de-facto orchestration platform, offers scalability and agility. However, managing its health and performance efficiently necessitates a robust monitoring solution. Prometheus, a powerful open-source monitoring system, emerges as a perfect fit for this role, especially when integrated with Kubernetes. This guide outlines a strategic approach to deploying Prometheus in a Kubernetes cluster, leveraging helm for installation, setting up an ingress nginx controller with metrics scraping enabled, and configuring Prometheus alerts to monitor and act upon specific incidents, such as detecting ingress URLs that return 500 errors. Prometheus Prometheus excels at providing actionable insights into the health and performance of applications and infrastructure. By collecting and analyzing metrics in real-time, it enables teams to proactively identify and resolve issues before they impact users. For instance, Prometheus can be configured to monitor system resources like CPU, memory usage, and response times, alerting teams to anomalies or thresholds breaches through its powerful alerting rules engine, Alertmanager. Utilizing PromQL, Prometheus's query language, teams can dive deep into their metrics, uncovering patterns and trends that guide optimization efforts. For example, tracking the rate of HTTP errors or response times can highlight inefficiencies or stability issues within an application, prompting immediate action. Additionally, by integrating Prometheus with visualization tools like Grafana, teams can create dashboards that offer at-a-glance insights into system health, facilitating quick decision-making. Through these capabilities, Prometheus not only monitors systems but also empowers teams with the data-driven insights needed to enhance performance and reliability. Prerequisites Docker and KIND: A Kubernetes cluster set-up utility (Kubernetes IN Docker.) Helm, a package manager for Kubernetes, installed. Basic understanding of Kubernetes and Prometheus concepts. 1. Setting Up Your Kubernetes Cluster With Kind Kind allows you to run Kubernetes clusters in Docker containers. It's an excellent tool for development and testing. Ensure you have Docker and Kind installed on your machine. To create a new cluster: kind create cluster --name prometheus-demo Verify your cluster is up and running: kubectl cluster-info --context kind-prometheus-demo 2. Installing Prometheus Using Helm Helm simplifies the deployment and management of applications on Kubernetes. We'll use it to install Prometheus: Add the Prometheus community Helm chart repository: helm repo add prometheus-community https://prometheus-community.github.io/helm-charts helm repo update Install Prometheus: helm install prometheus prometheus-community/kube-prometheus-stack --namespace monitoring --create-namespace helm upgrade prometheus prometheus-community/kube-prometheus-stack \ --namespace monitoring \ --set prometheus.prometheusSpec.podMonitorSelectorNilUsesHelmValues=false \ --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false This command deploys Prometheus along with Alertmanager, Grafana, and several Kubernetes exporters to gather metrics. Also, customize your installation to scan for service monitors in all the namespaces. 3. Setting Up Ingress Nginx Controller and Enabling Metrics Scraping Ingress controllers play a crucial role in managing access to services in a Kubernetes environment. We'll install the Nginx Ingress Controller using Helm and enable Prometheus metrics scraping: Add the ingress-nginx repository: helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx helm repo update Install the ingress-nginx chart: helm upgrade --install ingress-nginx ingress-nginx/ingress-nginx \ --namespace ingress-nginx --create-namespace \ --set controller.metrics.enabled=true \ --set controller.metrics.serviceMonitor.enabled=true \ --set controller.metrics.serviceMonitor.additionalLabels.release="prometheus" This command installs the Nginx Ingress Controller and enables Prometheus to scrape metrics from it, essential for monitoring the performance and health of your ingress resources. 4. Monitoring and Alerting for Ingress URLs Returning 500 Errors Prometheus's real power shines in its ability to not only monitor your stack but also provide actionable insights through alerting. Let's configure an alert to detect when ingress URLs return 500 errors. Define an alert rule in Prometheus: Create a new file called custom-alerts.yaml and define an alert rule to monitor for 500 errors: apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: name: ingress-500-errors namespace: monitoring labels: prometheus: kube-prometheus spec: groups: - name: http-errors rules: - alert: HighHTTPErrorRate expr: | sum (rate(nginx_ingress_controller_requests{status=~"5.."}[1m])) > 0.1 OR absent(sum (rate(nginx_ingress_controller_requests{status=~"5.."}[1m]))) for: 1m labels: severity: critical annotations: summary: High HTTP Error Rate description: "This alert fires when the rate of HTTP 500 responses from the Ingress exceeds 0.1 per second over the last 5 minutes." Apply the alert rule to Prometheus: You'll need to configure Prometheus to load this alert rule. If you're using the Helm chart, you can customize the values.yaml file or create a ConfigMap to include your custom alert rules. Verify the alert is working: Trigger a condition that causes a 500 error and observe Prometheus firing the alert. For example, launch the following application: kubectl create deploy hello --image brainupgrade/hello:1.0 kubectl expose deploy hello --port 80 --target-port 8080 kubectl create ingress hello --rule="hello.internal.brainupgrade.in/=hello:80" --class nginx Access the application using the below command: curl -H "Host: hello.internal.brainupgrade.in" 172.18.0.3:31080 Wherein: 172.18.0.3 is the IP of the KIND cluster node. 31080 is the node port of the ingress controller service. This could be different in your case. Bring down the hello service pods using the following command: kubectl scale --replicas 0 deploy hello You can view active alerts in the Prometheus UI (localhost:9999) by running the following command. kubectl port-forward -n monitoring svc/prometheus-operated 9999:9090 And you will see the alert being fired. See the following snapshot: Error alert on Prometheus UI. You can also configure Alertmanager to send notifications through various channels (email, Slack, etc.). Conclusion Integrating Prometheus with Kubernetes via Helm provides a powerful, flexible monitoring solution that's vital for maintaining the health and performance of your cloud-native applications. By setting up ingress monitoring and configuring alerts for specific error conditions, you can ensure your infrastructure not only remains operational but also proactively managed. Remember, the key to effective monitoring is not just collecting metrics but deriving actionable insights that lead to improved reliability and performance.

NoSQL databases provide a flexible and scalable option for storing and retrieving data in database management. However, they can need help with object-oriented programming paradigms, such as inheritance, which is a fundamental concept in languages like Java. This article explores the impedance mismatch when dealing with inheritance in NoSQL databases. The Inheritance Challenge in NoSQL Databases The term “impedance mismatch” refers to the disconnect between the object-oriented world of programming languages like Java and NoSQL databases’ tabular, document-oriented, or graph-based structures. One area where this mismatch is particularly evident is in handling inheritance. In Java, inheritance allows you to create a hierarchy of classes, where a subclass inherits properties and behaviors from its parent class. This concept is deeply ingrained in Java programming and is often used to model real-world relationships. However, NoSQL databases have no joins, and the inheritance structure needs to be handled differently. Jakarta Persistence (JPA) and Inheritance Strategies Before diving into more advanced solutions, it’s worth mentioning that there are strategies to simulate inheritance in relational databases in the world of Jakarta Persistence (formerly known as JPA). These strategies include: JOINED inheritance strategy: In this approach, fields specific to a subclass are mapped to a separate table from the fields common to the parent class. A join operation is performed to instantiate the subclass when needed. SINGLE_TABLE inheritance strategy: This strategy uses a single table representing the entire class hierarchy. Discriminator columns are used to differentiate between different subclasses. TABLE_PER_CLASS inheritance strategy: Each concrete entity class in the hierarchy corresponds to its table in the database. These strategies work well in relational databases but are not directly applicable to NoSQL databases, primarily because NoSQL databases do not support traditional joins. Live Code Session: Java SE, Eclipse JNoSQL, and MongoDB In this live code session, we will create a Java SE project using MongoDB as our NoSQL database. We’ll focus on managing game characters, specifically Mario and Sonic characters, using Eclipse JNoSQL. You can run MongoDB locally using Docker or in the cloud with AtlasDB. We’ll start with the database setup and then proceed to the Java code implementation. Setting Up MongoDB Locally To run MongoDB locally, you can use Docker with the following command: Shell docker run -d --name mongodb-instance -p 27017:27017 mongo Alternatively, you can choose to execute it in the cloud by following the instructions provided by MongoDB AtlasDB. With the MongoDB database up and running, let’s create our Java project. Creating the Java Project We’ll create a Java SE project using Maven and the maven-archetype-quickstart archetype. This project will utilize the following technologies and dependencies: Jakarta CDI Jakarta JSONP Eclipse MicroProfile Eclipse JNoSQL database Maven Dependencies Add the following dependencies to your project’s pom.xml file: XML dependencies> <dependency> <groupId>org.jboss.weld.se</groupId> <artifactId>weld-se-shaded</artifactId> <version>${weld.se.core.version}</version> <scope>compile</scope> </dependency> <dependency> <groupId>org.eclipse</groupId> <artifactId>yasson</artifactId> <version>3.0.3</version> <scope>compile</scope> </dependency> <dependency> <groupId>io.smallrye.config</groupId> <artifactId>smallrye-config-core</artifactId> <version>3.2.1</version> <scope>compile</scope> </dependency> <dependency> <groupId>org.eclipse.microprofile.config</groupId> <artifactId>microprofile-config-api</artifactId> <version>3.0.2</version> <scope>compile</scope> </dependency> <dependency> <groupId>org.eclipse.jnosql.databases</groupId> <artifactId>jnosql-mongodb</artifactId> <version>${jnosql.version}</version> </dependency> <dependency> <groupId>net.datafaker</groupId> <artifactId>datafaker</artifactId> <version>2.0.2</version> </dependency> </dependencies> Make sure to replace ${jnosql.version} with the appropriate version of Eclipse JNoSQL you intend to use. In the next section, we will proceed with implementing our Java code. Implementing Our Java Code Our GameCharacter class will serve as the parent class for all game characters and will hold the common attributes shared among them. We’ll use inheritance and discriminator columns to distinguish between Sonic’s and Mario’s characters. Here’s the initial definition of the GameCharacter class: Java @Entity @DiscriminatorColumn("type") @Inheritance public abstract class GameCharacter { @Id @Convert(UUIDConverter.class) protected UUID id; @Column protected String character; @Column protected String game; public abstract GameType getType(); } In this code: We annotate the class with @Entity to indicate that it is a persistent entity in our MongoDB database. We use @DiscriminatorColumn("type") to specify that a discriminator column named “type” will be used to differentiate between subclasses. @Inheritance indicates that this class is part of an inheritance hierarchy. The GameCharacter class has a unique identifier (id), attributes for character name (character) and game name (game), and an abstract method getType(), which its subclasses will implement to specify the character type. Specialization Classes: Sonic and Mario Now, let’s create the specialization classes for Sonic and Mario entities. These classes will extend the GameCharacter class and provide additional attributes specific to each character type. We’ll use @DiscriminatorValue to define the values the “type” discriminator column can take for each subclass. Java @Entity @DiscriminatorValue("SONIC") public class Sonic extends GameCharacter { @Column private String zone; @Override public GameType getType() { return GameType.SONIC; } } In the Sonic class: We annotate it with @Entity to indicate it’s a persistent entity. @DiscriminatorValue("SONIC") specifies that the “type” discriminator column will have the value “SONIC” for Sonic entities. We add an attribute zone-specific to Sonic characters. The getType() method returns GameType.SONIC, indicating that this is a Sonic character. Java @Entity @DiscriminatorValue("MARIO") public class Mario extends GameCharacter { @Column private String locations; @Override public GameType getType() { return GameType.MARIO; } } Similarly, in the Mario class: We annotate it with @Entity to indicate it’s a persistent entity. @DiscriminatorValue("MARIO") specifies that the “type” discriminator column will have the value “MARIO” for Mario entities. We add an attribute locations specific to Mario characters. The getType() method returns GameType.MARIO, indicating that this is a Mario character. With this modeling approach, you can easily distinguish between Sonic and Mario characters in your MongoDB database using the discriminator column “type.” We will create our first database integration with MongoDB using Eclipse JNoSQL. To simplify, we will generate data using the Data Faker library. Our Java application will insert Mario and Sonic characters into the database and perform basic operations. Application Code Here’s the main application code that generates and inserts data into the MongoDB database: Java public class App { public static void main(String[] args) { try (SeContainer container = SeContainerInitializer.newInstance().initialize()) { DocumentTemplate template = container.select(DocumentTemplate.class).get(); DataFaker faker = new DataFaker(); Mario mario = Mario.of(faker.generateMarioData()); Sonic sonic = Sonic.of(faker.generateSonicData()); // Insert Mario and Sonic characters into the database template.insert(List.of(mario, sonic)); // Count the total number of GameCharacter documents long count = template.count(GameCharacter.class); System.out.println("Total of GameCharacter: " + count); // Find all Mario characters in the database List<Mario> marioCharacters = template.select(Mario.class).getResultList(); System.out.println("Find all Mario characters: " + marioCharacters); // Find all Sonic characters in the database List<Sonic> sonicCharacters = template.select(Sonic.class).getResultList(); System.out.println("Find all Sonic characters: " + sonicCharacters); } } } In this code: We use the SeContainer to manage our CDI container and initialize the DocumentTemplate from Eclipse JNoSQL. We create instances of Mario and Sonic characters using data generated by the DataFaker class. We insert these characters into the MongoDB database using the template.insert() method. We count the total number of GameCharacter documents in the database. We retrieve and display all Mario and Sonic characters from the database. Resulting Database Structure As a result of running this code, you will see data in your MongoDB database similar to the following structure: JSON [ { "_id": "39b8901c-669c-49db-ac42-c1cabdcbb6ed", "character": "Bowser", "game": "Super Mario Bros.", "locations": "Mount Volbono", "type": "MARIO" }, { "_id": "f60e1ada-bfd9-4da7-8228-6a7f870e3dc8", "character": "Perfect Chaos", "game": "Sonic Rivals 2", "type": "SONIC", "zone": "Emerald Hill Zone" } ] As shown in the database structure, each document contains a unique identifier (_id), character name (character), game name (game), and a discriminator column type to differentiate between Mario and Sonic characters. You will see more characters in your MongoDB database depending on your generated data. This integration demonstrates how to insert, count, and retrieve game characters using Eclipse JNoSQL and MongoDB. You can extend and enhance this application to manage and manipulate your game character data as needed. We will create repositories for managing game characters using Eclipse JNoSQL. We will have a Console repository for general game characters and a SonicRepository specifically for Sonic characters. These repositories will allow us to interact with the database and perform various operations easily. Let’s define the repositories for our game characters. Console Repository Java @Repository public interface Console extends PageableRepository<GameCharacter, UUID> { } The Console repository extends PageableRepository and is used for general game characters. It provides common CRUD operations and pagination support. Sonic Repository Java @Repository public interface SonicRepository extends PageableRepository<Sonic, UUID> { } The SonicRepository extends PageableRepository but is specifically designed for Sonic characters. It inherits common CRUD operations and pagination from the parent repository. Main Application Code Now, let’s modify our main application code to use these repositories. For Console Repository Java public static void main(String[] args) { Faker faker = new Faker(); try (SeContainer container = SeContainerInitializer.newInstance().initialize()) { Console repository = container.select(Console.class).get(); for (int index = 0; index < 5; index++) { Mario mario = Mario.of(faker); Sonic sonic = Sonic.of(faker); repository.saveAll(List.of(mario, sonic)); } long count = repository.count(); System.out.println("Total of GameCharacter: " + count); System.out.println("Find all game characters: " + repository.findAll().toList()); } System.exit(0); } In this code, we use the Console repository to save both Mario and Sonic characters, demonstrating its ability to manage general game characters. For Sonic Repository Java public static void main(String[] args) { Faker faker = new Faker(); try (SeContainer container = SeContainerInitializer.newInstance().initialize()) { SonicRepository repository = container.select(SonicRepository.class).get(); for (int index = 0; index < 5; index++) { Sonic sonic = Sonic.of(faker); repository.save(sonic); } long count = repository.count(); System.out.println("Total of Sonic characters: " + count); System.out.println("Find all Sonic characters: " + repository.findAll().toList()); } System.exit(0); } This code uses the SonicRepository to save Sonic characters specifically. It showcases how to work with a repository dedicated to a particular character type. With these repositories, you can easily manage, query, and filter game characters based on their type, simplifying the code and making it more organized. Conclusion In this article, we explored the seamless integration of MongoDB with Java using the Eclipse JNoSQL framework for efficient game character management. We delved into the intricacies of modeling game characters, addressing challenges related to inheritance in NoSQL databases while maintaining compatibility with Java's object-oriented principles. By employing discriminator columns, we could categorize characters and store them within the MongoDB database, creating a well-structured and extensible solution. Through our Java application, we demonstrated how to generate sample game character data using the Data Faker library and efficiently insert it into MongoDB. We performed essential operations, such as counting the number of game characters and retrieving specific character types. Moreover, we introduced the concept of repositories in Eclipse JNoSQL, showcasing their value in simplifying data management and enabling focused queries based on character types. This article provides a solid foundation for harnessing the power of Eclipse JNoSQL and MongoDB to streamline NoSQL database interactions in Java applications, making it easier to manage and manipulate diverse data sets. Source code

By Otavio Santana CORE

In the paradigm of zero trust architecture, Privileged Access Management (PAM) is emerging as a key component in a cybersecurity strategy designed to control and monitor privileged access within an organization. This article delves into the pivotal role of PAM in modern cybersecurity, exploring its principles, implementation strategies, and the evolving landscape of privileged access. What Is a Privileged User and a Privileged Account? A privileged user is someone who has been granted elevated permissions to access certain data, applications, or systems within an organization. These users are typically IT admins who require these privileges to perform their job duties, such as system administrators, database administrators, and network engineers. A privileged account refers to the actual set of login credentials that provides an elevated level of access. These accounts can perform actions that can have far-reaching implications within an IT environment. Examples of privileged accounts include: Interactive login accounts: These are standard accounts used for logging into systems and performing administrative tasks. Non-interactive accounts: These accounts don't interact directly with the user interface but are used for automated tasks like running batch jobs or scripts. Generic/shared/default accounts: Such as the "root" account in Unix systems or the "Administrator" account in Windows systems, these are often shared among multiple users and have significant privileges across systems. Service accounts: Used by applications or services to interact with the operating system or other applications, these accounts often have elevated privileges to perform specific tasks and are not tied to a personal user's credentials. Popular Data/Security Breaches: What’s the Common Link? The common link between popular data and security breaches is often the exploitation of privileged accounts. Whether the perpetrator is a script kiddie or a seasoned cybercriminal, gaining control of privileged accounts is typically a key objective. This is because privileged accounts have elevated access rights and permissions that allow wide-reaching control of IT systems, potentially allowing attackers to steal sensitive data, install malicious software, and create new accounts to maintain access for future exploitation. Anatomy of a Data Breach: Privileged Access Is the Key to the Kingdom In the journey of a security or data breach, it all starts with an initial breach point: an exploited vulnerability, a phishing email, or a compromised password. This serves as the entryway for threat actors. However, their ultimate target transcends this initial breach: privileged access. This access isn't just any key; it's the master key, unlocking access to critical systems and data. Imagine someone gaining control of a domain admin account — it's as if they've been given unrestricted access to explore every corner of an organization's digital domain. This stealthy movement and the exploitation of privileged accounts highlight the significant risks and underscore the importance of vigilant security measures in safeguarding an organization's digital assets. Cost of a Data Breach In 2023, businesses worldwide felt a significant financial hit from data breaches, with costs averaging $4.45 million. This trend highlights the increasing expenses linked to cybersecurity issues. The U.S. saw the highest costs at $9.48 million per breach, reflecting its complex digital and regulatory landscape. These figures emphasize the crucial need for strong cybersecurity investments to reduce the financial and operational impacts of data breaches. Integrating Privileged Access Management (PAM) solutions can substantially enhance cybersecurity defenses, minimizing the likelihood and impact of breaches (Source). Common Challenges With Privileged Identities and How a Pam Solution Can Prevent Cyberattacks Just-in-time admin access: In any organization, admins possess broad access to sensitive data, including financial, employee, and customer information, due to their role. This access makes privileged admin accounts a focal point for security breaches, whether through deliberate misuse or accidental exposure. Just-in-time admin access within the realm of Privileged Access Management (PAM) refers to granting privileged access on an as-needed basis. A PAM solution facilitates this by enabling root and admin-level access for a limited timeframe, significantly reducing the risk of a compromised account being used to infiltrate critical systems. Securing admin access further through multi-factor authentication and user behavior analytics enhances protection against unauthorized use. Compliance visibility: As organizations continuously integrate new IT devices to enable business operations, tracking, securing, and auditing privileged access becomes increasingly challenging. This complexity escalates with multiple vendors and contractors accessing these critical systems using personal devices, leading to substantial compliance costs. A PAM solution provides organizations with control over privileged accounts through continuous auto-discovery and reporting. Acting as a central repository for all infrastructure devices, it simplifies compliance and allows data owners to gain comprehensive visibility over privileged access across the network. Cyber risk with privileged identities: Cyberattacks often correlate directly with the misuse of privileged identities. Leading cybersecurity firms like Mandiant have linked 100% of data breaches to stolen credentials. These breaches typically involve the escalation from low-privileged accounts, such as those of sales representatives, to high-privileged accounts, like Windows or Unix administrators, in a phenomenon known as vertical privilege escalation. The risk is not limited to external hackers: disgruntled employees with admin access pose a significant threat. The increasing prevalence of security breaches via privileged identities underscores the importance of understanding who possesses critical access within an organization. A PAM solution addresses this by enabling frequent rotation of privileged account passwords each time they are checked out by a user. Integrating multi-factor authentication with PAM solutions can further minimize cyber risks, including those from social engineering and brute-force attacks. Stagnant/less complex passwords: Various factors can contribute to vulnerable or compromised passwords, including the lack of centralized password management, weak encryption across devices, the use of embedded and service accounts without password expiration, and the practice of using identical passwords across corporate and external sites. Furthermore, overly complex enterprise password policies may lead to insecure practices, such as writing passwords on sticky notes. A PAM solution effectively secures passwords in a vault and automates their rotation on endpoint devices, offering a robust defense against hacking tools like Mimikatz. It promotes secure access by allowing admins to use multi-factor authentication to connect to PAM and subsequently to devices without direct exposure to passwords, thus significantly reducing risk. Uncontrollable SSH keys: SSH keys, which utilize public-key cryptography for authentication, pose a challenge due to their perpetual validity and the ability to link multiple keys to a single account. Managing these keys is crucial, as their misuse can allow unauthorized root access to critical systems, bypassing security controls. A survey by Ponemon on SSH security vulnerabilities highlighted that three-quarters of enterprises lack security controls for SSH, over half have experienced SSH key-related compromises, and nearly half do not rotate or change SSH keys. Additionally, a significant number of organizations lack automated processes for SSH key policy enforcement and cannot detect new SSH keys, underscoring the ongoing vulnerability and the need for effective management solutions. Implementing RBAC (Role Based Access Control) in PAM: By assigning specific roles to users and linking these roles with appropriate access rights, RBAC ensures that individuals have only the access they need for their job tasks. This method adheres to the least privilege principle, effectively shrinking the attack surface by limiting high-level access. Such a controlled access strategy reduces the likelihood of cyber attackers exploiting privileged accounts. RBAC also aids in tightly managing access to critical systems and data, offering access strictly on a need-to-know basis. This targeted approach significantly lowers the risk of both internal and external threats, enhancing the security framework of an organization against potential cyber intrusions. Core Components of a PAM Solution Credential vault: This is a secure repository for storing and managing passwords, certificates, and keys. The vault typically includes functionality for automated password rotation and controlled access to credentials, enhancing security by preventing password misuse or theft. Access manager: This component is responsible for maintaining a centralized directory of users, groups, devices, and policies. It enables the administration of access rights, ensuring that only authorized individuals can access sensitive systems and data. Session management and monitoring: This provides the ability to monitor, record, and control active sessions involving privileged accounts. This also includes the capture of screen recordings and keystrokes for audits and reviews. Configuration management: Configuration Management within PAM maintains the system's health by managing integrations, updates, and security configurations, ensuring the PAM aligns with the broader IT policies and infrastructure. Key Considerations for Selecting the Right PAM Solution Integration capabilities: Look for a solution that seamlessly integrates with your existing IT infrastructure, including other IAM solutions, directories, databases, applications, and cloud services. Compliance requirements: Ensure the PAM solution aligns with your organization's regulatory requirements and industry standards, such as HIPAA, PCI DSS, SOX, etc. Security features: Look for solutions with robust security features such as privileged session management, password vaulting, multi-factor authentication (MFA), and granular access controls to ensure comprehensive protection of sensitive assets. Scalability: Evaluate whether the chosen deployment model (cloud or on-premise) can scale to accommodate your organization's growth, supporting an increasing number of privileged accounts, users, and devices while maintaining performance and security. High availability and disaster recovery: PAM can be a single point of failure. Look for features that ensure the PAM system remains available even in the face of an outage. This includes options for redundancy, failover, and backup capabilities to prevent downtime or data loss. Implementing a PAM Solution Implementation involves several stages, from initiation and design to deployment and continuous compliance, with an emphasis on stakeholder buy-in, policy development, and ongoing monitoring. Initiate Needs assessment: Evaluate the organization's current privileged access landscape, including existing controls and gaps. Project planning: Define the project's scope, objectives, and resources. Establish a cross-functional team with clear roles and responsibilities. Stakeholder buy-in: Secure commitment from management and key stakeholders by demonstrating the importance of PAM for security and compliance. Design and Develop Solution architecture: Design the PAM solution's infrastructure, considering integration with existing systems and future scalability. Policy definition: Develop clear policies for privileged account management, including credential storage, access controls, and monitoring. Configuration: Customize and configure the PAM software to fit organizational needs, including the development of any required integrations or custom workflows. Implement Deployment: Roll out the PAM solution in phases, starting with a pilot phase to test its effectiveness and make adjustments. Training and communication: Provide comprehensive training for users and IT staff and communicate changes organization-wide. Transition: Migrate privileged accounts to the PAM system, enforce new access policies, and decommission legacy practices. Continuous Compliance Monitoring and auditing: Use the PAM solution to continuously monitor privileged access and conduct regular audits for irregular activities or policy violations. Policy review and updating: Regularly review policies to ensure they remain effective and compliant with evolving regulations and business needs. Continuous improvement: Leverage feedback and insights gained from monitoring and audits to improve PAM practices and technologies. Considerations/Challenges While Implementing a PAM Solution Developing a clear and concise business case: Articulate the benefits and necessities of a PAM solution to gain buy-in from stakeholders. This should outline the risks mitigated and the value added in terms of security and compliance. Resistance to change: Admins and users may view the PAM system as an additional, unnecessary burden. Overcoming this requires change management strategies, training, and clear communication on the importance of the PAM system. Password vault as a single point of failure: The centralized nature of a password vault means it could become a single point of failure if not properly secured and managed. Implementing robust security measures and disaster recovery plans is essential. Load-balancing and clustering: To ensure high availability and scalability, the PAM system should be designed with load-balancing and clustering capabilities, which can add complexity to the implementation. Maintaining an up-to-date CMDB (Configuration Management Database): An accurate CMDB is crucial for the PAM solution to manage resources effectively. Risk-based approach to implementation: Prioritize the deployment of the PAM solution based on a risk assessment. Identify and protect the "crown jewels" of the organization first, ensuring that the most critical assets have the strongest controls in place. Final Thoughts Privileged Access Management is integral to safeguarding organizations against cyber threats by effectively managing and controlling privileged access. Implementation requires a comprehensive approach, addressing challenges while emphasizing stakeholder buy-in and continuous improvement to uphold robust cybersecurity measures. DISCLAIMER: The opinions and viewpoints are solely mine in this article and do not reflect my employer's official position or views. The information is provided "as is" without any representations or warranties, express or implied. Readers are encouraged to consult with professional advisors for advice concerning specific matters before making any decision. The use of information contained in this article is at the reader's own risk.

What Is Patch Management? Patch management is a proactive approach to mitigate already-identified security gaps in software. Most of the time, these patches are provided by third-party vendors to proactively close the security gaps and secure the platform, for example. RedHat provides security advisories and patches for various RedHat products such as RHEL, OpenShift, OpenStack, etc. Microsoft provides patches in the form of updates for Windows OS. These patches include updates to third-party libraries, modules, packages, or utilities. Patches are prioritized and, in most organizations, patching of systems is done at a specific cadence and handled through a change control process. These patches are deployed through lower environments first to understand the impact and then applied in higher environments, such as production. Various tools such as Ansible and Puppet can handle patch management seamlessly for enterprise infrastructures. These tools can automate the patch management process, ensuring that security patches and updates are promptly applied to minimize application disruptions and security risks. Coordination for patching and testing with various stakeholders using infrastructure is a big deal to minimize interruptions. What Is a Container? A container is the smallest unit of software that runs in the container platform. Unlike traditional software that, in most cases, includes application-specific components such as application files, executables, or binaries, containers include the operating system required to run the application and all other dependencies for the application. Containers include everything needed to run the application; hence, they are self-contained and provide greater isolation. With all necessary components packaged together, containers provide inherent security and control, but at the same time, are more vulnerable to threats. Containers are created using a container image, and a container image is created using a Dockerfile/Containerfile that includes instructions for building an image. Most of the container images use open-source components. Therefore, organizations have to make efforts to design and develop recommended methods to secure containers and container platforms. The traditional security strategies and tools would not work for securing containers. DZone’ previously covered how to health check Docker containers. For infrastructure using physical machines or virtual machines for hosting applications, the operations team would SSH to servers (manually or with automation) and then upgrade the system to the latest version or latest patch on a specific cadence. If the application team needs to make any changes such as updating configurations or libraries, they would do the same thing by logging in to the server and making changes. If you know what this means, in various cases, the servers are configured for running specific applications. In this case, the server becomes a pet that needs to be cared for as it creates a dependency for the application, and keeping such servers updated with the latest patches sometimes becomes challenging due to dependency issues. If the server is shared with multiple applications, then updating or patching such servers consumes a lot of effort from everyone involved to make sure applications run smoothly post-upgrade. However, containers are meant to be immutable once created and expected to be short-lived. As mentioned earlier, containers are created from container images; so it's really the container image that needs to be patched. Every image contains one or more file system layers which are built based on the instructions from Containerfile/Dockerfile. Let’s further delve into how to do the patch management and vulnerability management for containers. What Is Vulnerability Management? While patch management is proactive, vulnerability management is a reactive approach to managing and maintaining the security posture within an organization. Platforms and systems are scanned in real-time, at specific schedules, or on an ad hoc basis to identify common vulnerabilities. These are also known as CVEs (Common Vulnerability and Exposures). The tools that are used to discover CVEs use various vulnerability databases such as the U.S. National Vulnerability Database (NVD) and the CERT/CC Vulnerability Notes Database. Most of the vendors that provide scanning tools also maintain their own database to compare the CVEs and score them based on the impact. Every CVE gets a unique code along with a score in terms of severity (CVSS) and resolution, if any (e.g., CVE-2023-52136). Once the CVEs are discovered, these are categorized based on the severity and prioritized based on the impact. Not every Common Vulnerabilities and Exposure (CVE) has a resolution available. Therefore, organizations must continuously monitor such CVEs to comprehend their impact and implement measures to mitigate them. This could involve taking steps such as temporarily removing the system from the network or shutting down the system until a suitable solution is found. High-severity and critical vulnerabilities should be remediated so that they can no longer be exploited. As is evident, patch management and vulnerability management are intrinsically linked in terms of security. Their shared objective is to safeguard an organization's infrastructure and data from cyber threats. Container Security Container security entails safeguarding containerized workloads and the broader ecosystem through a mix of various security tools and technologies. Patch management and vulnerability management are integral parts of this process. The container ecosystem is also often referred to as a container supply chain. The container supply chain includes various components. When we talk about securing containers, it is essentially monitoring and securing the various components listed below. Containers A container is also called a runtime instance of a container image. It uses instructions provided in the container image to run itself. The container has lifecycle stages such as create, start, run, stop, delete, etc. This is the smallest unit which has existence in the container platform and you can log in to it, execute commands, monitor it, etc. Container Orchestration Platform Orchestration platforms provide various capabilities such as HA, scalability, self-healing, logging, monitoring, and visibility for container workloads. Container Registry A container registry includes one or more repositories where container images are stored, are version-controlled, and made available to container platforms. Container Images A container image is sometimes also called a build time instance of a container. It is a read-only template or artifact that includes everything needed to start and run the container (e.g., minimal operating system, libraries, packages, software) along with how to run and configure the container. Development Workspaces The development workspaces reside on developer workstations that are used for writing code, packaging applications, and creating and testing containers. Container Images: The Most Dynamic Component Considering the patch management and vulnerability management for containers, let's focus on container images, the most dynamic component of the supply chain. In the container management workflow, most of the exploits are encountered due to various security gaps in container images. Let’s categorize various container images used in the organization based on hierarchy. 1. Base Images This is the first level in the image hierarchy. As the name indicates, these base images are used as parent images for most of the custom images that are built within the organization. These images are pulled down from various external public and private image registries such as DockerHub, the RedHat Ecosystem Catalog, and the IBM Cloud. 2. Enterprise Images Custom images are created and built from base images and include enterprise-specific components, standard packages, or structures as part of enterprise security and governance. These images are then modified to meet certain standards for organization and published in private container registries for consumption by various application teams. Each image has an assigned owner responsible for managing the image's lifecycle. 3. Application Images These images are built using enterprise custom images as a base. Applications are added on top of them to build application images. These application images are further deployed as containers to container platforms. 4. Builder Images These images are primarily used in the CI/CD pipeline for compiling, building, and deploying application images. These images are based on enterprise custom images and include software required to build applications, create container images, perform testing, and finally, deploy images as part of the pipeline. 5. COTS Images These are vendor-provided images for vendor products. These are also called custom off-the-shelf (COTS) products managed by vendors. The lifecycle for these images is owned by vendors. For simplification, the image hierarchy is represented in the diagram below. Now that we understand various components of the container supply chain and container image hierarchy, let's understand how patching and vulnerability management are done for containers. Patching Container Images Most of the base images are provided by community members or vendors. Similar to traditional patches provided by vendors, image owners proactively patch these base images to mitigate security issues and make new versions available regularly in the container registries. Let's take an example of Python 3.11 Image from RedHat. RedHat patches this image regularly and also provides a Health Index based on scan results. RedHat proactively fixes vulnerabilities and publishes new versions post-testing. The image below indicates that the Python image is patched every 2-3 months, and corresponding CVEs are published by RedHat. This patching involves modifying the Containerfile to update required packages to fix vulnerabilities as well as building and publishing a new version (tag) of the image in the registry. Let’s move to the second level in the image hierarchy: Enterprise custom images. These images are created by organizations using base images (e.g., Python 3.11) to add enterprise-specific components to the image and harden it further for use within the organization. If the base image changes in the external registry, the enterprise custom image should be updated to use a newer version of the base image. This will create a new version of the Enterprise custom image using an updated Containerfile. The same workflow should be followed to update any of the downstream images, such as application and builder images that are built using Enterprise custom images. This way, the entire chain of images will be patched. In this entire process, the patching is done by updating the Containerfile and publishing new images to the image registry. As far as COTS images, the same process is followed by the vendor, and consumers of the images have to make sure new versions of images are being used in the organization. Vulnerability Management for Containers Patch management to secure containers is only half part of the process. Container images have to be scanned regularly or at a specific cadence to identify newly discovered CVEs within images. There are various scanning tools available in the market that scan container images as well as platforms to identify security gaps and provide visibility for such issues. These tools identify security gaps such as running images with root privileges, having directories world-writable, exposed secrets, exposed ports, vulnerable libraries, and many more. These vulnerability reports help organizations to understand the security postures of images being used as well as running containers in the platform. The reports also provide enough information to address these issues. Some of these tools also provide the ability to define policies and controls such that they can block running images if they violate policies defined by the organization. They could even stop running containers if that's what the organization decides to implement. As far as mitigating such vulnerabilities, the process involves the same steps mentioned in the patch management section; i.e., updating the Containerfile to create a new Docker image, rescanning the image to make sure reported vulnerabilities don’t exist anymore, testing the image and publish it to image registry. Depending on where the vulnerability exists in the hierarchy, the respective image and all downstream images need to be updated. Let’s look at an example. Below is the scan report from the python-3.11:1-34 image. It provides 2 important CVEs against 3 packages. These 2 CVEs will also be reported in all downstream images built based on the python-3.11:1-34 image. On further browsing CVE-2023-38545, more information is provided, including action required to remediate the CVE. It indicates that, based on the operating system within the corresponding image, the curl package should be upgraded in order to resolve the issue. From an organizational standpoint, to address this vulnerability, a new Dockerfile or Containerfile needs to be developed. This file should contain instructions to upgrade the curl package and generate a new image with a unique tag. Once the new image is created, it can be utilized in place of the previously affected image. As per the hierarchy mentioned in image-1, all downstream images should be updated with the new image in order to fix the reported CVE across all images. All images, including COTS images, should be regularly scanned. For COTS images, the organization should contact the vendor (image owner) to fix critical vulnerabilities. Shift Left Container Security Image scanning should be part of every stage in the supply chain pipeline. Detecting and addressing security issues early is crucial to avoid accumulating technical debt as we progress through the supply chain. The sooner we identify and rectify security vulnerabilities, the less disruptive they will be to our operations and the lower the amount of work required to fix them later. Local Scanning In order to build Docker images locally, developers need to have tools such as Docker and Podman installed locally on the workstation. Along with these tools, scanning tools should be made available so that developers can scan images pulled from external registries to determine if those images are safe to use. Also, once they build application images, they should have the ability to scan those images locally before moving to the next stage in the pipeline. Analyzing and fixing vulnerabilities at the source is a great way to minimize the security risks further in the lifecycle. Most of the tools provide a command line interface or IDE plugins for security tools for the ease of local scanning. Some organizations create image governance teams that pull, scan, and approve images from external registries before allowing them to be used within the organization. They take ownership of base images and manage the lifecycle of these images. They communicate with all stakeholders on the image updates and monitor new images being used by downstream consumers. This is a great way to maintain control of what images are being used within an organization. Build Time Scanning Integrate image scanning tools in the CI/CD pipeline during the image build stage to make sure every image is getting scanned. Performing image scans as soon as the image is built and determining if the image can be published to the image registry is a good approach to allowing only safe images in the image registry. Additional control gates can be introduced before the production use of the image by enforcing certain policies specifically for production images. Image Registry Scanning Build-time scanning is essentially an on-demand scanning of images. However, given that new vulnerabilities are constantly being reported and added to the Common Vulnerabilities and Exposures (CVE) database, images stored in the registry need to be scanned at regular intervals. Images with critical vulnerabilities have to be reported to the image owners to take action. Runtime Container Scanning This is real-time scanning of running containers within a platform to identify the security posture of containers. Along with analysis that's being done for images, runtime scan also determines additional issues such as the container running with root privileges, what ports it's listening on, if it's connected to the internet, and any runway process being executed. Based on the capability of the scanning tool, it provides full visibility and a security view of the entire container platform, including the hosts on which the platform is running. The tool could also enforce certain policies, such as blocking specific containers or images from running, identifying specific CVEs, and taking action. Note that this is the last stage in the container supply chain. Hence, fixing any issues at this stage is costlier than any other stage. Challenges With Container Security From the process standpoint, it looks straightforward to update base images with new versions and all downstream images. However, it comes with various challenges. Below are some of the common challenges you would encounter as you start looking into the process of patching and vulnerability management for containers: Identifying updates to any of the parent/base images in the hierarchy Identifying image hierarchy and impacted images in the supply chain Making sure all downstream images are updated when a new parent image is made available Defining ownership of images and identifying image owners Communication across various groups within the organization to ensure controls are being maintained Building a list of trusted images to be used within an organization and managing the lifecycle of the same Managing vendor images due to lack of control Managing release timelines at the same time as securing the pipeline Defining controls across the enterprise with respect to audit, security, and governance Defining exception processes to meet business needs Selecting the right scanning tool for the organization and integration with the supply chain Visibility of vulnerabilities across the organization; providing scan results post-scanning of images to respective stakeholders Patch Management and Containers Summarized This article talks about how important it is to keep things secure in container systems, especially by managing patches and dealing with vulnerabilities. Containers are like independent software units that are useful but need special security attention. Patch management means making sure everything is up to date, starting from basic images to more specific application and builder images. At the same time, vulnerability management involves regularly checking for potential security issues and fixing them, like updating files and creating new images. The idea of shifting left suggests including security checks at every step, from creating to running containers. Despite the benefits, there are challenges, such as communicating well in teams and handling images from external sources. This highlights the need for careful control and ongoing attention to keep organizations safe from cyber threats throughout the container process.