An Approach To Synthetic Transactions With Spring Microservices: Validating Features and Upgrades

In fintech application mobile apps or the web, deploying new features in areas like loan applications requires careful validation. Traditional testing with real user data, especially personally identifiable information (PII), presents significant challenges. Synthetic transactions offer a solution, enabling the thorough testing of new functionalities in a secure and controlled environment without compromising sensitive data.

By simulating realistic user interactions within the application, synthetic transactions enable developers and QA teams to identify potential issues in a controlled environment. Synthetic transactions help in ensuring that every aspect of a financial application functions correctly after any major updates or new features are rolled out. In this article, we delve into one of the approaches for using synthetic transactions.

Synthetic Transactions for Financial Applications

Key Business Entity

At the heart of every financial application lies a key entity, be it a customer, user, or loan application itself. This entity is often defined by a unique identifier, serving as the cornerstone for transactions and operations within the system. The inception point of this entity, when it is first created, presents a strategic opportunity to categorize it as either synthetic or real. This categorization is critical, as it determines the nature of interactions the entity will undergo.

Marking an entity as synthetic or for test purposes from the outset allows for a clear delineation between test and real data within the application's ecosystem. Subsequently, all transactions and operations conducted with this entity can be safely recognized as part of synthetic transactions. This approach ensures that the application's functionality can be thoroughly tested in a realistic environment.

Intercepting and Managing Synthetic Transactions

A critical component of implementing synthetic transactions lies in the interception and management of these transactions at the HTTP request level. Utilizing Spring's HTTP Interceptor mechanism, we can discern and process synthetic transactions by examining specific HTTP headers.

The below visual outlines the coordination between a synthetic HTTP interceptor and a state manager in managing the execution of an HTTP request:

The SyntheticTransactionInterceptor acts as the primary gatekeeper, ensuring that only transactions identified as synthetic are allowed through the testing pathways. Below is the implementation:

@Component
public class SyntheticTransactionInterceptor implements HandlerInterceptor {

    protected final Logger logger = LoggerFactory.getLogger(this.getClass());

    @Autowired
    SyntheticTransactionService syntheticTransactionService;

    @Autowired
    SyntheticTransactionStateManager syntheticTransactionStateManager;

    @Override
    public boolean preHandle(HttpServletRequest request,HttpServletResponse response, Object object) throws Exception {
        String syntheticTransactionId = request.getHeader("x-synthetic-transaction-uuid");
        if (syntheticTransactionId != null && !syntheticTransactionId.isEmpty()){
            if (this.syntheticTransactionService.validateTransactionId(syntheticTransactionId)){
                logger.info(String.format("Request initiated for synthetic transaction with transaction id:%s", syntheticTransactionId));
                this.syntheticTransactionStateManager.setSyntheticTransaction(true);
                this.syntheticTransactionStateManager.setTransactionId(syntheticTransactionId);
            }
        }
        return true;
    }
}

In this implementation, the interceptor looks for a specific HTTP header (x-synthetic-transaction-uuid) carrying a UUID. This UUID is not just any identifier but a validated, time-limited key designated for synthetic transactions. The validation process includes checks on the UUID's validity, its lifespan, and whether it has been previously used, ensuring a level of security and integrity for the synthetic testing process.

 After a synthetic ID is validated by the SyntheticTransactionInterceptor, the SyntheticTransactionStateManager plays a pivotal role in maintaining the synthetic context for the current request. The SyntheticTransactionStateManager is designed with request scope in mind, meaning its lifecycle is tied to the individual HTTP request. This scoping is essential for preserving the integrity and isolation of synthetic transactions within the application's broader operational context. By tying the state manager to the request scope, the application ensures that synthetic transaction states do not bleed over into unrelated operations or requests. Below is the implementation of the synthetic state manager:

@Component
@RequestScope
public class SyntheticTransactionStateManager {
    private String transactionId;
    private boolean syntheticTransaction;

    public String getTransactionId() {
        return transactionId;
    }

    public void setTransactionId(String transactionId) {
        this.transactionId = transactionId;
    }

    public boolean isSyntheticTransaction() {
        return syntheticTransaction;
    }

    public void setSyntheticTransaction(boolean syntheticTransaction) {
        this.syntheticTransaction = syntheticTransaction;
    }
}

When we persist the key entity, be it a customer, user, or loan application—the application's service layer or repository layer consults the SyntheticTransactionStateManager to confirm the transaction's synthetic nature. If the transaction is indeed synthetic, the application proceeds to persist not only the synthetic identifier but also an indicator that the entity itself is synthetic. This sets the foundations for the synthetic transaction flow. This approach ensures that from the moment an entity is marked as synthetic, all related operations and future APIs, whether they involve data processing or business logic execution, are conducted in a controlled manner. 

For further API calls initiated from the financial application, upon reaching the microservice, we load the application context for that specific request based on the token or entity identifier provided. During the context loading, we ascertain whether the key business entity (e.g., loan application, user/customer) is synthetic. If affirmative, we then set the state manager's syntheticTransaction flag to true and also assign the synthetic transactionId from the application context. 

This approach negates the need to pass a synthetic transaction ID header for subsequent calls within the application flow. We only need to send a synthetic transaction ID during the initial API call that creates the key business entity. Since this step involves using explicit headers that may not be supported by the financial application, whether it's a mobile or web platform, we can manually make this first API call with Postman or a similar tool. Afterwards, the application can continue with the rest of the flow in the financial application itself. Beyond managing synthetic transactions within the application, it's also crucial to consider how external third-party API calls behave within the context of the synthetic transaction. 

External Third-Party API Interactions

In financial applications handling key entities with personally identifiable information (PII), we conduct validations and fraud checks on user-provided data, often leveraging external third-party services. These services are crucial for tasks such as PII validation and credit bureau report retrieval. However, when dealing with synthetic transactions, we cannot make calls to these third-party services.

The solution involves creating mock responses or utilizing stubs for these external services during synthetic transactions. This approach ensures that while synthetic transactions undergo the same processing logic as real transactions, they do so without the need for actual data submission to third-party services. Instead, we simulate the responses that these services would provide if they were called with real data. This allows us to thoroughly test the integration points and data-handling logic of our application. Below is the code snippet for pulling the bureau report. This call happens as part of the API call where the key entity is been created, and then subsequently we pull the applicant's bureau report:

@Override
@Retry(name = "BUREAU_PULL", fallbackMethod = "getBureauReport_Fallback")
public CreditBureauReport getBureauReport(SoftPullParams softPullParams, ErrorsI error) {
    CreditBureauReport result = null;
    try {
        Date dt = new Date();
        logger.info("UWServiceImpl::getBureauReport method call at :" + dt.toString());
        CreditBureauReportRequest request = this.getCreditBureauReportRequest(softPullParams);
        RestTemplate restTemplate = this.externalApiRestTemplateFactory.getRestTemplate(softPullParams.getUserLoanAccountId(), "BUREAU_PULL",
                softPullParams.getAccessToken(), "BUREAU_PULL", error);
        HttpHeaders headers = this.getHttpHeaders(softPullParams);
        HttpEntity<CreditBureauReportRequest> entity = new HttpEntity<>(request, headers);
        long startTime = System.currentTimeMillis();
        String uwServiceEndPoint = "/transaction";
        String bureauCallUrl = String.format("%s%s", appConfig.getUnderwritingTransactionApiPrefix(), uwServiceEndPoint);
        if (syntheticTransactionStateManager.isSyntheticTransaction()) {
            result = this.syntheticTransactionService.getPayLoad(syntheticTransactionStateManager.getTransactionId(),
                    "BUREAU_PULL", CreditBureauReportResponse.class);
            result.setCustomerId(softPullParams.getUserAccountId());
            result.setLoanAccountId(softPullParams.getUserLoanAccountId());
        } else {
            ResponseEntity<CreditBureauReportResponse> responseEntity = restTemplate.exchange(bureauCallUrl, HttpMethod.POST, entity, CreditBureauReportResponse.class);
            result = responseEntity.getBody();
        }
        long endTime = System.currentTimeMillis();
        long timeDifference = endTime - startTime;
        logger.info("Time taken for API call BUREAU_PULL/getBureauReport call 1: " + timeDifference);
    } catch (HttpClientErrorException exception) {
        logger.error("HttpClientErrorException occurred while calling BUREAU_PULL API, response string: " + exception.getResponseBodyAsString());
        throw exception;
    } catch (HttpStatusCodeException exception) {
        logger.error("HttpStatusCodeException occurred while calling BUREAU_PULL API, response string: " + exception.getResponseBodyAsString());
        throw exception;
    } catch (Exception ex) {
        logger.error("Error occurred in getBureauReport. Detail error:", ex);
        throw ex;
    }
    return result;
}

The code snippet above is quite elaborate, but we don't need to get into the details of that. What we need to focus on is the code snippet below:

if (syntheticTransactionStateManager.isSyntheticTransaction()) {
	result = this.syntheticTransactionService.getPayLoad(syntheticTransactionStateManager.getTransactionId(),
			"BUREAU_PULL", CreditBureauReportResponse.class);
	result.setCustomerId(softPullParams.getUserAccountId());
	result.setLoanAccountId(softPullParams.getUserLoanAccountId());
} else {
	ResponseEntity<CreditBureauReportResponse> responseEntity = restTemplate.exchange(bureauCallUrl, HttpMethod.POST, entity, CreditBureauReportResponse.class);
	result = responseEntity.getBody();
}

It checks for the synthetic transaction with the SyntheticTransactionStateManager. If true, then instead of going to a third party, it calls the internal service SyntheticTransactionService to get the Synthetic Bureau report data.

Synthetic Data Service

Synthetic data service SyntheticTransactionServiceImpl is a general utility service whose responsibility is to pull the synthetic data from the data store, parse it, and convert it to the object type that is been passed as part of the parameter. Below is the implementation for the service:

@Service
@Qualifier("syntheticTransactionServiceImpl")
public class SyntheticTransactionServiceImpl implements SyntheticTransactionService {

    private final Logger logger = LoggerFactory.getLogger(this.getClass());

    @Autowired
    SyntheticTransactionRepository syntheticTransactionRepository;

    @Override
    public <T> T getPayLoad(String transactionUuid, String extPartnerServiceType, Class<T> responseType) {
        T payload = null;
        try {
            SyntheticTransactionPayload syntheticTransactionPayload = this.syntheticTransactionRepository.getSyntheticTransactionPayload(transactionUuid, extPartnerServiceType);
            if (syntheticTransactionPayload != null && syntheticTransactionPayload.getPayload() != null){
                ObjectMapper objectMapper = new ObjectMapper()
                        .configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);
                payload = objectMapper.readValue(syntheticTransactionPayload.getPayload(), responseType);
            }
        }
        catch (Exception ex){
            logger.error("An error occurred while getting the synthetic transaction payload, detail error:", ex);
        }
        return payload;
    }

    @Override
    public boolean validateTransactionId(String transactionId) {
        boolean result = false;
        try{
            if (transactionId != null && !transactionId.isEmpty()) {
                if (UUID.fromString(transactionId).toString().equalsIgnoreCase(transactionId)) {
                    //Removed other validation checks, this could be financial application specific check.
                }
            }
        }
        catch (Exception ex){
            logger.error("SyntheticTransactionServiceImpl::validateTransactionId - An error occurred while validating the synthetic transaction id, detail error:", ex);
        }
        return result;
    }

With the generic method getPayLoad(), we provide a high degree of reusability, capable of returning various types of synthetic responses. This reduces the need for multiple, specific mock services for different external interactions.

For storing the different payloads for different types of external third-party services, we use a generic table structure as below:

CREATE TABLE synthetic_transaction (
  id int NOT NULL AUTO_INCREMENT,
  transaction_uuid varchar(36)
  ext_partner_service varchar(30)
  payload mediumtext
  create_date datetime DEFAULT CURRENT_TIMESTAMP,
  PRIMARY KEY (id)
);

• ext_partner_service: This is an external service identifier for which we pull the payload from the table. In this above example for bureau report, it would be BUREAU_PULL. 

Conclusion

In our exploration of synthetic transactions within fintech applications, we've highlighted their role in enhancing the reliability, and integrity of fintech solutions. By leveraging synthetic transactions, we simulate realistic user interactions while circumventing the risks tied to handling real personally identifiable information (PII). This approach enables our developers and QA teams to rigorously test new functionalities and updates in a secure, controlled environment. 

Moreover, our strategy in integrating synthetic transactions through mechanisms such as HTTP interceptors and state managers showcases a versatile approach applicable across a wide array of applications. This method not only simplifies the incorporation of synthetic transactions but also significantly boosts reusability, alleviating the need to devise unique workflows for each third-party service interaction.

This approach significantly enhances the reliability and security of financial application solutions, ensuring that new features can be deployed with confidence.