There are different ways to "monetize" a project. But they have one thing in common ― how money is transferred from the user's wallet to the organization's account. Today we will talk about how payment acceptance is organized in Badoo and what can be found on the payment gateway market. We warn you right away that you won't find any specific figures on the company's turnover in this article, but everything else will be equally interesting.
For us, billing is everything related to receiving money from users: price configuration, payment acceptance page, direct payment acceptance and processing, provision of paid services, various promotions, and, of course, monitoring all of the above.
Initially, as in all startups, we did not have paid services. The first serious steps towards monetization began back in 2008, despite the fact that the site was officially launched in 2006. France was chosen for the experiments, and payment was accepted only via SMS. Payment acceptance itself was organized on files. Each request was written to a separate file, which was then transferred by bash scripts from one folder to another, which meant changing the processing statuses. The database was used only for recording successfully processed transactions. This scheme worked successfully for a little over a year, after which it became difficult to maintain, and we decided to abandon the files and rewrite everything using the database.
The development of the new version was quite fast, since there were not many countries where paid services were available. But it was designed only for accepting payments via SMS. Because of this, we even still have some funny artifacts, for example, the MSISDN (phone number) and short code (short number to which a paid SMS is sent) fields in the processed payments table.
Now we accept payments almost all over the world. Every second, users try to pay for something on the site or in apps for all popular mobile platforms. And if you put it on a map, you get the picture "View of the Earth from space at night":
We have about 50 payment methods available, provided by different partners. The most popular ones are bank cards, SMS & Direct billing, and mobile app purchases.
Some of them are exotic, such as direct debiting from an Internet service provider's account or paying via a landline phone. And once we received a payment via regular mail!
All payment systems allow you to accept payments from their users. Such direct integrations are convenient to do as long as there are not very many of them and you connect well-known systems with well-established processes. But when you need to enter local markets, problems start to appear. It is becoming increasingly difficult to maintain the "zoo" of different APIs, regulatory requirements differ, a popular local payment system may refuse to work with foreign clients at low turnover, or signing a contract and settling legal problems may take a long time. Despite such difficulties, local payment systems can pleasantly surprise you with their conversion rate. For example, Holland, which we considered not very promising, after connecting the iDeal payment method, which is popular in this country, began to bring in 30-40% more money.
If there is demand, there will be supply. There are many aggregator companies on the market, or, in other words, payment gateways, whose goal is to combine all popular payment systems, including local ones, under a single API. By making one such integration, we can accept payments through dozens of payment systems around the world. You don't even need to create a payment page on your site, but use the ready-made one provided by the aggregator and customize only the design. Especially advanced companies allow you to upload your own CSS and JS files, change images, translate texts, and even register the resulting page on your subdomain, for example, payments.example.com. This gives you very rich opportunities for "customization", and the user gets the feeling that he does not leave your site. We don't use this feature at home, as we work with several aggregators at the same time, but for some it can be a very convenient solution.
Whether to use direct integration or via an aggregator depends primarily on the commission amount. The more your customers use the payment system, the more profitable it may be to save on commission and connect to it directly. The second important factor is API quality, usability, and stability. Here, aggregators allow you to smooth out rough edges, and sometimes provide a more stable service than a direct connection.
Separate are payments via SMS and direct debits from the mobile phone balance. They are very tightly controlled in many countries, especially in Europe. Local regulators or the state itself may impose special requirements on how the payment page should look or what the text of SMS messages sent should be. You need to keep track of changes to such requirements and make changes on your site in a timely manner. For example, in Belgium, there is a rule that a short number should be written in white font on a black background, and its price should be indicated next to it.
The type of SMS billing is also different-MO (Mobile Originated) or MT (Mobile Terminated). With MO-billing, everything is quite simple: the user sent an SMS to a short number, and we received the money. But for MT, there are several options. Payment is made not when the user sends an SMS message, but after they receive a special paid SMS from us. In other words, the fact of payment is considered to be a notification received from the aggregator that the paid SMS was successfully delivered. The main goal of this approach is to add an additional check before sending a paid SMS message to the user and prevent errors related to incorrect text in the SMS. To do this, the payment process takes place in two phases. First ― the user expresses a desire to pay for something, and the second is a confirmation. For example, the payment process might look like this:
Fortunately, there are also aggregator companies in the SMS payment market whose services are worth using. They deprive you of the "pleasure" of dealing with such details for a modest, and sometimes not so much, fee. Another nice bonus is that they often take on some of the end-user support responsibilities. Users start reporting their problems directly to the short number owner, i.e. the aggregator.
Badoo runs on a PHP + MySQL bundle, so we use the same technologies to process payments. The code is executed on a separate server group allocated from the shared pool. Internally, we have divided it into several more logical subgroups: servers for processing incoming requests, servers for background operations and collecting statistics, database servers, and servers for processing bank card payments. The latter are separated into a separate group, because they must comply with the PCI DSS security standard, developed with the participation of Visa, MasterCard, American Express, JCB and Discover for organizations that work or store bank cardholder data.
To process payments, we use two Percona MySQL database servers running in master-to-master replication. The main load goes only to one of them. The second one is used for "hot" replacement in case of an accident or for replacing the main one (for the duration of its maintenance, for requests from the monitoring system or for collecting statistics).
The entire billing system can be divided into several large parts:
Connecting a new payment method to the system consists in creating its plugin. It is responsible for all interactions between us and the payment gateway, which are of two types: when we are the initiator (pull request) and when the aggregator is the initiator (push request). For pull requests, HTTP is usually used as the protocol, pure or as a transport for JSON / XML, or SOAP. The recently popular concept of the REST API was not often encountered by us, only by new companies on the market, for example, the British GoCardless, or by old ones that decided to redesign their API, for example, PayPal. For push requests, SOAP is practically not used (of those who offer such a format for push notifications, only Qiwi is easily remembered). Basically, pure HTTP is used, or it is also used as a transport for all the same JSON and XML.
After implementing the API, the testing stage begins. There have already been articles on Habré about what our development and automation process looks like. But there are some special features for billing, mainly due to the fact that we have to test not just our code, but also interaction with aggregators. It is very convenient if they have a test environment that fully emulates real payment acceptance. If it is not present, we make "stubs" that emulate the behavior of the aggregator. This simplifies our manual testing and allows us to write autotests that check the entire payment process. Here is an example of what one of the stubs looks like.
After the test environment, you need to check how everything will work in real life, and make a real payment. However, SMS payments often require approval from regulators or operators, and this can take several months. In order not to upload half-finished code to production servers, we came up with such a thing as External Shot. This is our regular Shot, which is a directory with a task branch and is intended for testing it on production servers, but in addition to the local domain, it has an additional external address where anyone can go and see the changes made. For security reasons, such "shots" are not created for every task, but only in cases where it is really necessary. We give links to them to our partners, and they can check the changes made at any time of the day or night. This is especially true for countries located in the other hemisphere, where the time difference can be up to 12 hours.
After the new integration is uploaded to production servers, the stage of its operation and support begins. Technical support takes up approximately 60-70% of our time.
This includes, first, analyzing complaints from users. All simple situations are handled by the first-line support team, which also translates complaints from different languages to English for us. Therefore, only the most complex cases that really require developers ' attention come to us.
The second component of technical support is fixing bugs or making changes to existing integrations. Errors occur for various reasons. For example, due to inattentive reading of the documentation or gaps in it. Once, we even had to use chat logs with the aggregator developer instead, because the documentation for their new system was not yet ready. There were cases when the aggregator changed the interaction protocol or its parameters without notification. Another time, the acquiring bank turned off our gateway, and we had to urgently redirect traffic to another location. As it turned out later, it was an ancient server from the 80s, which, according to the bank, was not supposed to process anything at all. In general, you don't have to be bored, especially if you take into account that every minute of downtime is a lost profit.
To solve such problems, we write detailed logs of the app's operation. It includes not only errors, but also all interaction with aggregator systems or just important events that occur during query execution. Each request has its own unique identifier, which can be used to find all the records associated with it and restore the progress of its processing. This is especially useful when you have to deal with errors that have already occurred several weeks or months ago.
This is how billing is organized in Badoo. Of course, there are still many interesting topics that we plan to talk about in the future, such as monitoring, PCI DSS certification, and bank card payment processing. If you have any questions or suggestions on the topic of future articles, welcome to comment.
(c) Anatoly Panov
Lead Developer
What is "billing"?
For us, billing is everything related to receiving money from users: price configuration, payment acceptance page, direct payment acceptance and processing, provision of paid services, various promotions, and, of course, monitoring all of the above.
Initially, as in all startups, we did not have paid services. The first serious steps towards monetization began back in 2008, despite the fact that the site was officially launched in 2006. France was chosen for the experiments, and payment was accepted only via SMS. Payment acceptance itself was organized on files. Each request was written to a separate file, which was then transferred by bash scripts from one folder to another, which meant changing the processing statuses. The database was used only for recording successfully processed transactions. This scheme worked successfully for a little over a year, after which it became difficult to maintain, and we decided to abandon the files and rewrite everything using the database.
The development of the new version was quite fast, since there were not many countries where paid services were available. But it was designed only for accepting payments via SMS. Because of this, we even still have some funny artifacts, for example, the MSISDN (phone number) and short code (short number to which a paid SMS is sent) fields in the processed payments table.
Now we accept payments almost all over the world. Every second, users try to pay for something on the site or in apps for all popular mobile platforms. And if you put it on a map, you get the picture "View of the Earth from space at night":
We have about 50 payment methods available, provided by different partners. The most popular ones are bank cards, SMS & Direct billing, and mobile app purchases.
Some of them are exotic, such as direct debiting from an Internet service provider's account or paying via a landline phone. And once we received a payment via regular mail!
Bank payments
All payment systems allow you to accept payments from their users. Such direct integrations are convenient to do as long as there are not very many of them and you connect well-known systems with well-established processes. But when you need to enter local markets, problems start to appear. It is becoming increasingly difficult to maintain the "zoo" of different APIs, regulatory requirements differ, a popular local payment system may refuse to work with foreign clients at low turnover, or signing a contract and settling legal problems may take a long time. Despite such difficulties, local payment systems can pleasantly surprise you with their conversion rate. For example, Holland, which we considered not very promising, after connecting the iDeal payment method, which is popular in this country, began to bring in 30-40% more money.
If there is demand, there will be supply. There are many aggregator companies on the market, or, in other words, payment gateways, whose goal is to combine all popular payment systems, including local ones, under a single API. By making one such integration, we can accept payments through dozens of payment systems around the world. You don't even need to create a payment page on your site, but use the ready-made one provided by the aggregator and customize only the design. Especially advanced companies allow you to upload your own CSS and JS files, change images, translate texts, and even register the resulting page on your subdomain, for example, payments.example.com. This gives you very rich opportunities for "customization", and the user gets the feeling that he does not leave your site. We don't use this feature at home, as we work with several aggregators at the same time, but for some it can be a very convenient solution.
Whether to use direct integration or via an aggregator depends primarily on the commission amount. The more your customers use the payment system, the more profitable it may be to save on commission and connect to it directly. The second important factor is API quality, usability, and stability. Here, aggregators allow you to smooth out rough edges, and sometimes provide a more stable service than a direct connection.
SMS payments
Separate are payments via SMS and direct debits from the mobile phone balance. They are very tightly controlled in many countries, especially in Europe. Local regulators or the state itself may impose special requirements on how the payment page should look or what the text of SMS messages sent should be. You need to keep track of changes to such requirements and make changes on your site in a timely manner. For example, in Belgium, there is a rule that a short number should be written in white font on a black background, and its price should be indicated next to it.
The type of SMS billing is also different-MO (Mobile Originated) or MT (Mobile Terminated). With MO-billing, everything is quite simple: the user sent an SMS to a short number, and we received the money. But for MT, there are several options. Payment is made not when the user sends an SMS message, but after they receive a special paid SMS from us. In other words, the fact of payment is considered to be a notification received from the aggregator that the paid SMS was successfully delivered. The main goal of this approach is to add an additional check before sending a paid SMS message to the user and prevent errors related to incorrect text in the SMS. To do this, the payment process takes place in two phases. First ― the user expresses a desire to pay for something, and the second is a confirmation. For example, the payment process might look like this:
- we send an SMS to a short number, respond to an incoming SMS with a specific text or without it;
- we send an SMS to a short number and enter the received PIN code on the site;
- we enter the phone number on the site, get the PIN code, and enter it on the site.
Fortunately, there are also aggregator companies in the SMS payment market whose services are worth using. They deprive you of the "pleasure" of dealing with such details for a modest, and sometimes not so much, fee. Another nice bonus is that they often take on some of the end-user support responsibilities. Users start reporting their problems directly to the short number owner, i.e. the aggregator.
Technical Details
Badoo runs on a PHP + MySQL bundle, so we use the same technologies to process payments. The code is executed on a separate server group allocated from the shared pool. Internally, we have divided it into several more logical subgroups: servers for processing incoming requests, servers for background operations and collecting statistics, database servers, and servers for processing bank card payments. The latter are separated into a separate group, because they must comply with the PCI DSS security standard, developed with the participation of Visa, MasterCard, American Express, JCB and Discover for organizations that work or store bank cardholder data.
To process payments, we use two Percona MySQL database servers running in master-to-master replication. The main load goes only to one of them. The second one is used for "hot" replacement in case of an accident or for replacing the main one (for the duration of its maintenance, for requests from the monitoring system or for collecting statistics).
The entire billing system can be divided into several large parts:
- The core. This includes basic entities such as an Order, Payment, Services and their accounting and rendering rules, and various infrastructure items.
- Aggregator plugins. Everything that is responsible for communication between us and the payment system.
- Service selection and payment page.
Connecting a new payment method to the system consists in creating its plugin. It is responsible for all interactions between us and the payment gateway, which are of two types: when we are the initiator (pull request) and when the aggregator is the initiator (push request). For pull requests, HTTP is usually used as the protocol, pure or as a transport for JSON / XML, or SOAP. The recently popular concept of the REST API was not often encountered by us, only by new companies on the market, for example, the British GoCardless, or by old ones that decided to redesign their API, for example, PayPal. For push requests, SOAP is practically not used (of those who offer such a format for push notifications, only Qiwi is easily remembered). Basically, pure HTTP is used, or it is also used as a transport for all the same JSON and XML.
After implementing the API, the testing stage begins. There have already been articles on Habré about what our development and automation process looks like. But there are some special features for billing, mainly due to the fact that we have to test not just our code, but also interaction with aggregators. It is very convenient if they have a test environment that fully emulates real payment acceptance. If it is not present, we make "stubs" that emulate the behavior of the aggregator. This simplifies our manual testing and allows us to write autotests that check the entire payment process. Here is an example of what one of the stubs looks like.
After the test environment, you need to check how everything will work in real life, and make a real payment. However, SMS payments often require approval from regulators or operators, and this can take several months. In order not to upload half-finished code to production servers, we came up with such a thing as External Shot. This is our regular Shot, which is a directory with a task branch and is intended for testing it on production servers, but in addition to the local domain, it has an additional external address where anyone can go and see the changes made. For security reasons, such "shots" are not created for every task, but only in cases where it is really necessary. We give links to them to our partners, and they can check the changes made at any time of the day or night. This is especially true for countries located in the other hemisphere, where the time difference can be up to 12 hours.
Support and operation
After the new integration is uploaded to production servers, the stage of its operation and support begins. Technical support takes up approximately 60-70% of our time.
This includes, first, analyzing complaints from users. All simple situations are handled by the first-line support team, which also translates complaints from different languages to English for us. Therefore, only the most complex cases that really require developers ' attention come to us.
The second component of technical support is fixing bugs or making changes to existing integrations. Errors occur for various reasons. For example, due to inattentive reading of the documentation or gaps in it. Once, we even had to use chat logs with the aggregator developer instead, because the documentation for their new system was not yet ready. There were cases when the aggregator changed the interaction protocol or its parameters without notification. Another time, the acquiring bank turned off our gateway, and we had to urgently redirect traffic to another location. As it turned out later, it was an ancient server from the 80s, which, according to the bank, was not supposed to process anything at all. In general, you don't have to be bored, especially if you take into account that every minute of downtime is a lost profit.
To solve such problems, we write detailed logs of the app's operation. It includes not only errors, but also all interaction with aggregator systems or just important events that occur during query execution. Each request has its own unique identifier, which can be used to find all the records associated with it and restore the progress of its processing. This is especially useful when you have to deal with errors that have already occurred several weeks or months ago.
This is how billing is organized in Badoo. Of course, there are still many interesting topics that we plan to talk about in the future, such as monitoring, PCI DSS certification, and bank card payment processing. If you have any questions or suggestions on the topic of future articles, welcome to comment.
(c) Anatoly Panov
Lead Developer
