by Andrew Beveridge, CEO eCOGRA
As the online gaming industry continues to mature and address the player’s need for fair gaming through independent testing and certification, the debate continues regarding source code and Total Gaming Transaction Review (TGTR) testing, an advanced form of output based testing.
We can all agree that testing and third party verification is a healthy debate for the industry to explore, and there are definitely strong advocates for each method.
Certainly, the primary objective for both testing methods is to ensure a fair gaming environment for online players. Beyond that, operators must verify that governments are receiving the right allocation of taxes, and the games are performing according to applicable rules.
This article will explore TGTR testing in the online gaming environment. First let’s take a look at the differences between the traditional, land-based gaming environment and online gaming.
In the traditional land-based casino environment, a single game is typically installed and hard coded onto a fixed device. In this case, the random number generator (RNG) – the key to computer-based casino games, whether real-world slot machines or online slots and table games – and the machine form a single, integrated unit of hardware that is sealed from the outside world.
The RNG is a program within the host computer’s microprocessor that determines the randomness of a game whether it’s the spin of a slot machine or the cards dealt or drawn in a hand of video poker, among other games.
In order to assess the fairness of a game, regulators or the independent testing agencies that they hire, review the game’s source code to ensure it operates as specified. The source code is the arrangement of instructions written by computer programmers, which is converted from a human-readable form into some kind of computer-executable form, and ultimately results in the executable programs that make up the gaming software. Upon approval, the source code and console are locked and certification is awarded. Once the software is reviewed and burned into the EPROM chip of the gaming unit, it cannot be physically changed. EPROM is a special type of memory that retains its contents until exposed to ultraviolet light.
After the game is certified, land-based game developers and software providers do not have access to the code unless an adjustment is required. This could include installing new graphics, or making changes in rules.
Once the gaming machine is operational, changes to the games do not happen very often. In the event an adjustment is required, the source code testing process would be performed again.
To minimize the chance that an operator could switch one EPROM chip for another, regulators have access to on-site surveillance cameras and officials from the Gaming Board and the testing agency should be present any time the hardware is unsealed. In addition, regulators usually require casinos to keep detailed records for any time the gaming machines are opened.
By analyzing the source code of the game in the land-based environment, regulators or the testing companies that they hire, can be confident that the game performs as specified. Source code testing has worked well in traditional gaming.
Online Gaming Environment
Online gaming is a much more complex and dynamic environment. In this industry, the player is the only one with control over the hardware on which the games are being played, typically the player’s personal computer.
The gaming server, the computer that runs the game, is the most critical component in online gaming.
In addition to housing the RNG, the gaming server links into a highly sophisticated transaction processor and controller that routes the millions of messages coming into the system each minute. The server provides each message with a response, creates a complete record of all messages (in and out), and generates summary information – all in real time.
As with companies such as Microsoft, Oracle and Sun, gaming software providers invest large sums to develop and refine their proprietary server products – their core competitive advantage in the online gaming industry. In fact, the gaming server is the culmination of the intellectual property that is their primary asset. It’s no secret that this information is closely guarded.
In the online gaming environment, software developers work constantly to improve the end user’s experience through speed and efficiencies, increase security and reduce maintenance costs by refining the servers.
Unlike a traditional gaming machine, a network system has many more points of ongoing failure, which require constant monitoring and adjustments through regular system maintenance. This could include making upgrades to the system or games, installing a new security patch from an operating system vendor such as Microsoft or Sun, fixing a bug or restoring a network fault.
In the online gaming environment, technicians require access to the gaming server at all times. Because of the frequency of changes to the software in a live operating environment, source code testing, which verifies the software’s performance only at a single point in time, is simply not suited to online gaming.
Total Gaming Transaction Review (TGTR)
TGTR is an advanced form of output-based testing that has been developed by independent auditors and the industry’s leading software providers over the past eight years specifically for the online gaming industry, and to determine the randomness of the RNG and verify payout percentages (the percentage of winnings to wagered amounts).
Recognizing the need to shift from traditional gaming testing in the more complex online gaming environment, TGTR was developed as a controls-based approach that focuses on data input and data output. TGTR is a continuous review that incorporates testing of the entire system rather than just a segment of the source code.
The process starts with a thorough on-site inspection by independent auditors to ensure that software is being developed, implemented and maintained in a manner representative of best practice standards. The historical performance of the software will also be taken into account. The data analysis process does not begin until it is established by the auditors that the developer is consistently adhering to the appropriate standards.
TGTR utilizes the fact that each casino can produce a record of every transaction over a period of time. This information permits a thorough analysis of every historical transaction for every game by the independent auditors, resulting in publicly available monthly payout percentage reports and bi-annual RNG reports.
A continuous process, TGTR verification is based on actual transactions and actual game play irrespective of volumes, and can be applied during ongoing system changes and development.
To verify the integrity of raw data and ensure that it has not been manipulated or corrupted in any way, the TGTR percentage payout review process requires the following: recording of every single bet and payout transaction from each casino throughout the period of review directly onto dedicated servers (independent of the casino’s gaming servers); rigorous validation and balance checks of these transactions against the data on the casino’s live gaming servers; final reconciliation of bet values and payouts against actual total players’ account closing balances on the live gaming servers; and trend analysis regarding transaction volumes and player behavior to provide assurance that the information appears reasonable and consistent.
By continually checking both the input and output of the system, it is feasible to test that the system is meeting regulatory requirements while providing operators and software developers with complete operational flexibility. It is also possible to verify that all transactions have been accurately recorded by checking each transaction for its completeness, along with spot testing of transactions.
In addition, the auditors ensure that wagering activity, including wins, is distributed among an acceptable population of players.
Through further analysis of the summary data and subjecting the vast amounts of data to rigorous generally accepted statistical testing methods, one can further confirm the integrity of the RNG and of the system as a whole.
eCOGRA views TGTR verification as a solution for the online gaming industry that goes far beyond trying to simply match the security of a real-world slot machine. We recognize that it’s the fairness of the final outcome that is most important to the players, and the regulators who protect them.
TGTR is enacted in the eCOGRA review and certification process once there is adequate assurance that full source code testing, implementation, version and change control are in place with the necessary security controls at the software provider, and all have been operating properly for a reasonable period of time.
There are those who may argue that TGTR will only highlight a problem in the software after it has already occurred in a live environment. For this reason, eCOGRA requires a thorough ongoing review of the operating environment at the software provider and operator levels, and the controls and entire processes around the implementation and testing of changes to the software. This review ensures that the possibility of anything occurring which will adversely affect the output of the system is within acceptable limits.
An additional advantage that TGTR holds over source code testing is the appropriate assurance of fairness in slots games. While source code testing is able to confirm that an RNG is performing correctly, it is not able to confirm what the payout percentage of a particular game should be. A slots game with a payout percentage of 50 percent that passes the RNG test will not be considered fair by the players.
Overall, TGTR is a solution that allows for rapid advances in technology, reduces development and regulatory costs, provides for easier dispute mediation and is easily implemented using existing methods from the broader commercial world.
Recognizing the benefits of TGTR, the Isle of Man, Gibraltar, Malta, Kahnawake and the Philippines have agreed that operators using software which is already subject to the eCOGRA’s generally accepted practices (eGAP), testing principles and methodologies, will not be required to submit to the review and monitoring processes that the regulators employ through additional third parties.