GLI explains the compliance process for a gaming device submitted to one of its laboratoriesTuesday, 26 January 2010
Submissions made to GLI for compliance testing of Class III gaming products— the slot machines with random number generators on board typically found in most land-based casinos—come to each of GLI's labs, while the main testing lab for such devices is the company's global headquarters in New Jersey.
"Ninety percent of our submissions arrive into NJ, but we do accept submissions in each of our offices directly, so we have a small team here in Las Vegas to handle this," says Ian Hughes.
"With today's technology, a lot of those submissions are made electronically, but we still get some machines delivered here in Las Vegas. Our administrative staff will log them into a database, obtaining electronic signatures for the equipment. That goes into a bar folder on the database. Engineers will then pick that up. We have a job scheduling system to make sure nothing gets lost and to make sure we keep track of our turnaround times and notifications out to the manufacturer.
"We also have several portals that the regulators and manufacturers use to keep track of that whole process. It's a bit like DHL tracking, where you send a package and it's in transit in, say, Los Angeles and somebody has signed for it. We know exactly where the equipment is, and its progress at every stage."
"We believe we are the only gaming lab in the world that offers interoperability testing and virtual interoperability testing on a global scope like this," adds Christie Eickelman.
"We have it in our South Africa office, in our Europe office, and we have it in Las Vegas. We have more than 500 employees worldwide now that can handle this in our 13 offices. That gives us a large competitive advantage."
This remote testing and remote database interrogation may seem somewhat at odds with the idea of a gaming laboratory as a place where technicians in white coats rummage inside the belly of a slot machine. Ian Hughes points out that remote testing is complementary to, rather than a substitute for, tests on the physical machine itself.
"The point is that when a submission takes place, the machine itself is always physically in our custody somewhere at one of our 13 facilities around the world," he states.
"So if some issue arises during interoperability testing or diagnostic analysis via GLI Link, we, as a laboratory, still have the ability physically to examine the machine."
"We're very transparent in our approach," adds Ms Eickelman.
"Whatever GLI office a supplier visits, they will see the same processes and approach. We don't just open offices, we open labs. We become fully committed to the region that we're working in. It's really easy for a compliance testing company to say, 'Oh we have an office in such-and-such a place' when it could just be an executive suite with one person working part time and the mail goes there. The question then is, 'Where's the testing done?'"
When GLI physically receives a machine in Las Vegas, it goes into one of the site's engineering 'pods'. These are booths to one side of the main lab floor where teams of engineers can go through a list of checks.
"Worldwide, we have more than 500 employees, just over 70 of whom are here in Las Vegas," explains Ian Hughes.
"We keep our teams in small groups to ensure they are well managed and that nothing gets lost in the flow. We have a technical manager that oversees about 20 engineers. And then within that group, there are another two senior engineers, and ten engineers under each senior engineer— good small team groups.
"We would love to have one of these systems in every one of our labs around the world, but it's just too costly. So we have designed an internal solution to that, and that's GLI Link.
"Thanks to advances in technology by the system manufacturer, we're now able to convert what is normally a local area network environment into a wide area network environment. We do this via our proprietary setup, GLI Link.
"Basically, we're routing in each of these systems into a panel here. Then we're making it available via other panels. There's one in every one of our offices. It's in our Nevada office, and its' the same set up in New Jersey, Macau, Colorado, Netherlands and in Australia, etc.
"During your visit today, we have machines that are all connected to our system based in Golden, Colorado.
"It's just a normal PC—no server, no application, nothing running. It's hooked into GLI Link, as too are these machines. Now because it's G2S, there's no interface card. There's a card reader, but there is no slot machine interface board. The Ethernet comes directly into the machine, and the machine itself is the interface.
"The server's in Colorado, so an engineer in Las Vegas has a view of the terminal in Colorado. If we open the door of a slot machine in Las Vegas hooked up to the system, that event is sent down GLI Link to Colorado and our engineer can see it in real time.
"The engineer can also take game content from the server in Colorado, push it onto a cabinet in Las Vegas, and it can be up and running. So it's basically wide area network gaming via GLI.
"If we did an event [such as opening a cabinet] and it didn't come through on the system, how would we know the problem is with the game rather than with GLI Link? We need to know it's not our problem before we start going off to the manufacturer and saying, 'Hey, that event didn't come through'. So there are a lot of diagnostics that our IT dept have brought into GLI Link to do message tracing and all those sorts of things, so that we can go back to the manufacturer and say: 'We're absolutely sure that the problem lies with such and such'."
Despite the logistical benefits of testing via GLI Link, there are some jurisdictions that require testing to be conducted in a specified location of the regulator's choosing—namely [the US state of] Missouri and Singapore. In those cases, where a machine to be tested is physically delivered to the Las Vegas lab, it is assigned to a 'pod'—a closed off area to the side of the 80,000 square ft. laboratory floor. At the pod, a small team of engineers examines the machine.
Ian Hughes explains: "We want to do everything we can to make the regulator feel the most comfortable with the testing we are doing for their individual jurisdiction."
Inside the pod
"In each of the 'pods,' we have eight to ten machines," says Mr Hughes.
"The pods are pretty much identical. We can configure any pod pretty quickly by unplugging the interface from the machine, and then via a system of overhead routing across our lab floor, connect it back to one of the systems. We have the management systems of all the major slot manufacturers on site.
"Typically, the engineers will have their gaming machine or target platform in their cubicle, along with all the development tools they need to interrogate and test the game," reveals Mr Hughes.
"They will run through the prescribed test scripts, which are all ISO1725 [International Organization for Standardization] accredited, and it's all done online.
"The device submission is entered in our database, and our QA team in New Jersey, along with any one of our offices and the supplier, can see it. We also do concurrent testing, so if, for example, a manufacturer has complicated communications in their equipment, mathematics and pay table, we can actually split those functions off to three different groups, and those engineers can be working on the same submission package at the same time. This has allowed us to significantly reduce the time the product is in GLI."
"Our main laboratory floor has two purposes. We do device testing here for some of the larger terminals, including multiplayers and tables that can't physically fit in the cubicle," explains Mr Hughes.
"You will see a lot of the larger multi terminal machines stationed around here. The engineers are actually performing device testing. They will come out of their work spaces onto the machine."
Some of the work GLI does in this testing phase is 'nuts and bolts', i.e., checking whether a machine pays out correctly. Other parts of the work relate to new challenges and opportunities that have been created by digital technology.
"Regulators and operators want to make sure that machine pays out correctly to the player. That's always been the case," states Ian Hughes.
"I, as a player, want to know that if I put ten dollars in I have a fair chance of winning that advertised prize that's on the pay glass.
Interoperability—a hot topic
"As well as that aspect of regulatory compliance, interoperability is becoming more and more important to the industry. It's the area of most risk for the operators, because as you've got more and more player accounts on membership cards, or held via ticket in ticket out systems, any failure in communication caused by interoperability issues can have a direct impact on an operator's bottom line. It can also have regulatory implications in regard to the operator's duty of fairness to the player," adds Mr Hughes.
"With feature-rich immersive technology, increasingly on a single cabinet, games can be played within the basic game. Not only that, but with player loyalty card technology, it's now possible for game content and preferences to be ported from one machine to another. That means there's increased connectivity between the machine and the back end server. That makes interoperability testing a really key part of compliance testing. There are two ways that we test that: the first is protocol testing," continues Mr Hughes.
Game to System
"That's about making sure it complies to the published communication protocol. The G2S [Game to System] standard [developed by the Gaming Standards Association] is approaching 2,000 pages of information, so we have to go through that and make sure the device complies to each of those standards. We are the only independent test laboratory certified by GSA to test and certify G2S protocols."
"GLI is the only independent test lab certified by GSA to test to G2S standards, in the world. It's a very tough process to go through. We had to prove that our engineers really knew what they were talking about with G2S technology and that they're involved in every facet of its design. We are very happy to have that skill set available, and it's increasingly relevant. G2S is what's going into CityCenter [MGM MIRAGE's new Las Vegas property]," adds Mr Hughes.
Interoperability—the testing process
To test for interoperability on a piece of equipment, GLI must plug the submitted device into a test board linking it with the machines hardware, software and systems of all the major gaming manufacturers.
"We have the typical SAS [Slot Accounting System] environment on the lab floor that you would see operating in any casino in North America. We have on site over a dozen online systems used by the suppliers," explains Ian Hughes.
"We take an interface board, which normally fits inside a slot machine, and we hook it up to the machine to be tested. We don't actually put it physically inside the body of the cabinet, in order to make the testing process easier.
"For example, if we have a Bally machine on a Bally system, we will make sure that all the other suppliers' systems can 'talk' to that Bally machine and system. It's then a question of going through the list of suppliers. We plug in an IGT interface board to the Bally machine, and then a Konami board, and so on.
"So we're leaving the machine that's being tested exactly where it is, and the system where it is, and just putting it on a new board. It keeps everything nice and secure and means we don't need to move stuff physically around. We do that interoperability testing on every single system, and sometimes on two or three different versions of a system."
'Casino floor' stress testing
GLI doesn't rely only on theoretical testing in the booth. After the interoperability phase, the machine is taken out on the main lab floor where it's put through its paces as if it were on a live casino floor.
"After we've done the simulation testing over in a booth with the device engineers, now we've brought it onto the main lab area we refer to as the 'casino floor', to create a real life environment, and make sure everything's working," explains Ian Hughes.
"On our lab floor, we can also test player tracking systems, accounting etc. We can test server based gaming applications here as well, including production server based gaming where the game is actually being played on the server rather than on the game cabinet—i.e., 'thin client' true server based gaming. We have approved such a system for use at MGM [GRAND Macau] in Macau.
"As I'm speaking to you now on the lab floor, one of our engineers is testing an IGT Advantage® system," says GLI's Ian Hughes.
"He's looking at the machine data feed, reconciling what he's done on the machine, making sure it's all communicated on the system, making sure there's no loss of data. We'll go through a checklist including some cashless transactions, electronic funds transfer, advance funds transfer, promo credits, downloadable credits, and noncashable credits. We will go through all the test groups, and all the test groups are ISO certified independently internally. We have several different supervisory groups for this work in house—a compliance group and a QA group to make sure that our test engineers are doing exactly what they're supposed to be doing in the order in which they're supposed to be doing it."
Every GLI test site, including Las Vegas, is physically secure from outside interference or sabotage, stresses the company.
"Internal doors inside the facility are electronically controlled and require numbered passwords," states Ian Hughes.
"Engineers need an individual password in order to work on a particular device. Even the skylights to the building have bars on them. It's a highly secure environment. It gives the regulators and the suppliers a high degree of confidence that what we're doing here is very secure.
"If a manufacturer has a new product they don't want anybody else to see, there are private testing suites on site. It could be a licensed product such as a game based on a movie, etc. We can lock it in that private room and only they [the supplier] and senior GLI staff have access.