Introduction of Automated Camera Testing Equipment

SoMA is a fully automatic multimedia quality testing device assisted by a robot arm. It can support mobile phones (Android/iOS/Window phone), tablets, motion cameras, surveillance cameras and other devices. Fully automated camera functional testing, stress testing, API testing, shooting performance testing, image quality testing and so on. Customers can also integrate their own testing methods and judgment algorithms in a short-time. If combined with electric guides, it can even move around in open laboratories, shooting and switching different scenes.  It can ensure effective testing in the development process. On the same device, different functions can be combined to achieve t efficient way. For example: Camera API compatibility test, and then functional test, and then performance test can be carried out to find bottlenecks as soon as possible. Finally, press and quality testing, as well as image quality testing by taking photos. Even though testing may be divided into different stages due to the development process, all test layers can be used in the development process to avoid errors from scratch.

The Problem: Testing Subjectivity, Inconsistent Results

Testing of multimedia features commonly requires an extensive amount of manual work and the utilization of single tools. Therefore, test results are based on human observation and vary significantly depending on the particular testers. This can result in inherent testing subjectivity, a problem that can lead to unreliability, production delays and unnecessary costs .

The Solution: Objective Testing, Precise Results

SoMA’s  computerized testing methods revolutionize built-in camera testing by enhancing objectivity,  effectiveness and quickening the development of the product being tested.In other words, SoMA offers you the ability to have greater quality, consistency and reliability for your product.

SoMA's automated test systems provide increased load capacity and speed for the basic tests that must be performed during the product development process. The computerised system also improves the accuracy of the testing process, which inevitably provides accurate results comparable to those previously obtained. In other words, you are able to perform more tests more frequently and more quickly, while obtaining accurate test results with minimal error. In addition, SoMA makes it possible for you to repeat the same tests and compare past results effectively.

Early Problem Detection

Frequent, fast, accurate, easy-to-run tests generate effortless and instantaneous problem detection.  SoMA gives you the considerable advantage of identifying potential problems and defects early on in the development of the product. Early problem detection promotes quick and easy correction, ensuring substantial quality, robustness and reliability of your product.

API Testing

API testing is more than a traditional API testing. Normally, the API testing concentrates on verifying the compatibility of the API methods (or functions) and testing the parameter ranges of each method. Sofica’s API testing also offers the possibility to compare the API compatibility and robustness between different devices.

As the following steps describes, Sofica’s API testing verifies also the dependencies between methods and also verifies the robustness by testing illegal method scenarios.

Three steps of API testing

1. Parameter testing: Compatibility verification of the methods by testing the smallest and largest input parameter values, as well as random values between them. Furthermore, negative testing is done by providing illegal input parameter values to the method interface and verifying that the correct error code is returned.

2. Correlation between inputs: The API testing verifies the correlation between methods. Normally, there are two kinds of dependencies between methods: callbacks and set-get method pairs. Testing verifies that correct GET values are returned for each SET input. Also the correct response of illegal input values is verified. In addition, the callback logic also is checked by detecting the response (callback) to each call.

3. Method scenarios: The API testing verifies the method sequences and the stability of the system when illegal method sequences are executed. This has revealed to be a very effective and stressful test to the camera systems and several robustness improvements are done after the sequence tests.

The API testing also offers the possibility to compare the API compatibility and robustness between different devices. When the existing API tests were executed against commercial devices, the pass rates were as low as 57% and severe crash issues were identified.

API Testing Available for All Platforms

The API testing can be implemented against every API in all platforms. Currently, the API testing has large reference implementation against Android Camera Hardware Interface. The API testing contains about 200 different tests against this interface. Camera Hardware Interface is the device driver interface of Android camera system and it offers C++ API. However, the same kind of tests can be done, for example, against Java-API.

Sofica's camera test facility also allows the execution of external tests such as Android Compatibility Test Suite (CTS) tests. While the API tests verify the device driver interface, the CTS ensures application layer compatibility. Together, these features create a very comprehensive testing entity for the entire camera system.

Functional Testing

Functional testing of the algorithms requires the capture and verification of the real images. The functionality of different camera algorithms and parameters can be tested by measuring their influence on the captured image. The testing procedures uses mathematical analysis to detect the differences between reference and captured images.

The mathematical analysis with the automatic test environment enables efficient testing of both individual functionalities and the combination of several features.

The main purpose of the functional testing is to verify the correct functionality and generate repeatable measurements between test executions and across devices. However, functional testing alone is not sufficient to determine the absolute quality of the camera system. Qality testing provides these additional measurements capabilities.

Standards-Based Image Quality Testing

Integrate Imatest IT software and test charts with SoMA test routines to add well-known image quality testing analysis.

Supported Functional Tests		SoMA Hardware and Software Included
Image & Video Resolution		Camera Obscura
Image & Video Content Verification		Manipulator Arm: Align the camera to the test card Location equipment and test card Automatic Test Cycle
Image EXIF Data Correctness
Auto Focus, Auto Exposure, Auto White Balance
Dynamic Range
Flash & zoom		Image Quality Test Card
Face, Smile, and Blink detection		Illumination Source
Audio/Video Synchronization		4K Display for Camera Functional Verification
Color Effects		Automatic Analysis Software
IR Hotspot		Image Quality Algorithm Software (imatest)
Image Stabilization		More than 200 Tests Perform (Android)
Panorama		Standard support for 12 devices

Stress Testing

Stress testing, as the name suggests, aims to identify the limits of the camera system, by testing the robustness and stability of the system. Furthermore, it reveals cumulative defects of the system and analyzes the memory usage.

Sofica simplifies the creation of large stress test entities, consisting of API, functional, performance and quality tests. This enables, for example, the generation of performance statistics from long duration tests.

Random testing is one way to search for unexpected defects. Different tests are randomly executed against camera system and the corresponding test execution schemes and logs are saved. If defects are detected, the saved test flow can be executed again and the defects can be analyzed more closely.

Stress testing, like all Sofica testing, automatically saves data in the event of a crash, which helps to resolve the cause. The crash data files comprise process information, file handlers, system log, memory information and tombstone log.

Performance Testing

Performance testing reveals bottlenecks in the camera system by measuring the execution times of different camera functions and combinations of functions. High resolution camera sensors can cause critical performance issues when megabyte sized images are processed in a very short time.

Performance testing creates timestamps when the camera API calls are made and when the corresponding response is received or callback method is executed. The execution time is simply the difference between the timestamp values.

Performance testing also measures and records the execution time of each camera function. For example, the compression time of the encoder can be measured as well as the shutter speed with different auto focus settings. These measurements are quite straightforward, however, different combinations of features may change the performance values significantly. With performance testing it is possible to measure different combinations and identify the critical ones.

In general, performance tests do not require reference images. However, there are some features which affect the performance and the image capture circumstances should be static to ensure objective test results. As an example, the execution time of the auto focus and auto exposure depends on the environment.

Performance testing also enables the study of performance trends between test executions. This facility is valuable for early detection of performance degeneration in feature development.

Performance testing allows benchmarking and comparison between different devices. When tests were executed on a sample of commercial devices, significant differences were found. For example, the shutter speed varied from 50 to 225 ms.

Quality of Image and Video

The quality testing enables standards based testing of camera and video processing algorithms in a compact and isolated test environment. Our computational based quality analysis guarantees that the camera and video processing algorithms will be tested in an objective, reliable and repeatable way in every R&D phase.

Video Quality Testing

Quality of the processed video stream is the most common problem when developing new multimedia products. Video processing algorithms can introduce distortion to the video signal, so testing video quality in every R&D phase is important. Video quality testing helps to find anomalous behaviour of encoders and decoders.

Examples of Supported Video Quality Tests

• 1. Audio/Video synchronization test
• 2. Resolution test
• 3. Framerate test

Integrate Imatest software with SoMA to test auto-focus, auto white balance, and auto exposure metrics for video recording. These metrics were created to address important video quality aspects that affect the user experience.

Image Quality Testing

Quality test camera algorithms to fine tune the captured images.

Imatest software and test charts can be integrated into the SoMA workflow for device image quality analysis. Assess the quality of the captured images and adjust camera algorithms to get desired results.

Imatest algorithms and proprietary SFRplus test charts build upon the ISO standards for robust and comparable measurements. Imatest iterates their software frequently to include modern standards, such as the ISO 12233:2014 standard for resolution and spatial frequency and CPIQ camera characterization metrics.

For more information about the lighting system and its many options, please contact us. If you would like to learn more about a complete lab setup, please visit our Image Quality Lab Setup page.

Follow ColorSpace for  industry news/events/solutions