May 24th (UNLISTED) - Success Definition and Operationalization

A few weeks ago, we were tasked with creating a document on "Success Definition and Operationalization;" basically, a list of goals that would define whether our project was a success or not. These will be listed below, along with our analysis of our success in those regards: successful goals will be defined by green text, and failed/uncertain goals in red.

-----------------------------------------

1.) Users can easily connect and organize devices within the dashboard

• Automated connectivity removes mundane tasks from user workload
• Device Groups in “folder system” allows easy management of devices
• Admin users can manage/group users to have specific permissions

This success definition primarily focused on the Dashboard's functionality, and whether users responded well to those specific features. This was indeed the case, as all of Ombitron's testers reportedly had no problems understanding their capabilities and completing the tasks given to them.

-----------------------------------------

2.) Users of all skill levels can monitor the status of all connected devices quickly and accurately

• Users will need only basic training to be acquainted with the system
• Users can scan the Detail View for metrics information and graph performance overviews
• Minimal effort to discover errors, measured by TLX Task Load for 50 connected devices
• System outperforms competition with direct visualization of device metrics vs. sole list view

Unfortunately, we were unable to obtain any further testing data or metrics from Ombitron aside from a few quotes, making it impossible to critically analyze the results of Ombitron's formal testing. We could do some testing of our own to make up for this, though our data will not be significantly useful; the target audience of company workers and businessmen will not be the ones we test with. That lack of context will hamper efforts as students and friends don't have the same mindset, nor interest, for such a product. On the other hand, if they can use it well, it speaks to the effectiveness and simplicity of the Dashboard for non-specialists, which might be good point to make. 
That... might not be a bad idea. We may or may not conduct a few more user testing sessions with our Dashboard. If we do, expect an extra blog post about that.

We do know that the visualization of the Folder View provided some users great clarity in testing, so that one goal was a success. Those who preferred the List View used that, but more casual users found the Folders more appealing and clear. Having both options allows the Dashboard to be more adaptable depending on the preference of the user.

In the meantime, we have requested any of the following information from our client:

• What confidentiality is in place that prevents us gaining the any of the information requested in these other questions?
• Who was tested: businessmen, stakeholders, random employees, target users, other? Not looking for names, just benign tester information. How many, and for how long?
• What were their tasks? How many tasks, and how complex?
• Which features would be used in these tests?
• How many testers passed all tasks? Were there any difficulties in any of them? Which task took the longest? The shortest?
• What questions were asked by testers?
• Did the facilitator provide any assistance during testing? If so, what?
• Were there any specific testing metrics used? SUS, TLX Task Load, custom forms, or other?
• Are there feedback sheets or transcripts we can get a look at?
• Which companies "demoed" the site? If that information is unavailable, at least what was their size (startup, corporation, multinational) or industry (tech, industrial, venue, etc)?

-----------------------------------------

3.) The results are used by Ombitron for current or future projects

• Web 10.0’s dashboard as the baseline for Ombitron’s other products and services
• High SUS reported when used by client companies (CenturyLink Arena, others)

The second bullet of this Success Definition falls under the same problems as the one above.

The first, however, is very, very much a success. Ombitron is using our design moving forward, and the demoed design has been ours from the beginning. We expect continuing development based on our ideas for this product and ones like it in the future!

-----------------------------------------

So hopefully that covers it. The lack of testing data is a setback, but hopefully our own testing sessions will give us the stats we need to provide context as to whether our goals have been achieved or not.