This has been a busy year for SENSEI. That’s a bit of an understatement, really. In the past 18 months the way data is collected, manipulated, stored, and retrieved has changed completely. But that’s not all! Administrative users also have a new toolbox to track and manage data, better ways to share and edit saved views, and a set of reports to make sure hardware devices in the field are operating correctly. This barely scratches the surface of the hard work performed by the development team, and the generous hours of support and feedback from the rest of Cascade and SENSEI customers.

SENSEI 3 Makes its Debut

The biggest achievement for SENSEI this year was the release of SENSEI 3. The major version change marked the transition to an entirely different data pipeline. This means that the path data travels from where it enters the system to where it is displayed to the user is brand new. The development team benefits from better tracking and error logging, as well as transparency and redundancy in the pipeline, but SENSEI users get the biggest benefit: increased speed! Initial tests showed that graphs in SENSEI 3 loaded up to ten times faster than before! Since then, small changes have reduced load times even further.

But the fun doesn’t stop there! Measurable data isn’t the only information stored in SENSEI; it also tracks metadata about measurables, sites, users, and more. This information has moved into the same faster architecture as the measurable data and is now accessed through “services”, which are small programs designed to do specific tasks. Generally, a service will retrieve information more quickly because it will only look at a small part of SENSEI rather than the entire dataset. New services are being added to SENSEI regularly as the migration to more modern systems continues.

Another powerful aspect of SENSEI 3 is the new calculation engine. This tool creates an environment where calculations can do just about anything as the library of operations expands. This release also necessitated a new interface for entering and editing calculations. Training users on this interface was an important part of the transition–one that is still ongoing. The writing style itself may be more verbose, but the increased speed and flexibility are worth the extra characters.

Other benefits of SENSEI 3 are the improvements to data tracking and monitoring, and redundancy in the system. Previously when a file arrived in SENSEI it would trigger a script that both transformed its content and put the data into a database. When something went wrong with this script no errors or warnings were given, and the file remained in limbo. Now an arriving file is given an ID and a series of scripts perform individual tasks that are tracked on a central dashboard. If one of these tasks fails to perform, the system produces a warning and may retry the file, depending on the type of error. There are multiple copies of each of these tasks so if one breaks, the system can continue to operate. This prevents the single point of failure in the old system.

Finally, SENSEI 3 is built from the ground up to be modular, scalable, flexible, and replaceable. The world of software is changing more rapidly than ever before and SENSEI needs to be able to adapt and leverage new technology to deliver the best user experience possible. It also needs to be ready to integrate new tools and features so they feel like they were always part of SENSEI, not something stuck on the end. And perhaps most importantly, bugs and other issues need to be easy to diagnose and as simple as possible to solve without affecting the entire system.

Administrative Toolbox Gets a Power-up

SENSEI’s Administrative users are the heart and soul of the software. They are expert users and like to push the boundaries of what is possible. This helps inform and drive SENSEI’s development. As a combined development and operations team SENSEI must balance multiple priorities, which can lead to delays in addressing customer service requests. To help administrative users diagnose and resolve issues on their own, SENSEI released a number of tools that allow self-service auditing of various systems in SENSEI. They range from data tracking to meter health to saved view management.

With the number of hardware devices in the field rapidly approaching the thousands, and a team of four to keep track of everything, the need for meter health and down hardware was growing. Enter a set of reports that cover: days since last reported point, gaps between values, periods where only zeros were reported, and values that lie well outside the expected range. Now there is one location to check if there are any issues with Cascade’s installed devices, which: reduces downtime, decreases issues discovered by end users, and streamlines error checking. On top of these reports, SENSEI has a new endpoint for devices sending data to its servers. It gathers information about the device itself and can communicate when settings change or errors occur. Full transition of devices pointing to this location is ongoing.

Saved views are the driving force in SENSEI’s Explore tool. Every SENSEI customer has a set of views that show energy consumption in their facility. Most of these views are shared among many users, having originally been created by an administrative user. If one of these views needed to change, the administrative user would have to delete that view, create a new view, and re-share that view with the end users. This year SENSEI introduced the saved view edit tool, which allows a user to edit a saved view directly, including with whom the view is shared. The user can even create multi-axis plots that are not accessible through Explore.

Finally, when SENSEI 3 added tracking for data processing, administrative users needed a tool to audit this information and reprocess any data that did not succeed. The file status tool surfaces all incoming files and is searchable by source, date, name, and more. This allows users to see, for example, if a customer has uploaded this month’s production data, or why a particular file failed to import. Once a file imports into SENSEI it is broken into individual data points and processed through a series of operations. The operations report lets the user follow a particular point to see how the values on the graph are determined.

Permissions to Land

Possibly the most important feature of SENSEI? Making sure users can only see what they are allowed to see. If SENSEI doesn’t get this right, nothing else matters. The original permissions system was very simple and easy for the administrative users to manage. However, it made many assumptions based on what it was given, and the possibility for mismatched permissions increased as the complexity of customer data and feature requirements grew.

A new permissions system would need to be both flexible and easy to use; two requirements seemingly in direct conflict! In addition, it would need to support the current method of permissions during the transition period. The system that emerged is indeed quite complex, but by defining user roles, default relationships, and rules for inheritance, the administrative user sees a simple UI and needs only know the definitions of the roles to assign the desired permissions to their customer users. The complexity of the system allows developers to create relationships that mimic the old system to satisfy the transition requirements.

While the new permissions system is already in place, it is not yet implemented across all of SENSEI. The implementation process will continue through 2017 as additional SENSEI modules are rebuilt to support it. It is currently applied to all the new administrative tools described in the previous section, and will be supported in all new pages going forward.

Not to Mention …

While SENSEI offers numerous ways to view customer data, sometimes more flexibility is required to show the information a customer needs. To accomplish this, SENSEI sends data to a reporting database where the schema can better support specific reports created in 3rd party software platforms. This Extract, Transform, Load (ETL) process runs frequently enough to keep the reports updated, but due to its long runtime cannot run as frequently as the data arrives, usually every 15 minutes. With the release of SENSEI 3, the ETL runtime decreased dramatically. This opened opportunities for optimization that did not exist before, as well as the ability to run tasks more frequently.

Besides the new tools in the SENSEI toolbox, administrators also have a set of existing tools to manage customer metadata, such as user, site, and company details. These pages have been the same since SENSEI’s initial release, but with all the changes to the software since then it was time for an update. This also coincided with the deployment of the new permissions system, which needs to be deeply integrated with any administrative tools. While still under development, the new administrative pages have a brand new look and feel to them and are full of more ways to connect with customer information. They will make metadata management a breeze once completed, and streamline new site and company creation.

Here’s to a Great 2017

This year had a clear focus on administrative users. These folks help Cascade’s customers understand their data and get the most out of SENSEI. Without them many customers would be lost in an unfamiliar software platform, surrounded by scary graphs and endless streams of data. That is why SENSEI developed new tools to make administrators’ lives easier, allowing them to better serve the customer. This theme will continue through the beginning of 2017 as the rest of the metadata management administrative pages are upgraded. However, there are big plans for customer-facing modules in SENSEI as well! Updates to Act and Manage are both in the pipeline, and new Dashboards are high in everybody’s mind.

The SENSEI team was sad to say goodbye to three of its members in 2016, but happy to welcome a new face as well! Good developers who share Cascade’s values are hard to come by, but clearly they are out there. In fact, the SENSEI team is currently hiring! Check out the link at the bottom of the page for more information.