Diffusion 6.6 – Enhanced Data Wrangling for Real-Time Data Streams

Plus new Python SDK, Support for MQTT Protocol and Apache Kafka Adapter

Push Technology, the pioneer and leader in real-time data streaming and messaging solutions, today announced powerful new functionality for the company’s Diffusion Intelligent Event-Data Platform, that consumes, enriches and delivers data among applications, systems and devices. The new release expands the range of Diffusion’s unique Data Wrangling capabilities which provide developers with the tools to manipulate and transform event-data – in-flight – via topic trees, topic views and time series with actual or custom time stamps.

Data Wrangling
Data wrangling is the process of transforming and enriching raw data streams for optimized data delivery to a receiving application or device. This process is particularly relevant to event-driven architectures, which sit at the heart of modernization and digital transformation initiatives worldwide. The platform’s new functionality enables users to consume and mix disparate raw data sources and extract pertinent pieces of data to form new data streams for delivery of exactly and only the data required by recipients.

For development teams, Data Wrangling can be the most challenging and time-intensive aspect of application development because in-depth knowledge in this area is not often part of the in-house team’s expertise.  The Diffusion platform unburdens development and helps speed applications to market.

Diffusion 6.6 introduces capabilities to augment event-data associated with a topic via topic view inserts. With topic view insert, you can now merge data from topics other than the selected source topic into JSON reference topics. You can insert whole values, or partial data specified with JSON pointers. Advanced features include the ability to derive paths from values and chaining multiple inserts.

Diffusion 6.6 also introduces two major enhancements to time series topics. You can now update a time series topic via the standard topic update API, treating a time series topic as if it were a single topic with the same event type as the time series. This means that when you update time series topics, you can now use features like update constraints, update streams and the AddAndSet operation.   6.6 enables loading persisted or historical data into a time series topic and also allows setting a custom timestamp on historical data, instead of one based on the current time.

New Python SDK
Diffusion 6.6 includes a new Python SDK to accommodate the growing adoption of Python by the developer community and market feedback. Python is popular for general web development, scientific computing/data science, machine learning, and FinTech applications.  The Python SDK supports subscribing to topics, and request-response messaging.

Apache Kafka Adapter
Diffusion’s Data Gateway makes it easy to consume both static and streaming data from a wide array of sources and provides capabilities to prepare the data for wrangling before an event-data is created and made ready for distribution.

Using the new Kafka Adapter, organizations can now efficiently and securely extend Kafka solutions over the Internet, streaming real-time data to millions of end-user applications. In addition, customers can easily manage the high-volume of data across geographically dispersed regions.

Push’s earlier standalone Kafka adapter used the Kafka Connect framework and was available via GitHub. The new adapter does not use Kafka Connect, and is instead based on Kafka Producers and Consumers. The adapter translates data between Diffusion topics and Apache Kafka events. It is designed for quick and easy Kafka integration with Diffusion.

Support for MQTT Open Source Protocol
By supporting MQTT with Diffusion, software development teams can now bring state-of-the-art event-driven architecture to their IoT and Mobile solutions. The low code features of the Diffusion platform significantly reduces software development effort and the overall cost of deployed solutions.

Diffusion implements MQTT 5.0, the latest version of the specification. Both the TCP and WebSocket transports are supported, and connections can be secured using Transport Layer Security (TLS). Diffusion treats MQTT as a first-class protocol and acts as a session broker for MQTT clients in the same way as it does for Diffusion SDK clients. Each Diffusion server can host tens-of-thousands of MQTT sessions; and servers can be configured in clusters to scale to an arbitrarily large number of connections. MQTT clients can be monitored and managed by Diffusion SDK clients, the management console, and external tools connected via the Prometheus or JMX gateways.

Sean Bowen, CEO of Push Technology, said: “Diffusion’s new Intelligent Data Wrangling functionality sets the platform light years ahead of the alternatives and assures organizations’ event data is optimized and actionable. Development teams can take advantage of the powerful data wrangling features of the Diffusion platform and assure efficient and reliable real-time data delivery, with secure access control to applications, devices and systems.”