News

Real-Time AI Data Processing Spikes Despite Skills Gap

• By John K. Waters
• 06/11/2019

It's common wisdom these days that artificial intelligence (AI) and machine learning technologies will eventually find their way into virtually every software system and application. But companies processing data in real time for AI and machine learning use cases appear to be kicking adoption into high gear, despite concerns about a dearth of developer expertise.

A recently published report provided by Lightbend based on a survey of more than 800 IT professionals ("2019 Steaming Data and the Future Tech Stack") concluded, among other things, that "the use of stream processing for AI/machine learning applications increased four-fold in two years," and "those already using streaming for AI/machine learning expect this trend to continue with even broader use in the coming year."

To be precise, companies processing data in real time for AI and machine learning use cases jumped from 6 percent in 2017 to 33 percent in 2019, according to the survey results.

"Artificial intelligence is more than just speculative hype," the report's authors wrote. "Fifty-eight percent of those already using stream processing in production AI/maching learning applications say it will see some of the greatest increases in the next year."

Driving this increase: "The sharp rise in real-time processing for IoT pipelines, ETL and integration of different data streams indicates that organizations need to extract insights from their data and leverage advanced analytics (such as AI/maching learning) as quickly as possible."

Stream processing is big data-related techology used to query a continuous data stream and detect conditions in real time (or close to it). The aim is to provide access to the data before it goes into long-term storage -- more or less the opposite of batch processing -- and dramatically reduce the latency between gathering and acting on data. (Hazelcast has published a useful white paper on this topic here.)

The upsurge in adoption notwithstanding, survey respondents also said developer skills were lagging. "Developer experience, familiarity with tools, and technical complexity are barriers to adoption," the report concludes. "Concern about scalability, latency and other technical challenges increases as the number of workloads utilizing stream processing rises."

The survey, the second on fast-data trends sponsored by Lightbend in partnership with The New Stack, sought "a better understanding of trends around cloud-native, real-time streaming data applications."

"If you go back just a couple of years, you can see that this is not exactly new," Lightbend CEO Mark Brewer told Pure AI. "It's something people have been trying to accomplish with data they have access to for quite a while. What's different now is we have the ability to take data as it's actually happening, say, in the clickstream, or it could be real-time data coming from other sources, like Twitter or somewhere else. This isn't just data that's stored somewhere. It's contextually aware and happening now. And you should be able to get value from that data much earlier. And I'd argue that that data value is much higher."

Brewer also noted that the developer skills gap around AI and machine learning isn't just about understanding how to write the code.

"It's actually a combination of things," he said. "There is a skillset gap, but it's not just about knowing which technologies to use. It's about understanding how to apply them. Developers are having to become much more like data engineers -- or even explicitly data engineers. They're having to deal with the data more directly and in the application context, as opposed to an external query."

Lightbend is the company behind the Scala JVM language and developer of the Reactive Platform. Several modern frameworks are written in Scala, including Spark, Kafka and Lightbend's own Akka. Not surprisingly, Brewer argued that these technologies are right for this use case.

"I think was probably fairly obvious that many of these technologies happen to be built using Lightbend software," he said. "They're built in Scala, and they use Akka. The reason for that is functional programming has enabled developers to deal with data in a much more native way, via collections libraries in Scala, for example. And the fact that it's Type-Safe gives you much more confidence that when you're manipulating data, you're not going to run into errors or problems."